Take a look at RDS2216 with RaidOwl
Yeah cool, if you really need that much throughput but if you do, you're going to be paying a lot of money - they're a lot. They're probably five, six, seven times the price of this one. We wanted to make it a deal that you just could not resist. A massive amount of nvme u.2 storage, 100-Gig networking using a low power ARM chip - all of that for under, you know, $2,000. Hey Brett it's nice to see you. Hey
Druvis how you doing? Really good, glad you could speak to us! It's always nice to hear different perspectives from people doing different... specializing in different things. And I looked at your YouTube channel, looked really good. I saw you do a lot of this storage stuff so maybe you can tell our viewers a little bit more about your channel and what you do.
It was funny to hear you say 'specialize' and things, because I don't really consider myself to specialize in much. But I'm just a home lab channel. I started with just one server, dabbling and home labbing, and progressively got into more virtualization stuff, more storage stuff, networking, and my channel has grown along with my home lab to where now I have a 42U rack in the garage... I have a 12U sitting up there in my studio and I just got another 22U rack sitting there next to my desk. So it's grown and uhah... basically like you said a lot of storage stuff - that's kind of been my niche now... It's kind of been what I've been interested in,
so when you reached out to send me one of these, I was, I was totally down to check it out. And, well, how do you get into like storage? Because it's to me, it's pretty specialized. I have on my own little home lab, but you know currently I don't have massive like storage there.
So it starts with convincing yourself that you need a bunch of storage, which some people find easier to do than others I guess... For me I had a really easy excuse saying like I'm a YouTuber, I have all this footage, I need somewhere to store it, right? So that took care of the mass storage aspect and then since I'm editing videos and I want to keep all that, all my footage and stuff, on a centralized server, I need high-speed networking, I need fast storage. So I'm definitely operating above what I need which is 99% of the time in my videos, it's way overkill, but, basically, convincing yourself that you need it, and then you quickly go down the rabbit hole and realize thousands of of dollars later, terabytes later that you... you got
a bit more than you need, but it's... it's fun. Yeah, I guess this doesn't apply as much to me as I'm... I'm blessed to have all the storage at my office, so... Hah, yeah... And, yeah, so the high-speed internet as well. Besides using all this storage for video, what else do you do at your home lab? What, what sort of things pick your interest? What do you have running right now? Oh, um in terms of storage or just in general? In general I think? It's just interesting to... Um... so a bit of a hodge-podge I'd say. So my storage... I have a main setup for my bulk
storage is a TrueNAS system, my high-speed NVMe storage is actually running in a Windows Server virtual machine. Basically, because I was playing around with RoCE and trying to get the fastest transfers possible over the network, um, for SMB. So that's what's running my high-speed networking at the moment, um, my backup system is a Synology device just because their software and infrastructure is just so easy, and then that shoots off to an off-site location on my other Synology system where, you know, everything got my 3-2-1 backup going. In terms
of what I'm running, your kind of basic stuff, a couple of websites, reverse proxy, nginx proxy manager, Plex of course, or, you know, Jellyfin for those Plex haters out there. I'm hosting my own AI server/virtual machine with a couple of GPUs in there. I think I have two RTX Ada 4000s - I think that's what they are - a couple of those, and uh T4 or whatever. But I think it's like 56 gigs of usable VRAM and I'm just running a little AI instance to play around with, uh, open web UI, open, is that the (?) one of those, and Ollama, and it's, it's working great. It's really fun to play with, but, yeah, just a bunch of random stuff depending on what I... what I'm
interested in that week, that month. I'll spin it up try it out and move on to the next thing. It's pretty good to play around with all these different technologies. It's that, I find that, if you if it really picks your own interest then it will be much better for creating your videos as you're just going to be genuinely talking about your experience and what you found, all the difficulties, and everything. So that's really cool. It would be very nice uh to see you uh do something with the RDS. Oh yeah 100%! Yeah, this thing has been solid, so I don't know if you want to get in and talk about like the specs or how this thing came about, but when you sent it to me, a couple of things stood out, obviously the green color, so I guess we can get into that, maybe, in a bit or if you want to answer that now - why it why you guys chose to go with the uh the green color? But overall it's a pretty cool system, I guess the first question I have for you is, you know, MikroTik is the the, known as the networking, make a bunch of cool networking hardware, and then all of a sudden you see a storage solution, so, I guess, what is, why did that come about and, like, what is the optimal use case for this device? Alright, well, the first about the color - to be honest I don't know the the exact moment when somebody decided it needs to be green, but... I like it it's cool. I've heard I've heard people say that it's like because
it's uh it saves uh you know like resources and money and it's like greener, better for the planet if you... instead of having a separate server running the whole time you can just get one of your high-end routers to do that stuff it could be good for the planet. Personally, I would say, that it's... it's good, green is good color because the product is a little bit green, it's just very fresh, and it could use a little bit of ripening. But it is something that I think is going to be
very valuable for very specific scenarios and we might not know all the exact locations where we might discover that it's useful for someone... But to speak more about how it came about, I could give uh give you basically two takes on it one is the story and the other is sort of more technical. So if you... I'm going to show you my screen... So this website gdn.lv is a website for a Genomic Data Network project that Mikrotik has taken part in. And here's you can see the members, here's... you can see our founder John. He's taken interest in a lot of these science/research and
healthcare... projects that we can sort of help in. And what we usually do is we help with the network. Here you can see a map, well it's a simplified map, of an existing sort of science research network, very high-speed connections. As it expanded we found that the genomics project requires a lot of storage, because genes as you know they they can take up very large amounts of storage, so trying to help with this project we had this idea that we could, maybe, create an affordable server of our own, and that's actually how it started, but, realistically, the the next bottleneck is the the server speeds of the of the researchers, and the people that work with the Genomics Data Network - so that's going to be their next challenge. So to put it shortly, how it came about is that we wanted to do this sort of ethical charitable thing and here we are.
So now we're at the technical side and um if we look at the block diagram it looks a lot like our flagship routers. You have all the networking part of of a high-end router - you have a good switch chip, some high-speed data lines and all connects to the CPU that is also found in our flagship CCR2216 router. That obviously is not your typical data server CPU, it's just something that has a really good price-to-performance ratio, so it doesn't add just multiple thousands of of dollars as cost to the user, so that's why we've chosen this architecture, and then if you look at the bottom of the block diagram you see that there's these PCIe lines that we simply tapped into. So let me let me just double check this... So on the block diagram right we have PCIe gen-3 16x lanes, correct? Yes, that's correct, so PCIe 3 - that will be 1 Gigabyte per second that you can push through there in both directions or that's 8 GBs per second and for 16x lanes that would work out as the max maximum throughput you can get there is 128 Gbits per second. Now, the drives are grouped up by four, but they get two PCIe lanes each. Some
people have been upset that you only get 2 GB per second per drive, but it's not how you're supposed to be using it - you're supposed to do RAID. So you'll never be using just one disk at a time, and if you set up the RAID properly... the speed, the speeds that we have been getting and our testing have been somewhere between 40 and 50 Gbits per second with, with RAID set up. And that's if you push the data to the server from another server using NVMe-over-TCP. So you mentioned something that was was interesting, and one of the main things about this is that you, you mentioned before, you said, it's not meant to, I mean we looked at it, we said, you know, PCIe 3 16x lanes to the whole thing. I think, when a lot of people look at NVMe storage
devices they're going to say oh I need four lanes of PCIe Gen 4 to each of my drives. And, yeah, cool, if you really need that much throughput... but if you do you're going to be paying a lot of money for a server like that. I've seen them they're a lot! They're probably five, six, seven
times the price of this one, so this is honestly the first time I've seen a piece of hardware with this much NVMe capability or the ability to add a bunch of capacity of NVMe drives with a price under 2,000 USD, which is crazy, so that's one thing that stood out here, was the price. Yes definitely, that's basically, this is basically... if if you took a CCR2216, lowered the price by like 700 bucks or something and added all this storage functionality on top for free. So we wanted to make it a deal that you just could not resist and this is what... I mean on the back you're getting you're getting a whole, like... This is a high bandwidth switch,
network switch with storage built-in, like, it's... it's a very very interesting device, so yeah you get your your dual 100-gig, you get what 8x 25-gig? Did I get that right? No, 4x 25-gig and 4x 10-gig which, I mean - you have a lot of network throughput through there, and then, yeah, like you said, add some storage on top of it. It's not just... Price - it's wild! It does have this powerful switch chip, but it's not just limited to switch functionality, it actually... Well, it can do layer3 offloaded to the switch (chip), but it can also function purely off of this CPU, well, provided that you're not copying a lot of files at the same time. So I think... I think that's where one of these sort of scenarios where people are going to find this is very useful - if you want to do, like, very large backups during the night, but you need a high-end router during the day, so you can, when you're not using it at night, you can just use it for this massive backup storage. Yeah, and another thing that stood
out was you're utilizing like... software wise - it's running RouterOS and then you've kind of incorporated a new, I guess, feature set within that, called ROSE. Is that correct? Yeah, so ROSE stands for a RouterOS Enterprise and it's this idea behind it, is that we we're going to try to take these enterprise features and bring them to users that normally would be priced out of having those. So, yeah, when I got this and and was playing
around in RouterOS... So I'm going to let you know right off the bat, I'm not by any stretch of the imagination an expert on RouterOS, and I was dabbling around in there, and looked into the documentation on the the ROSE, and the RDS features, I went to set up a RAID configuration... It was actually really easy within the disks UI within RouterOS, and then going in using the terminal to set up the NVMe-over-TCP setup, I exported a RAID-0, I just have two drives in here for now, just to test it, I have more on the way, but I set up a little RAID-0 configuration, imported it for to to be... imported into my client on the client's side and that all went super easily. I mean it was like three lines and on the other end I had a full NVMe-over-TCP setup, and over a 100-Gig connection, which I mean that's crazy to think about - all within this tiny package. Yeah, that's exactly the idea that we want to take this all this stuff that normally you couldn't get, or you couldn't afford to get and you can just use it - you have a router with all these advanced functionalities on top.
Did... I guess, a question, a follow-up question I have is - were there people... I know you guys talk to customers, some smaller, some larger enterprise customers... Was that something you were hearing from them that, you know, kind of spurred this was like 'Man it would be cool to just have some storage built into this!' or was this just kind of all on Mikrotik side, like, 'you know what... this is this is something we want to pursue'? Yeah, so well, the initial... the push was the GDN. Okay. But but there have been people who
have said that 'oh it's a shame that RDS...' and sorry 'it's a shame that uh CCR2216 and CCR2116...' - that they don't have more M.2 slots or they don't have traditional SSD slots for SATA or whatever, and so this is the the answer. So now you do! Yeah, so now you have 20 slots.
So you're -you're set. Yeah, good to go. Yeah, I'm trying to get... I ordered a couple of drives and I'm, I'm working with um a vendor to see if they want to supply a bunch of drives for this to test it out. But for now, the only two I had that were 7 mm that would fit in here were like little adapters for m.2 drives, so I'm getting some actual u.2 native drives in soon. So I'm excited to actually test it out. Yeah, if you're going to buy this and fit it with all these drives,
it's probably the drives that's going to be the expensive part, isn't it? Oh yeah, 100%. And I mean that's the thing, that's that it's hard with NVMe storage because, yeah, this is under $2,000 which is a deal, but like you said NVMe drives are not cheap they're getting reasonable... You can find, you know, re-certified used u.2 drives for a decent price. Now I guess more so than a few years ago where they were just not obtainable by the normal person. I just bought a couple of used ones for like I think 50 bucks each, so it's not unobtainable, as it was a few years ago, but, yeah, if you're looking to get like mass storage in NVMe, you better, you better have a big bank account. So this just got released for sale, like, what the last week or so, but I know you guys have probably been in talks with customers, and people who have been interested in buying these... I guess, who are those people and and what are they using it for?
Well as a manufacturer we sell primarily to distributors or clients that want to order like very large volumes, so we're not actually selling to the end users. And it's going to take us a while before we get the feedback from the end users because they'll, probably, they'll tell something to the distributor and then they'll tell us, and that kind of stuff. It's hard to say what exactly people are going to use it for, but I think there's going to be people with servers that simply need storage, that simply need more storage, but they don't necessarily need the fastest possible storage. Or it's going to be these sort of edge computing situations where
people want to store large amount of data closer to these end users - and so this could be used for that. Yeah, I mean I've talked to people like in my Discord and whatnot who have sent me links to this that have just said 'man this thing is super cool', you could probably count on one hand with a couple of fingers missing how many devices are out there in the wild that you can actually buy that have a massive amount of NVMe U.2 storage 100-Gig networking combined with, you know, multiple high-speed networking options, using a low power ARM chip, in a 1U chassis and all of that for under, you know, $2,000, which you know it's a... it's subjective about what's expensive, but for what you get, you know, I don't know that there's something out there that mimics what this brings to the table. So a lot of people are very interested just to get their hands on it
and play with it. Like the the amount of stuff you can do with it is it's pretty awesome. Yeah, I need to play with it myself, I've actually, I have a sample just like next door and it's fitted with 20x one-terrabyte drives, so I can really test it and make sure that... and just see if the speeds that we're reporting 40-50 - is it actually feasible for like every day well not every day, but for situations that people might use this for. So it would be nice to do another video where I kind of push it to the limit and see what it can actually deliver in different situations, but uh the challenge is like how do I even generate this this throughput, because I need uh 40-50 Gbits per second of data flowing into it or out of it, somewhere to some other place, so I need like a pretty powerful server to do that. I'm not sure if I'm going to get my hands on an entire server,
so maybe I'll have to get creative how I can attain that speed, but, yeah, it would be very interesting. Maybe (you) have some ideas what-how how I could test this? Yeah, I mean I, bare bones, I've had this running for, before I pulled it out of the server rack, for like less than 24 hours, but I ran a super bare bones test, like I said - I threw two m.2 super generic, whatever I had laying around NVMe drives into their little sleds and and put them in here, spun up a a RAID-0 configuration and then did a NVMe-over-TCP connection to another server and ran an FIO test on that. And just with that I think I was getting like 18-gigabit which is not optimal not optimized in any way so to see that right off the bat with nothing else being done was pretty cool. I guess, I was thinking if you had it fitted with 20 drives, like, what would you do? How would you test? It, oh man, I guess, there would be in a lot of cases you know there's your your synthetic like just running a various different FIO tests, different Q depths, different jobs, random versus sequential, you know, if you want to test like straight throughput or you want to test the iops and the benefit of using NVMe storage? You know you can use your synthetic stuff with FIO, you can do your straight, you know, real world tests, like, you know, use cases transferring large files for, you know, if you have a media server, if you want to use this to host your your... all your storage, which you're going to be transferring back and forth a lot, editing off of (it), you can do that you can set up like an iSCSI. If you don't want to do
NVMe-over-TCP, different ways of setting that up. Or see how those perform against each other? But, yeah, synthetic, I'd run my my gambit of FIO tests and then when I typically get these in, I just run it through, you know, my real world stuff, which is basically how I use it on a day-to-day basis. Which is a lot of editing and big file transfers. Yeah, the big file transfers is probably the simplest thing that... I I'll just get some massive file and just drop it in. I mean that's always the funnest one, right? You just transfer a file over and watch the number, and see how big it can get. Yeah, can you elaborate on the FIO tests, because I'm not sure what that means to be honest. Oh FIO, it stands... it's a uh soft or a storage benchmarking tool where you can give it a bunch of different options on how to test that either block-device or a file system. So when I set up my uh NVMe-over-TCP, I had it on the client side, so I ran FIO. I
told it to run a sequential write test, so the fastest... just throughput, and I basically said, you know, run for 30 seconds, Q depth of 64, your block size, I do have a... I don't say I have it, but there's there's a tool that builds on top of FIO.... Can't remember the name of it, I used it in one of my videos, but it's kind of a FIO plus where you can give it a whole gambit, a whole range of different tests, and just click the button, and it'll run all those tests on that drive or block-device or whatever, and spit out like a nice graph that can show you, like, you know, different iops at different at different block sizes, different number of jobs, different Q depths, and (it's) just a really cool tool, so once this is kitted out, I would probably use that and just push it to the limit, and see what what kind of optimizations, what type of different, you know, flags are set to where this thing can can get you the most speed or iops or whatever you're wanting to look at. Cool, you should send me the link to that that f... FIO p... plus was it? Yeah, it's not called that, it's called um... I'll send you a
link to the video I used it in because I kind of go through how to how I used it but I also send you it's it's on GitHub somewhere, I can send that to you. It's really cool. Cool I'll check it out. So with Mikrotik now getting into the storage game and having that kind of RDS-Rose environment within RouterOS, if this does become popular, and you see these fly off the shelves, and people start asking for, for more of this, has there been any talk of like making a dedicated operating system or dedicated subset of RouterOS specifically for storage or is that kind of... no one's even talked about it, yet? Not really, it's... we don't intend to do like a separate thing we're just going to improve the existing package and we're going to improve the graphic interface for managing the package. And there will be new features added to the ROSE package as well, but I think it's pretty good bet that there will be someday... there will be another RDS with a much more capable CPU and it's probably going to have a PCIe 4... I would
imagine, and I bet you'll be able to get some really good speeds with that one. But it's still, it's still just... Take it as a speculation for now, but... but I really hope that will come about not too far in the future. Yeah, I'm excited to see what comes out of this and for a gen one product, I think this is pretty nifty. Yeah, well, thank you for this conversation and I'm really looking forward to the video that you'll do on your YouTube channel. We'll put a link at the end of this video. Yeah, no, I'm excited, stay tuned... I have some fun stuff to do with this, so I'm excited to see what comes out of it.
2025-03-23 03:25
