Building a 26TB Offsite Backup Server! ft. Tailscale
Despite many protests from long time viewers, I still make Youtube videos And just like many other creators, I have terabytes of video footage. Mostly of me fumbling my lines. But if you’ve been watching my videos for some time now, You’ll notice that I reuse quite a lot of B-Roll footage, And so do other youtubers. Sorry for ruining the magic. Now, some video creators have dozens of external hard drives, some have a dedicated off-the-shelf NAS. Personally, I store it on my home server. It’s got two 16 TB hard drives in a MergerFS array, as well as another 16 TB drive dedicated to Snapraid parity data.
So I can theoretically lose one drive out of three and not lose any data. And this data is very important to me. Having all of the original footage from my videos is very useful, It lets me reference my previous videos in my new content, without having to resort to downloading my own videos from Youtube in a super compressed format with subpar quality. Besides, there are plenty of situations where I would film something for a video, and then just not end up including it in the final product. If I still have the original file, I can actually use it for another video in the future, instead of it just going to waste. So how do we keep important data safe? That’s right, backups.
You’ve probably heard of the 3-2-1 rule when it comes to backups. You should have 3 copies of your important data. On two different types of media one of which is off site. Meaning, in this case, as Aaron Paul puts it, … Now, these days, the most popular type of offsite backup is cloud storage.
Be that just putting your data manually into Dropbox or Google Drive, or using fancy backup software like Restic or Duplicati and sending your data to an S3-compatible storage endpoint. Like Backblaze B2 And that’s what I do for some of my other important data, such as personal documents and my password manager database. But when it comes to video footage… Well, I shoot these videos in ProRes, and the entire archive folder, with all of the raw footage that I’ve accumulated over the years, is currently at 14 TB.
So, is it really feasible to store that much data in the cloud? Well, surprisingly, yes! AWS offers an S3 Tier called Glacier Deep Archive, which currently goes for $0.0018 dollars per gigabyte of storage. So if my math is correct, at 14 terabytes, this would set me back 25 dollars a month. As well as a one time fee of 70 dollars, should I ever need to retrieve the data. And as far as business expenses go, 25 dollars a month is not that much. After all, it’s backups of very important data we’re talking about here. The real problem with the cloud are transfer speeds.
Germany is not really known for its fast Internet, and at the moment, I got 250 mbits down and 50 mbits up at my house. Uploading 14 terabytes of data at 50 mbits per second will take me 25 days. And downloading it will take 5 days. Plus, depending on AWS for my business is not something I want to do. You see more and more companies being burned by depending on the public cloud infrastructure, and I prefer to avoid using the cloud if i can.
So, is there a different way? Yes! And all we’re gonna need is another NAS, a few hard drives, and parents with Internet connection. Now some people might say that I shamelessly stole the idea of offsite backups to my parents house from Hardware Haven But that’s just not true. In his video on offsite backups, Colten backed the data up to his parents house, whereas me, well, I’m gonna be backing the data up to my parents house. Which are two completely different things.
Me and Colton do however have the same sponsor for our parents NAS videos, and that is Tailscale. I guess that similarly to how Squarespace became the de-facto podcast sponsor, Tailscale is trying to become the official sponsor of off-site backups to Youtubers’ parents. So definitely be on a look out for more parents NAS videos sponsored by Tailscale. In all seriousness, the idea actually came from a few comments under my first Parents NAS video that I did in 2022. And just like the previous build, this new machine is also going to double as a backup target for my parents vacation photos. So first things first, let’s talk about the build itself! For the motherboard, I’m gonna be using Asrock N100DC-ITX from my last video.
For 130€, you get a motherboard plus CPU combo, which is fairly power efficient, and supports hardware video transcoding in Plex and Jellyfin. Which my parents probably won’t need, but you might. Now to be fair, the only reason I bought the N100DC-ITX is the fact that it has a PCIe x2 slot, which would be perfect for my personal use, since I need 10 gigabit networking.
But then the idea of offsite backups kind of resurfaced in my mind, and instead of buying a more appropriate motherboard and creating more e-waste, I decided to use the one I already had. The N100DC-ITX is actually a pretty lousy NAS motherboard out of the box, since it doesn’t have a 24-pin ATX input and the stock SATA power connnector will not be able to handle more than two drives. If I were choosing a motherboard for this build from scratch, I’d go with the ASUS N100I-D D4 instead. It only has a PCIe x1 slot, but unlike the board from Asrock, it has a standard 24-pin power connector. However, this motherboard doesn't So I’ve hacked together a DC jack to 4-pin ATX power adapter in my last video.
But, after some of you guys convinced me that the terminal block to DC jack adapter I’ve used in the build is really not a good idea for a permanent project like that, I decided to go full goblin mode, and simply soldered the 4-pin adapter to the motherboard itself. Is this good soldering? Hell no. But it seems to hold pretty well. And in case you’re wondering what soldering iron I’m using, it’s called Pinecil, and I’m gonna leave a link in the description, hashtag not sponsored. I’ve also covered the actual 12v connector on the motherboard with some electric tape So that in case I die, and some poor soul has to maintain this machine, they will at least know that they should not plug anything in there. Hopefully.
For RAM, I’m gonna be using a single 16 gig stick of AEGIS DDR4 memory. The N100 only has a single channel of RAM, and most N100 motherboards come with one memory slot. 16 gigs should be more than enough for running a few docker containers – and this build isn’t really going to be doing anything else. And by the way, according to Intel, 16 gigs should be the maximum amount of RAM you can put into N100 But despite what Intel's official specs say, I’ve heard people say that this motherboard can handle 32 gig sticks just fine as well. So if that was a deal breaker for you – you might want to reconsider.
For the case, I’m gonna be using Jonsbo N2 in white. It has 5 hot swappable hard drive slots, supports SFX power supplies, and is really easy to work on. The motherboard part is completely separated from the rest of the system, which means that you won’t have to remove the hard drives to work on the system, like we did in the Fractal Node 304 or Streacom DA2, for example. The mounting system for storage is actually pretty clever. It doesn’t use caddies to keep the hard drives in place and instead, you’re basically supposed to attach these rubber handles to your drives.
Then, you can simply slide them into the bays, and if you ever want to pull a hard drive out, you just pull on the handle. Apart from that, it’s just a really well built and good looking case. It’s made of powder coated aluminium, and is pretty unassuming for a piece of computer equipment like, it doesn’t scream “nerd alert”, if you know what i mean. So it should have no issues blending into my parents furniture. Now me being me, I couldn’t just give them the stock case. So, I’ve printed a full depth fan adapter for the hard drive cage.
Jonsbo N2 comes with a 15mm thick fan, which is quiet enough, but it could be even quieter. With this mod, we can use a standard thickness 120mm fan, like this one from Arctic. This fan is gonna push more air while producing less noise, which is always good. For power supply, I’m gonna be using this Corsair SF450 Unfortunately, this PSU is not being sold anymore, but if you’re looking for a power efficient alternative – take a look at my power supply efficiency spreadsheet, which I’m gonna link in the description.
As you can see, this power supply has clearly seen some shit, but no worries, that’s just a result of trying to fit it in tight holes that it shouldn’t fit into. And by that, of course, I mean the Supermicro SCC833. On the inside, it’s fully intact and works with no issues whatsoever.
For bulk storage, I’m gonna be using a random mix of 8 and 6 TB drives from Western Digital. Since we’re gonna be using MergerFS, our drives don’t have to be of the same capacity, so I could basically just use any high capacity drives that I had lying around. This will give us 28 terabytes of usable storage, which is more than enough for photos and backups.
Finally, for boot drive, I’ll be using this 256 gigabyte SATA drive that i had lying around. Nothing special about it. Now some of you guys mentioned that this motherboard doesn’t come with IPMI or any kind of remote management functionality.
And I agree. Some way to manage this system remotely would be nice, since there are almost 300 miles between me and my parents. So this is where BliKVM PCIe comes into play.
I’ve already made a video about this little PCIe card, so make sure to check it out if you want to know more about it, but long story short, it’s a KVM board that’s powered by the Raspberry Pi Compute Module 4. The Raspberry Pi itself is running an operating system called PiKVM which then uses an HDMI capture card on the KVM board to show us the HDMI output from the computer that it’s connected to. It also connects to the ATX pins, which can then be used to hard shutdown or reboot the computer remotely from the WebUI. Finally, the board also connects to a USB header on the motherboard, and this lets us control the mouse and the keyboard, and also sideload bootable ISOs and make our system boot from them. And the best part is that it doesn’t even need the PCIe slot to work.
it’s completely decorative, and as long as you have a physical PCIe slot in your case, that you can screw the board into it’s going to work. I did my best to try and cable manage everything, and I’ll let you be judge of how well it turned out. I’ve added a small Noctua fan to the top section of the case, so that there’s a little bit of airflow around the motherboard. And since I’ve no idea how to use 3d modelling software, I simply used strong double-sided tape to mount the fan to the case.
Let’s hope that it holds If you watched my previous video about this motherboard, you’ll know that we actually need to jump the ATX connector manually with a piece of wire, since this motherboard doesn’t have a 24-pin ATX plug. And well, since the SF450 is a modular power supply, we can actually omit the ATX cable altogether and simply jump the pins on the PSU itself, with a custom made sacrificial 4-pin connector. Yes, I know that it looks super cursed, but hey, it works! And here’s what the entire build cost me, including the hard drives. Now. Is this a lot of money? Yes! But you have to realize that unlike many people, I buy hardware to make videos about it, and I then recoup the costs either with the AdSense revenue, or sponsored segments. Some hardware I didn’t even have to buy, because the company sent it to me for free.
And obviously, most people will not have that privilege. Personally, instead of letting the hardware sit on the shelf, I’d rather put it to good use, even if in some cases, it’s totally overkill. But the good news is you probably don’t need to spend one and a half grand on an offsite backup system. Most of the money went into hard drives, and unless you also have 14 tb of very important data to back up, you could probably get away with a couple of 2 tb SSDs at most.
as for the actual build – literally any old PC you might have lying around will do just fine. As long as your parents are okay with integrating it into their house interior. Alright, rant over. After assembling the rest of the system, it was finally time to configure the software on it.
And as you can probably already see by the length of the next chapter, I have a lot to say about that. On the previous iteration of the parents NAS, I used plain old Ubuntu Server that was configured using Ansible. That’s what I was using on my own personal home server at the moment so it kinda made sense to also use the same setup for my parents. And that’s what I’m doing this time as well. Only instead of Ubuntu, or even something like TrueNAS or Unraid, I’ve actually been using a little obscure operating system that you’ve probable never heard about called NixOS.
That’s right, I’ve been using NixOS on my home server for about a year now, and I’ve even started using it on my home router a few months ago. My storage server is a pretty simple setup, with ZFS for the root file system, as well as for my 6TB SSD array, and Mergerfs + SnapRAID for my hard drive storage. I’m using Podman containers for almost all of the services that run on it, and it also runs the usual stuff like SMB. I’ve also written a small bash script for Telegram notifications, which basically tells me the status of scheduled backup jobs, ZFS scrubs and Snapraid operations.
Finally, there are also email notifications about Smart errors, which will definitely come in handy for my parents NAS. As to why I still haven’t made a video about it? Well, as some of you may know, NixOS is a hell of a rabbit hole. And I personally feel like I’m nowhere near the level where I can make a somewhat educational video about NixOS and teach you things about it. I also just don’t think that a Youtube video is a good format for that.
My infrastructure code changes almost every week, and there’s no way for me to go back to a video that I did, and add those changes to it. The entire configuration for my Nix-based machines, including my home server, my router, my parents NAS, my VPS, and even my Macbook to some extent, is on Github, at github.com/notthebee/nix-config so if you’re curious, you can go and take a look at it. But once again, I’m a complete amateur when it comes to Nix, and my configuration only reflects it.
There is a lot of very questionable code in there. In short, I’m sorry if the way I describe my software setup is very surface level, and no, I probably won’t do a deep dive on any of that stuff because one, I don’t know shit about shit, and two, I dont’ think that an hour long video that is frozen in time is a good medium for describing a quickly changing and evolving base of code. All of that aside though, I’ve been really happy with NixOS when it comes to using it in my homelab. And making a configuration for my parents was basically just a matter of copy pasting the configuration for my home server and removing 80% of the services. Fundamentally, it’s the same setup – ZFS for the boot drive, MergerFS and Snapraid for the hard drives, Podman for web applications and Samba for sharing the files.
So I basically modified the configuration, and installed NixOS using the liveCD Which, thanks to declarative partitioning, courtesy of disko, was actually a breeze. For the backup software, I decided to go with Borg. It allows for encrypted, compressed and deduplicating backups over rsync, and the setup, at least on NixOS, was really easy.
Like, here’s the server portion and here’s the client portion. Basically, the server just has to know which SSH keys should have access to the repo and the client has to be told which directories should be backed up, which SSH private key to use and some other paramters, like backup frequency, compression technology and encryption. On NixOS, no additional SSH configuration is required, and the whole thing just works (tm) And yes, the backups are encrypted not just in transit, but at rest, as well. Which makes borg work super well for a use case where you and your friend both have homelabs, and both want to set up offsite backups, but don’t necessarily want the contents of those backups to be stored on your friend’s hard drives unencrypted. So with the backup procedure itself taken care of, This leaves us with just one small question. How are the systems gonna access each other? Well, traditionally, you’d have to punch some holes in your firewall, set up port forwarding and slash or create a peer-to-peer VPN between the two machines.
However, I decided to use today’s sponsor, Tailscale, to connect the two machines together. Full disclosure, Tailscale did sponsor today’s video, but I’ve already been using it in my homelab for two years, as you can see here. If you’ve never heard about Tailscale, it’s basically a zero-configuration mesh VPN software based on Wireguard. You install it on the machines that you want to connect together, enable the service, and that’s it.
All of your systems are now connected in a virtual private network with no need to forward any ports, and with automatically assigned domain names based on their hostnames. But that’s not all You can also use any of the connected systems as an exit point for other systems, you can expose subnets, and if you’re working in a team, Tailscale provides a really robust ACL system, which lets you control who can access which machines and even limit the ports that can be accessed. And if you want to avoid juggling public SSH keys for all of your team members, Tailscale SSH lets you easily integrate SSO und multi-factor authentication into your SSH workflow, making onboarding new users and offboarding the former ones a breeze.
Tailscale is completely free for personal use, with up to 3 users and 100 hosts, and you can check out their business pricing at tailscale.com/wolfgangschannel Configuring Tailscale was pretty easy, and there’s even an official blog post by Xe Laso (probably butchered that) which outlines the configuration for NixOS. This service makes sure that the machine is authenticated with the Tailscale network and re-authenticates the system if necessary. And after starting the service on the new NAS… Voila! It’s now integrated into my Tailscale network.
Let’s verify that the two machines can ping each other, and… Yes! they can. Now even though Tailscale is amazing, I still decided to work out a plan B, which would theoretically let me access the system even if it’s physically turned off, or if I need to change some BIOS settings or reinstall the OS. It consists of the PiKVM, which I’ve talked about earlier, and a Wireguard server running on the FritzBox router. Yes, FritzBox, which is a very popular router brand in Germany, basically lets you set up a Wireguard server right on the router. The set up is super easy, and even takes care of port forwarding for you.
I’ve also set up dynamic DNS using DuckDNS, and now I can still tunnel into my parents network, and get a full KVM access to the NAS machine, even if the Tailscale service isn’t running. But let’s go back to our backups. Now that the two machines are connected in a virtual network, where do we point our backup software? Well, thanks to Tailscale’s MagicDNS, i can simply use the device’s hostname to access it. So for our backup job, i can simply use `aria`, which is the hostname of my parent's NAS, and that will work just fine. No need to hardcode the IP address.
The backup job itself is basically a systemd service, which creates a backup archive with the suffix “failed”, and only “promotes” it to an actual archive if the job succeeded. All of this is set up by NixOS automatically. As you can see, even verifying the backup, with no new data being copied, takes a very long time, since it’s 14 terabytes of data we’re talking about But what about the photo backups? Well, after some of guys suggested Immich, I decided to try it out I’ve set it up as a podman container, using NixOS’s oci-containers module. and it works pretty well! As for sending the pictures from my parents phones to the NAS, I’m still using Syncthing, even though Immich theoretically has an Android app that takes care of photo transfer.
The reason is simple – photo gallery apps change. A couple of years ago, PhotoPrism was all the rage, and now it’s Immich. Two years from now , it might be something else. Syncthing however, has pretty much been with us for more than 10 years and so far, it’s been rock solid for my use case. I’ve set it up to only run when my parents phones are charging, and that way, there’s no extra drain on the battery. So now that I’ve told you about the hardware and the software side of things, let’s compare the cost of my NAS solution with the cost of S3 Deep Glacier Archive – over the span of 5 years.
As I already mentioned, storing 14 terabytes of data in the Deep Glacier Archive would set me back around $25 a month. The raw footage for a single video can take anywhere between 200 gigabytes and 1 terabyte. But sometimes I make less videos, and sometimes I’ll make a simpler video with very little B-Roll.
So let’s take the minimum value of 200 gigabytes. On average, I make 12 videos a year, which means that every month, there will be 200 gigabytes added to the overall stored data. Let’s also charitably assume that Amazon is not going to increase the prices for their S3 products at all.
I mean, it’s not like they’re known for that kind of thing, right? And so, after crunching the numbers, in May of 2029, provided that we don’t all get enslaved by AI, I would have paid 2196 dollars, or 2057 euros! For storing 26 tb of video footage. I mean, that makes even my 1500 euro build look like a pretty good deal in comparison. And since the NAS already has 27 terabytes of useful storage, I would theoretically not even have to buy any new hard drives! Buuut that’s just the upfront cost for hardware. What about energy prices? Well, our NAS currently consumes 17W at idle, which we’ll get back to later, but if we include the very lengthy backups, during which the hard drives actually have to spin up, let’s say that it pulls 30W on average, which would translate into 21.6 killowat hours per month. Now assuming that my parents pay 30 cents per killowatt, and assuming that that price goes up by about one cent every 10 months, after 5 years, this NAS would have cost them 434 euros and 80 cents. Which, if we add this to the upfront cost for our hardware, is still cheaper than paying for the cloud.
But apart from just the costs, this solution has plenty of other advantages. If I end up actually needing the data from S3, I’d have to download it over the Internet with my parents NAS though, I can simply grab a direct attached storage unit, take a train over to my parents, and transfer the data physically. And I also get to see my parents as a bonus. Your mileage may vary though, and depending on your relationship with your parents, this might be a minus and not a plus. This NAS can also double as a photo backup solution, and can theoretically run any self-hosted service that my parents might need in the future. Like Paperless, Jellyfin, Calibri, Booksonic, and so on. The S3-based solution on the other hand, would only take care of my backups.
Now, in order to truly follow the 3-2-1 rule, I’d actually have to backup both to my parents NAS, AND to S3. Having three copies of data will take care of the “3” part, and since the S3 Glacier Deep Archive data is actually stored on tape, I would also have my data on 2 different types of media And even though this is the golden standard of backups, it’s also a helluva lot of data. So for now, I think I’ll just stick to 2-1-1 instead of 3-2-1. However, you probably don’t have 14 terabytes of data to back up, so having something like a Raspberry Pi and an external SSD at your parents house as well as backing up your data to Backblaze B2 is definitely a good idea.
Backblaze B2 is free for up to 10 gigabytes of storage, and they don’t even ask you for your credit card. I have been using it for my less storage-intensive backups for years now. And it’s gotten me out of some pretty scary situations like losing my password manager database. But yeah, I think that’s all I have to tell you guys for today. This video will actually be the first one that I’ll be backing up to this NAS.
and we’ll see how this backup solution fares in the long term. I could say that I’ll make an update video about it in a year or so, but that would be a lie, because I’m actually a very lazy person. Anyway, thank you guys for watching, and as usual, I’d like to thank my patrons
2024-05-31 18:22
