Building a 26TB Offsite Backup Server! ft. Tailscale

Building a 26TB Offsite Backup Server! ft. Tailscale

Show Video

Despite many protests from long time  viewers, I still make Youtube videos And just like many other creators,  I have terabytes of video footage. Mostly of me fumbling my lines. But if you’ve been watching  my videos for some time now, You’ll notice that I reuse  quite a lot of B-Roll footage, And so do other youtubers.  Sorry for ruining the magic. Now, some video creators have  dozens of external hard drives, some have a dedicated off-the-shelf NAS. Personally, I store it on my home server. It’s got two 16 TB hard  drives in a MergerFS array, as well as another 16 TB drive  dedicated to Snapraid parity data.

So I can theoretically lose one drive  out of three and not lose any data. And this data is very important to me. Having all of the original footage  from my videos is very useful, It lets me reference my previous  videos in my new content, without having to resort to  downloading my own videos from Youtube in a super compressed format with subpar quality. Besides, there are plenty of situations  where I would film something for a video, and then just not end up  including it in the final product. If I still have the original file, I can  actually use it for another video in the future, instead of it just going to waste. So how do we keep important data safe? That’s right, backups.

You’ve probably heard of the 3-2-1  rule when it comes to backups. You should have 3 copies of your important data. On two different types of media one of which is off site. Meaning,  in this case, as Aaron Paul puts it, … Now, these days, the most popular type  of offsite backup is cloud storage.

Be that just putting your data  manually into Dropbox or Google Drive, or using fancy backup software  like Restic or Duplicati and sending your data to an S3-compatible  storage endpoint. Like Backblaze B2 And that’s what I do for some  of my other important data, such as personal documents and  my password manager database. But when it comes to video footage… Well, I shoot these videos in ProRes,  and the entire archive folder, with all of the raw footage that  I’ve accumulated over the years, is currently at 14 TB.

So, is it really feasible to  store that much data in the cloud? Well, surprisingly, yes! AWS offers an S3 Tier called Glacier Deep Archive, which currently goes for $0.0018  dollars per gigabyte of storage. So if my math is correct, at 14 terabytes, this would  set me back 25 dollars a month. As well as a one time fee of 70 dollars,  should I ever need to retrieve the data. And as far as business expenses go,  25 dollars a month is not that much. After all, it’s backups of very  important data we’re talking about here. The real problem with the  cloud are transfer speeds.

Germany is not really known for its fast Internet, and at the moment, I got 250 mbits  down and 50 mbits up at my house. Uploading 14 terabytes of data at 50  mbits per second will take me 25 days. And downloading it will take 5 days. Plus, depending on AWS for my business  is not something I want to do. You see more and more companies being burned by  depending on the public cloud infrastructure, and I prefer to avoid using the cloud if i can.

So, is there a different way? Yes! And all we’re gonna need is  another NAS, a few hard drives,   and parents with Internet connection. Now some people might say that I shamelessly stole   the idea of offsite backups to my  parents house from Hardware Haven But that’s just not true. In his video on offsite backups, Colten  backed the data up to his parents house, whereas me, well, I’m gonna be backing  the data up to my parents house. Which are two completely different things.

Me and Colton do however have the same sponsor  for our parents NAS videos, and that is Tailscale. I guess that similarly to how Squarespace  became the de-facto podcast sponsor, Tailscale is trying to become the official  sponsor of off-site backups to Youtubers’ parents. So definitely be on a look out for more  parents NAS videos sponsored by Tailscale. In all seriousness, the idea actually came from  a few comments under my first Parents NAS video that I did in 2022. And just like the previous build, this new machine is also going to double as a  backup target for my parents vacation photos. So first things first, let’s  talk about the build itself! For the motherboard, I’m gonna be using  Asrock N100DC-ITX from my last video.

For 130€, you get a motherboard plus CPU combo, which is fairly power efficient, and supports  hardware video transcoding in Plex and Jellyfin. Which my parents probably  won’t need, but you might. Now to be fair, the only  reason I bought the N100DC-ITX is the fact that it has a PCIe x2 slot, which would be perfect for my personal  use, since I need 10 gigabit networking.

But then the idea of offsite backups  kind of resurfaced in my mind, and instead of buying a more appropriate  motherboard and creating more e-waste, I decided to use the one I already had. The N100DC-ITX is actually a pretty  lousy NAS motherboard out of the box, since it doesn’t have a 24-pin ATX input and the stock SATA power connnector will  not be able to handle more than two drives. If I were choosing a motherboard  for this build from scratch, I’d go with the ASUS N100I-D D4 instead. It only has a PCIe x1 slot, but unlike the board from Asrock, it has a standard 24-pin power connector. However, this motherboard doesn't So I’ve hacked together a DC jack to  4-pin ATX power adapter in my last video.

But, after some of you guys convinced me that the terminal block to DC jack  adapter I’ve used in the build is really not a good idea for  a permanent project like that, I decided to go full goblin mode,   and simply soldered the 4-pin  adapter to the motherboard itself. Is this good soldering? Hell no. But it seems to hold pretty well. And in case you’re wondering  what soldering iron I’m using, it’s called Pinecil, and I’m gonna leave a  link in the description, hashtag not sponsored. I’ve also covered the actual 12v connector  on the motherboard with some electric tape So that in case I die, and some poor  soul has to maintain this machine, they will at least know that they  should not plug anything in there. Hopefully.

For RAM, I’m gonna be using a single  16 gig stick of AEGIS DDR4 memory. The N100 only has a single channel of RAM, and  most N100 motherboards come with one memory slot. 16 gigs should be more than enough for  running a few docker containers – and   this build isn’t really going  to be doing anything else. And by the way, according to Intel, 16 gigs should be the maximum amount of RAM you can put into N100 But despite what Intel's official specs say, I’ve heard people say that this motherboard  can handle 32 gig sticks just fine as well. So if that was a deal breaker for  you – you might want to reconsider.

For the case, I’m gonna be  using Jonsbo N2 in white. It has 5 hot swappable hard drive  slots, supports SFX power supplies, and is really easy to work on. The motherboard part is completely  separated from the rest of the system, which means that you won’t have to remove  the hard drives to work on the system, like we did in the Fractal Node  304 or Streacom DA2, for example. The mounting system for storage  is actually pretty clever. It doesn’t use caddies to  keep the hard drives in place and instead, you’re basically supposed to  attach these rubber handles to your drives.

Then, you can simply slide them into the bays, and if you ever want to pull a hard  drive out, you just pull on the handle. Apart from that, it’s just a really  well built and good looking case. It’s made of powder coated aluminium, and is pretty unassuming for  a piece of computer equipment like, it doesn’t scream “nerd alert”, if you know what i mean. So it should have no issues  blending into my parents furniture. Now me being me, I couldn’t just give them the stock case. So, I’ve printed a full depth fan  adapter for the hard drive cage.

Jonsbo N2 comes with a 15mm  thick fan, which is quiet enough, but it could be even quieter. With this mod, we can use a standard thickness  120mm fan, like this one from Arctic. This fan is gonna push more air while  producing less noise, which is always good. For power supply, I’m gonna  be using this Corsair SF450 Unfortunately, this PSU is not being sold anymore, but if you’re looking for a  power efficient alternative – take a look at my power  supply efficiency spreadsheet, which I’m gonna link in the description.

As you can see, this power supply  has clearly seen some shit, but no worries, that’s just a result of trying to  fit it in tight holes that it shouldn’t fit into. And by that, of course, I  mean the Supermicro SCC833. On the inside, it’s fully intact  and works with no issues whatsoever.

For bulk storage, I’m gonna be using a random  mix of 8 and 6 TB drives from Western Digital. Since we’re gonna be using MergerFS, our  drives don’t have to be of the same capacity, so I could basically just use any high  capacity drives that I had lying around. This will give us 28 terabytes of usable storage, which is more than enough for photos and backups.

Finally, for boot drive, I’ll be using this 256  gigabyte SATA drive that i had lying around. Nothing special about it. Now some of you guys mentioned that  this motherboard doesn’t come with IPMI or any kind of remote management functionality.

And I agree. Some way to manage  this system remotely would be nice, since there are almost 300  miles between me and my parents. So this is where BliKVM PCIe comes into play.

I’ve already made a video  about this little PCIe card,   so make sure to check it out if  you want to know more about it, but long story short, it’s a KVM board that’s  powered by the Raspberry Pi Compute Module 4. The Raspberry Pi itself is running  an operating system called PiKVM which then uses an HDMI capture card  on the KVM board to show us the HDMI   output from the computer that it’s connected to. It also connects to the ATX pins,   which can then be used to hard shutdown or  reboot the computer remotely from the WebUI. Finally, the board also connects  to a USB header on the motherboard, and this lets us control  the mouse and the keyboard, and also sideload bootable ISOs  and make our system boot from them. And the best part is that it doesn’t  even need the PCIe slot to work.

it’s completely decorative, and as long as you have a  physical PCIe slot in your case, that you can screw the board into it’s going to work. I did my best to try and cable manage everything, and I’ll let you be judge  of how well it turned out. I’ve added a small Noctua fan  to the top section of the case, so that there’s a little bit  of airflow around the motherboard. And since I’ve no idea how  to use 3d modelling software, I simply used strong double-sided  tape to mount the fan to the case.

Let’s hope that it holds If you watched my previous  video about this motherboard, you’ll know that we actually need to jump the  ATX connector manually with a piece of wire, since this motherboard doesn’t  have a 24-pin ATX plug. And well, since the SF450  is a modular power supply, we can actually omit the ATX cable altogether and simply jump the pins on the PSU itself, with a custom made sacrificial 4-pin connector. Yes, I know that it looks super cursed, but hey, it works! And here’s what the entire build  cost me, including the hard drives. Now. Is this a lot of money? Yes! But you have to realize that unlike many people, I buy hardware to make videos about it, and I then recoup the costs either with the AdSense revenue, or sponsored segments. Some hardware I didn’t even have to buy,  because the company sent it to me for free.

And obviously, most people  will not have that privilege. Personally, instead of letting  the hardware sit on the shelf, I’d rather put it to good use, even if  in some cases, it’s totally overkill. But the good news is you probably don’t need   to spend one and a half grand  on an offsite backup system. Most of the money went into hard drives,   and unless you also have 14 tb of  very important data to back up, you could probably get away with  a couple of 2 tb SSDs at most.

as for the actual build – literally any old PC  you might have lying around will do just fine. As long as your parents are okay with  integrating it into their house interior. Alright, rant over. After assembling the rest of the system, it was  finally time to configure the software on it.

And as you can probably already see  by the length of the next chapter, I have a lot to say about that. On the previous iteration of the parents  NAS, I used plain old Ubuntu Server that was configured using Ansible. That’s what I was using on my own  personal home server at the moment so it kinda made sense to also  use the same setup for my parents. And that’s what I’m doing this time as well. Only instead of Ubuntu, or even  something like TrueNAS or Unraid, I’ve actually been using a little obscure   operating system that you’ve  probable never heard about called NixOS.

That’s right, I’ve been using NixOS on  my home server for about a year now, and I’ve even started using it on  my home router a few months ago. My storage server is a pretty simple setup, with ZFS for the root file system, as well as for my 6TB SSD array, and Mergerfs + SnapRAID for my hard drive storage. I’m using Podman containers for almost  all of the services that run on it, and it also runs the usual stuff like SMB. I’ve also written a small bash  script for Telegram notifications, which basically tells me the  status of scheduled backup jobs, ZFS scrubs and Snapraid operations.

Finally, there are also email  notifications about Smart errors, which will definitely come  in handy for my parents NAS. As to why I still haven’t made a video about it? Well, as some of you may know,  NixOS is a hell of a rabbit hole. And I personally feel like I’m  nowhere near the level where   I can make a somewhat educational video  about NixOS and teach you things about it. I also just don’t think that a Youtube  video is a good format for that.

My infrastructure code changes almost every week,   and there’s no way for me to go back to a  video that I did, and add those changes to it. The entire configuration for my Nix-based  machines, including my home server,   my router, my parents NAS, my VPS,  and even my Macbook to some extent, is on Github, at so if you’re curious, you  can go and take a look at it. But once again, I’m a complete  amateur when it comes to Nix, and my configuration only reflects it.

There is a lot of very questionable code in there. In short, I’m sorry if the way I describe  my software setup is very surface level, and no, I probably won’t do a deep  dive on any of that stuff because one, I don’t know shit about shit, and two, I dont’ think that an hour  long video that is frozen in time is a good medium for describing a quickly  changing and evolving base of code. All of that aside though,   I’ve been really happy with NixOS when  it comes to using it in my homelab. And making a configuration  for my parents was basically   just a matter of copy pasting the  configuration for my home server and removing 80% of the services. Fundamentally, it’s the same  setup – ZFS for the boot drive, MergerFS and Snapraid for the hard drives, Podman for web applications and  Samba for sharing the files.

So I basically modified the configuration,  and installed NixOS using the liveCD Which, thanks to declarative partitioning,  courtesy of disko, was actually a breeze. For the backup software,  I decided to go with Borg. It allows for encrypted, compressed  and deduplicating backups over rsync, and the setup, at least on NixOS, was really easy.

Like, here’s the server portion and here’s the client portion. Basically, the server just has to know which  SSH keys should have access to the repo and the client has to be told which  directories should be backed up, which SSH private key to use and some other paramters, like backup frequency,  compression technology and encryption. On NixOS, no additional SSH configuration is  required, and the whole thing just works (tm) And yes, the backups are  encrypted not just in transit, but at rest, as well. Which makes borg work super well for  a use case where you and your friend both have homelabs, and both  want to set up offsite backups, but don’t necessarily want the  contents of those backups to be   stored on your friend’s hard drives unencrypted. So with the backup procedure itself taken care of, This leaves us with just one small question. How are the systems gonna access each other? Well, traditionally, you’d have to  punch some holes in your firewall, set up port forwarding and slash or create a peer-to-peer  VPN between the two machines.

However, I decided to use today’s sponsor,  Tailscale, to connect the two machines together. Full disclosure, Tailscale  did sponsor today’s video, but I’ve already been using it in my  homelab for two years, as you can see here. If you’ve never heard about Tailscale,   it’s basically a zero-configuration  mesh VPN software based on Wireguard. You install it on the machines that you want  to connect together, enable the service, and that’s it.

All of your systems are now connected  in a virtual private network with no need to forward any ports, and with automatically assigned  domain names based on their hostnames. But that’s not all You can also use any of the connected  systems as an exit point for other systems, you can expose subnets, and if you’re working in a team, Tailscale provides a really robust ACL system, which lets you control who can access which   machines and even limit the  ports that can be accessed. And if you want to avoid juggling public  SSH keys for all of your team members, Tailscale SSH lets you easily  integrate SSO und multi-factor   authentication into your SSH workflow, making onboarding new users and  offboarding the former ones a breeze.

Tailscale is completely free for personal  use, with up to 3 users and 100 hosts, and you can check out their business pricing  at Configuring Tailscale was pretty easy, and there’s even an official blog post  by Xe Laso (probably butchered that) which outlines the configuration for NixOS. This service makes sure that the machine  is authenticated with the Tailscale network and re-authenticates the system if necessary. And after starting the service on the new NAS…   Voila! It’s now integrated  into my Tailscale network.

Let’s verify that the two  machines can ping each other, and… Yes! they can. Now even though Tailscale is amazing, I still decided to work out a plan B, which would theoretically let me access the  system even if it’s physically turned off, or if I need to change some BIOS  settings or reinstall the OS. It consists of the PiKVM, which  I’ve talked about earlier, and a Wireguard server running  on the FritzBox router. Yes, FritzBox, which is a very  popular router brand in Germany, basically lets you set up a  Wireguard server right on the router. The set up is super easy, and even  takes care of port forwarding for you.

I’ve also set up dynamic DNS using DuckDNS, and now I can still tunnel  into my parents network, and get a full KVM access to the NAS machine,  even if the Tailscale service isn’t running. But let’s go back to our backups. Now that the two machines are  connected in a virtual network, where do we point our backup software? Well, thanks to Tailscale’s MagicDNS, i can simply use the device’s  hostname to access it. So for our backup job, i can simply use `aria`,  which is the hostname of my parent's NAS, and that will work just fine. No  need to hardcode the IP address.

The backup job itself is  basically a systemd service, which creates a backup archive  with the suffix “failed”,   and only “promotes” it to an actual  archive if the job succeeded. All of this is set up by NixOS automatically. As you can see, even verifying the  backup, with no new data being copied, takes a very long time, since it’s 14 terabytes of  data we’re talking about But what about the photo backups? Well, after some of guys suggested  Immich, I decided to try it out I’ve set it up as a podman container,  using NixOS’s oci-containers module. and it works pretty well! As for sending the pictures from  my parents phones to the NAS, I’m still using Syncthing, even though  Immich theoretically has an Android app that takes care of photo transfer.

The reason is simple – photo gallery apps change. A couple of years ago, PhotoPrism was all the rage, and now it’s Immich. Two years from now , it might be something else. Syncthing however, has pretty much  been with us for more than 10 years and so far, it’s been rock solid for my use case. I’ve set it up to only run when  my parents phones are charging, and that way, there’s no  extra drain on the battery. So now that I’ve told you about the  hardware and the software side of things, let’s compare the cost of my NAS solution  with the cost of S3 Deep Glacier Archive – over the span of 5 years.

As I already mentioned, storing 14 terabytes   of data in the Deep Glacier Archive  would set me back around $25 a month. The raw footage for a single video can take  anywhere between 200 gigabytes and 1 terabyte. But sometimes I make less videos, and sometimes  I’ll make a simpler video with very little B-Roll.

So let’s take the minimum value of 200 gigabytes. On average, I make 12 videos a  year, which means that every month,   there will be 200 gigabytes  added to the overall stored data. Let’s also charitably assume that Amazon is not   going to increase the prices  for their S3 products at all.

I mean, it’s not like they’re known  for that kind of thing, right? And so, after crunching the numbers, in May of 2029, provided that  we don’t all get enslaved by AI, I would have paid 2196 dollars, or 2057 euros! For storing 26 tb of video footage. I mean, that makes even my 1500 euro build  look like a pretty good deal in comparison. And since the NAS already has  27 terabytes of useful storage, I would theoretically not even  have to buy any new hard drives! Buuut that’s just the upfront cost for hardware. What about energy prices? Well, our NAS currently consumes 17W  at idle, which we’ll get back to later, but if we include the very lengthy backups, during which the hard drives  actually have to spin up, let’s say that it pulls 30W on average, which would translate into  21.6 killowat hours per month. Now assuming that my parents  pay 30 cents per killowatt,   and assuming that that price goes up  by about one cent every 10 months, after 5 years, this NAS would have  cost them 434 euros and 80 cents. Which, if we add this to the  upfront cost for our hardware,   is still cheaper than paying for the cloud.

But apart from just the costs, this solution has plenty of other advantages. If I end up actually needing the data from  S3, I’d have to download it over the Internet with my parents NAS though, I can simply  grab a direct attached storage unit, take a train over to my parents, and transfer the data physically. And I  also get to see my parents as a bonus. Your mileage may vary though, and depending  on your relationship with your parents,   this might be a minus and not a plus. This NAS can also double  as a photo backup solution, and can theoretically run any self-hosted  service that my parents might need in the future. Like Paperless, Jellyfin,  Calibri, Booksonic, and so on. The S3-based solution on the other hand, would only take care of my backups.

Now, in order to truly follow the 3-2-1 rule, I’d actually have to backup  both to my parents NAS, AND to S3. Having three copies of data  will take care of the “3” part, and since the S3 Glacier Deep Archive  data is actually stored on tape, I would also have my data on  2 different types of media And even though this is the  golden standard of backups, it’s also a helluva lot of data. So for now, I think I’ll just  stick to 2-1-1 instead of 3-2-1. However, you probably don’t  have 14 terabytes of data to back up, so having something like a Raspberry Pi and an  external SSD at your parents house as well as backing up your data to Backblaze B2 is definitely a good idea.

Backblaze B2 is free for up  to 10 gigabytes of storage, and they don’t even ask you for your credit card. I have been using it for my less  storage-intensive backups for years now. And it’s gotten me out of  some pretty scary situations like losing my password manager database. But yeah, I think that’s all I  have to tell you guys for today. This video will actually be the first  one that I’ll be backing up to this NAS.

and we’ll see how this backup  solution fares in the long term. I could say that I’ll make an update  video about it in a year or so, but that would be a lie, because  I’m actually a very lazy person. Anyway, thank you guys for watching, and as usual, I’d like to thank my patrons

2024-05-31 18:22

Show Video

Other news