Data Hoarding in an RV - Updated

wretchedghost
Data Hoarding in an RV - Updated

In my previous article of https://blog.lanlocked.xyz/post/data-hoarding-in-an-rv/, I documented how I was using a Synology DS1520+ as my NAS. After fighting with it and the main array of weird issues Synology desires to implement, which I won’t get into here, I have recently sold the setup and gone with a much more robust system, or systems, in my RV.

I have built three new server systems, but first lets start with what hardware I am currently running.

My current onsite, offsite, backups, and cloud schema

Onsite

Physically in my Toy Hauler RV “garage” is my office setup, complete with bed, restroom, and clothes and storage, etc. The large desk is my home gaming rig setup while the smaller one is my daughter’s desk she uses for drawing and gaming.

  • Workstation (wretchedbox)
  • Laptop/Workstation (Daughter’s Alienware which i purchased by trading bullets to a guy at work)
  • Proxmox server (cprox)
    • Jellyfin server (lxc-jelly)
    • Omada TP-Link controller (lxc-omada)
    • Pihole local DNS/Ad blocker (lxc-pihole)
    • Seedbox of “distros” (vm-s33d)
    • etc
  • Debian 12 ZFS backup server (wretchedzfs)
  • Nvidia Shield (home media center)

My workstation

My workstation named wretchedbox specs:

OS: Arch Linux (BTW)
Kernel: arch-zen
Shell: bash
MOBO:
CPU: AMD Ryzen 9 5900X (24) 3.70GHz 12-Core 24-Thread
GPU: AMD ATI Radeon RX 6700XT
RAM: 32GB Corsair Vengeance RGB Pro 2x16GB DDR4 3600HMz C18
EXT4: 
WM: i3
Theme: vimix-dark-amethyst [GTK2/3]
Icons: vimix-amethyst-dark [GTK2/3]
Terminal: urxvt
Terminal Font: bitstream vera sans mon

The case is a Cooler Master Qube 500.

Servers

The Proxmox server labeled cprox consists of:

OS: Proxmox 8
Kernel: x86_64 Linux 6.2.16-3-pve
Shell: bash
MOBO: Asus Prime x370-Pro
CPU: AMD Ryzen 5 5600X 6-Core @ 12x 3.7GHz
RAM: 64GB Corsair (CMK32GX4M2B3200C16) Unbuffered (Unregistered) 4x16GB DDR4 3200MHz
NIC: NetXtreme II BCM57810 10Gb
ZFS:
	rpool (main boot and root partition)
		mirror
			2x 1TB CT100BX500SSD = 2TB usable
			
	fast0 (fast SSD storage for containers)
		raidz1
			4x 500GB Samsung 870 Evo = 1.2TB usable
	superfast0 (superfast NVMe storage for VMs)
		mirror
			2x 1TB Samsung 980 Pro = 1TB usable
	tank0 (proxmox pool)
		mirror
			2x 18TB Seagate Exo = 18TB usable
	vault (bulk storage and backups)	
		raidz1
			6x 12TB Seagate Whitelabel = 50TB usable

This server runs all my containers and VMs as listed above.

The second server labeled wretchedzfs builtin in a Jonsbo N5 12HDD Bay is built on:

OS: Debian 12 bookworm
Kernel: x86_64 Linux 6.1.0-32-amd64
Shell: bash 5.2.15
MOBO: Gigabyte Z390 UD
CPU: Intel Core i5-9400 @ 6x 4.1GHz
RAM: 64GB Fujitsu F4-3200C16-32GVK 2x32GB DDR4 Synchronous 2666HMz
NIC: NetXtreme II BCM57810 10Gb
ZFS:
	tank0
		raidz2
			10x 4TB Seagate IronWolf = 26.7TB usable

This server is only for backups running on ZFS

Offsite:

Workstation

Workstation intel NUC Extreme 9 (whitebox)

OS: Arch Linux (BTW)
Kernel: x86_64 Linux 6.14.4-zen1-2-zen 
Shell: bash 
MOBO: Inetl NUC9i5QNB
CPU: Intel Core i5-9300H @ 8x 4.1GHz
GPU: Nvidia GeForce GTX 1050 Ti
RAM: 32GB SK Hynix 2x16GB SODIMM DDR4 Synchronous @ 2667HMz
NIC: NetXtreme II BCM57810 10Gb
WM: i3
Theme: vimix-dark-amethyst [GTK2/3]
Icons: vimix-amethyst-dark [GTK2/3]
Terminal: urxvt
Terminal Font: bitstream vera sans mon
EXT4:

Server

czfs (non-window) Debian 12 server used for offsite backup:

OS: Debian 12 bookworm
Kernel: x86_64 Linux 6.14.4-zen1-2-zen 
Shell: bash 
MOBO: Gigabyte B365M DS3H
CPU: Intel Core i5-9600 @ 6x 4.6GHz
GPU: Nvidia GeForce GTX 1050 Ti
RAM: 32GB CMW16GX4M2C3200C16 Synchronous Unbuffered (Unregistered) 4x8GB @ 2133MHz
NIC: NetXtreme II BCM57810 10Gb
ZFS:
	tank0
		raidz2
			6x 8TB Seagate IronWolf = 28TB usable

Cloud

Linode Debian 12 server (linode0) (web server) basic tier ($5 a month) which I recommend for a cheap simple server. I have setup domain name hosting between four domains and many more sub-domains.

What I have learned from my previous build

Over a year ago I posted a page on my data hoarding endeavors in an RV. At the time we were mostly thinking we would only be in the RV for about a year but here we are about to go onto three years where my needs of data have changed. I have made many changes from my first post and wanted to do an update.

Since that time I was using a Synology 5 bay as my main NAS but found it very lacking in my desires to run VMs, containers, hardware acceleration, and other things of the sort.

I built a ten HDD bay server rig that runs Proxmox and is my main storage system. I have it backing up to my offsite NAS which is the Synology 5 bay NAS from before using rsync and syncthing. Syncthing is used for most of the datasets or folders I need backed up but also those that don’t change too often or that are not huge.

My ZFS datasets

  • archive
  • datacenter
  • linux
  • games
  • multimedia
    • Media
      • Shows
      • Movies
      • Books
      • Kid Movies
      • Youtube
  • proxmox
    • cprox (current server)
    • wretchedprox (0ld server. Used for archiving)
  • users
  • windows

Between these eight datasets/folders I house all of my files, media, etc that I have collected across the years.

archive is for long archival storage files. Its mostly made up of timestamped tarballs of the other datasets. datacenter-20240523.tgz for example and large_files/enwiki_2022-0101-pages-articles-multistream. Most the backups are made manually by myself with no schedule or scripting done at the moment.

datacenter is the current files directory for all of my computers/servers. Here will house datacenter/prox, datacenter/syno1, or datacenter/lxc-pihole1, which contain the home directory and or the etc directory of those servers.

games is for all my games I own. Some steam games and my GOG collection, emulators, etc.

linux is where I put anything Linux related. Isos, img files, binarys, etc. Since I am a hoarder I have some distros all the way back from 2012 here. I don’t see myself ever installing a distro that old but if I had to or wanted to I could do it without any internet :)

One important thing to do if you did want to run your own old distros is to take a mirror of the apt repo which you can tell your distro of choice (in this case, debian based) to pull files from your internal server instead of the internet which would allow you to run programs if an outage occured or if you didn’t want a computer to touch the internet directly but have a internal server making sure things are good before installing to the air-gapped workstation or server.

proxmox is self explanatory and is where proxmox will use ZFS local directory or NFS directory if it was on a NAS server in the LAN. All the container files and iso images are placed here for whenever I need to run a new VM or LXC. I have a plan to run all proxmox root folders here, one per proxmox instance I have installed, but as of right now the main proxmox houses for all instances, with its pros and cons of doing it this way.

users This is where my most precious things are held. Family photos, notes, personal files, financials, and my syncthing Sync folder that I share with all of my devices.

Windows This folder is the least used as it houses old files and isos of Windows 7, 8, 10, Server 19. I also place programs like firefox, and ccleaner portable version so that I don’t have to download them from the internet.

How I keep it all synced up

I use some simple tools to keep everything linked up. Between my two servers in the RV, those are synced with a custom script using syncoid as the base since they both run ZFS. Updates on files and datasets can be backed up from cprox to wretchedzfs in as little as a few seconds. Here is an example of one dataset:

40 01 * * 2,4,6 /usr/sbin/syncoid --recvoptions=w -r --compress=zstd --no-sync-snap 192.168.10.12:vault/enc/proxmox tank0/encrypted/proxmox >> /var/log/syncoid.log 2>&1 || echo "Replication failed" | mail -s "Syncoid failure - proxmox" <my-email>@gmail.com # Backup of proxmox dataset from cprox to local. Create a log under /var/log/syncoid.log and email to my email only if failed. No news is good news.

The options used above are as followed:

  • –recvoptions=w is needed for when the dataset is encrypted
  • -r is recursive
  • –compress=zstd isn’t actually implemented on syncoid yet but one day it might be. Should default to lz4.
  • –no-sync-snap Some people might or might not want this. From the Github: This argument tells syncoid to restrict itself to existing snapshots, instead of creating a semi-ephemeral syncoid snapshot at execution time. Especially useful in multi-target (A->B, A->C) replication schemes, where you might otherwise accumulate a large number of foreign syncoid snapshots.

Pushing updates or changes is much more involved due to me being on Starlink with its abyssmal upload throughput to my offsite locations. Normally with a large update or change, I will use an external HDD or in my case a spare HDD, place on a dock, and transfer the updated files from my server to that HDD. Then doing the same on the offsite location but pushing the changes using rsync with the –delete flag.

One example of this scenario:

  1. Place HDD in dock on cprox (source)
  2. Use cryptsetup –luksOpen to open encrypted HDD
sudo cryptsetup --luksOpen /dev/sdx <name-you-want-to-use>
sudo mount /dev/mapper/<name-used> /mnt/hdd # /mnt/hdd must exist
  1. Copy changed files or datasets to HDD.
sudo rsync -avP /mnt/vault/enc/<dataset-or-files> /mnt/hdd/
  1. Unmount and re-lock
sudo umount /mnt/hdd
sudo cryptsetup --luksClose <named-used-above>
  1. Head to location where czfs (destination) is
  2. Run step 2
  3. Copy files or datasets from from HDD to czfs
sudo rsync -avP /mnt/hdd/<dataset-or-files> --delete /mnt/tank0/enc/
  1. Run step 4

Incrementals of rsync will now run much faster for smaller changes. I have not attempted yet but running the above syncoid cronjob on the offsite server should run much faster once the initial snapshot has been run.

How its all tied together

I use a few more tools that can help keep things synced up and easy to get to from virtually anywhere in the world. Using Tailscale is a marvelous tool. There are options to self host a tailscale setup but I have yet to dive into that one.

Tailscale for the un-initiated is a wireguard cloud hosted network that can link all of you devices together on the same subnet over the internet. Suffice it to say there is much more invovled than that but in this post its good enough to know that it will link up many devices together on their own public IP address. It will also setup a domain name based on the hostname of the computer or server you setup meaning you can ssh via hostname over the internet.

Another great tool is Syncthing. This tool allows you to sync two more devices beit computer, server, or phone and sync folders between them. I have found that conflicts can exist between these devices which are smart enough to not overwrite the file but create a separate file labeled conflict. For the most part transfering smaller files work great. Larger files can be troublesome which is where rsync is better used for this task.