Here is my single consolidated home server for NAS and VM host.
Specs are 5900x/128GB with 36TB HDD & 1.5TB SSD, a LSI 9607-8i, and Mellanox ConnectX-3 40Gb.
It runs Fedora, and all services are each in their own full Fedora VM.
I manage the VMs using virt-manager, and I keep template VMs for different purposes.
Configuration is generally the same as what I document here: Infrastructure - Divested Computing
Largely through Brace: Divested Computing Group / Brace · GitLab
I take a bit of a different path for network storage, it is solely OpenSSH based with dedicated nologin/chrooted users with only public key authentication.
I plan to write a guide up on that eventually, but here are the basics:
Script to create a user, use groupadd sftponly once only, then this:
#!/bin/sh
cd /mnt/share-1;
useradd -G sftponly -s /bin/nologin -d /mnt/share-1/$1 $1;
chown -v root:$1 $1;
chmod -v g+rx $1;
cd $1;
mkdir .ssh data backup sync;
chown -v $1:$1 . -R;
systemctl restart sshd;
You’ll also need this for Fedora’s selinux policy:
semanage fcontext -a -e /home /mnt/share-1
restorecon -rv /mnt/share-1
The sshd_config bits
Match Group sftponly
ChrootDirectory %h
ForceCommand internal-sftp
#brace
AllowTcpForwarding no
GatewayPorts no
PermitListen none
PermitOpen none
PermitTTY no
PermitTunnel no
PermitUserRC no
X11Forwarding no
#Don't require 2FA
AuthenticationMethods publickey
ChallengeResponseAuthentication no
additionally if you want to allow use of borg:
Match Group borgonly
ChrootDirectory %h
ForceCommand borg serve --restrict-to-path $HOME --append-only
AllowTcpForwarding no
PasswordAuthentication no
PermitEmptyPasswords no
X11Forwarding no
and on the client end for /etc/fstab:
user@server:/data/ /mnt/remote fuse.sshfs rw,defaults,allow_other,noauto,x-systemd.automount,_netdev,noatime,nosuid,nodev,reconnect,dir_cache=yes,max_conns=4,ConnectTimeout=20,ServerAliveInterval=10,Compression=no,Ciphers=aes256-gcm@openssh.com,IdentityFile=/home/user/.ssh/id_ed25519.whatever,uid=1000,gid=1000 0 0
Other things:
- I personally use restic however for all backups of all machines.
- I use ZFS on the drives, I have a cheat sheet here: https://divested.dev/misc/zfs.txt
- All the hard drives are used enterprise SATA/SAS 6TB ones from eBay, they’re only $28-35 and I’ve only had about 1/13 fail on me so far. Even if one fails, it is only $30 to replace, as opposed to expensive larger sized drives. But you’ll have to do the TCO analysis for your usecase.
- If you’re buying LSi cards, be sure to avoid fakes, there are some videos on YouTube comparing them.
- Be sure you allocate enough PCIe bandwidth to your cards, do the math before buying, etc.
- The Mellanox CX3 is an extremely cheap way to get high speed 10g/40g networking, they’re both $30 on eBay. Combine it with a quiet/efficient 10g ICX6450 ($80) or louder/power hungry 10/40g ICX6610 ($50) for switching. Or just direct attach even, great thread here: Brocade ICX Series (cheap & powerful 10gbE/40gbE switching) | ServeTheHome Forums
- Make backups, always make backups, do not push them off.
- Make backups of your backups.
- And actually verify your backups work!
- For passing folders into a VM, virtiofs is amazing for most use cases, something that you should try if you haven’t.
