I’ve been running a small home server for several years now, and I’ve started to wonder whether it’s time to admit that I’m not as knowledgeable in this area as I’d like to be and should seek some more experienced opinions.
I know every setup is different and depends heavily on the applications and services being run, but I’m looking for a general idea and some perspective. What does your base system look like?
For example, I’m using Proxmox as a hypervisor and running several Debian LXCs on top of it.
There is another post in the community but in my opinion it’s a bit too generic. While every setup is built around individual requirements, I’m curious whether most experienced privacy security conscious self-hosters eventually settle on a similar foundation. Not sure if the consensus nowadays is to run BSD instances for example.
I have my two main servers with AlmaLinux. one is just for me and my personal data, the other is my media server for the seven seas, and apps for my friends to access.
I also have a few Raspberry PIs with DietPi as servers for things like my WireGuard server, backup AdGuard Home instance, PeaNut for UPS monitoring, and PiKVM
I use umbrel. It’s probably not the most secure or private and it’s kind of stupid to use because most everything I host is in Portainer.
But it’s nice to use because it handles all the network side and it’s all GUI. I would recommend it because it will give you a confidence boost, but you’ll quickly realize it’s not usable with many of the regular installable apps because you can’t customize them enough.
Portainer is like a GUI docker app afaik and I use AI to help customize the docker compose files. I do host a few things that are installable from the Umbrel appstore.
Some of my favourite things that I host are an onion service Monero node, a snowflake proxy, and Ente Photos.
I might switch to proxmox at some point because I imagine it has better security, but it would probably be a lot of learning when I already have a functioning setup.
You can easily go too simple or too complex. Whatever you’re comfortable with handling would be the right choice for you, I think.
For the last few years I’ve had a (closed) laptop with Manjaro running a bunch of docker compose-based containers (bewCloud, Gitea, etc.). The compose files (and any specific configs) are git-versioned, and the data is rsync’d daily to a Hetzner storage box and a local external disk. I find that easy enough to manage, maintain, etc.
First, I’d like to thank everyone for their input.
I completely understand the argument that the “best” setup is often the one you’re comfortable maintaining. Reliability and operational familiarity probably matter more than chasing the perfect stack.
That said, I’m still curious about the foundation underneath these setups. In your case, for example, you’re running Manjaro as the host OS. Do you feel that has been a good long-term foundation for a server?
One thing that sparked this question is that Privacy Guides, according to their recommendations, isn’t very enthusiastic about Manjaro on the desktop side. I realize server requirements are different, but it makes me wonder whether there is any broader consensus around server operating systems.
For example, when people are building self-hosted infrastructure today, do they tend to gravitate toward Debian, Alpine, AlmaLinux, Rocky Linux, one of the BSDs, or something else entirely? I’m interested in understanding whether experienced privacy and security conscious self-hosters eventually converge on certain foundations, even if the services they run are completely different.
Edit: I wish PG would have a new recommendation page for servers.
Of course, everyone has different requirements and needs when choosing a Linux distro for their server. Most people may choose and settle with the distros you mentioned or derivatives of those distros (like DietPi), or Proxmox for specific use cases for long term. While I an privacy and security conscious, I eventually settled with AlmaLinux since it is in the same family as Fedora, and hasn’t caused me any issues unlike Fedora. While AlmaLinux works for me, it may not work for others. It usually depends on trial and error, distro-hopping, and seeing what distro works after some time
I am also currently using Proxmox. On my todo list is migration to Incus:
works on every Linux distribution (vs. Debian only)
Community-driven project under non-profit Linux Containers organization (vs. for-profit Proxmox Server Solutions GmbH company and paid subscription for early security updates)
Maintained by same team that already engineered LXD
Similar feature set as Proxmox VE, given home-lab case
Growing popularity: For example recent TrueNAS versions already include it under the hood for VM and container management
Disadvantage - depending on your skills - is, that it currently is CLI only mainly. They also started to include a simple web interface, forked from Canonical/LXD, but which arguably is not as mature as the Proxmox one.
Basically where I’m at, too. Household needs are mostly frontends and Immich, so old laptop running Docker compose-based containers does the job perfectly.
Great question. I don’t know why PG isn’t very enthusiastic about Manjaro, maybe I am missing something there. I’ve used many distros before and felt that it’s easier to maintain rolling release ones (they’re generally more stable and predictable, much fewer “big updates” that tend to break things unexpectedly). Manjaro just seems to have stuck as the best one for my setup (laptop and apps), probably. I have it with Gnome though it’s not necessarily my favorite UI, it’s just easier for anyone else in my family to go in and extract data if I die, for example (I have some clear instructions on how to easily move things back to private clouds, since it’s unlikely someone else will manage this long term in that case). So, it’s personal and practical reasons, really, which will highly vary per person and situation.
Base should be a well supported os your familiar with. Generally avoid derivatives as they can have patch lag. But systems built on top of others without said patch lag, like proxmox, are fine. This base should have nothing outside of the base server distro that it doesn’t need. If you go with proxmox, you should check out level1techs forum.
If you have more nodes than your laptop and server, you should use configuration mangement like ansible. Even if its just your server and router. There are many reasons for this. The ansible folder acts as verified documentation of your home labs state. Its less data to backup an installer and a few playbooks than system images. You can manage the ansible folder with git or your choice of version control and have a history, with branches if you want. Ansible is easy to use and works over ssh. It does not need an agent running on the managed node. And it can manage windows. So its great for home labs.
If you have a gaming machine, i wouldn’t ansible that unless its running from your hypervisor.
For a router, a base distro is more manageable and versatile than a router os. In your case, I’d probably go with debian since thats what your familiar with. The bsds are nice too. PF is easy to use. Pick one with a team of people whos job it is to keep up with patches. There are ansible collections for pfsense and openwrt.
My laptop runs qubes-os. Home server changes linux distro once in a while but usually fedora, debian or ubuntu. Arch is on my to try list. Its mostly a development server. An old machine runs bazzite for gaming, blender, and movies.
As you mentioned “immutable OS”: Nice about Incus is that it is just a package to install (on Debian: apt install incus). Unlike Proxmox it doesn’t depend on a bundled OS and architecture (ARM and RISC builds available, too). They still provide a minimal base OS IncusOS , described as:
IncusOS is an immutable OS solely designed around safely and reliably running Incus. It uses modern security features like UEFI Secure Boot and TPM to provide a safe boot experience and seamless full disk encryption.
To be honest, major upgrades on Proxmox incl. Debian customizations like LVM over LUKS always felt painful and risky, so it is nice to have choices.
If I am not mistaken, proxmox disks can be directlry migrates over to Incus via QCOW2 exchange format (given same cpu arch).
I don’t think the specific distribution matters a lot as long as it’s a major Linux distribution. I use Debian for what it’s worth. I don’t use Proxmox or other convenience tools, and usually avoid virtual machines. I run most services either bare metal or in Podman containers. When I occasionally need a VM, I just write a short script that launches QEMU.
Yes, it was hypervisor version (IncusOS) that I was looking at specifically.
I used to spin up VMs through virt-manager on my workstation (not server) for many different programs, but since moved to QubesOS as it just does the dirty work for you.
IncusOS appears a similar solution that is not too bloated. My initial concerns would be lack of coverage online.
Proxmox for me, when I last tried it just seemed like it was trying to do too much and I had a lack of control.
Cockpit (developed by Fedora but available on most OSes) is pretty good as a fairly lean web GUI for managing server and VMs. You still need to fall back to the command line but it’s just easier opening and mounting LUKS volumes, for example.
Talking of sandboxing, anyone remember SandboxIE, or Appguard? The days of trying to tame Windows
Replied to this a few days ago but it didn’t go through I guess, typing it again
Currently on Debian with Docker, but I found this article, and MicroOS with Podman seems amazing! You can enable automatic updates for both the server and pods/containers, the Podman containers are rootless so way less attack surface in case any maintainer of your containers gets hacked, and it even rolls itself back if an update makes it not boot , since it is immutable AND rolling release
My primary home server runs Proxmox, with several Debian VMs, each running their own isolated application. Notably, I do not use LXCs because the process isolation is not as robust as a fully-fledged VM. It has some small downsides, but is easier for me to reason about, and more consistent with the environment that most installation instructions offer (that of a bare machine, running only the specified application).
I avoid Docker wherever possible, and if necessary just install application binaries directly. Docker can hide a lot of complexity and configuration that I’d rather know about (since I’m running the service, I need to have an idea of how to fix it). Additionally, the process isolation of Docker is even worse than LXCs. There have been many, many issues with Docker applications being able to escape the sandbox.
My secondary home server is on a Raspberry Pi 4B 4GB that I got second-hand, and it obviously cannot easily run Proxmox—so for this box, I use only traditional process isolation (i.e. file permissions, systemd hardening). This box is also behind CG-NAT, so I have to use a WireGuard tunnel to a public server where the NGINX proxy for the services on the raspi is hosted (I use a $2/mo IONOS box, though I could use my other server… something for me to ponder).
Doing things without Docker, LXCs, or .deb packages is a bit more manual, but I’ve been doing it for a while and it’s pretty much second nature at this point. Install the binary, create the /etc and /var/lib directories, write the systemd service, voíla. It allows much more granular control over each service, and I can ensure that it is as hardened as it can be.
I feel you are into something here. I never looked this setup very much and I think many may be overlooking it as well. Thanks for sharing the article, it was very interesting.
Something you can do easily on any OS. Auto-updates are easy. What’s more work is implementing secure automatic decryption of user data during boot with attestation or something like clevis/tang.
Can also be done on any OS. Also I would rather use gvisor for untrusted containers, which is more painless to use with other solutions like Docker.
Nice goody, but not as important as it sounds. I haven’t had an unbootable Linux system in years, despite doing automatic updates.
I really don’t see what microos has to offer.
If I was using an immutable OS, I would rather use one with strong integrity guarantees via UKI plus verity protection for everything important, including security ciritical config files.
