I have been looking into Cosmos which is an open source home cloud / homelab implementation that looks very interesting in how quickly it could speed setup up. However, it has a limited amount of users it supports and runs with a very hefty freemium model.
Are there better FLOSS options? One of my requirements is making setting up SSO very easy. I’ve looked at Yunohost and CasaOS, any opinions there?
Anyone? 
I don’t really have a good answer but these all-in-one homelab OSes have always kinda sketched me out. I would probably go for something more established like Unraid (not FOSS) if you want to go this route. HexOS (also not FOSS) could also be an option but AFAIK it requires a cloud UI for now.
Otherwise I’ve recently discovered Linux Containers - Incus - Introduction which is sort of like Proxmox but I haven’t had the time to experiment with it yet.
Quite a lot of them are done in a docker, but some are as you said, they run on a custom Ubuntu or Debian distro (imo not a good idea for a server in the first place) and the big benefit is they trivialize things like SSO. I can see a reason for it, a user can easily muck up the security setup of these server apps if done improperly.
If I were to choose an OS it’d probably be Fedora CoreOS or something, mostly because its already tooled for what all servers are now anyways (docker/podman containers).
Maybe theres a NixOS config file for a server setup with all the stuff preset? 
Why not just use TrueNAS or Proxmox respectively?
Unraid and HexOS are supposedly easier to configure and use than TrueNAS and Proxmox.
I just mentioned Incus because it seems interesting and somewhat more flexible since you can bring your own distro.
If you need a gui, CasaOS seems decent, but it seems to take a lot of storage, my container stack of about 10 containers was 150gb.
Fedora and friends have cockpit which uses podman, its basically a gui, but you still need to know what to do.
Unraid works well, but sometimes containers break or dont launch, its not super easy.
I mostly just do containers via compose, I spend most of my time in the terminal, and compose files are basic usually.
If you need sheer security, straight docker using compose or docker run.