TL;DR: Proposal for a way to define multiple-machine systems in a immutable way, without requiring by-hand layouts. I’ve been reading a lot about Nix and NixOps recently, and although I’m still leaning towards Kubernetes overall as my system configurator of choice (technically k3s, but that’s not so relevant for this), I’ve realised there’s a gap in the market for better immutable infrastructure. Note that this is a half-formed proposal, not a plan to do things, as I’ve already got way too many things on my plate, but I felt it was worth sharing the idea with others if anyone feels like picking this up, or doing something derived from it.
The goal is as follows: I want to be able to declare things like “I want X replicas of this service running, on different nodes” and “don’t put too many things on one node so the load gets well shared”, while not having to layout by hand which nodes the services go on, as they’re cattle not pets. We can do this with Kubernetes right now, but at the cost of a system that tends towards being hard to debug when things go wrong, and things can go very wrong. It also has the limit that you have to containerise everything and isn’t really suited to managing local node setups (i.e. everything you need installed before you run the Kubernetes agent, like say ntp).
Nix OTOH, is absolutely designed for the local node setup, and NixOps would let us then setup multiple machines. However, NixOps is designed towards a “here’s a set of Nix expressions defining a series of specific machines”, and isn’t designed for any of the “choose where stuff goes for me” bits. These machines aren’t full-blown pets as they’re still automatically configured, but they’re not really fully cattle either.
We need a 3rd option: something with the best bits of both. Maybe it can be done with a variant of NixOps that defines a “basic node” config (stuff you want everywhere, like a specific kernel, ntp, maybe some firewall stuff), and then a series of “service” configs, that probably contain an actual service config and maybe any other packages/config that service needs, along with some metadata (e.g. replication count, memory/CPU requirements). It does need some sort of thing that decides “where does that particular service go” especially if the node that was carrying it goes down. Maybe we have some little service running on each node with Raft consensus to decide what goes where, but all compiling down to Nix configurations for ease of debugging. It would also need to be able to cope with the upgrade case, plausibly by using the consensus system to replace a subset at a time of the services (or maybe some magic with unique names for services so we can run old version + new version at the same time)
Ideally, we also have something that lets us say spit out Terraform config for the more infrastructure things (NixOps supports AWS, Hetzner and GCE; Terraform supports basically everything and so it’s just easier that way), and then we can have a single config language that defines everything about our system’s setup other than the source code. And without needing to write HCL!
I would also note that I don’t regard Nix as the definitive option on system config languages. I think the immutable infrastructure approach is great, but the Nix language is kinda awful. I’m somewhat more fond of Guix, but I’ve used Nix here as the example as it’s the 800-pound gorilla in this space currently. I’m not sure what “better” looks like, other than liking the approach of using an existing language (as Guix does with Guile Scheme) rather than building half of one along the way like Nix does.
Thoughts anyone?