Continuing on my series about the infrastructure of a personal project. Last month I talked about the Terraform work and this month, it’s the Mitogen work.
But first, some context. For this step, I needed a configuration management tool. Something to say “here’s all the stuff on a node before it gets as far as Kubernetes” e.g things like NTP, Wireguard to make a little private network, and of course all the Kubernetes node stuff. Now, there’s a bunch of tools out there for that, and I’ve used a fair number of them: Puppet, Chef, Salt, Ansible etc. I was however getting a bit annoyed at all of them for one reason and another (mostly to do with their generally terrible startup times for rapid “make a small change and retest” scenarios and/or strict enforcement of a broken model about how things must be done), and I decided I’d have a go at building my own. Nothing too general, just something fairly specific.
I figured to begin with that I’d just write a bunch of Python to do this (not wanting to write a bunch of shell scripts for something of this size), but then I’ve got the fun of how do you make sure it’s kept in sync between the nodes, etc, etc. I then remembered reading about Mitogen at some point and wondered if it might work here. The front page does say “ultimately it is not intended for direct use by consumer software” and intends it more as a low-level API, but I’ve regarded it as pretty friendly. Basically, it’ll do all the magic of forwarding on your local Python code (provided anything running remotely is Python 2.4 compatible) and I could then do everything with Python :) This has worked really well - startup time is practically zero and although I’ve had to write a bunch of helpers to do standard stuff like “download from URL with specified hash for result”, I’ve enjoyed the trade-off between the extra work v.s. the flexibility to get things just how I like them.
Ok, so I’ve got the basics working, now I need to make them into Kubernetes nodes. I started by using k3sup but ran into a lot of issues with it (mostly due to it making bad assumptions about the “external” interfaces given I had both wireguard and regular network setup), and figured I’d rather do things my own way. This meant starting from Kubernetes The Hard Way, translating into my own automation along the way, and eventually also running through “Calico the hard way” for the overlay networking.
Despite all these hoops, it did eventually work. I was able to do things like generate all the SSL certs for intra-Kubernetes bits locally, use Mitogen to transfer files back and forth (e.g. to dump the network + Wireguard config of all the nodes so they could all find each other), there’s a nice firewall setup using ferm that mostly just locks down everything on the external networks, but lets the nodes talk among themselves over Wireguard fairly freely. This has been a pretty high level of overkill, but I’m planning on keeping it around, barring the Kubernetes bits, but that’s a problem I’ll talk about next month when I detail the k8s folder.
Comments
With an account on the Fediverse or Mastodon, you can respond to this post. Since Mastodon is decentralized, you can use your existing account hosted by another Mastodon server or compatible platform if you don't have an account on this one. Known non-private replies are displayed below.