Tuvix: Experimental hermetic build engine with Starlark

19 Mar 2022

DevOps  Rust 

Tuvix is an experiment I've been fiddling around with recently, building a hermetic build engine. By hermetic, I mean that it requires you to specify all the dependencies of a build as opposed to just getting them automatically from the OS. This has the advantage that you know what all the dependencies are and don't get randomly tripped up by something not existing in CI or disappearing at some point in the future. It also means you can potentially do really cool things like "have a cache of build artifacts that always perfectly knows about all the things that your build depends on" and then share those artifacts among people while avoiding the "but what if machine X has a different compiler?" problems.

It's called Tuvix both because I really like that Voyager episode and because after Nix and Guix I've obviously got to name my one something ending "ix". Those are the two biggest influences on things, although after getting as far as I have, I have much more sympathy than I used to for the Bazel folks. My original goal was "compile zsh" (as it's my shell of choice), which although it's a pretty simple task outside of this sort of structure, I really should have realised that it's harder than that. As it is, I've gotten to the point of having a hermetic build that starts the configure step, but then runs into problems with a lack of the compiler and I haven't gotten as far as "and now I hermetically build/bootstrap gcc" yet but I think I've learnt enough to stop the experiment.

What I've learned from all of those things is what a good hermetic package manager would look like, in large part by stealing from all the others

However, getting from here to "and here's a replacement for those" is a full time job for several people for the next few years, and given a lack of those people (heck, I don't have the spare time to expand this out further) I'm probably going to have to live with those suboptimal choices. I suspect one of these years I'm going to finally bite the bullet and try using Nix, if only because it's the most used tool in this space.

I've worked with a few new things along the way which are worth talking about:

Previously: Cavalcade: PostgreSQL-backed AMQP broker Next: Parallel for serial_test