I'm intermittently working on a personal project I might eventually launch into the world (the same one that's the subject of the infrastructure posts). It uses Celery for task management, and in the process of setting up the infrastructure I wanted to move from a single RabbitMQ node to a multi-node setup to avoid a single point of failure. There was just a few problems here: firstly the RabbitMQ clustering guide which quite frankly is a thing of fear and terror (partially because I've tried to do it before and remembering such fun as needing to bring up nodes in a very specific order...). Secondly, Quorum Queues which are the recommended "you're doing replication with queues in RabbitMQ" queue aren't supported by Celery.
The sensible thing here would be to have done something with mirrored queues and wait for Celery to fix the issue. As I'm writing this post, I obviously didn't do that. Instead, some part of my brain went "hey, you've already setup CockroachDB so it's not like you've got to solve the distributed systems problems again. You've just got to write an AMQP broker. How hard can it be?". It would have been really hard, except that someone already wrote an AMQP protocol crate, which made this plausible without having to build all the AMQP spec. Combined with the Tokio tutorial not being HTTP-centric, and instead implementing a frame-based protocol (Redis in their case, but adaptable to the AMQP case), I was able to get something vaguely working pretty fast.
3 weeks later, we have Cavalcade. It's a hacky prototype of an AMQP broker that stores all it's data in a PostgreSQL-compatible DB (with integration testing on both PostgreSQL and a single-node CockroachDB setup). It's not fully compliant, but it's sufficiently compliant to make Celery able to submit and run tasks, which is enough to get me past my core problem. It's not very fast, but using sqlx it's definitely fast enough for some Celery tasks, which gets me past the core problem here. Patches to improve either of these are welcomed!