Originally posted at https://tech.labs.oliverwyman.com/blog/2016/10/31/potboiler/
Over the last couple of years I’ve been reading and talking about a lot of things related to distributed systems. This is a common train of thought around here, and after working on this on and off for the past 18 months or so (the version you’re seeing here is in fact version 3 having repeatedly changed my mind about how it should work), I’m placing my own stake in the sand. Potboiler is an AP Event Sourcing system. More specifically, it’s an MVP/research prototype of said, with known issues and is not even slightly suitable for production use. However, I think the shortcuts I’ve taken aren’t fundamental problems of the design but trade-offs to get to something sufficiently usable to play with and get a sense if the design works. There’s a couple of larger problems still to be solved, but I’ll address those in a minute.
Potboiler actually has multiple event logs. Each node has a single node-specific log that it can append arbitrary events to (currently blobs of JSON), and it also stores a local copy of all the other logs it knows about as well, but can only append events that other nodes tell it belong there (which may be the originator node for that event, or another intermediate node). An individual log is linear, and strictly append-only, but to aid ordering of events each has a Hybrid Logical Clock timestamp applied. Nodes tell the other nodes they know about when they get new log events in and they also periodically poll the other nodes they know about in order to do reconciliation of any events that failed to get through the first time. They also periodically poll other nodes for what nodes they know about in order to try and build a fully connected topology of all nodes in the network. Nodes also have clients which they tell about new events. The intention is for there to be a Potboiler node per server (or in less ideal situations one within each presumed-reliable network subset) that needs any of it’s data, and so have shared fate with the other services that use it.
I’ve also built the first client of Potboiler, a Key/Value store that uses Potboiler to distribute update events. All of the keys are arbitrary unicode strings, and the values are CmRDTs. The only current one is a variant on Last-Writer-Wins Register with Hybrid Logical Clocks for timestamps, but I plan to add OR-Set soon. By default, there exists a table called “_config” using LWW for values. If you add an entry to _config, the key name indicates a new table to create, and the value is a config (currently just {“crdt”:”LWW”}). There’s also a simple browser for said K/V tables.
Trade-offs to get to this point are as follows:
Major problems remaining:
Over the next few months I plan on adding more clients, hopefully without having to change the core node code, and attempt to further demonstrate that a CRDT-based AP approach to data is the right way to go (despite some of my colleagues opinions to the contrary).