TigerBeetle's Spectacular Jepsen Report - with Joran Greef

**Joran Greef** (0:00)
That was the one bug that he found, but otherwise he didn't break TigerBeetle's strict usability. And also what's interesting is what was a first, he added new nemeses or storage fault injectors into Jepsen. If we wrote to a data file on one machine, he would take that right and move it to another place in the data file or move it onto another machine's data file. So he was literally playing with the cups. So this was just a normal Jepsen test, and then now he was also playing with cups, and TigerBeetle had to survive, and it did. Not many people know this, but he actually could also use antithesis, so he could use their DST against us.
And he didn't find anything like that, which is the interesting thing.
It was the human. It was Kyle that found all the interesting findings in his report was pure Kyle. Kyle already gathered everything he could, but again, skill of the human, he was phenomenal.

**Richard Feldman** (0:58)
Welcome to Software Unscripted. I'm your host, Richard Feldman. Today, I'm talking with Joran Greef, founder of TigerBeetle, which is known for having developed a legendarily reliable open source distributed database for financial transactions. We talk about how they achieved such a high level of quality using things like Zig, deterministic simulation testing, or DST for short, their unorthodox Tiger style programming style, and even subjecting their database to the also legendary scrutiny of Kyle Kingsbury and Jepsen, which has absolutely dismantled the quality claims of other popular databases in the past, but which earned TigerBeetle a glowing review. We also get into broader topics of software quality, the economics of open source, and where the real value comes from for software in general. I want to give a massive thank you to everyone who's been supporting Software Unscripted on Patreon. If you enjoy these episodes and you'd like to become a supporter too, you can get ad-free episodes by signing up at patreon.com/softwareunscripted.
Thank you to Mailtrap for sponsoring this episode of Software Unscripted. If you don't know Mailtrap or you only know them for email testing, they are a modern email delivery system for developers. They support things like straightforward integration into your code with native SDKs or they have a security compliant API and also SMTP access. In their free tier, you get 4,000 emails monthly, so it's pretty easy to try out. And they also have 24-7 support where you talk to real humans, not AI chatbots. If that's the type of thing you're interested in because you do email delivering at your business, check out mailtrap.io to learn more. And now here's Joran Greef.
All right, Joran, welcome back.

**Joran Greef** (2:24)
Hey Richard, so good to be back. I think it's been about two years.

**Richard Feldman** (2:30)
Yeah, and something that happened in the last two years since we talked was the Jepsen Report coming out about TigerBeetle, which was incredibly impressive because I always thought of Jepsen Reports as being something that would just skewer databases. I think about the famous MongoDB example just left and right, like here's this problem and that problem and this guarantee didn't hold up and that correctness issue and this security flaw.
But yours was not like that at all. I mean, they did their usual deep dive and I was really impressed by the result. Which, how did you feel about getting that result?

**Joran Greef** (3:05)
Yeah, thanks. I really appreciate it. So I was also really happy. I should say I could tell you how I felt during the audit.
That was quite something because you do feel like you skewered. Carl is so good. He has an instinct for where to go in the African Safari to find the lion, to find the bugs. And he was just so good in building a reference implementation of our state machine. He did almost the most comprehensive implementation that he's ever done for any report because he didn't just try to break TigerBeetle's strict theorizability, just to break the consensus. Normally, he achieves that for almost everything because that's one of the hardest problems in a distributed system is the distributed system, so the consensus protocol. And you just have to find a way to break that and you break everything. And so he checks linearizability and all these things. But he actually didn't only try to check that. He tested almost TigerBeetle's entire product surface area or entire API. And he re-implemented everything in a second independent implementation, and then differential fuzz both. So he did actually find one bug on the read path, so it didn't affect durability. Query engine didn't always return all the results. Maybe the tail would be less, and then it paginated so you'd go back and then you'd get it.