RAG & Beyond: Semantic Storage and Retrieval

**Erik Torenberg** (0:00)
Turpentine is a network of podcasts, newsletters, and more covering tech, business, and culture, all from the perspective of industry insiders and experts.
We're the network behind the show you're listening to right now.
At Turpentine, we're building the first media outlet for tech people by tech people. We have a slate of hit shows across a range of topics and industries, from AI with Cognitive Revolution, to Econ 102 with Noah Smith. Our other shows drive the conversation in tech with the most interesting thinkers, founders, and investors, like Moment of Zen and my show Upstream. We're looking for industry leading hosts and shows along with sponsors. If you think that might be you or your company, email me at erikaturpentine.co. That's E-R-I-K at turpentine.co.

**Anton Troynikov** (0:46)
We are conditioned to think about data as this static thing. It's sitting somewhere and it has a particular instance in time, and then we access that instance in time, and then the next time it might be different, but it's still essentially mentally we think of it as static. I really think of these things more like a loop.
It's like a control system loop almost, where data is actually a engine. It's something that is interacting with the outside world mediated through computation, and then it's constantly adapting and it's improving itself.

**Nathan Labenz** (1:17)
Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week, we'll explore their revolutionary ideas, and together we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my co-host, Erik Torenberg. Hello and welcome back to The Cognitive Revolution. Anton Troynikov is the founding CTO of Chroma. We first had Anton on the show back on episode 5, which was released in early March, before GPT-4's release. And he impressed me then and since as a real intellectual force.
Anton is someone for whom the fundamentals seem to come easily, who codes all the time, and for whom it's natural to think in higher dimensions and abstractions. He's also a super quick wit, a prolific and at times provocative commenter, and amazingly, at the end of the day, still a bit of a doubter, certainly more of a doubter than I am when it comes to LLM reasoning abilities.
In March, I treated my conversation with Anton as an opportunity to speak to an expert tutor on core concepts of embeddings and retrieval, and I tried to explore how he thinks about navigating the latent space as a way to develop my own intuitions and to help you develop yours. If you haven't heard that episode and want something more foundational from Anton, definitely check that one out. This time around, I've wanted to talk about everything that's happened since, including the fact that RAG, or Retrieval Augmented Generation, has been LLM's first big commercial application hit, with tons of companies spinning up all manner of document-backed bots for all sorts of labor-saving purposes, and in the process, nearly as many companies spinning up and deploying a vector database for the very first time. We talked about what's working today in Retrieval, and also what's next, including for Chroma as a business.
Now coming in, I was honestly wondering if Chroma and other vector database startups might struggle to grow businesses, given how incumbent database providers are now racing to implement their own vector stores. But Anton did a good job of reminding me of one of my own maxims, that there's a good chance that we're all still thinking too small.
His observation that most of the data stored in Chroma has never been in a database before suggests that there will be plenty of growth to go around, at least at the infrastructure layer for a while yet. And his plan to bring more and more value into the database so that retrieval becomes as simple as dumping text into a text box, not unlike how people use chatbots today, seems like one that can drive Chroma to a great outcome, even as lots of incumbents get into the game. I just wanted to take one moment to say that I'm really quite proud of this content. This is a focused but fast moving conversation conducted at an expert level with a genuine intellectual leader currently working as a wartime CTO. I've learned an incredible amount from the process of making this show, and I'm grateful that thousands of you from an incredibly diverse range of backgrounds have become regular listeners. I also want to thank the team at Turpentine, including our producers Vivian Meng and Natalie Toren, our editor Graham Bessalou, and of course Erik for putting the whole thing together. Working with Turpentine does make for an unbelievably convenient operation, in which I am 100% focused on understanding, communicating, and interviewing as well as I possibly can.