Vector Databases and the Power of RAG

**Edo Liberty** (0:00)
I can't remember the year exactly, but fairly quickly, it became obvious that there are some really deep, interesting questions to be answered. And by the way, I think the main frustration is that those are really hard questions, like the tools that we have to analyze classical machine learning just doesn't work anymore. It's like we've been studying grains of sand, and suddenly we need to like study how dunes behave.
You know, it's like a whole different phenomena.

**Derrick Harris** (0:32)
Hi, this is Derek, and you're listening to the a16z AI Podcast, where we dig into all things artificial intelligence with our in-house team of experts, as well as the founders, engineers, and researchers working at the state of the art.
In this episode, I'm joined by a16z partner Satish Taluri and Pinecone founder and CEO Edo Liberty. Pinecone sells a vector database, so we definitely dig into that technology, including its value outside of just generative AI. But Edo has been around machine learning for about two decades, including time spent running research teams at AWS and Yahoo! as far back as 2009 So you can also consider this a trip down machine learning memory lane.
As a reminder, please note that the content here is for informational purposes only, should not be taken as legal, business, tax, or investment advice, or be used to evaluate any investment or security, and is not directed at any investors or potential investors in any a16z fund. For more details, please see a16z.com/disclosures.

**Edo Liberty** (1:43)
First of all, I started going into this topic in my PhD, that was started in 2003, where really the foundations of machine learning theory were being really formalized, and it really started way before that. I mean, it started, frankly, in the 70s. We as a community started to really understand how optimization and functional analysis and online algorithms and other topics, and even graph theory, started really merging and really contributing to how machines can learn.
At the end of the day, we focused on the same things, and making machines smarter, and being able to learn, and remember, and react, and be smart, and predictive, and so on. The set of techniques was just math and basic coding. Like, there wasn't any... If you devised a new model, you had to spend a few months really building the gradient descent and the optimization and everything by hand. Like, you didn't have, whatever, tensile flow and auto-differentiation.
Everything was manual, right? But we were really obsessed about exactly the same problems and just how to run things faster, better, how to get more accurate data, how to train bigger models. And yeah, I mean, that line has continued for my entire career. Of course, we've seen many orders of magnitude improvement on every possible front of this problem, but every one of them was hard-earned by, not by me, by many, many, many brilliant, hardworking scientists and engineers.

**Derrick Harris** (3:14)
Right, what did a big model mean in 2003?

**Edo Liberty** (3:17)
Well, I'll give you an example. I'll answer with a very short story. One of the problems that pushed me into a topic called dimension reduction and big data at the time was I was working on what's called hyper spectral microscopy. You scan an image not in RGB, like only in three spectra. There are very, very specialized microscopes that actually look at hundreds of spectra. So this is like wavelengths of light.
You can identify different chemicals and structures a lot better. So think about whatever detecting cancerous cells and so on. Back then, every image was, I forget it was a few hundred spectra, and each one of those was a layer, was like a few tens of megabytes.
And so the whole thing was gigabytes of data, and it just didn't fit in memory.
Like it couldn't fit a single image in memory of a desktop. Like we had like 64 megabytes or whatever on the desktop. You would kick under the table all the time. Forget about big data. We couldn't load one image, right? Let alone thousands of images, there would be the output of like a single experiment.

**Derrick Harris** (4:25)
On the grand scale of things, was the bigger shift in maybe the AI and machine learning space, the algorithms and the models and everything on that front, or was it, I mean, it seems like an equally big part of it was just having the infrastructure, like the compute resources.

**Edo Liberty** (4:39)
Yeah, but frankly, that has been one of the many, many, just lucky breaks that I had in my career. A lot of the techniques for reducing cost, improving efficiency, improving speed, and so on, that we had to just build from scratch, the kind of cramming the bits and the bytes.