Ep 2: Databricks CTO Matei Zaharia on scaling and orchestrating large language models

Patrick Chase (0:04)

Welcome to Unsupervised Learning, Redpoint's AI podcast. I'm Patrick Chase. I'm joined by Jacob Effron. We are so excited to have Matei Zaharia on the show. Matei is the CTO and co-founder of Databricks. In this episode, we dive into the founding story behind Databricks, some of the latest research at Stanford, including the Demonstrate, Search and Predict framework that his group is building and open-sourcing, how Databricks is using large language models, the flaws that large language models have, especially when it comes to search and how to fix them, open source versus closed source, and some fun stories like Matei getting a $50 Amazon gift card for weeks of work in the early days of Databricks.

I think it's a super awesome show. Thanks for joining us.

Today, we are so excited to welcome Matei Zaharia to the show.

If there was a Mount Rushmore for AI technologists, Matei would definitely be on it. He is the creator of Apache Spark, one of the most widely used AI infrastructure projects and is the founder and CTO of Databricks. Databricks famously built a data platform around Spark that has reached incredible scale. They passed a billion in ARR in August of last year, raised $3.5 billion in funding from A16Z, NEA, CO2 and more. Their last round of funding put the company at a $38 billion valuation. If that wasn't enough, Matei is also a professor at Stanford focusing on computer systems and machine learning and one of the leaders of the Stanford Dawn Project, which is a five-year research project to democratize AI. Matei, thank you so, so much for joining us. We're really excited for the show.

Matei Zaharia (1:46)

Thanks a lot for having me.

Patrick Chase (1:47)

Maybe to start, it'd be awesome to talk about your journey building Databricks. It started as your research project with Apache Spark and then now it's used by more than 7,000 organizations worldwide, including 40% of the Fortune 500, which is just really incredible scope that you guys have reached. How did you come to start Databricks?

Matei Zaharia (2:05)

Yeah, for sure. So it basically started out of some of my research as a PhD student, as you mentioned. So I was doing my PhD at UC Berkeley and I was really interested in large-scale data-intensive computing, which was just starting to become a thing back then with web companies like Google and later Facebook, Yahoo back then, all these companies that were indexing the whole web and building search engines and other products. And I was excited about bringing this kind of technology to all users, like storing, collecting data, storing that isn't that expensive. So why can't everyone do really large-scale computations? And indeed, there was this trend that in most areas of industry, most areas of science, you could start collecting lots of types of data. And then I was also excited about letting people on more sophisticated algorithms on them because that's what I saw at the early MapAge users. You don't want to just do analytics, especially when you have so much data. It's perfect for machine learning. It's perfect for unsupervised learning, when deep learning became popular and so on.

So we started first this open source project that very quickly grew. It was clear that there's demand for something. And we decided that we should start a company to really have an impact in this space because you can only get so far by just having an open source project with no one backing it.

And the other bet we made that was a bit controversial at the time, that turned out to be right, is we decided to go cloud only.

So you can only use Databricks in the cloud on one of the three major cloud providers as a SaaS service on top of that. And this was 2013, so it wasn't as obvious that banks and other really big enterprises would move to the cloud, but it turned out to be the right bet and it really focused the company on delivering an excellent experience there with things you can only do in the cloud.

And it helped us land as one of the big options if you want to do these things in there.

Patrick Chase (4:14)

Yeah, amazing. It's super cool just to see how far it's come. I remember when you were just getting started, I actually was on the machine learning team at LinkedIn, and we were some of the early adopters of Apache Spark, and I think you and Reynolds and you guys came and did a talk. And it's just amazing the technology that you've built and then how far it's come. And I remember a big part of it was kind of for these sophisticated algorithms, you're moving to training things, to doing things in memory, so iterative computation would be a lot faster than for Hadoop.