Why the Frontier Ecosystem must be Open — Matei Zaharia and Reynold Xin, Databricks

**Swyx** (0:03)
Matei and Reynold from Databricks, welcome to Latent Space.

**Reynold Xin** (0:06)
Thanks for having us.

**Matei Zaharia** (0:07)
Yeah, thanks so much.

**Swyx** (0:09)
Thanks for taking time out. You have your Databricks, Data AI Summit going on. You were just telling me how the first summit that you guys ran was just 50 people.

**Reynold Xin** (0:17)
Yeah, it was a little meetup at Berkeley, I think.

**Matei Zaharia** (0:19)
Yeah, we would do these tutorials and just teach people Spark. Yeah.

**Swyx** (0:24)
Obviously, now it's like, I think the headline number is like 100,000 people around the world, 30,000 in person. It's a crazy community. Well, I mean, I just saw the keynote.
Ali is just, did you know, was it obvious back when that Ali would be such a great CEO, like such a great presenter?

**Matei Zaharia** (0:43)
What do you think? I think among our group of founders, it was clear that I think he'd be the best at this. Yeah, it turned out great.
I mean, he's ramped up on so many topics. Going to a company, he would just go in and like study it and talk to all the experts. Like even if he can't hire the person, you know, learn enough about like finance and sales and whatever it was and go from there. Yeah.

**Reynold Xin** (1:09)
I mean, he's obviously very high IQ and very high EQ, but it wasn't like Ali today is quite different from Ali from like 10 years ago. I think there's a lot of work that he put in to get to this point.

**Swyx** (1:21)
I mean, to me, the most appealing thing about him is that he's funny. And like, it's hard to make jokes about data warehouses, security and what have you. Oh yeah, that's for sure. So you guys launched a whole bunch of things. I'll just go to name check briefly the stuff because we're not going to cover everything. Omnigent, your baby.
LTAP, your baby. Your dream engine.
We're also going to cover Genie, cover CustomerLake, you acquired Panther, OpenSharing, and there's Unity AI Gateway. A lot of these, I think, are things that you would expect Databricks to do. It's like part of the roadmap. Everyone in your category has similar things. But I think probably the two of you are leading the two most unique and differentiated initiatives in the landscape. Maybe we'll start with Omnigent and we'll go into it. I do think that a lot of people are exploring this MetaHarness concept.

**Matei Zaharia** (2:21)
What led you to it? Yeah. There were actually a couple of converging lines, which I think is a good sign that you need something new.
On the one hand, there's all the coding agent, Infra Internally. We have a really great Dev Infra team.
They built something called Isaac, that's basically like a wrapper on Claude Code and Codex, and let you use them either on the web in like sandboxes or just on your dev machine or on your laptop or whatever. Then they were adding all kinds of stuff there. We saw all the more advanced engineers were building their own workflows with tons of agents, and they were building their own UIs and stuff even on top of that. And then the other one was like us building agents. We ship this data science agent called Genie on the research team, which I co-lead basically. We also build a lot of internal ones for various things, and then we have all the customer ones. And all of them were running into this thing of like, I need to switch model and harness and so on every few months. Plus, the agent is completely useless if you can't share sessions with someone and have history and have search and all this like layer on top of it for collaboration. I thought a bit about it from both contexts. And at first people thought it was weird. They're like, why are you doing coding agents and custom agents in the same thing? But I said it's basically the same problems. And you just want to build the stuff that lets you deliver the agent, maybe control it if you care about security, and make it portable across things. And then we prototyped some things as experiments. We saw, yeah, actually we can make it work. And then we sort of built that for real.

**Swyx** (4:06)
I'm wondering if this kind of, let's call it architecture, maps to anything in your careers in the past. I always think about how a lot of things actually just tie back to operating systems.
A lot of operating systems tie back to databases or the other way around.