DuckLake 1.1 solves the absurd complexity of data lakes at AI scale

SPEAKER_1 (0:03)

DuckDB is an analytical SQL database that can run in process. Simple, fast, and portable. Visit duckdb.org to learn more.

Jennifer Riggins (0:14)

Hey, y'all, I'm Jennifer Riggins for The New Stack Podcast. Today, we have an episode sponsored by Amazon Web Services and DuckLabs, and we're talking with Pedro Holanda, who is a principal engineer at DuckLabs, and you could say a DuckDB contributor, but he's been there from the very beginning when it was a prototype. So we're really happy to have you. Bem-vindo.

Please tell us how your day is going so far, and how's this whole being bought by Amazon feeling?

Pedro Holanda (0:47)

First of all, happy to be here. Thanks for having me. Yeah, I mean, it's a little bit of a change, I would say.

It does, I think especially like in resources, it feels pretty cool because now we have all these Amazon machines, a bunch of tokens, we can go a bit crazier than before. But otherwise, it hasn't changed that much as well. Like we're still kind of ourselves and behaving as we were before, still doing everything open source and having a bunch of fun with it.

Jennifer Riggins (1:17)

Yeah. And in that time, you released 2.0.

DuckDB 2.0, also known as Cinnamon Teal, which I've learned is a type of duck, which logically is colored that way. So that just came out on October 21st. You being someone that's been with it since it was a prototype, what's the thing that excites you most and you're most excited for developers to know about it?

Pedro Holanda (1:44)

Well, I would say that this has been the DuckDB version of the most features we ever released.

And it has been quite a while since DuckDB 1.0. 1.0, the focus was more like, oh, let's make it stable, let's have our storage settle down so people can actually start using it for serious things. But DuckDB 2 was more like, let's really push the performance and features to the next level. So personally, the thing that I'm most excited about is AsyncIO because it's something that I worked quite a bit to get to DuckDB 2.0. Maybe I can explain it a little bit. The basic idea is that when you're querying or requesting data from remote sources like S3, for example, without AsyncIO, your process just gets stuck waiting for data to be downloaded. But with AsyncIO, you have different groups of threads that just keep downloading it, so you're not really waiting to be processing it. So it gets things much faster.

And I have heard of some companies that use it, that have actually been using the DuckDB 2 before the release, just because of AsyncIO, because it made that much of a difference. So yeah, I was very happy to be able to get that out, and all our major data sources use it, so Parquet, CSVs, the DuckDB file formats, the Arrow file formats, so it's really widespread in the system.

Jennifer Riggins (3:09)

It's a lot, and I see, we're at your office today, I see that 2.0.1 is coming soon. So y'all are working hard and hitting the ground, I guess the air running, flying.

The ducks, you know, you look smooth, but they're going fast under the water, aren't they, they say? So yeah, but to need all this, have we made lake houses more complicated than they need to be?

Pedro Holanda (3:34)

Yeah, so the lake house is a different story than DuckDB, and the thing is, maybe we could start off like, why lake houses, right? Why do people even care about it?

Jennifer Riggins (3:45)

How does a Duck Lake relate to DuckDB? How does that all come together?

Pedro Holanda (3:51)

So the main reason is that these lake house formats, they exist so people can exchange their database systems without having to do migrations, right? You don't really want to be stuck to one company because your data is on their proprietary formats or whatever. You want something that's open, that anyone could read a specification and implement it and then be able to interact with your data without you having to modify it.

So there has been lake house solutions out there. That's not really something new, right? Like you had Iceberg, you had Delta. I think Iceberg maybe originated from Netflix, even in 2017 or something like this. And their whole point of what they were doing was that your whole data should be able to be stored on S3. So just file formats, right? So the metadata, the way that you can traverse and figure out which data files you care, is just JSONs and avros, and your data is stored as Parquet files. What we realized is that at some points, they added a little database there, because the whole point is that you had no databases, but it turns out that to have transactional control, so assets, you did need a database system to keep track of which table has its last version.