LLMs Everywhere: Running 70B models in browsers and iPhones using MLC — with Tianqi Chen of CMU / OctoML

**Alessio** (0:09)
Hey, everyone, welcome to the Latent Space Podcast. This is Alessio, partner and CTO and resident of Decibel Partners, and I'm joined by my co-host, Swix, writer and editor of Latent Space.

**Swyx** (0:20)
Hey, and we are here with Tianqi Chen, or TQ as people call him, who is assistant professor in ML Computer Science at CMU, Carnegie Mellon University, also helping to run Catalyst Group, also chief technologist of OctoML. You wear many hats.
Are those your primary identities these days?

**Tianqi Chen** (0:41)
Of course, of course. I'm also a very enthusiastic open source, so I'm also a VPN Pearson member of the Apache TVM project and so on. But yeah, these are the things I've been up to so far.

**Swyx** (0:53)
You also created Apache TVM, XGBoost and MXNet, and we can cover any of those in any amount of detail. But maybe what's one thing about you that people might not learn from your official bio or LinkedIn on the personal side?

**Tianqi Chen** (1:07)
Let me say yes. Normally, when I do a real lab coding, even though I'm trying to run all those things. So one thing that I keep a habit on is I try to do sketchbooks. I have a book, like real sketchbooks to draw down the design diagrams and the sketchbooks that keep sketching over the years and now I have like three or four of them.
And it's kind of usually a fun experience of thinking the designs through and also seeing how open source project evolves and also looking back at the sketches that we had in the past to say, you know, all these ideas really turn into code nowadays.

**Alessio** (1:43)
How many sketchbooks did you get through to build all this stuff? I mean, if one person alone build one of those projects, it will be a very accomplished engineer. Like you build like three of these.
What's that process like for you? Like is the sketchbook like the start? And then you think about the code or like?

**Tianqi Chen** (1:59)
Yeah. So usually I start sketching on high-level architectures, and also in a project that works for over years, we also started to think about new directions, like alternative AI language model comes in, how it's going to evolve. So normally I would say it takes like one book a year, roughly at that rate. It's usually fun to, I find it's much easier to sketch things out and then gives a more like a high level architectural guide for some of the future items.

**Swyx** (2:28)
Have you ever published this sketchbooks? Because I think people would be very interested on, at least on a historical basis, like this is the chart where XGBoost was born.

**Alessio** (2:36)
Yeah, not really.

**Tianqi Chen** (2:37)
I started sketching like after XGBoost. So that's the kind of missing piece, but a lot of design details in TVM are actually part of the books that I try to keep a record of.

**Swyx** (2:49)
Yeah, we'll try to publish them and publish something in the show notes. So maybe you can grab a little snapshot for visual aid.

**Alessio** (2:57)
And yeah, talking about XGBoost, so a lot of people in the audience might know it's a gradient boosting library, probably the most popular out there. And it became super popular because many people started using them in like a machine learning competitions.
And I think there's like a whole Wikipedia page of like all state-of-the-art models that use XGBoost and like it's a really long list.
When you were working on it, so we just had Tree Dao, who's the creator of Flesh Attention on the podcast and I asked him this question. It's like, when you were building Flesh Attention, did you know that like almost any transform race model will use it? And so I asked the same question to you. When you were coming up with XGBoost, like could you predict it would be so popular or like what was the creation process? And when you published it, what did you expect?

**Tianqi Chen** (3:40)
We have no idea. Like actually the original reason that we built our library is that at that time, deep planning just came out. Like that was the time where Alex and I just came out. And one of the ambition mission that myself and my advisor Carl Schoestrand, then is we want to think about, try to test the hypothesis. Can we find alternatives to deep learning models? Because then there are other alternatives like support vector machines, linear models and of course tree based models. And our question was, if you build those models and feed them with big enough data, because usually like one of the key characteristics of deep learning is that taking a lot of data, right? So we will be able to get the same amount of performance. That's a hypothesis we're setting on our test. Of course, if you look at now, right, that's a wrong hypothesis.