#15 - Jeremy Howard

Balaji Srinivasan (0:00)

Jeremy, welcome to The Network State Podcast, and we've been friends or friendly online, I think, for a while. You are the founder of fast.ai, which is this incredible course that's online. We both taught large online courses, so we kind of have talked about that. You're the founder of answer.ai. Before that, I think you were at Kaggle, right? And you're Australian, you have an interest in biomedicine, and I think we're also into, I mean, peace and trade, broadly, internationalism, and so on. Give me the spiel.

Did I nail everything? Or give me Jeremy on Jeremy.

Jeremy Howard (0:34)

Yeah, no, pretty much. I mean, I'll say maybe fast.ai, most people know us for the course, because that's how most people interact with us, but that was only one quarter of it. So fast.ai was all about trying to avoid a kind of massive centralization of power and inequality due to what my wife and I saw in 2012 as likely to be a rapid growth of AI.

Balaji Srinivasan (0:57)

So similar to OpenAI's mission.

Jeremy Howard (0:59)

In theory, except we actually were open.

Balaji Srinivasan (1:02)

At least that was their initial mission.

Jeremy Howard (1:04)

So we basically decided to get AI into the hands of as many people as possible, including people with few resources. So we did a lot of research to figure out how to make AI more accessible, because at that time only five labs in the world. The techniques to actually use AI in practice were not published. They were kind of like little recipes. So my wife Rachel actually asked earlier when he was presenting in 2012 or something about some of his work, and it's like, okay, so how did you actually do that bit? What weights did you use? What fine tuning methods did you use? He's like, oh, we don't publish any of that. That's our bag of tricks. So we were like, okay, this is not okay.

This technology is going to change the world, and it requires a bag of tricks. You have to go to Stanford to learn. So we figured out all the tricks and built a lot more tricks of our own. And then everybody then tried to make it all about money. So then Google eventually started creating TPUs and stuff, instead of saying like, oh, you can't. I remember Jeff Dean saying, there's no point trying to do stuff with AI unless you're at Google, because only we have the skill to compute. Yeah. And we beat them in a global competition to train ImageNet.

Balaji Srinivasan (2:22)

I mean, at Kaggle?

Jeremy Howard (2:23)

No, at that fast AI.

Balaji Srinivasan (2:26)

Oh, really? I didn't actually know that.

Jeremy Howard (2:27)

Yeah. Yeah, there was a global competition called DawnBench. And we competed against Intel. They had like a class of 1000

Balaji Srinivasan (2:33)

DawnBench?

Jeremy Howard (2:36)

D-A-W-N-Bench, D-E-N-C-H.

Balaji Srinivasan (2:38)

By the way, I love I'm friendly with Jeff Dean. I think he's amazing.

Yeah. Yeah. So that's actually pretty impressive. I mean, I'm sure he was impressed that you're able to do so much. Oh yeah.

Jeremy Howard (2:46)

No, he was great about it. You know, they published a post, they published a paper and they credited us. And that's not hard feelings, you know, but we just wanted to say like, no, you don't have to be a rich Google person to be successful.

Balaji Srinivasan (3:03)

Actually, maybe you can talk about that because like, that's a little surprising to me because, you know, obviously DeepSeek has brought costs down recently, but back then, was it, did you, like, obviously Google had massive amounts of clean data and huge compute resources and so on. Why could, how could the student projects be competitive with Google during Dawnbench?

Jeremy Howard (3:27)

Because these big labs suffer from being over-resourced. So, in fact, not as bad now, but particularly around that time and for the next few years at Google, you were explicitly rewarded for using more compute. Where else we were like, hey, we don't have much money, like we made no revenue, we had no grants, it was just my wife and I put our own money into Fast AI.

Balaji Srinivasan (3:52)

How were they rewarded for using more compute?

Jeremy Howard (3:54)

So they were basically, if you could use more TPUs, that's like a good tick on your performance.

Balaji Srinivasan (4:00)

No, really?

Jeremy Howard (4:01)

Yeah.

Balaji Srinivasan (4:01)

Wow, okay.

Jeremy Howard (4:02)

So, you know, we came along and said, hey, like, so for example...

Balaji Srinivasan (4:08)

Is it because they wanted people to use the TPUs since they were a new product?