The AI magic words

Ryan Donovan (0:08)

Hello, and welcome to The Stack Overflow Podcast, a place to talk all things software and technology. I'm your host, Ryan Donovan, and today I'm reporting from the floor at AI4 conference, and talking about what is the place of books in developer education in the age of AI.

And my guest for that today is Tim O'Reilly, founder and CEO of O'Reilly Media. So welcome to the show, Tim.

Tim O'Reilly (0:30)

All right. Thanks. Well, first off, I will say that O'Reilly is about a lot more than books these days. And in a lot of ways, we define ourselves, we have for the last 25 years as a technology transfer company, learning from the leading edge to the people who want to follow.

Ryan Donovan (0:48)

Right.

Tim O'Reilly (0:49)

And we've done that through books originally, that was our original business we've done through conferences, although we stopped down our physical conferences at the beginning of the pandemic and haven't looked back, and through our online learning platform.

And increasingly today, we're using their content in our books to actually shape not just human learning, but also the learning and the actions of agents. So we can start there. But let's start back with what role a book plays. I think particularly nonfiction books. First of all, the idea of book is a very loosey goosey thing.

Ryan Donovan (1:24)

What is a book?

Tim O'Reilly (1:24)

A novel is a very different type of book than the books that O'Reilly publishes. What we publish, in a sense, are a user interface to a body of knowledge.

If you think about it, even in LM, it knows everything. But you have to know what to ask. And when you engage with the work of a particular author, you're kind of learning for how they frame the questions and how they answer them. And one of the key skills that I've always thought of as, when we were trying to develop books long before AI, was putting things in the right order. So that just when somebody was at the right point in their learning journey, they needed to know this thing. That kind of knowledge interface design is still important.

Ryan Donovan (2:10)

And I know it's been a point of pride for a lot of engineering folks, engineering leaders to get a O'Reilly book out.

Tim O'Reilly (2:18)

That's right. Because obviously it is a way of saying, I am somebody who is a recognized expert in this field. And absolutely that signaling function is still there. But we're really leaning in, in our business, into this idea of human expertise and how it still matters in AI. A lot of people in the AI field are bitter lesson field.

For those of you who are listeners who might not know the bitter lesson, it's a 2019 paper by Richard Sutton, in which he looked at the history of AI. This is pre-LLMs.

Ryan Donovan (2:49)

Yeah.

Tim O'Reilly (2:50)

But just he was looking at that chess and Go. We thought that it would really help to train the models on human expertise, but it didn't. Effectively, it was a way we got going. But in the end, AlphaZero trained, so self-trained from the beginning on Go actually beat the version that was trained on human experts. And I think the bitter lesson is a good general business principle.

It's not just an AI thing. We're in a world where scale and big data and computation do actually do a better job on many tasks. I think my first experience of the bitter lesson, way before Richard Sutton was in 1993, we launched the first catalog of the World Wide Web called GNN, and then Yahoo followed a year later, and we were curating, we were publishers. So we said, okay, we'll find the best sites on the web, we'll make a catalog, and Yahoo tried to catalog everything, and that overrun them, and then Google came along, and they use what we now would call the bitter lesson to basically just build a custom catalog for every query. So it's a real thing, but human expertise still matters. Let me just tell you a little bit of some of this research we're doing. We're calling it O'Reilly Expert Intelligence, and we found in our own work internally that referencing the O'Reilly MCP and saying, what would the O'Reilly experts say about this problem? It gives actually way better results for many classes of problems.

Ryan Donovan (4:18)

It's sort of a wisdom of the crowds.

Tim O'Reilly (4:20)

That's part of it. It's the wisdom of the crowds, but it's also the particular shaping of the query. I wrote a post two or three years ago called Magic Words.