Wharton AI Researcher: If AI Froze Today, It Would Still Rewrite Work

Ethan Mollick (0:00)

On the academic research side, this stuff can write a good paper at this point. I think people are too eager to assign bright lines to AI. Like, it can never do creativity or judgment, it does creativity and judgment. So the AIs are trained to use English because it lets us have some observability into what they're thinking. It's not a lucky thing, it is a actual design decision. If we just had Fable 5.1 right now in Astro, that's enough to change almost all of human work in the next 10 years. It's good enough.

Brian Keating (0:24)

This is Ethan Mollick. He runs a generative AI lab at Wharton, and he measures what AI does to real work with field experiments, not just theoretical forecasts. He's going to name the four human skills he thinks still hold an edge, and one of them is the thing that we scientists claim to have and can almost never define. We start with mathematics, where the experts forecast missed by a megaparsec. Getting towards the mathematical angle of things, just in the recent days, a couple of days ago as we're recording now in mid-October, there was an announcement of 700 papers and different levels of theorems and proofs and axiom development, released by OpenAI three weeks earlier.

There was an announcement that it had solved the Millennium Problem, which is a huge deal. The pace of change is just astonishing. I have my arguments against it being as useful in physics and the hard physical sciences as it is in math, because I do think math is much closer to a language, and what these things are are large language models coupled to GPUs. But I have my doubts about whether or not they can truly do physics in the way that your colleague at Penn, my friend Mark Devlin does it, or I do it. So my question is, right now, solving Olympiad-level math problems and Millennium problems is happening, but also simple mistakes are happening.

I wonder if the jagged frontier that I want you to define is going to be kind of a permanent feature, the jaggedness, or is it going to become a phase transition where we do trust it, it doesn't think there's 10 Rs and strawberry, and it doesn't do that anymore, typically. What is the underlying problem? Is the uniqueness or the unevenness of things that we see now, is that temporary or is it a permanent feature of how these machine intelligences work?

Ethan Mollick (2:11)

Really great question. So the jagged frontier's idea we came up with in our first academic paper where we were studying the effects of AI on people in consulting business, and what we identified was the idea that AI is good at some things you won't expect, bad at some things you won't expect, and you can't necessarily tell in advance what it's good or bad at in a particular way.

That means your expertise matters, in knowing what this is. Now, math is really interesting because it was the most jagged of parts of the intelligence. When my colleague Phil Tetlock, who runs a famous forecasting institute, predicted the chance of the AI would get a gold medal at the International Math Olympiad right before Chachibutihima, the odds were under 5% or something like that. And of course, the AI got a gold medal. But for a year, it was getting the hours of strawberry wrong. There was a bunch of technical breakthroughs around how AI reasoning works, but also it just turned out the models were better than we thought they were. And then last year, the same forecasting group, including a group of industry experts, were asked to predict, last September, what the chance is that the AI would help solve the millennial problem by September 2026

A year later, I think the odds were 1.7% that that would happen, right? So like as an indicator of how quickly the jagged frontier changes, right? Leave aside all the other effects on math, that's a really good indicator. I think AI remains jagged. And part of what you got at with your question was one of the reasons why. So AI works better when we have definite things to train on, right? So math, coding, whether they're right answer works great. And by the way, that's why writing lags so much because Claude writes the same kind of way because there isn't an objective measure of writing to judge against.

Brian Keating (3:41)

Well, that's a load-bearing argument that we can-

Ethan Mollick (3:43)

Thank you.

Brian Keating (3:44)

Thank you.

Ethan Mollick (3:45)

It's really doing the work here.

Brian Keating (3:46)

Yes.

Ethan Mollick (3:47)

If it was a year ago, we would have said that's a rich tapestry of answers. On the other hand, I'm an economic sociologist by training. I work at a business school, right? I probably was quantitative papers, large-scale data sets.