Do AI Tokenomics Matter More Than Model Benchmarks? with Chris Potts

Sam Charrington (0:00)

We've started to enter a phase of AI that's not just about making models smarter. It's also about making them economically sustainable.

As reasoning models consume more tokens, context windows continue to grow, and agents become embedded in more products and workflows, the economics of these systems are becoming impossible to ignore. That's given rise to a new conversation around tokenomics, how we think about the costs, incentives and trade-offs shaping the next generation of AI.

One person who's been thinking deeply about this is Stanford professor and Big Spin co-founder Chris Potts. His recent work argues that measuring AI progress requires looking beyond model benchmarks to ask a different question. What are our tokens actually buying us? Here's Chris explaining how he thinks about tokenomics.

Chris Potts (0:48)

Another interesting moment to be in, as we're all being made aware of the true costs of all this AI usage. The analogy here is like it used to cost me $20 to take a ride share to the airport, Uber or Lyft, and now it costs $90. But it's more like $20 to like $500 or something, right? I think what's happening is that the big providers are testing the waters on charging us the true costs plus whatever profit they need to make as they all try to gear up for IPOs and so forth.

And in turn, that is very quickly leading people to ask questions like what is the return on investment for all these tokens that we have purchased? And it's a very tricky area to be in because what does it mean to think about value in this context?

Sam Charrington (1:31)

I'm Sam Charrington and this is The TWIML AI Podcast. For over a decade, I've been exploring the ideas and innovation shaping the future of AI through conversations like this one that help you understand what's real, what's next and what matters. Let's jump in.

I want to say thanks for coming on. I've been looking forward to this conversation. And I think where I'd love to start us off is to really dig into your background and how it got you to where you are now.

Chris Potts (2:13)

Yeah, my background is in linguistics.

Linguistics proper, not even natural language processing. I did my PhD on, among many other things, swears, what swears are like, why we swear, what information they encode, what kind of taboos exist around them, and so forth.

And that was actually the trigger that got me into NLP because I wanted a lot of data of people swearing. I wanted to know what the context was like, what their intentions were. So, I turned to corporate, and from there, you start using NLP toolkits to add structure to those corporate. And then after a few years, maybe you're writing your own tools for doing that work. And then when you look back after 18 years or whatever it's been, you're just an AI person or an NLP person. But that is the true story. And I feel like if I had to, I could trace the lineage of every one of my current projects back to my fascination with why we care when someone drops an F-bomb.

Sam Charrington (3:12)

So are you an F-bomb dropper or did you come at it from the perspective of trying to understand these others?

Chris Potts (3:20)

I think very infrequently in my life, for my linguistics class, Semantics and Pragmatics, which is about linguistic meaning, on the final day, we always do a class on swearing. And I review the history and we kind of tie all the course themes together. My handouts for that are full of swears. But I only swear once in the lecture. I present the result that people remember things better if the utterance contains a swear, because it has a kind of emotional resonance, very primitive reaction. And so in that moment, I pick some fact from the course, some trivial thing, and I restate it with a swear and then I say, all of you will remember this for eternity.

But other than that, I'm very shy about it in the class.

Sam Charrington (4:04)

That's funny. I'm sure there is loads of research on this, but I grew up in New York City, and as a New Yorker, I think that swearing is just kind of part of my natural language and way of communicating. And I married a Midwestern girl, and she doesn't tolerate it at all. She doesn't do it, she doesn't tolerate it, she won't tolerate it for me. And it made for, we've been married for 30 years, so I adapt quickly apparently. But for a long time, it took a lot of restraint to change that way of communicating, particularly when I'm communicating about something that I'm excited about or emotional about, or want to convey the importance of.