Gemini 4 Argon and the Benchmark That Matters: Customers

Nate B. Jones (0:00)

You can stop reading those AI leaderboards. I promise you, you're not gonna miss out. Benchmarks are dead. Google's new model Argon sits on top of one, and almost no one can use it right now, so no one can tell you whether it's actually useful. The AI that is winning right now is AI that is built by people who obsessively use it and fix what breaks. That's the new benchmark, customer obsession. And it's the same test for the AI that your company is building. Right now, if you go through the different labs and the different models out there, we see a really clear map of lab to customer, except for one instance. So I'm going to go through. Anthropic, they focused on enterprise. They focused on how you can professionally use AI to succeed.

OpenAI, they have a growing enterprise arm, and they also have a very, very big consumer play.

If you look over at Meta, Meta is deep on consumer, and Muse is the latest AI release there, and their whole motion, their whole push from Zuck is all about AI for consumers. What about Google? Well, Google makes a ton of money from compute. Google obviously serves a lot of people with search, and Google has Gemini. What I don't understand is how Argon supports those use cases.

Why do we need that kind of cutting edge model? And that got me thinking about the importance of having tight customer product relationships in the age of AI. And that customer obsession is something I would love to see more of from Google, because I'm someone who thinks that the world is better off with more AI, not less. And especially in a world where we're talking about powerful AI, you want that power in the hands of more people, not less. So if you're one of those people who's worried about AI, and you're like, I don't know that I trust Anthropic, I don't know that I trust OpenAI, I want other players, well, you want another player in the game? Great. Let's have Google in the game, too. I would love there to be more players, because I think that more players, generally speaking, make things safer and better for us consumers. So I want Google to succeed. And I say this with love, but I don't yet see how Argonne is connecting with customers. And I think that that leads me to some larger questions for all of us. If you work at a company, this is for you. If you work in enterprise and leadership, this is for you.

You need to understand, in a world where intelligence is evolving every single week, how do you make money with AI? And especially, how do you make money with AI when the underlying model you're using is probably going to keep evolving? You know, when I was in product, for a long time, we would just release software. And the software that we released, we could update it, you know, once a month, once every two months, and we would have a roadmap, and that's great. I don't know that roadmaps work anymore. I think a lot of us in product would say they don't. And the larger problem is that you have to build a product with an evolving intelligence underneath that also needs to help users understand what is actually going to happen to their worlds as AI changes in a good way, not in a doomer way. That's a really tall order. It's a tough product challenge. It's a challenge that literally a trillion dollars is getting thrown at right now. But if you don't start with understanding who your customer is and what they're doing, you're not going to get very far. If you think about this benchmark news, the benchmark news doesn't matter. Let's peel that back. Let's look at what's underneath. I think the best AI products are being built by people who obsessively use them and notice what goes wrong and who make those failures matter. And this is a business discipline as much as it's an engineering discipline. And I want to share an example because I ran into a very practical use case when I was using AI myself. So I shared a few weeks ago that I used Muse to review my personal subscriptions, and I talked about that in an earlier video. But I wasn't satisfied with stopping there because I also have business subscriptions. And the same problem existed across more than one part of my life, and the information was scattered across lots of different accounts and lots of different services. So on a plane flight that I took recently, I went further. I used ChatGPT's DOTS tool to connect to all of the different sources I needed. I could connect to Mercury. I could connect to other financial sources, and I could have OpenAI's built connectors help me bring all of my subscription material together, along with email to double check it. And I could use DOTS to process all of that information and to build me a financial model of subscriptions across both personal and business all in one place, which I'd never seen before. Afterward, I could take that spreadsheet and feed it to Muse and have Muse be like my hunter killer drone. Like Muse went through and helped me action against all of those subscriptions and make good decisions against it, partly because Muse is faster than DOTS right now, partly because Muse is able to really use that speed to navigate the web and get things done directly, so I can just delegate stuff like a cancellation. I actually had someone reach out to me when I shared that I was doing this and say, why would you use Muse to do that? You can literally click through and do it yourself. And I said, yeah, I can. But it's awfully nice when I am strapped for time to be able to hand a job to an AI and have it just get done. And so when we're talking about customers and we're talking about what matters, I use that story because I think it illustrates the importance of actually using the AI.