Your Ranking Model Is Optimizing the Wrong Thing with Zoe Weil

Sam Charrington (0:01)

Security teams don't need another list of alerts. They need to know which vulnerabilities are actually reachable, exposed and safe to fix.

That's what hyperscale remediation from Blitzy does. Blitzy understands your entire software state, then finds known and previously undiscovered vulnerabilities in your codebase. It traces each one through the execution paths, checks the dependencies and validates the fix. Your engineers approve the plan, and Blitzy delivers batches of tested pull requests ready to review. Less alert noise, faster remediation, and confidence that fixes are validated as safe before you review them. Visit blitzy.com to see it in action. That's blitzy.com.

TikTok is known for getting you to buy things you never knew you wanted. Online retailers would love that kind of impact, but they don't have TikTok's data. So they try to show the shopper what they're looking for, and hope that sales will follow.

Optimizing for what you can measure is not unique to e-commerce. AI teams often settle twice. They lean on signals that are easy to collect, like a click or a thumbs up, and they optimize for metrics that are easiest to evaluate, like relevance or benchmark accuracy. Then they hope it all adds up to what they actually care about. If you're building a rag system, a co-pilot or an agent, you may be making the same bet. Zoe Weil's answer, at least in retail, is to look between the clicks. The company she co-founded and leads, Sequen, is built on signals most systems ignore. How long a shopper hesitates, how fast they scroll, and how many times they go back and forth between two products before adding one to the cart. And it judges the results not by what people click, but by what they buy.

I push back on two things, whether this is just age-old upsell and cross-sell with better data and where it starts to look like surveillance. But she made a point about the playbook most teams follow that I hadn't heard put this bluntly.

Zoe Weil (2:02)

The biggest innovators in the space are social media companies, and they publish a lot in this, right? But they have a wealth of data that none of our customers have.

So if you're a scientist at Wayfair or at Priceline, you're going to read these papers and you're going to try to implement it. And you know what? Oftentimes, you're successful at implementing it, except guess what? It doesn't drive the same outcomes for you as it does for them, because you don't have the data that they have. And Sequen's big unlock is that we're finally able to get you that unlock without you having that data.

Sam Charrington (2:45)

I'm Sam Charrington, and this is The TWIML AI Podcast. For over a decade, I've been exploring the ideas and innovations of shaping the future of AI through conversations like this one that help you understand what's real, what's next, and what matters. Let's jump in.

Zoe Weil (3:09)

So anything that you see on the Internet is ranked, right? Whether it's a pair of pants that you're looking for at a retailer, or hotels that you're looking for for your next travel, content, media, even the ads you see online are powered by an auction. And these seem maybe to most people to be unrelated, but every dollar of online spend and consumption goes through a ranking model somewhere.

And we power those ranking models for the world's largest consumer companies.

Sam Charrington (3:45)

Walk me through kind of the highlights of what you all are doing, what makes your approach unique, and with an eye towards or ear towards the kind of technical innovations.

Zoe Weil (4:01)

Unsurprisingly, when people think about ranking, they think about information retrieval. In fact, I went to graduate school for information retrieval, which is literally the study of search systems, right? How do I retrieve the most relevant items or documents for a given query, whether that's in a search bar or some kind of relevance module somewhere out there? And historically, and again, quite, I would say intuitively, the wisdom here has been that utility should be measured by whether or not I'm able to retrieve documents that are most semantically relevant to a user's query, right? That's a very technical side of it. And that works incredibly well. And I would say by and large, maybe it's not a solved area of research, but there have been so many advances on it. It's been pretty remarkable. I mean, the Frontier Labs have it pat down.

Or down pat, I forget what the idiom is, but they're pretty good at it. It's not really on its own and interesting, I would say, area of research. However, in a commercial context, modern recommender systems, if you think about TikTok, Instagram and the like, they're not really recommending content anymore, and they're definitely not trying to help people find things. Instead, they're bending the end user's will in subtle ways over time to make them want things that they otherwise wouldn't want, right? So that destination vacation, the latest fashion that you might not need, whatever it is that you can think of that people end up just buying on Instagram or TikTok, they do that not because Instagram and TikTok are really good at helping you find what you're looking for. No, they're very, very good at influencing your behavior. And the reason that they're able to do that is that billions of people use their apps and are frankly addicted to those apps. They open their apps 10 to 21 times a day. And when billions of people do that, you have an insane amount of data on them. And so you're able to create these incredibly immersive, I would even say addictive experiences that are hyper-personalized and dynamic. And in fact, the apps react in real time to the user's feedback.