Measuring the performance and business impact of AI agents - with Todd Olson, Founder and CEO Pendo

**Ray Rike** (0:08)
Welcome to today's episode of the AI to ROI Podcast. Today, I am joined by Todd Olson, founder and CEO of Pendo. We'll be covering four topics with Todd today. First, the impact of AI agents in measuring productivity for digital workers. Second, the adoption and the ROI measurements of agentic software. Third, transitioning to outcome-based pricing. Is it really possible at scale in the AI world? And fourth, the SaaS and AI convergence. So with that, Todd, can you pick a moment to give a brief background of your journey to becoming a guest here on one of the first episodes of the AI to ROI podcast?

**Todd Olson** (0:48)
Well, thanks Ray, and it's great to be here. I'm honored to be one of the first guests of this podcast, and it's a super exciting topic and one that I'm very passionate around. And look, so I am the CEO, co-founder of Pendo. Pendo is a roughly 12-year-old software company whose focus is on improving experiences in software applications. Our products essentially embed themselves within applications itself, capture analytics and how people are engaging in using those analytics, and then surface-back insights to teams to make it better. And we also complement it with tools to actually improve the experience, teach users how to use it more effectively. Our goal is like driving adoption of technology so that it ultimately, honestly delivers on the ROI and promise, which I think connects very nicely with this podcast.

**Ray Rike** (1:34)
Well, I love it. And Todd, you know, it was over three years ago, you were one of my early guests on the Metrics That Measure Up podcast, and we were talking about how product analytics was so key to the evolution of product-led growth. But now, come on, it's 2025, we talk about agentic AI and native AI applications and AI everything, right? So let's start because I saw your agent analytics announcement a few months ago, and I'm like, this is exactly what Fortune 1000 and early stage companies alike need, and that is how are their AI investments delivering ROI. So can you share a little bit about what agent analytics are in your vision?

**Todd Olson** (2:14)
Absolutely. Yeah, thanks for that.

**Ray Rike** (2:15)
Yeah.

**Todd Olson** (2:16)
No, I think you're completely correct. When we talk to our customers and large enterprises, we heard the same thing. Everyone has a mandate right now to run experiments, to try AI, to actually launch AI in the organizations. But what we also learned is that people didn't have a really effective way to measure whether those pilots, whether those AI investments were actually yielding success for the users, henceforth the ROI. We step back and really, if you think about this problem, it's very similar what we would do for traditional applications. We're capturing and measuring data and how people are using traditional applications and feeding it back to make it better. What we needed to do is to expand what we collect to get more insights into what the agent and the user are doing so we can start essentially providing insights for those types of applications.
That was the thesis around agent analytics. What we do is we leverage the existing installation we already have. You can then direct us towards the gener of AI agents or agentic interfaces you have, register them with Pendo, and we'll start collecting the actual conversational information. By having that and processing that, we can start understanding what are the types of questions people are asking? How good are the responses? What percentage of your user base is leveraging what types of themes? Are they coming back and asking the same questions because we're getting good answers? Are they asking different questions? Aka, what's the retention of certain prompts, themes, areas in this agent? We're also looking at the broader context of the traditional application. When do people go to the agent and when do people use the traditional parts of the application?
What's that experience like? Do people come in, get frustrated, go to an agent, and then go back to the traditional side? What's that hybrid workflow look like? Essentially, the vision is giving people more quantitative data on how these things are working. Are they working on it? Then hopefully, giving them the tools to make it better. We'll tell them, you're really bad at this. Or people are really frustrated when they ask these types of questions, which should be the action and impetus people need to go fix it for what the users actually want.

**Ray Rike** (4:36)
Yeah, one of the things you see, like the most recent MIT Nanda report, where 95% of proof of concepts don't get deployed into full production, even though they didn't really define full production, right? I had data that said it was like 78%.