Episode: How to prove ROI from AI product investments

**Prasanna Vaidya** (0:06)
Welcome to GydeBites. I'm your host Prasanna Vaidya. Building an AI product is no longer the hard part.
It's quite easy these days because the code is generated by coding assistants, the assistants, the PRDs are generated by AI agents and so on and so forth. But proving its value, it's still very, very hard.
Organizations are embedding AI into their products, their workflows, their customer experiences at an unprecedented pace. So everyone is embedding AI into their workflows. Everyone is doing API calls, AI calls tool for making the product smart, intelligent and so on. But when the leadership asks, what's the return on this investment? Many teams struggle to provide a concrete and clear answer to this. So how should organizations measure success of their AI initiatives? And what separates these initiatives that create lasting or concurrent business value from those that do? To explore these, I am joined by Mike Akeroyd, who is SVP of AI and Product Management at MASA. He has over 20 years of experience in product leadership. He has driven development and growth of digital products across healthcare, fintech, e-commerce and more. He has also led large scale initiatives spanning AI, machine learning, personalization and digital transformation. Mike, welcome to GydeBites. I am so grateful to have you here.

**Mike Akeroyd** (1:37)
Thank you for having me. I appreciate it.

**Prasanna Vaidya** (1:39)
All right, Mike. So let's get it rolling straight away. Why do you think so many AI product investments fail to show clear business value or ROI?

**Mike Akeroyd** (1:51)
Well, I think it's because they get funded as an experiment, but then you try to judge them as an investment. And honestly, it's just because AI is so powerful. Think about how product managers operated five, ten years ago. How did they do this? So if you wanted to work on a product or a project, you need to build a PowerPoint deck, write a doc, build a business case, maybe do some wireframes, some concept mocks, approach your leadership, go to a committee, get an investment to build that prototype. Now you can skip all of that right there and go right to the prototype. And it's about having something cool to show. So for example, that if you wanted to build a CS chatbot several years ago, you need to go out there and say, this is what we want to be able to do. We're going to build this chatbot to reduce AHT because it's handling some of the workload upfront. Then from there, we're able to do more efficient tasks on the back end with those CS agents.
Whatever that value is, you're defining that in the business case. Now, you're just starting with, hey, we have our knowledge base, embedding that into an LLM. Look how cool this is. Can we start expanding on that?
That's not the right approach. I think that it's just really not connecting the value of what you're trying to build. You're just starting from a demo and expanding there.

**Prasanna Vaidya** (3:17)
Well, that's an interesting perspective because, I mean, for the last two, two and a half years, what has been very exciting is the demos are always great, right? I mean, the demos are fascinating. Everyone is bowed and so on. Mike, but then what should leaders actually measure before investing in an AI initiative?

**Mike Akeroyd** (3:44)
Well, I think it's still trying to understand your business from the ground. So it's not just your upstream all the way down to your downstream metrics. It's really just trying to understand what those technical details are. So your costs, your revenue, your volume, your variability of a certain metric, your risks.
I know that seems very high level, but let me try to ground this a bit. So at Amazon, one of the things that I was working on was the address book.
We knew that if a customer came in, typed in an address, and we tried to match that to a source of truth. If there was an error there, we would pop up a UX, and if the customer said no to our recommendation, we knew what the delivery failure rate would be. If they accepted it, we knew what the delivery failure rate would be for that. We know that it would be better.
So it's not just about the delivery failure rate though.
It's what is the cost of a delivery failure? So we know that, okay, that is a reattempt cost, that is the cost of a concession, that's the cost of a third-party additional fee. So all those things are tying together. And if I introduce a machine learning model to this, and I know that this improves this right here, I could then tie that to delivery success was this, now it's this, and now I know that the value that I'm creating is that delta by introducing this machine learning model to this. So tying it all the way down to a cost, a volume, a risk, variability, that is how you need to start understanding what those metrics are in your business before you start building the AI model.