Tech Titans: Lessons From Shipping Felix AI to Market with Mario Ciabarra, CEO of Quantum Metric

Mario Ciabarra (0:00)

It's not hard to create a multi-billion-dollar success. It's like execution day in and day out. It's just hard work. And you better love the hard work, or else it's probably not going to work out. You're going to end up like the kids at the soccer game that want to quit after the first game. I think our mission statement is, we help companies build cultures, maniacally focused on winning the hearts of their customer.

Joel (0:25)

I'm actually curious, because I got an e-mail the other day that said, it was a market research report, and they said one out of every eight transformation AI type projects actually sees the light of day and succeeds. And I was like, well, it's like a 90% failure rate.

So I'm curious with that information, first of all, that's where we could start. Is that true? Do you see a lot of people failing to achieve these GenAI projects? You do?

Mario Ciabarra (0:51)

Okay. I think that in itself is a myth. I think it's way higher, because I think we, like any new technology where, wow, like this could solve everything. And it's a panacea. And then we start to get, I think Garner calls it this trough of disillusionment, et cetera, like there's this hype cycle. And what is it that we see working?

I think Sam Altman of OpenAI, as we all know, he knew early on, I was in a conference where he shared, I think the early successes, 24, 25, will be in call center technology. Like he knew it. And I think part of it is because it's like the paradigm shift. Like if you're a technologist listening to this podcast, you probably can relate with this concept of, like, you know, deterministic outcomes. You know, if I give a program the same inputs, I'll end up with the same outputs. And that's not generative AI. We know this in our own personal consumption of generative AI. We ask the same question seven times, we get seven different answers. Sometimes they're pretty similar, sometimes they could be on different ends of the spectrum. Sometimes a yes can be a no when we ask it a very direct question. And that doesn't make sense in our programming paths. And like the fundamentals of what we learned about technology doesn't make quite sense. And so I think like there was this, wow, it can do anything perspective and so we can apply it anywhere. And I think we've all come to realize that's not true.

Where can we apply it? And so we saw this first wave of, I'll just use the words like co-piloting, because we realized like the machine can't do it perfectly. So we're going to need a human and loop and a human plus the machine can do better. And so we're trying to figure out like what are the applications where this drives real economic benefit in organizations. And so I've seen a lot of people say that, you know, like GenAI, like, hey, we have GenAI X or Y, or we rebranded, or we pivoted to GenAI. And most of those, I agree with a strong taste of skepticism because there's a lot of people saying they have something that does something with GenAI that doesn't really have economic benefit. And it works great in a single demo.

And so I, you know, like I look at like the, the bar to pass of like, is this one of the successes? Does it work at a high consistency rate for the objective it was trying to do? Not did it work in the demo?

Because people always show me good demos. I think one of like the, the veils of GenAI, it's so easy. It's like, it's like putting two Lego blocks together, like you can get it to work so quickly.

But you can't see does it work consistently without you testing over and over in different datasets. And I think that's one of the things that, you know, if people walk away from this podcast thinking, it's not about getting the two Lego blocks together and you can get it to work, because like linking an API into like Gemini is super simple. We could be done by the end of the hour. And you could say, wow, Mario works. But it's really about that patience for, how do you test and determine its consistency? Like, does it really work the majority of the time? And I think that's like the big leap, bringing a GenAI project to market. Can you get it to work in all different sorts of circumstances for whatever project you're trying to solve for? So we had this idea, we tested the story of it working, and it resonated strongly with executive leaders. And then we iterated, iterated. And I think some of the takeaways that I would have had were like finding the right data set to get to the right answer in the most consistent way, was probably the most secretive piece of our success. It's really hard to figure out what goes in really changes on what comes out.