𝐀𝐠𝐞𝐧𝐭 𝐏𝐞𝐫𝐟𝐨𝐫𝐦𝐚𝐧𝐜𝐞: 𝐁𝐞𝐲𝐨𝐧𝐝 𝐀𝐜𝐜𝐮𝐫𝐚𝐜𝐲

Monika Aggarwal (0:01)

You're listening to Agent Sense, the podcast where we keep AI simple, practical and grounded. Hosted by Monika Aggarwal and Frank Chavez.

Frank Chavez (0:16)

Welcome to Agent Sense, where we make enterprise AI practical. I'm Frank.

Monika Aggarwal (0:21)

And I'm Monika.

Frank Chavez (0:22)

Last month, I called a banking support line with a simple account question.

An AI agent picked up the call and understood the question.

It even got me with the right answer. The problem was it took 15, maybe 20 minutes to get there. But then I had already solved the problem myself. A correct answer but a broken user experience. So how do you know if your AI agent is actually performing well? Traditional software is easy to measure using response time or availability. But agents are different. They can give the right answer, but still fail the user being too slow. So today we're joining by Jay Migliaccio, Technical Product Manager for IBM Watson X-Orchestrate, who has worked in various early-stage software companies in product management and sales roles. He is currently joining, building agentic solutions for enterprise customers. Jay, welcome to Agent Sense.

Jay Migliaccio (1:19)

Thank you, Frank. It's great to be here.

Frank Chavez (1:21)

Yeah, let's start with a question. When we have a team saying our agents need to perform better, what should they actually measure?

Jay Migliaccio (1:31)

Frank, I think the age-old wisdom holds true here. You want to start with the outcome in mind, right? So your users, they want correct answers quickly. And for users, the definition of quickly or performance would be, how long did it take to get my answer? Solution architects, on the other hand, they want scale. For the architect, the definition of performance would be, how many user questions can I answer at once?

In addition to the time it takes the agent to respond, you should also consider accuracy, cost, user experience, and business outcomes. The important shift is from how quickly did the agent answer, to a specific question, to did the agent accomplish what it was supposed to accomplish?

Monika Aggarwal (2:14)

So the response which the agent gives could be correct, but if the task did not get completed on time, like in Frank's scenario when he was talking to his account, customer care, or the experience was poor, it still means, so let's say an agent is 95 percent accurate, right? That sounds really impressive. Why do you think that number still might be misleading?

Jay Migliaccio (2:48)

Great question, Monika. You have to remember that accuracy measurement alone can hide other important failure modes.

An agent may answer correctly, but use unnecessary tool calls, take too long, consume too many tokens, or fail on more complex workflows. We need to evaluate the complete execution, what the agent understood, what decisions it made, what tools it used, what actions it took, and whether the outcome was successful.

Frank Chavez (3:16)

It seems that it's not only the configuration or the prompt, it means that the outcome is also important for evaluating the performance. Jay, for the product management and engineering perspective, what metrics do you put on an agent performance dashboard?

Jay Migliaccio (3:31)

That's another great question. Here for Watson Orchestrate product, we learned from our customers that building and deploying secure agents is not enough. To be successful with agents, enterprises need a holistic view of their agent.

For me, I would group the metrics into four areas. First, outcome metrics, task completion and success rates. Second, quality metrics, accuracy, relevance, hallucination rates, and whether the agent followed the expected process. Third, operational metrics, latency, tool calling efficiency, message failures, and cost. And lastly, fourth, user metrics, satisfaction, feedback, escalation rates, and continued usage. Looking at these together gives a much more realistic picture of the agent.

Monika Aggarwal (4:13)

You're absolutely right, Jay. I think that is the right way to look at the agent performance holistically.

We really need to look at all of these metrics together. So for a customer service agent, for example, did it complete the task? Did it get the right answer? And what was the overall experience of the user would be key, I would think. So now let's say, you know, as a head of customer operations, you hear that the agents are underperforming. How do you know whether to change the prompt, the model, the tools, the workflow, the data? You know, how would you start as a product manager of this large agentic platform provider? How would you actually provide guidance on troubleshooting this?

Jay Migliaccio (5:04)

The shorter answer is to define your evaluation strategy before you launch. Set your performance objectives and select measurable outcomes. For example, I need my agent to be X percent accurate, or I need my agent to respond in X number of seconds or minutes. Then once you have these goals and metrics, use the outcomes to create a baseline.