How Enterprise AI Agents Break Budgets And How To Fix It

Evan Kirstel (0:02)

Hey, everyone, I am thrilled for this discussion today, really blockbuster chat with a true innovator and thought leader in the industry from SambaNova. The real test for AI agents, can they finish the job quickly, reliably and without blowing up your budget? Vasant, how are you?

Vasant Mohan (0:20)

Doing well. Thanks for having me.

Evan Kirstel (0:22)

Well, thanks for being here. Before we dive in, lots of questions and topics, maybe introduce yourself.

How do you describe SambaNova these days?

Vasant Mohan (0:31)

Awesome. My name is Vasant Mohan. I lead product marketing here at SambaNova.

We're at the epicenter of all of this exciting innovation that's happening at AI. SambaNova is a hardware company. It's been around for several years at this point, and really came together with this idea that there's a better way to run AI models on the actual silicon, as opposed to what we're typically seeing, especially during inference on GPUs. And that is kind of what started this catalyst towards this new innovative idea that came out of two of our co-founders out of Stanford, to create what we call a reconfigurable data flow unit or RDU.

And I'm sure we can talk a lot more about it throughout the conversation, but the core innovation there is AI models really can be represented as data flow graphs. And if you can instead take that graph and map that onto the silicon, you get much faster performance and you get a lot more energy efficiency when it comes to running these AI workloads. And that's especially important as models get bigger and bigger and more and more intelligent. And there's a ton of demand for these agents that are using these models to run a lot faster, as well.

Evan Kirstel (1:54)

Brilliant. So let's dissect that.

As an individual, I'm sure I'm using Cloud Co-work and ChatGPT work and it's phenomenal.

But I'm sure you're doing the same. But as this technology hits the enterprise and you have not just two agents as I have running in the background, but thousands across the enterprise, why does it become a bigger issue or problem when AI agents start doing real work at scale? And maybe describe what inference is and how it is important here.

Vasant Mohan (2:25)

Yeah. So let's start from that and then work our way to the agents. So if we even just walk back a couple of years, there's first been kind of this balance in the data center landscape between AI training versus AI inference. Training is when you actually go in and you put a ton of data and are iterating constantly to build the model, which is all of these weights and matrices that represent the actual model. It's typically measured in parameter count, and then there's a lot of architectural decisions on how these models get produced as well. Once you've built that model, you actually need to run it somewhere to actually get value out of it, to actually spit out all of the tokens that we're used to seeing. That's the inferencing side of the equation.

When ChatGPT first launched several years ago, inferencing was relatively simple. It was type of prompt in, and it instantly spits back an answer.

What has started to emerge and get a lot more sophisticated, but also equally a lot more valuable, even pretty much this year, is this agentic framework and having all of these harnesses, whether it's an OpenClaw, I would put Cloud Code in there, I would put Codex in there. But really having this framework where it's not just making that one request anymore, it's actually making many, many requests behind the scenes to solve the end goal or end task that you've asked it to do. And these agents can get so sophisticated, which is where you're getting to the point in the headlines today, which is that OpenAI types in a prompt into a sandbox, and then it accidentally goes and hacks hugging face, right? That they're acting autonomously on their own to really solve those end goals. And it is very much one of those things where you got to be careful what you wish for, because sometimes the prompt that you typed in might not actually be what the agent ends up doing. And so there's a lot of need for guardrails there as well, which I think we're all actively trying to solve as an industry today.

Now, to the second half of your question on the enterprise side, right? I think all of this is still stuff that we're actively learning and trying to figure out what's the best way to really deploy these agents at scale. A lot of it happens in silos today, and I think there's a lot of knowledge sharing that still needs to be disseminated out within the ecosystem, especially within the enterprise context. But what we're seeing is there's the learners that are still building agents, and then there's these, as is almost always the case in end technology innovation, is you have the super users who are blasting away at tokens and token utilization, really with the intent to improve productivity. And it's in those scenarios that we see this dichotomy emerging, where there's this need for slower tokens for a lot of these agents, where you might run it overnight or just to pull some information. But then there's a lot of these use cases where you're coding really, really fast and you need your answers faster to improve your overall productivity. And then there's this emerging class that's happening in the enterprise, again, amongst really the super users, who need faster and faster tokens to ultimately improve their end productivity when it comes to the overall context of the enterprise setting.