Modern AIOps:
What It Takes to Build Reliable AI Products

**Jesse Hall** (0:06)
Welcome, everyone, to the MongoDB Podcast Live. I'll be your host today, Jesse Hall, and joining me today is Karthik Kalyanamaran00.
Sorry for butchering your last name. He is coming to us from Langtrace, and so we have a great discussion lined up for us today. He's the co-founder and CTO of Langtrace AI, and we're going to discuss how engineering teams are building reliable AI products, they're doing things a little bit differently than the rest. And we also are going to dive into what modern AI ops stacks look like for high-performing AI engineering teams. So looking forward to this. Karthik, give us a quick background. Who are you?
Where did you come from? And what do you want to talk about today?

**Karthik Kalyanamaran** (0:55)
First of all, thank you so much for having me here, Jesse.
It's a pleasure. My name is Karthik. I'm the co-founder and CTO of Langtrace AI. Langtrace is an open-source LLM application observability platform. Prior to Langtrace, I was at Coinbase. I was the engineering lead for the observability team at Coinbase. And been in the industry for close to 10 years, graduated with a master's degree in computer engineering from Texas A&M University. Started my career at VMware building on-premise software.
Worked at a few YC startups along the way. Done zero to one a bunch of times. And also solved problems at scale at companies like VMware, HPE, and Coinbase. So that's my short background.

**Jesse Hall** (1:50)
Yeah.
It seems like the typical startup background for the majority of it, right? So tell us a little bit then about Langtrace. What is the high level? What is Langtrace?

**Karthik Kalyanamaran** (2:03)
Yeah. In very simple terms, Langtrace helps developers figure out what's happening in their AI stack. So as we all know, LLMs are very popular and a lot of developers have started adopting LLMs for building either features in their existing products or they're building fully AI-powered products. One of the challenges with building with LLMs is its non-deterministic nature.
Langtrace helps you understand what's happening behind the scenes, because if you query a model 10 times with the same question, there is no guarantee that it's going to give the exact same response all the time, and you would want to exactly know what's happening and what response the model is giving and why it's giving a particular response to basically understand and improve the accuracy of your product. That's where Langtrace comes in and helps you out. Langtrace is fully open source. It's built on top of OpenTelemetry standards. In simple terms, again, what it means is we have two components. One is the Langtrace SDK. We have support for Python and TypeScript. We also have the Langtrace client where you visualize all your traces, sort of like a dashboard. Because we are built on top of OpenTelemetry standards, you could adopt Langtrace just using the SDK and send the traces over to another OpenTelemetry compatible observability vendor, which most of the observability vendors are today.
That's what Langtrace is all about.

**Jesse Hall** (3:49)
Nice.
We'll dive a little bit more into the details of Langtrace, but let's talk a little bit about how you got to Langtrace. I think there was quite a bit of a journey from where you came from, from Coinbase, HP, VMware, etc. Then there was a bit in between that and Langtrace. Let's talk a little bit about that. What was the origin story of Langtrace?

**Karthik Kalyanamaran** (4:13)
Great question. We started the company back in 2022, me and my co-founder, we used to work at Coinbase prior to starting Langtrace. Like I said, we were part of the team that was responsible for doing observability and maintenance on all the blockchain nodes. Essentially, it's a key piece of infrastructure that accepts all the transactions flowing into Coinbase and broadcasts them over to the public blockchain.
The criticality of this piece of infrastructure is such that if any of the nodes go down, like let's say Ethereum node goes down, none of the Coinbase customers would be able to send and receive Ethereum over Coinbase.
One, because blockchains are built on different standards, typically in traditional software stack, you have proper standards around tracing and telemetry and whatnot. But in the case of blockchain, every crypto token is built differently, and because of that, the standards did not exist. It was a lot of fun to do observability for this piece of infrastructure where standards did not exist and tooling was far and few. We had to use tools that were built for the Web2 world to help us with doing observability for blockchains. Our key insight here was, why not go build observability that is custom-made for the blockchain world? That's how we got started back in August of 2022 We left Coinbase, we raised a seed round from Redpoint Ventures, and we started the company. Over the course of close to 12 to 15 months, we built a product, and around the same time, ChatGPT launched, and like every developer out there, I was pretty excited. Our entire team was excited to get hands on it. Copilot came about, we started using Copilot. Within our observability product for blockchains, we built a couple of features. One is a chatbot. Essentially, what we did at that time was we built a RAG system. Because all blockchain software was open-source, we semantically chunked the open-source code, ingested them into a vector database, put an LLM on top of it, and exposed a chatbot in our product. That's one thing. Secondly, for the logging layer within our observability product, we had this idea where, because every blockchain is different, if a person is managing 10 different blockchains, there's no chance that they would be able to tell exactly what's happening if they see an error or a warning in the logs.