Understanding Open Telemetry and Observability w/ Splunk's Spiros Xanthos

**Dan** (0:00)
In the digital world, data is everything. The internet runs on it. The global economy relies on it.
We monitor, analyze and act based on data. To explain the new age of open telemetry and data tracking, we're joined by Spiros Xanthos. Spiros is the founder of Omnition, and VP of Product Management, Observability and IT at Splunk. Now, let's go behind the data.

**SPEAKER_2** (0:32)
This episode is sponsored by Linear B. Give your dev team the power to improve with team-based metrics, high-risk code alerts, and the world's first project board based on real-time Git activities.
Sign up free at linearb.io.

**Dan** (0:47)
Spiros, thank you so much for joining the pod today.

**Spiros Xanthos** (0:51)
Glad to be on the podcast, Dan. Thank you for inviting me.

**Dan** (0:54)
It's awesome to have you here. We have a bunch of stuff that we're gonna talk about today, ranging from your work that you're currently doing at Splunk. You have an open-source project that you're working on, open telemetry.
But what we're gonna start out with is your somewhat recently acquired startup Omnition. What I see has been going on in the community for a while is kind of the microservice architecture approach, which has a lot of pros to it. You can split up your teams, you can release faster. There's less dependency per team. But when it comes to tracing and finding the root cause of errors, that's something that's pretty tough.
Could you talk to us about how you and your acquired company have kind of changed the game in this tracing space?

**Spiros Xanthos** (1:51)
I guess I can probably start by maybe describing why is it so much harder. Obviously there are many benefits like you described, faster velocity, independent teams, independent releases, all of that. But what has happened is as we have moved towards microservice-based architectures, and also as the underlying infrastructure has moved to containers and Kubernetes and serverless, what we introduced is actually a very dynamic infrastructure that changes multiple times a day.
Typically when I make a new release, I recycle the entire container.
So a lot of churn at the infrastructure level, and then a lot of distributed systems problems at the application level. It's not longer about understanding what has happened within, let's say, my monolithic application, or maybe my interaction of my application with a database. It's about all the interactions that are happening across microservices, and I guess the underlying infrastructure that is itself dynamic. So we're in a situation where we have a highly dynamic application infrastructure that changes multiple times a day and highly distributed on top of it, right? So it's very, very difficult to reason about a problem when it happens. Most of the issues we're facing are, let's say, unknown unknowns because of the many, many, many kind of failure patterns that you can encounter in a situation like this. So that was essentially our observation that the existing tooling that was available before, whether it's for monitoring or APM, was not cutting it any longer. In some sense, in the past, we have been using, let's say, metrics, custom metrics often, and maybe logs to do monitoring and availability troubleshooting.
Oftentimes when an application became important, oftentimes associated with the amount of revenue it produced, we might have introduced APM, which was heavy and expensive. And maybe APM gave us the ability to understand our performance and complement, let's say, the performance and availability troubleshooting I might have been doing with metrics and logs.
Our realization at Omnition was that actually distributed tracing was not just a better APM. It wasn't necessarily the APM for distributed systems, but it was much more fundamental. You need actually distributed tracing because it has the full context of how the application is executing to be able to do even basic monitoring and troubleshooting, right? Without that, you might be able to know how the application behaves end to end, and you might be able to know how an individual service behaves. And you might be able to troubleshoot an individual service, let's say using logs or metrics. But in reality, most of the problems are because of all these interconnected pieces. And distributed tracing is the, let's say the piece of telemetry that gives you now the fabric that connects everything together, does what we call context propagation. So for every request, I can follow how it's executing and what pieces of infrastructure is touching or backend systems or databases. And it's actually fundamental, right? Without that, it's very, very, very, very difficult to do even basic availability, troubleshooting or monitoring, right? So that was our realization, right? So we thought that distributed tracing is not the next generation APM. It is actually a lot more fundamental in being able to monitor and troubleshoot distributed systems and applications.