The Future of AI Security is Here with Adam Wenchel, CEO of Arthur.ai

**Erik Torenberg** (0:00)
Turpentine is a network of podcasts, newsletters, and more, covering tech, business, and culture, all from the perspective of industry insiders and experts.
We're the network behind the show you're listening to right now.
At Turpentine, we're building the first media outlet for tech people by tech people. We have a slate of hit shows across a range of topics and industries, from AI with Cognitive Revolution to Econ 102 with Noah Smith. Our other shows drive the conversation in tech with the most interesting thinkers, founders, and investors, like Moment of Zen and my show Upstream. We're looking for industry-leading hosts and shows along with sponsors. If you think that might be you or your company, email me at erikaturpentine.co. That's E-R-I-K at turpentine.co.

**Adam Wenchel** (0:45)
It's a pretty manual review process that can take months. If there's a problem, like someone's exploiting a weakness in the model, oftentimes the easiest thing to do is to put in a rule up in front of the model, because you can do that in a couple of days, whereas it might take you literally six, eight, 12 months to get a new model.
Metrics like helpfulness or readability or a concision, how often does a model hedge? How often does it hallucinate? These are the kinds of metrics that I think you need to start to really think about to build a system that's most helpful to the people using it and that provides the most value in your organization.

**Nathan Labenz** (1:21)
Hello, and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week, we'll explore their revolutionary ideas, and together we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my co-host, Erik Torenberg. Hello, and welcome back to The Cognitive Revolution. Today, I'm excited to share my conversation with Adam Wenchel, CEO of Arthur.ai, a leading provider of AI security solutions that says simply, We make AI better for everyone. Now, if you listen to this show, you know that companies of all sizes are racing to implement LLMs for their revolutionary speed and efficiency. But of course, they're also worried about the risks stemming from their unpredictable behavior. And this is where Arthur comes in. Their tools, including Arthur Shield, which the company describes as the first firewall for LLMs, and also Arthur Bench, which they describe as the most robust way to evaluate LLMs, help their enterprise customers in such high-stakes compliance-centric sectors as finance, healthcare, and computer security to monitor LLMs in production to detect problems and to prevent harmful outcomes.
In our conversation, Adam, who started Arthur as an AI security company in 2018 before GPT-2, shares his unique perspective on the AI security landscape, drawing on years of experience building commercial AI systems. He describes the sorts of attacks he originally set out to detect and defend against, explains how priorities have changed for boards and executives with the surge in LLM adoption, and outlines the techniques that Arthur has developed specifically for LLMs, including using one LLM to evaluate another in context.
Along the way, we touch on benchmarking, performance metrics, standards for responsible use, and the future of AI governance. Adam believes that effective security systems will accelerate beneficial applications of AI, and his insights are directly relevant for any organization implementing AI today.
As always, if you're enjoying the show, we'd love a review on Apple Podcasts or Spotify, or simply a share on social media. This is the best way to help others find the show. Now, for an authoritative overview of the nascent field of LLM security, I hope you enjoy this conversation with Adam Wenchel, CEO of Arthur.ai. Adam Wenchel, welcome to The Cognitive Revolution.

**Adam Wenchel** (3:57)
Hey, thanks for having me on, Nathan.

**Nathan Labenz** (4:00)
My pleasure.
So, I'm excited about this conversation. You are the CEO of Arthur.ai, which is a security company and increasingly also kind of a performance management company, I think, and we're going to get into all that. But as a way to sort of contextualize just how fast AI is moving and how even folks who have demonstrated foresight like yourself are kind of sometimes reacting to developments as they come at you. I'd love to hear a little bit of the background of the company. I understand that you started it in 2019
Obviously, that's a pretty different era of AI technology versus the one that we're in today.
So maybe for starters, give us a little bit of your background, especially what you saw at that time that motivated you to start a company and then a little bit about kind of how AI has surprised you and how you've reacted to that in the time since.