Underwriting Superintelligence: How AIUC is using Insurance, Standards, and Audits to Accelerate Adoption while Minimizing Risks

**Nathan Labenz** (0:00)
Hello, and welcome back to The Cognitive Revolution. Today, I'm speaking with Rune Kvist and Rajiv Dattani, co-founders of the AI Underwriting Company, who aim to unlock enterprise AI adoption by certifying and insuring AI agents. Their core insight is that security and progress are mutually reinforcing. Just as good brakes, seat belts, and airbags are required if you want to drive 70 miles an hour, rigorous standards for AI agent behavior and reliability are critical for society's effort to realize the potential of increasingly powerful and autonomous AI systems. Their approach, which has already won support from an impressive list of industry leaders, including Cognition, ADA, Intercom, and many more, combines three elements. Frequently updated technical standards that codify the latest best practices, periodic audits to verify adherence on an ongoing basis, and insurance to align incentives and provide financial protection when things do still go wrong. As always, in this conversation, we get into the many details of making this work, including the fact that today's insurance policies generally don't explicitly address AI risks, creating ambiguity about coverage for AI incidents. The AIUC-1 standard that they've developed in partnership with technology and security leaders across a wide range of industries, which covers data and privacy, security, safety, reliability, accountability, and societal risks. Their approach to auditing client companies, which combines analytical evaluation of technical safeguards with systematic red teaming. How the data generated by red teaming helps insurers price risk in domains where historical loss data is either limited or non-existent. How they plan to structure financial incentives so as to avoid a race to the bottom, such as that which affected the credit rating agencies in the run-up to the 2008 financial crisis. And how, even though the market may not be able to effectively insure against the largest scale existential risks from AI, the government, as the de facto insurer of last result, still benefits tremendously from the governance that this model creates. Overall, I have to say, I really like this approach, so much so that I did participate in the company's seed round alongside leading investors, including Nat Friedman, Emergence, and Terrain. Certainly, there is a place for government regulation and also for unencumbered experimentation, but I think a private sector approach to AI reliability, powered by enterprise customers' desire to move with all-responsible speed, and designed to align all parties' financial incentives with consumer and public safety, seems much more likely to get the important details right. Not just once, but over and over again, as both the technology and the market mature. For now, I hope you enjoy this conversation about how the AI underwriting company seeks to build AI confidence infrastructure and ensure the intelligence age. With co-founders, Rune Kvist and Rajiv Dattani.
Rune Kvist and Rajiv Dattani, co-founders of the AI Underwriting Company, welcome to The Cognitive Revolution.

**Rune Kvist** (3:00)
Thanks for having us.

**Nathan Labenz** (3:02)
I'm excited for this conversation. You guys are doing some really interesting innovation at the intersection of trying to make sure that everybody can have all their AI goodies, which I am very excited about getting, but also making sure that we keep things on the rails. I think this is obviously extremely important as we head into a brave new AI world. So I'm excited to dig into everything that you're doing, the standard that you've put out, and the future that you guys envision. Maybe for starters, I think you've used a pretty provocative analogy in some of your writing and the way that you've introduced the company. Maybe tell us what the downside scenario would be for AI if we don't do a good job of this.
Obviously, there's like extreme, extreme extinction, but you've got sort of the nuclear outcome, which is a scenario where we don't get the goodies, but we still maybe get some of the bad stuff, and then maybe you could compare to an industry where things have gone well that you take inspiration from.

**Rune Kvist** (4:01)
Totally, and maybe before we get to the nuclear outcome, a lot of the concerns that we're seeing in the world today relate to today's risks that are much more, in some ways, mundane, data risk, safety and security risk related to brand disasters and data leakage. Some of these age-old concerns that become much more severe in a new environment where there are many, many more vulnerabilities, and frankly, people understand the vulnerabilities much less. But actually plays out into the future. We'll move to more and more agentic systems that are going to be way smarter, and that opens up new kinds of nuclear scenarios. One has been talked about, a bunch of that relates to imagine terrorists that want to afflict a huge amount of harm if they have access to brilliant scientists that can help them create, for example, a new virus. They will be able to do that way, way cheaper. So you can imagine new kinds of COVIDs coming out, and they will be way easier for adversaries to create. You can also start to imagine what a misaligned kind of system looks like, where today that might look like a slight deception or sick fancy, but as they get smarter and smarter, they'll do some of the things that humans can do. One of the things I sometimes get questions about is like, will as they get smarter, will they still fail? And I think there are a lot of analogies to humans where a really smart CEO will no longer kind of mess up basic logic, but that enables them, their IQ enables them to run kind of end run levels of fraud. So as the kind of agents get smarter, you get different kinds of scenarios. And when we think that really opens up for nuclear scenarios, a different category of scenarios altogether is the geopolitics, some of which may not have to happen, like relate directly to AI. But if you really get national security is believing that AI will give their nation a decisive strategic advantage that might lead them to preemptively attack their opponents in ways that may look like regular military escalation, but is really prompted by AI.