Security Founders Talk Shop About Generative AI

**Dean De Beer** (0:00)
Every prototype in AI works perfectly, right? You grab your API key from whatever service out there, write a little script, give it data, and you get an incredible response. You sit there going, well, security is done, we don't have to worry about it, we've given it to the computers. That's once, it works once really well.
Being able to do that again and again, being able to do that accurately, ensuring that in security, you don't have the luxury of being close or good enough, or avoiding or dropping data, or excluding a record. We don't have that luxury.

**Derrick Harris** (0:33)
Hi, this is Derek Harris, and you're listening to a bonus episode of the a16z AI podcast. Last month, we held an event in our San Francisco office, featuring presentations by a handful of security startup founders, some of whom had a lot to say about generative AI. The new threats that surface is yes, but also some advice on building security products that take advantage of it. We begin with some clips from a presentation by Command Zero co-founder and CTO, Dean De Beer, who discusses some things to consider when deploying large language models and security products. And then move to Doppel co-founder and CEO, Kevin Tian, sharing some novel examples of how generative AI is both a new tool for and a target of phishing attacks. And finally, I sat down with Resourcely co-founder and CEO, Travis McPeak, after his talk, to discuss how he sees AI finally spurring some real improvements for software developers and architects. So you'll hear some clips from that as well.
As a reminder, please note that the content here is for informational purposes only, should not be taken as legal, business, tax or investment advice, or be used to evaluate any investment or security, and is not directed at any investors or potential investors in any a16z fund. For more details, please see a16z.com/disclosures.
Dean from Command Zero had a lot to say about best practices for adopting LLMs as a security vendor, but the big one was simply not to get cocky after a successful trial run. As he puts it, security teams don't have the luxury of nailing at once. They need to be thorough and accurate again and again and again.

**Dean De Beer** (2:14)
Every prototype in AI works perfectly, right? You grab your API key from whatever service out there, write a little script, give it data, and you get an incredible response, you sit there going, well, security is done, we don't have to worry about it. We've given it to the computers.
That's once, it works once really well. Being able to do that again and again, being able to do that accurately, ensuring that in security, you don't have the luxury of being close or good enough or avoiding or dropping data or excluding a record. We don't have that luxury. So being able to build those things around it, that's where it becomes a little more difficult. Our use cases, which at the time seem pretty straightforward, like what question should I ask next? What lead should I follow? Why should I follow this lead?
Are there any other steps I should take? What are the outcomes? They all seem really simple until you're required to repeat that again and again for different data that you might never see. We're not in our customer environments in the sense that I'm sitting there looking at the content that they're looking at, but I have to be sure that when I look at it, I can provide them an accurate outcome and I can do it in an automated fashion.
No hallucinations, structure of repeatable outcomes, right? You're dealing with non-determinism, you're dealing with models that, like any human, they want to respond, they want to give you an answer. So how do you control those or how do you prevent those responses? How do you minimize them? How do you wrap structure around it?

**Derrick Harris** (3:34)
Dean De Beer also brought up the issue of model selection, specifically concerns like which model or models to use, where and how to host your models, and how to optimize for inference cost and performance, or whether to use a generative AI model at all.

**Dean De Beer** (3:49)
First things first, you have to go shopping for a model. First, one size does not fit all. GPT-4 is certainly a hammer and you can use it for all things. I'd certainly not recommend it once you start trying to productize and do these things at scale with minimal latency. Your model decisions are going to be driven by your use cases. Are you looking to at least pull the security use cases, extract from an event the command line strings and analyze that script?