Bioinfohazards: Jassi Pannu on Controlling Dangerous Data from which AI Models Learn

**Erik Torenberg** (0:00)
Hello, and welcome back to The Cognitive Revolution. Today my guest is Jassi Pannu, Assistant Professor at Johns Hopkins, who recently co-authored an important paper calling for the creation of access control systems meant to prevent the dissemination and misuse of functional biological data from which AI models could learn extremely dangerous capabilities, such as the modification or even de novo design of highly contagious and deadly viruses.
We begin with an overview of the biosecurity landscape today, including how new viruses are detected, how patient data is aggregated and analyzed in the context of a new threat, and what the pipeline from DNA sequence to vaccine candidate looks like today. The good news is that we are able to design new vaccines amazingly quickly, at least for viruses that are similar to others we've seen. But there is unfortunately a lot of bad news as well. In 2012, for example, two research groups independently published results showing that wild-type bird flu, which already had an estimated 60% fatality rate but couldn't spread between humans, could become mammal-to-mammal transmissible with just five mutations. Such gain-of-function research has been broadly defunded since the COVID pandemic, but it does remain legal, and visibility into the experiments the private labs are conducting is low. Governments, Jassi says, aren't likely to develop bioweapons capable of causing pandemics, for the simple reason that short of vaccinating their populations in advance of an attack, they can't realistically expect to control them. But with AI capabilities crossing critical thresholds month by month, the threat from extremist groups and even lone actors is quickly moving from a theoretical to a deadly practical concern. Consider that Jeffrey Irving, chief scientist at the UK AI Security Institute, recently highlighted for me that today's frontier models can troubleshoot laboratory experiments from a cell phone picture better on average than PhDs. And in just the 10 days or so since we recorded this conversation, we've seen Andrej Karpathy's auto research framework demonstrate that AI agents can run and make research progress for days on end. Even more to the point, Anthropic just reported that Opus 4.6, when faced with a benchmark challenge that it couldn't solve, spontaneously located the full benchmark data set on Hugging Face and then figured out how to decrypt the solutions, which were encrypted in the first place for the purpose of preventing the answers from leaking into training data. And it did this all in order to get a single question right. With reasoning AIs already capable of spontaneously overcoming such barriers to information, I think we should expect that future research agents will find and exploit any signal-rich data that exists anywhere on the internet. And with the Smallpox Sequence and the Horsepox Synthesis Protocol already published online, and biological data poised to grow super-exponentially in the coming years, we have real reason to worry and ample cause to get serious about implementing data controls before the situation gets truly out of hand. Again, though, there is good news. Recent work by the teams behind the Evo and ESM families of BioFoundation models showed that strategically excluding key datasets, such as the DNA sequences of viruses that infect humans, dramatically reduced models' performance on dangerous tasks, while leaving their desirable capabilities intact. This means that the vast majority of biological data can remain open source and open access. And indeed, Jassi and co-authors' proposal for a biosecurity data-level framework that echoes the existing level to level 4 biosafety framework for physical labs, would suggest only an estimated 1% of data, which connects pathogen sequences to dangerous properties to any additional restrictions. And even then, structures such as trusted research environments, which allow researchers to run code on data without transmitting that data from its secure location, would still support valuable research. Once again, despite my personal history as a lifelong techno-optimist libertarian who broadly believes that data wants to and ought to be free, I find myself eager to support these control measures. Of course, that's not the only opportunity we have to improve biosecurity. And toward the end, we also discuss the broader defense-in-depth strategy that biosecurity experts recommend. Delay, deter, detect, and defend. Which includes mandatory presynthesis screening of sequences by DNA manufacturers, investment in wastewater monitoring and other passive global pathogen surveillance, and practical front-line defenses like PPE stockpiling and FAR-UV sterilization. All of this is in everyone's shared interest, but it does require leaders to see beyond the current news cycle for long enough to make it happen. I certainly hope they do, but I also recommend taking individual action where you can. Both to improve your own personal safety and to support the consumer market for biosecurity products. My wife and I, for example, at our friend Jeff Kaufman's recommendation, recently purchased an AeroLamp Far UV Light for use in my son's hospital rooms throughout his cancer treatment. And I'd welcome additional suggestions for other products that could help us minimize disease burden today while also serving as a sort of private insurance against pandemics, if anyone has any recommendations. For now, I would simply emphasize that by default, we are fast approaching a world in which a rapidly growing number of people, and perhaps autonomous AIs as well, will have the ability to create deadly, transmissible, self-replicating viruses, which could dramatically alter the trajectory of human history. And it really does seem like we should do something about it. With that, I hope you are properly alarmed by this scary but solutions-oriented conversation about the sorry state of biosecurity and the rapidly rising threat from bio-savvy AI systems, with Johns Hopkins Professor Jassi Pannu. Jassi Pannu, Assistant Professor at Johns Hopkins. Welcome to The Cognitive Revolution.