Reading Minds from Shared Latent Space

**SPEAKER_2** (0:03)
Everyone is talking about ChatGPT right now, but are you actually using it to the max? How do you use ChatGPT? It's an interview show where the people at the forefront of technology show you how they use ChatGPT in their work and their lives. Host Dan Shipper talks to programmers, writers, founders, academics, tech executives, and others to walk through all of their ChatGPT use cases, including historical chats step-by-step.
They even use ChatGPT together live on the show to build apps, analyze their leadership qualities, read more deeply, and do the best work of their lives. Listen to How Do You Use ChatGPT from Dan Shipper and the team at Every, wherever you get your podcasts.

**Paul Scotty** (0:48)
The first step that we're going to is get these complicated machine learning things that are currently only able to be done in an academic setting with a huge data set to work with very limited data. Hopefully, do things in real time, be able to do things with very constrained data sets and still get high quality results. And furthermore, be able to generalize this to different kinds of outputs that are more relevant than just perception. So for instance, we are collaborating with the University of Minnesota to try to do mental imagery reconstruction, like visualizing an image and recreate that. You can imagine that there's a lot of different pipelines that would be relevant here, like memory decoding, stuff like that.
We're working, for instance, with Princeton to collect new data. We're trying to have more collaborations with academic institutions and the broader open science community.
And in that sense, we are open for people to help out. If you think you can help us with these projects, we have several projects going on right now on the MedArch Discord. It's an opportunity to work on some cutting edge research together.

**Nathan Labenz** (1:55)
Hello, and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week, we'll explore their revolutionary ideas, and together, we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my co-host, Erik Torenberg. Hello, and welcome back to The Cognitive Revolution. With everything in AI going exponential all at once, I'm often amazed at what manages to fly under the radar. Last year, for example, we had Tanishq Matthew Abraham, founder of MedArc, on the show to talk about MindEye, an AI system that literally reads minds in the form of fMRI brain scan data and reconstructs the image that the patients had seen at the time of the scans. Of course, there were some limitations to this technology. At the time, it took 30 to 40 hours worth of scans to create a new model for a single human observer. Today, however, my guest is Paul Scotty, lead author on the recent follow-up, MindEye2, which advances this technique by creating shared subject models, still with remarkably little data and compute required, that can be fine-tuned to a new subject and deliver remarkably high-quality image reconstructions with just one hour of new scans.
In this conversation, Paul shares the details of how they pulled this off, helping me understand the nature of the fMRI data that they used, what the visual cortex is believed to understand, and how much that can vary across individuals, how they overcame that anatomical variation to create a shared subject plate in space, how improvements in foundation models like Stable Diffusion XL are making everything easier, faster, and cheaper, the opportunities that this creates for new clinical and scientific applications, the prospects for a neuroscience foundation model trained on millions of hours of data spanning multiple scan modalities, the current limits and future potential of brain reading technology more broadly, and how to think about the risks and governance challenges that may arise as this technology matures.
While in the grand scheme of things, we're still quite early in the development of neural decoding technology, and a general purpose consumer-grade brain-computer interface remains science fiction for now, Paul's work, especially considering the very minimal incremental compute that was required to produce it, shows not only that an AI-powered understanding of the brain is increasingly feasible, but also that all the investments currently being poured into foundation model research and training are starting to pay off in all kinds of unexpected ways across the full range of human endeavor.
As always, if you find this sort of pioneering work as fascinating as I do, please take a moment to share this episode with a friend. I think everyone deserves to know that AI can read human minds. And as always, you're welcome to reach out with feedback either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. With that, I hope you enjoy this deep dive into AI-powered decoding of human brain activity with Paul Scotty of MedArc and Stability AI. Paul Scotty, welcome to The Cognitive Revolution.