Neural Nets and Nobel Prizes: AI's 40-Year Journey from the Lab to Ubiquity

**Anjney Midha** (0:00)
I think the period between the initial excitement around neural networks in the 1980s and the deep learning breakthroughs in 2012 were marked by a bunch of really important foundational contributions from folks like, of course, Hinton, but also Jan LeCun and Schmidhuber, because they continued to work on sort of neural network approaches during this time, making gradual progress. And I think a few of the milestones that come to mind for me was one, of course, the CNN moment, where Jan LeCun and others in the 90s came up with these convolutional neural networks that proved incredibly effective for image recognition tasks. And then shortly after that, Schmidhuber and Hockreiter did this really important work around long short-term memory networks, LSTMs. And then building on that, Hinton and a few others developed these techniques for pre-training deep networks layer by layer, which was the foundation for unsupervised pre-training. The most notable benefit of those techniques was to help overcome the difficulties in training very deep networks. And then, of course, the star of the show became GPUs that could accelerate all the matrix map that these neural network computations need. And so while I think no individual technique during that AI winter, as I'd prefer to call it the AI autumn, led to widespread adoption of neural networks immediately, they kind of set the stage for the entire deep learning revolution that we're in the grips of right now.

**Derrick Harris** (1:18)
Welcome once again to the a16z AI Podcast. I'm Derek Harris, and I'm joined once again by a16z general partner Anjney Midha to dive into an interesting artificial intelligence topic. In this case, it's the spate of Nobel Prizes, five of them to be exact, in the fields of physics and chemistry awarded to AI researchers. Because it's a great jumping off point to explain how we arrived at our current state, we focus much more on the physics prize, which was awarded to John Hopfield and Jeff Hinton for early work on artificial neural networks dating back more than 40 years. Although Hinton, in particular, was also an instrumental figure in the deep learning movement of the early 2010s, which provided a direct line to today's foundation models and mass adoption of AI tools. We discuss the connections between neural nets, computer science, and physics, why the last AI winter was more of an AI autumn that actually laid the groundwork for some huge advances, and how we might see other fields and scientific disciplines adopt the factory-like approach to building AI models that has proven remarkably effective for AI labs. We end with a discussion of how to rejuvenate leading-edge AI research inside universities and the increasingly important role of independent builders, teams, and open-source creators in driving important systems-level advances in AI and in software in general. Enjoy! As a reminder, please note that the content here is for informational purposes only, should not be taken as legal, business, tax, or investment advice, or be used to evaluate any investment or security, and is not directed at any investors or potential investors in any a16z fund. For more details, please see a16z.com/disclosures.
So Jeff Hinton and John Hopfield won the Nobel Prize for Physics, and relatedly, Demis Hassabis and John Jumper from Google DeepMind won the Prize for Chemistry this year from some of their work on Elphafold. So at a high level, Anj, what does it mean for the field of AI to see five researchers win the Nobel Prize in a single year, and at the risk of using a locally sourced cliche? Is this confirmation that AI and computer science maybe overall is eating other scientific fields?

**Anjney Midha** (3:20)
One school of thought would be that this represents a watershed moment for AI. It's hugely validating for AI's importance across different scientific disciplines, that it signals this sort of AI's growing impact and integration into a bunch of other fundamental research areas. And I think the contrarian view or the opposing view to that is that these awards were surprising because they sort of dilute the meaning of field-specific prizes and may reflect hype more than scientific merit. And maybe the Nobel committee was jumping on the AI bandwagon, and actually they potentially overshadow much more important work in traditional physics and chemistry. I'm more sympathetic towards the former view, which is that I do think this represents a sort of crossing the chasm moment for AI. It signals how AI is moving from niche technology to mainstream scientific tooling. And I find the explore versus exploit framework relevant here, where I think basically we're seeing the fruits of sort of decades of exploration in AI now being exploited across multiple scientific domains. And so I think it's a huge win both for science and for AI in the following sense. Artificial intelligence and computer science in many ways is a meta-discipline. It's the study of general purpose computational methods that benefit specific fields and application. I think it's very exciting that we are recognizing the value of a meta-science like computer science and artificial intelligence to a bunch of other disciplines. So my hope is it ignites a lot more adoption of these tools in those fundamental science domains. And I think that we all win if it results in a lot more efficiency in the scientific method.