Five Decades of Neural Networks with Geoffrey Hinton

**Tom Mitchell** (0:00)
Welcome to Machine Learning How Did We Get Here? I'm Tom Mitchell. Today's episode is an interview with Geoff Hinton, one of the pioneers in the field of neural network learning. Geoff started out early, as you'll hear, in the 70s, 1970s, and has continued working in neural networks ever since. During the period of the 1990s and early 2000s, when neural networks were really in disfavor in the field of machine learning, Geoff nevertheless persisted, and he co-led the triumphant return of neural networks in the form of deep networks in the 2010ish period. In 2018, Geoff, along with Joshua Benjio and Yan LeCun, received the Turing Award in Computer Science. That's the highest award given in the field of computer science to researchers. In 2024, Geoff, along with John Hopcroft, were awarded the 2024 Nobel Prize in Physics for their work on artificial neural networks. I hope you enjoy the episode.
I'm pleased to have with me today Geoff Hinton, one of the pioneers of machine learning. Geoff, great to see you again.

**Geoffrey Hinton** (1:45)
Thanks for inviting me.

**Tom Mitchell** (1:47)
What I'd like to do today is get two things, two types of things from you. One is your own personal history and how you got into this field and what happened after you did. And the second is kind of your perspective on the whole field of machine learning, AI and how things are turning out.

**Geoffrey Hinton** (2:11)
So when I was in high school, I had a very smart friend who was a very good mathematician and read widely, unlike me. And he came into school one day and talked about how memories might be distributed over the brain rather than localized in a place like a hologram. Because this would have been 1966 and holograms have just come out. And that got me interested in how our memories are represented in the brain.
And I've been interested in that ever since.

**Tom Mitchell** (2:39)
Now, when I met you, we were both at Carnegie Mellon. It was 1986 when we really got to do some work together, teach course together. What, how did you get from 1966 up till 1986? What was the answer?

**Geoffrey Hinton** (2:58)
I'm slightly rocky. So I went to university, I studied physics, chemistry and physiology. And in physiology, in the last term, they're going to teach us how the central nervous system worked. And I was very excited. And they taught us how action potentials are conducted along an axon, which wasn't what I meant by how it worked. And so I switched to philosophy, that was even less useful. And then I switched to psychology, which was completely hopeless. And then I became a carpenter. And after I've been a carpenter for about nine months, I met a real carpenter and he was so much better than me, I decided it'd be easier to be an academic. So I went to graduate school in Edinburgh with Longett Higgins, who had published interesting stuff on using neural nets for our memory. Unfortunately, around the time I arrived, Winograd's thesis came out and he switched his allegiance to symbolic AI and gave up on neural nets. And so I spent five years as his graduate student with him trying to persuade me to give up neural nets and he never succeeded.
In the end, he was very helpful to me, but for a long time there was a lot of argument about how I should really be doing symbolic AI and all this neural net stuff was complete nonsense. And everybody else in Edinburgh believed that neural nets were nonsense. With actually a couple of exceptions. There was a postdoc called David Willshaw who had done associative memory. He had basically done something quite like Hopfield nets, but a long time before Hopfield. And Aaron Sloman was a visitor for a while and he was more sympathetic.
But basically, they all knew it was rubbish. And they would explain to me how neural nets can't even do recursion. So because everybody believed in recursion then, I actually figured out how to do true recursion in a neural network, and implemented it on a machine with, I think by then it had 192 kilobytes of memory, and it was only shared by 40 people. But it had a huge disk that had two megabytes, so you never ran out of memory because it used virtual memory. And I actually implemented a little neural net that did true recursion. That is, in the recursive call, it used the same neurons and the same connection strengths for the recursive call as it did for the high-level call. Now, to do that, of course, it had to offload all the parameters of the high-level call into some short-term memory, and onto a stack eventually. And I figured out how to implement a stack with associative memory in a neural net. So I had this little neural net running that was doing full recursion in neural nets, and that was the first talk I gave. And people were very puzzled. They said, why would you want to do recursion in the neural net? I mean, it's so easy to do in POP2, which was our unfortunate bastard child of Pascal and Lisp. Well, I don't think Pascal existed then. So, yeah, so I keep meaning to go back to that.