@HPCpodcast-110: John Gustafson on AI, HPC, Making Every Bit Count – In Depth

John Gustafson (0:04)

So, if what you're paying for is kilowatt hours, going to half precision is a factor of four, at least. They call it quantization error in AI. You really didn't need to make up a new word for that, it's called rounding error. We're getting all these sparse matrix problems that now solve with 32-bit pauses that used to require a 64-bit floating point to do.

If you're doing something like nuclear, that's where I would reach for interval arithmetic. There's one place where you need to add absolute confidence. That would be the best possible benchmark for high-performance computing because you make FFTs faster, you're going to make everything faster.

SPEAKER_2 (0:38)

From orionx.net, this is the At HPC podcast. Join Shahin Khan and Doug Black as they discuss supercomputing, AI, quantum technologies, and the applications, markets, and policies that shape them. Thank you for being with us.

Doug Black (0:54)

Hi everyone, I'm Doug Black. This is the At HPC podcast and with me is my co-host Shahin Khan of orionx.net.

Shahin and I are very pleased today to have with us as our special guest, John Gustafson, who is a very well-known computer scientist and businessman and an HPC pioneer. He's known for, among other things, the invention of Gustafson's law, the introduction of the first commercial computer cluster, and the invention of new number format and computational systems. John is currently chief scientist at VQ Research and is a visiting scholar at Arizona State University. He has held senior positions at AMD Intel Labs and other companies. John, does that about cover it, or would you like to?

John Gustafson (1:44)

That covers it perfectly, thank you. Yes, the academic people think I'm a businessman and the business people think I'm an academic.

Doug Black (1:51)

Yeah, he's also the author of some very influential books in the field of computer science that we'll get into.

A very hot topic these days in AI and then AI for Science and how that works with HPC is numerical precision. You've been looking at it for decades. As I say, it's become a very hot topic.

John Gustafson (2:13)

Yes.

Doug Black (2:13)

Could you share with us some of your most recent thinking or activity in this area?

John Gustafson (2:18)

Well, first of all, Moore's Law is not getting us more speed through smaller transistors anymore.

And really, we've been communication bound for decades.

So the only way I see forward is to get more information per bit. If we're going to communicate a bit, it better mean as much as it possibly can. And we've been very sloppy about how we communicate real numbers in particular. We're pretty sloppy about integers too. But the real number crisis is really hitting the AI community. And now that they're using megawatts for months to try to train their systems, they're starting to realize that they're spending an awful lot of electricity just communicating bits back and forth. So anything we can do to get more information about real numbers into a small number of bits is going to save millions of dollars, maybe billions.

Doug Black (3:03)

Shahin, do you want to dive in with a follow up?

John Gustafson (3:05)

Yeah, Shahin, you mentioned mixed precision. And that phrase is used a lot. But I think I'd rather you could say right sizing precision. So don't use any more precision than you need. But I also am wary of putting a big burden on people to pick the precision. I think we need to let the computer pick the precision, and do as much of it automatically as possible, because programming is hard enough as it is. But people have a certain accuracy in their inputs, and they have a certain accuracy they want the outputs.

And our habits have been to just throw 64 bit floating point at everything in between. And that's really very, very wasteful. And I'm finding increasingly that I can certainly get it down to 32 bits and still get very acceptable answers. And that's more than a doubling of speed and a halving of the energy and power required.

Shahin Khan (3:55)

Well, so you're taking it to the next level by saying, not only do I have a better way of representing numbers, I can do it dynamically so that you will get just as much as you need and not more.

And of course, the title of your latest book is Every Bit Counts. And that's very consistent with that.

John Gustafson (4:18)

Yes, every bit counts and every bit pattern counts. If you take a look at a current day floating point, you say, what are all the possible bit patterns? And what is this complete set of numbers that represents?