Haters think AI agents can't write GPU code? This'll ROCm

Ryan Donovan (0:08)

Hello, and welcome to The Stack Overflow Podcast, a place to talk all things software and technology.

My name is Ryan Donovan, and today we're talking about the programming on the GPU, and not only just programming on it, just doing it with AI. My guest for that is Anush Elangovan, who is the VP of Software over at AMD. So welcome to the show, Anush.

Anush Elangovan (0:30)

Thanks for having me, Ryan.

Ryan Donovan (0:31)

My pleasure. Before we get into today's topic, can you tell us a little bit about how you got into software and technology?

Anush Elangovan (0:37)

Well, that's a very long story, but...

Ryan Donovan (0:39)

TLDR, TLDR.

Anush Elangovan (0:41)

TLDR.

I started programming when I was 10 or something like that back in India, and did a lot of computer graphics back in the day. Then I did my computer science, focused on networking, network security, and then that slowly led to operating systems, worked on Chrome and Chrome OS, worked on gesture recognition, and then about 10 years ago, we started nord.ai, that became, 13 years ago, that became, I was focused on ML compilers as AI was taking off, and we were building software that could adapt and be applied to various different accelerators, and then AMD acquired us about three years ago, and we've become the core software engine for AMD.

Ryan Donovan (1:20)

AMD was originally known as a CPU provider, and it's getting into making GPUs and integrated CPU-GPU sets. With the GPU, you need to be able to program on the GPU, and that's where AMD has ROCm as a direct general purpose GPU programming language, right?

Anush Elangovan (1:37)

Yes.

Ryan Donovan (1:37)

Can you tell us a little bit about ROCm and how it works?

Anush Elangovan (1:41)

So ROCm is AMD's software layer, and like you said, AMD grew from CPUs, GPUs, FPGAs with the Xilinx FPGAs, but each piece kind of had like a board support package, so it was something to unlock hardware.

It wasn't a platform that stayed on top of all of these hardware, accelerators and components that we had. So ROCm has become this unified toolchain and layer that allows for customers to program on our hardware, and it looks unified as much as from toolchains, debuggers, etc. But it provides you the core building blocks for you to build and innovate on, and all of it is open source. That's the key part of it, which is different from CUDA, etc., right? Where we want the ecosystem to flourish. So that's a core philosophy that all the source code is open. And then it allows you to iterate fast, right? You're not limited by the speed at which we can provide you support on some closed source binaries. We want you to see what the source is, and we want you to innovate and then build on top of it. So that's the core of ROCm.

Ryan Donovan (2:47)

We love open source here at Stack Overflow, but obviously the ROCm language and toolchain has to fit with the AMD chip designs. How much leeway is there for open source developers to push their own features or fork the code?

Anush Elangovan (3:02)

As of about a year ago, we started this effort to not just make source code open, make the entire development process open, right? So today, if you found an issue or you had a bug or you want to add a feature, you can go to ROCm, GitHub, make the feature, submit it, and it will get rolled into tomorrow's Nike. It is a pure open source project, including the build process and everything included within.

Ryan Donovan (3:27)

I'm not super familiar with GPU programming. I have seen a little bit of CPU programming. How much direct hardware access are you doing there?

Anush Elangovan (3:37)

It's very direct. It is all direct to ISA. One of the advantages that AMD has is we even publish the ISA spec.

I know in CPU land, it's given that you know, what are the x86 instructions? You can write an assembler, you can write a compiler. GPUs are not like that other than for AMD GPUs.

We actually document the ISA for each of the GPUs that we launch. And we just launched the MI-455 last month. And as soon as it was launched, we launched the ISA. So anyone in the world can pick up the compilers, pick up the tools, program to our GPUs. The GPU programming itself, if you go a little bit more into the technical details, it's a little different where it's called SIMT. It's like a group of threads. So you're programming, you have a very highly parallelizable groups of threads, thousands and thousands of threads, doing kind of similar-ish work. So it's like good for heavy parallelism, versus CPUs and processes and threads. You could have more fungibility of what runs. You could have a SQL server and a web server and something else. Whereas in a GPU, you're usually saying, oh, I'm doing a matrix multiply and I'm going to decompose the matrix like this, and here are a group of thousand threads that are going to compute these individual components, and then I'm going to group it and run on 500 and group it and reduce it to 200, and then find an answer. So it's good for heavily parallelizable work that is predominantly seen in HPC workloads, scientific workloads, and of course, AI workloads.