Reiner Pope – Chip design from the bottom up

**Dwarkesh Patel** (0:00)
I'm back with Reiner Pope, who is the CEO of MatX, which is a new AI chip company. Last time we were talking about what happens inside a data center. Now, I understand what happens inside an AI chip. How does a chip actually work? Full disclosure, by the way, I am an angel investor in MatX. So, hopefully, you have designed a good chip.
Also, if you're listening to this on an audio platform, it's much preferable to watch this Blackboard Lecturer on a platform where you can see what's happening. So, switch over to YouTube or Spotify.

**Reiner Pope** (0:31)
So, I'll start with the very smallest fundamental unit of chip design, then we'll build up into what an overall actual production chip, what are the components of that. Yeah. At the very bottom level of a chip, the primitives that we work with are logic gates, which are very simple things like AND or NOT. Then these are connected together by wires that have to be laid out physically as metal traces on a chip. The main function that AI chips want to compute is multiplication of matrices, and really inside that is the fundamental primitive is multiply accumulative just like of pairs of numbers. So, we're going to sort of demonstrate what that calculation looks like by hand, and then sort of infer what a circuit would look like for that.
It'll turn out to be sort of easiest if I do a multiplication accumulator of something like a four-bit number with another four-bit number, and then we're going to, the actual clearest primitive is actually multiply accumulate. So, there's a multiply these two terms, and then we're going to add in, so product of these two terms, and then we're going to add in an eight-bit number.

**Dwarkesh Patel** (1:51)
Can I ask a clarifying question? Why is this the natural primitive for whatever computation happens inside a computer?

**Reiner Pope** (2:02)
Yeah. So, there's a few reasons for this. It's a little bit more efficient, but the reason it's natural for AI chips is that, if you look what's happening during a matrix multiply, the what is matrix multiply in very short, it is there's a for loop over i and over j and over k of output i, k plus equals to input i, j times other input j, k.
And so multiply, accumulate happens at every single step of a matrix multiply.

**Dwarkesh Patel** (2:48)
Makes sense.

**Reiner Pope** (2:49)
And then the other observation is that the precision will almost always be higher in the accumulation step than in the multiplication step.
This is maybe specific to AI chips, but you're multiplying low precision numbers, but then when you accumulate, errors accumulate quickly and so you need more precision here. So this is why we've chosen to do a four bit multiplication and an eight bit addition.

**Dwarkesh Patel** (3:09)
Let me make sure I understood that.
There's two ways to understand that. One is that the value will be larger than the inputs, and the other is that if it was a floating point number, it would be, maybe that part is less intuitive to me, but it's maybe the same principle.

**Reiner Pope** (3:25)
It is really the same principle.
I guess the separate principle is that, as you are summing up this number, you are summing up a whole bunch of numbers and so you've got a lot of rounding errors accumulating. Whereas, in this case, there's only one multiplication in that chain and so there's not a lot of rounding errors accumulating in the multiplication.

**Dwarkesh Patel** (3:43)
Why are you summing up a whole bunch of numbers? It's just two numbers, right?

**Reiner Pope** (3:45)
I mean, the summation happens, it's repeated many times.

**Dwarkesh Patel** (3:49)
Yeah, any errors accumulate. I see.

**Reiner Pope** (3:51)
Yeah, makes sense. How would we perform this calculation by hand? As a human, we would probably separate it into two, but we can do it all in one using long multiplication. So the multiplication term first, we're going to multiply this number, this four-bit number here by every single bit position in the other four-bit number. So we write that out. First, 1001 multiplied by this bit position, that is this number itself. Then shifted across by one, we're multiplying by zero, that gives us an old zeroes number. Shifted across even one more to multiply by this one, we get 1001 Then finally, for this last bit position, we get an old zeroes number again.
So this gives us a bunch of terms that we're going to have to add for the multiplication. Then while we're doing that summation of this, we might as well add in the actual accumulated term as well. So we just copy that directly across.