The Age of Inference: Etched Startup Proves the Next AI Hotspot

**Josh** (0:00)
So at Limitless, we are always in pursuit of Alpha, in pursuit of the thing that's around the corner, the interesting investment opportunity, and I'm very excited to say that we found a new one. And it comes in the form of this company named Etched, and it comes in the form factor of this thing called Inference. Over the weekend, I know we spent a lot of time while people were shooting off fireworks, reading about this one small company that's intention is to change the way that we look at Inference forever. In fact, they're just a couple of 24-year-olds who have already dethroned Nvidia across a series of benchmarks. This is in the world of Inference and I think a lot of people aren't really paying too much attention to. A lot of people are still focused on pre-training and they think Nvidia GPUs are the be-all end-all, but we've seen this trend popping up of companies like Google, building their custom accelerators, companies like Amazon. And now we have a series of startups you might have heard like Cerebris or Grok, which was acquired by Nvidia for a tremendous amount of money. Cerebris just went public a few weeks ago and there's a lot to unpack here, both as an economic opportunity but also as just a really cool opportunity for a new frontier in AI, which is this accelerated inference chip architecture that everyone seems to be gunning towards.

**Ejaaz** (1:03)
Now, in order to understand everything we're going to describe in this episode, I think it's important to start off with what on earth is inference? So typically when you use a model, you write a prompt, you send that prompt, and suddenly, magically, an answer appears from the LLM, which whether you're using Cloud or ChatGPT. Now, what happens when you click Enter is the following. Your prompt gets sent to a server. Server rack has a bunch of different AI chips, commonly known as these GPUs. That's what you associate Nvidia with. What this GPU does is it firstly, reads your entire prompt and processes it. This is known as something called prefill. So the chip does this. It reads your prompt, processes it, and then the second thing that it does is it draws on memory that it has for your entire conversation, the context that you have about you, the context that you gave it in previous prompts in that exact same conversation, and then it starts generating a response to you, token by token, one at a time. This is something called decode. So typically, if you interact with AI chips, what's happening on the back end is this prefill and decode type process, and that's what generates your answer at the end, and that's effectively what inference is as an output. So now, when we're talking about chips in general, you think of Nvidia, you think of GPUs, and you think, okay, well, these chips are primarily used for AI training. And recently, as you just mentioned, Josh, a bunch of these different AI labs have announced that they're building their own AI chip. And the question then becomes, what does this AI chip specifically optimize for? And the answer is very simple, it's inference. And this has been getting more busy over the last couple of months. OpenAI recently announced that they're building their own Jalapeno chip. You've got Anthropic that's rumored to be building their own in terms with Samsung. You've got Cerebus and Grok, as you just mentioned. And now we have this new startup called Etched, which is building a brand new chip which competes pretty effectively with Nvidia's ability to perform inference.

**Josh** (2:59)
Yeah, there's a key difference between training and inference. Training is that thing that happens one time, and it normally takes months long. So when you hear a company is training the new GPT, or train the new quad model, that's what it is. It's using this pre-training. Inference is a totally different animal. And a really interesting thing that I learned about doing research about this is that the amount of inference demand, as of I believe two years ago, was about one third and it was two thirds of pre-training. Now that's flipped to be two thirds inference, one third pre-training and it's expected by the end of this year even, it's going to be heading towards 80 percent, which is a really interesting stat considering that Nvidia holds roughly 75 percent of the total chip share, but they're not actually optimized for this. And that is a signal because Nvidia share has actually gone up in terms of how many, what percentage of the AI world is using Nvidia GPUs, when the reality is, is that they're not optimized for this type of inference. So as inference demands are going higher, Nvidia share is also going higher, but Nvidia chips are not optimized for this inference. And that's a signal showing that it's really the only thing available. No one's actually figured out how to build these custom chips at scale. Therefore, the entire market must buy Nvidia chips. But there's this new series of companies that's been sprouting up, like Edge that we're going to talk about, that is going to solve this problem and their intention is to do so at a scale that's large enough to offset this kind of asymmetry that we're seeing in the market.