Junchen Jiang, Tensormesh CEO on Faster, Cheaper LLM Inference

**Junchen Jiang** (0:00)
English or Chinese?

**Grace Gong** (0:02)
English.

**Junchen Jiang** (0:03)
Okay, okay, okay.

**Grace Gong** (0:04)
Okay, one sec. Let me just make sure that I'm connected to this page.
We are live. Hi, Junchen. Welcome to Venture with Grace.

**Junchen Jiang** (0:15)
Hi, Grace. Thanks for having me here.

**Grace Gong** (0:18)
To start off our show, I want to give a quick shout out to our amazing sponsor. This episode is brought to you by Nebius, the ultimate cloud for AI innovators. Nebius provides AI infrastructure you can count on, combining reliability and speed with flexibility and engineering support, unmatched by hyper-scholars. AI leaders like Meta, Shopify, and Higgsfield already partnered with Nebius to run their AI workloads. Plus, venture-backed startup can save up to 150K on compute costs when they apply for access. Visit nebius.com or nebius.com/startup to learn more.
Okay. I want to start with your, maybe we could start from your journey from being a researcher to founder. What were some core lessons that you've learned as a researcher from Google and Microsoft before you started your own company?

**Junchen Jiang** (1:06)
My background, is that a question, right?

**Grace Gong** (1:10)
Yeah. You have worked at Microsoft, Google before you become a professor, and what were some core lessons?

**Junchen Jiang** (1:19)
The core lessons here from my research life and from my industry experience, is that a lot of the amazing technologies that eventually take over industry, has its root in academia, but to make it real, to make it real, to make it practical, you have to be in industry to scale it, to have a sustained agenda, to push it to millions of users, and of course, you have to have the investment, to make it real, to have a team dedicated to pushing this. This is not something I mean, most time, actually, it's quite rare these days, you can have, you can build a real prototype with just a Ph.D.,
a group of Ph.D. students, it's pretty hard.

**Grace Gong** (2:15)
For sure. I mean, I feel like it's very surprising to come from you, since you are the ultimate researcher yourself, and you're a professor at University of Chicago. Maybe we could start with, since Tensormesh started as an open source project, and then now you guys are a company, as well as you essentially are the ultimate spokesperson about KVcache. Why don't we start with, what were some problem that you found during your research, and how do you transform the research itself into an open source project?

**Junchen Jiang** (2:54)
First of all, KVcache was not something we have been doing for decades. KVcache is a concept built in transformer architecture. It was proposed as part of the transformer, I would say probably eight, nine years ago. But when transformer started to take over the world, that was the Chachi PT moment, remember, three and a half years ago now. At that moment, we are not working on large language model. But some of the students would not mean to look at this problem, realized that there was this very interesting data problem in large language model inference. Back then, people were just looking at Chachi PT or large language model as an application. They look at this and they say, how do I use this application? But we were looking at the serving stack, the system, the software that runs large language model, and look at this and realize, there's this very interesting piece of data called KB Cache, that is what a model can directly read. I mean, it was created by the model and the model can directly read this data.
It's like the knowledge, it's like the memory, it's like the understanding produced by the model on any human input data. The model translates the human input data into this tensor shape called KB Cache. Now, why we thought that data was so interesting was, if you look back into the history of Internet, the most important piece of software or hardware is the communication of data between human. Ultimately, you're building a data pipeline between human. When we look at large-language model, the ultimate data pipeline between models or directly shareable between models is the KB Cache. That's model's internal understanding. If you want one model to understand the text, and if the text has already been read by another model, you want the model to directly get understanding rather than reading the text over again.
To share the KB Cache, you have to store it, you have to distribute it, you have to send the KB Cache over the network. That was the problem we envisioned three years ago.
At the beginning, we started writing papers about this, and soon we realized industry didn't even have a way to store the KB Cache in the first place. We open-sourced the project and then got a lot of attention and eventually we started a company to evangelize this idea of KB Cache-centric systems.