The evolution and promise of RAG architecture with Tengyu Ma from Voyage AI

**Sarah** (0:05)
Welcome to No Priors. Today we're talking to Tengyu Ma, Assistant Professor of Computer Science at Stanford, and the co-founder and CEO of Voyage. Voyage trains state-of-the-art components for next-generation retrieval systems, including embeddings models and re-rankers. We're really excited to talk about his research and the RAG debate today.
Welcome Tengyu.

**Tengyu Ma** (0:23)
Yeah, thanks so much. Thanks for having me here. We're looking forward to the debate.

**Sarah** (0:28)
Yeah, why don't we start with just a little bit of an overview of like your research agenda to date? Because I think uniquely it covers a broad range of fields within and around deep learning from like theory to RL to embeddings and optimizers. So can you talk a little bit about sort of how you pick the directions you have?

**Tengyu Ma** (0:46)
Yeah, so I think most of the papers I wrote have some theoretical thinking in it. I guess maybe that's the commonality. And besides that, I think I worked on quite a few topics as you mentioned, ranging from the theoretical understanding, mathematical proofs of deep learning systems, to all the way to practical large language models, reinforcement learning, deep reinforcement learning. And these days recently, I think what we are working on more centralized to efficiency of training the large language models and the reasoning, improving the reasoning tasks for large language models.
So my vision is that in the future, the efficiency is very important because we are running out of data and compute. So we have to either use the data much better and use the compute much better. And also reasoning tasks seems to be a pretty important direction and also in some sense, kind of like a risky direction in the sense that we don't know exactly how fast we can solve those challenging reasoning questions yet.

**Sarah** (1:53)
Can you mention a few of the like key papers or work that you or students in your lab have done just so our listeners can look them up?

**Tengyu Ma** (2:03)
In the very early days, I think I worked on some of this matrix completion optimization for matrix completion. That's like 10 years ago. And then I moved on to embedding models, like sentence embeddings, vector embeddings. One of the papers we wrote is a very actually simple paper where we averaged the word embeddings to get sentence embeddings. And then we did some of these transformations using PCA to make the performance much better. That was even before Transformer came out.
And then I think I moved on to Transformers, large language models, and contrastive learning, which is the new way of training the embedding models.
Especially the direction started with some of the papers on using contrastive learning for images and we work on improving those and understanding why contrastive learning can work. And recently we work on optimizers for large language models. For example, one of the papers we got last year was Sophia, which we found where we found that we have a new optimizer which can improve the training efficiency by 2x for pre-training.

**Sarah** (3:09)
This is great. Adam is very old at this point.

**Tengyu Ma** (3:12)
Yeah, he's 10 years old now. I think that's the interesting part about it. So optimizers, I think people have tried in the last 10 years for so many times, there were so many papers published which has improvements over Adam in various cases. But so far, Adam is still the default algorithms for training large English models. And that's why we thought that it's the time to really, we spent a lot of time on this. I think I started probably around 2018, 2019, and I asked a few students to work on this. And finally, we had one paper out after a few years, after a few failed projects and failed ideas.
And recently, I think one of the Facebook friends actually used this in their large scale multi-modal training. And they found that on that scale, I don't know exactly how many parameters they are, but I assume it's kind of more than 100 billion parameters.
They found that on that scale, there is a 1.6x improvement in the efficiency of the training. So that's like a $10 million versus $16 million.

**Sarah** (4:20)
That's super exciting, yeah. I think Sophia has an opportunity to be really impactful.
You started a company last year taking leave from Stanford. Given your work has been theoretical, but with practical applications, what drove you to do that?

**Tengyu Ma** (4:41)
I think I came to Stanford partly because there's a very strong industry connection here at Stanford compared to some of the other universities.
And also probably entrepreneurship is just part of my career plan anyways. And in terms of the timing, I felt that this is the right timing in the sense that the technologies are more and more mature so that it seems that the commercialization is the right timing right now. So for example, I think one story I have is that I look up some of my slice deck for my lectures at Stanford CS 299 seven years ago when I started to teach at Stanford. And that point machine learning, we have a lecture with Chris Ray and the machine learning on applied machine learning. So how do you apply machine learning industry? And there are seven steps there. So the first step is you define your problem, the second step is you collect your data and you choose the loss function, you trim it and you iterate so on and so forth. So it's pretty complicated at that point. And now the foundation model arise to power and in a new foundation model era, the only thing you have to do is that you have to, someone will trim the foundation model for you and then you trim the prompt and you add a retrieve argument generation on top of it. And that's pretty much, that's it. So applying machine learning AI to industry environment is much, much easier than seven years ago.