How to train a Million Context LLM — with Mark Huang of Gradient.ai

**Alessio** (0:05)
Hey, everyone, welcome to the Lit in Space Podcast. This is Alessio, partner and CTO and residence at Decibel Partners, and I'm joined by my co-host, Swix, founder of Small AI.

**Swyx** (0:14)
Hey, and today we're in the remote studio with Mark Huang from Gradient. Welcome, Mark.

**Mark Huang** (0:18)
Hey, glad to be here. It's really a great experience to be able to talk with you all. I know your podcast is really, really interesting, and I always am listening to it every time you guys have a release.

**Alessio** (0:30)
He's not a paid actor. He said that out of his own will.

**Swyx** (0:35)
We'll give you the check later. So you're unusual in the sense that you and I go back to college.
I don't exactly remember where we overlapped, but we both went to Wharton. It went into the sort of quantitative developer realm.

**Mark Huang** (0:46)
Yeah, exactly. Kind of crazy, right? So it all goes full circle. I was a quant for quite a few years and then made it out into Silicon Valley, and now we intersect again when it kind of feels like more or less the same, right? Like the AI wars, the trading wars back in the day too, to a certain extent, in the grab for talent.

**Swyx** (1:07)
I think there's definitely a few of us ex-finance people moving into tech and then finding ourselves gravitating towards data and AI. Seems like you did that. You were at a bunch of sort of quant trading shots, but then as you moved to tech, you were lead data scientist at Box and staff ML scientist at Splunk. And then before working on the startup, they eventually became Gradient. You want to tell that story?

**Mark Huang** (1:28)
Yeah, I think part of the reason why I came over from the quant finance world is to get more collaboration, learn about what big data and scaling, machine learning really looks like when you're not in this bubble, right? And working at Box, I worked mostly in a cross-functional role, helping product analytics and go to market. And then at Splunk, it was a lot more specific role where I was helping with streaming analytics and search and deep learning and for Gradient, really why we started it was, whether it was in finance or whether it was in tech, I always noticed that there was a little bit more to give in terms of what AI or ML could contribute to the business.
And we came at a really good time with respect to wanting to bring the full value of what that could be into the enterprise. And then obviously OpenAI created this huge vacuum into the industry to allow for that, right? So I myself felt like really, really empowered to actually ship product and ship stuff that I could think could really help people.

**Alessio** (2:35)
And maybe just to touch a little bit on Gradient, I know we have a lot of things to go through, Gradient, Llama3, context extension. There's a lot, but what exactly is Gradient? You have an awesome design on your website. It's like really retro. And I think people that are watching fall out on Amazon Prime right now can maybe feel nostalgia just looking at it.
What exactly is it? Because I know you have the Foundry, you have the H&S SDK. There's like a lot of pieces into it.

**Mark Huang** (2:59)
Yeah, for sure. And appreciate the call out for the design. I know my co-founder, Chris, I spent a lot of thought in terms of how he wanted the aesthetic to look like. And it reminds me a lot about Mad Men. So that was the initial emotional shape that I felt when I saw it. Quite simply, like Gradient, we're a full stack AI platform.
And what we really want to do is we want to enable all of the, you know, RPA workloads or the codified automation workloads that existed in enterprise before. We really want to enable people to transition into more autonomous, agentic workflows that are less brittle, feel more seamless as an interface too, able to empower what we really think the new AI workforce should look like.
And, you know, that kind of required us to build a fairly horizontal platform for those purposes.

**Alessio** (3:49)
We had this discussion at our AI in Action Club on Discord, like the minimum viable agent, or like kind of how you define an agent. In your mind, what is like the minimum thing that you can call actually an agent and not just like the for loop, you know? And how do you see the evolution over time, especially as people adopt it more and more?