How Hugging Face’s Investor Spots Winners

Dan Shipper (0:00)

The Frontier duopoly is over. This week, we got new Frontier models from Google and from Meta. Google launched Gemini Argon 4, and Meta launched Wonder, which comes in standard and full, so you can run Wonder full. You know, that's their sense of humor. Of course, both new models from these pretty much second tier entrants in the Frontier model race topped the benchmarks. But our hands-on testing inside of Every, running these models for knowledge work, for coding, for design, for writing, revealed something a lot more nuanced and different from what the public benchmarks show. To us, it seems pretty clear that Muse is benchmarked, which means they trained it to look really good on public benchmarks, but those gains don't necessarily translate into your everyday work. So it looks like it scored really well on Sweet Bench, but in our testing, for example, on my Senior Engineer benchmark task, where we ask a model to rewrite a Vibe-coded slap codebase, it scored about a 50% across a few different runs, which is solidly middle of the pack territory. So it's also kind of a mid-writer. It buries the lead in the stuff that it writes, and it falls prey to some basic benchmark tasks like my McDonald's test, where I ask a model to do literary analysis of a very serious book, and then I say, that reminds me of McDonald's, and I see whether the model takes the bait. Muse takes the bait. So Muse might take the bait, but Gemini kind of takes the cake. I'm so sorry.

Gemini 4, it's one of those models where I saw the benchmarks and I was like, this is also really benchmark maxed, but it's good. It's a good model. I used it for writing all weekend, and it is without a doubt my favorite new writing model. It beats Astra, which was my previous favorite. It definitely beats Claude. It has an incredibly good feel for structuring information, and I used it all weekend to write this 8,000-word piece, How Codex Happened, which is coming out on Thursday.

And I think it probably made me two times more productive. Like, truly a game-changing model for writing coding. Not very good. It got like a 20% on the senior engineer benchmark task, and generally it seems better than previous Gemini models, but especially in the anti-gravity harness, it's not really something you're gonna want to switch to unless you're using it specifically for writing. Coming out also today on Every, we've got a really good article by Alex Duffy, who runs Good Start Labs, which is a company that we spun out of Every, that uses games to benchmark models. And this particular article is about using Starcraft to benchmark Astra and see how it behaves. The answer is it cheated. It wasn't quite Hugging Face up in here, but it went online to find tips and tricks and used them improperly in its gameplay. And that is really interesting. And again, my piece, How Codex Happened, it's an 8,000 word story about how opening I fell behind in the coding race and then came back with Codex written by me from a bunch of first person interviews and also a little bit from my perspective because I was there for a lot of it. That's coming out on Thursday. It's going to be awesome. I am psyched for this. I wrote it mostly with Codex and a lot of Gemini 4 Argon. Last but not least, we have an interview coming up with Natalia Quintero. She's our head of AI consulting and she interviewed Matt Hartman, who is the managing partner of Factorial Capital. Matt is a really interesting investor.

He came out of Betaworks in New York City. He was involved in their Hugging Face investment. He's just been in AI since before AI was cool.

They get into a lot of really interesting questions. If anyone can vibe code your app, where are the software modes? How do you build a software business now? And I think it's a great interview. Enjoy.

Natalia Quintero (4:03)

Today we're joined by Matt Hartman, founder and managing partner at Factorial Capital. Matt spent nearly a decade at Betaworks, where he was a partner. And while he was there, he wrote the first check into a tiny chatbot company called The Every Podcast.

He wrote to a tiny chatbot company called Hugging Face, which Nvidia bought last month for $13 billion. In 2023, he left to start his own fund, Factorial, a venture fund with a very unique structure, which we'll learn about today. And anytime that I talk to Matt, I think of him as a builder and a tinkerer. Every time we catch up, he's built something really interesting for himself, or his fund, or his sons. So welcome, Matt. So excited to have you on the show.