OpenAI's Model Cracked Hundreds of Open Math Problems

SPEAKER_1 (0:00)

Hey, everybody.

Beth Lyons (0:02)

Welcome, welcome. Good morning. Later morning for me, earlier morning for Andy. Today is October 7th, it's a Wednesday, and this is episode 828

Hey, Andy. I'm Beth Lyons. Yes, I am.

SPEAKER_1 (0:17)

And with me in the studio today is Andy Halliday.

Andy Halliday (0:20)

Good morning. I'm well, thank you.

Beth Lyons (0:23)

That's awesome.

We were talking right before the, but right before we went live, wanting to make sure that we are up on the things that have happened more recently. And we've been saying lately that we haven't heard from Mistral. But we, but there is an announcement about Le Chonk. Le Chonk is the new model. It's open source or going to be. Is it available? I have not gone that far.

Andy Halliday (0:57)

I'm not sure about that. Because I have actually in this, in the Le Chonk news piece, there's multiple Western non-China open models that have been released. Three of great importance. And Le Chonk is one of them. Now, what's I think interesting about Le Chonk is it's a one trillion parameter model. It's very big. They trained it with, I think, several, I think around 4,000 GB 300 chips that they have in their own data centers over there in Europe.

And it's very inexpensive. It's called Mistral Large 4 is the official name for that model, but they call it internally Le Chonk, which in French is like, it's a big mamma jamma. Okay.

Beth Lyons (1:50)

So and Jeff is so probably Andy is pronouncing this correctly.

SPEAKER_1 (1:56)

I am I am not pronouncing it correctly.

Beth Lyons (2:00)

So thank you, Jeff.

SPEAKER_1 (2:03)

Yeah, yeah, you got it. Oh, there you go.

Andy Halliday (2:07)

Okay. Anyway, it's it's very inexpensive. Now, if you're going to use Astra, Astra is $10 input, $50 output. Le Chonk is $1.36 input and $4.18 per million tokens output. So vastly cheaper. And one of the leaders of that organization told Axios that the company believes Le Chonk is the world's best open weight model, exceeding the performance of any of the Chinese models. Now, it's only available right now through a moderated API.

And eventually, on October 27th, it's 20 days away, it'll be released after reinforcement learning and safety testing. So, no, it's not really widely available now, but testers can have access to it. Yeah, so that's one of these major new open models. And, you know, it sounds like it may be really superior to what's currently available to enterprises to use as a foundation model for their own inbuilt systems that don't depend on the cloud, frontier cloud providers and the costs associated with that. But there are two others of importance. One is a company, a US company called Reflection that introduced Beam. Beam is half the size of Le Chonk. It's a 500 billion parameter model. It's a mixture of experts model with 23 billion parameters active. By the way, Le Chonk activates 49 billion. So they're basically doing the same kind of mixture of experts architecture, but on a much larger scale for the Mistral 4 version.

Now, Reflection says Beam is competitive with China's GLM 5.2. But Turing Post says, well, why are you comparing yourself to that? Because GLM 5.3 is already out.

And then it also says that it approaches Alibaba's Quinn 3.8 Max on some coding and agent tasks, but it uses less computing firepower to do that. So Turing Post says, hey, they're the shade casters on this one. They said, this is unimpressive because you, in putting forward the benchmarks that you perform on, you left out six that were omitted where Quinn and DeepSeq results exceed the performance of the reflection model. So while this is a pretty substantial open source model initiative that's happening from within the United States, their initial release, this Beam model, is not all that impressive. To give you some numbers on that, for example, on DeepSeq V41 Flash does 78 or does 74.2 on DeepSwee and this model does 44 So it's lame by comparison on the benchmarks. But again, it's their first effort, and hopefully they'll come forward. The company reflection notes quickly that the advantage compared to frontier open models is really high efficiency at inference time. So it's going to be much more efficient than the closed frontier models are.

That's number two. The third one just was released yesterday by Google, and that is their Embedding Gemma 2

So Google is open sourcing Embedding Gemma 2 Now this is a very special model. It's an open model, so you can get the weights, for private search on local devices. So it's a local first personal search system for on-device retrieval-augmented generation. So you can have a local model that not only can do really effective search across your local device, but it can also do text generation. So a combination just for the purposes of sort of understanding what information you have accessible to you on your local device, whether that be a Mac or a phone, and accessing, obviously Google will make this, provide access to all of the tools that are in your Google App Suite, right? Including Drive and Gmail, etc. Which seem to be the focus of what many of the other companies are trying to enable. And there's news about that with respect to Claude. Yeah. So we'll talk about that in a bit. But speaking about open models generally, what you want to have, if you're going to be a major enterprise and you're going to try to restrict the expense that is associated with being a Claude shop or an OpenAI shop, what you want to do is you want to have the very best foundation model, open foundation model that you can start with. And you're not likely to choose one, in my opinion, you're not likely to choose one that is kind of behind the game. You want to have the very best open source system as the foundation. And then you're going to fine tune it by putting post training against that model for all of the sort of company or service or product specific things that you want to have applied there so that that model is in your shop, it's running on your hardware or on a cloud service provider that you retain to run your servers.