AI Model Fatigue Sets In For Frontier Labs - OpenAI is DEAD

Eli the Computer Guy (0:00)

Coming from CNBC, Model Fatigue sets in as AI labs race to rattle out new versions at frantic pace. So this is one of the things that I've been warning about for a long time now at this point, that basically all of these Frontier Labs keep rolling out models damn near on a monthly basis, and that frankly it is just too fast for most organizations and companies to be able to keep up with. And when you start actually looking at implementing these models into your systems and deploying them to your user base, what you find out is that you simply cannot keep up with the speed of these models basically rolling out, and it is becoming a real issue. When I was at this conference called AI4, I got to talk to a founder, I think it was the CTO of one company, and the founder and CEO of another company, and I was talking about them with these new models coming out. And the interesting thing is one of the CEOs said that normally they skip a number of models coming out because every time a model comes out, he essentially has to assign three full time engineers to being able to test, implement, and deploy the model into their systems, right? Since these are non-deterministic systems, they don't necessarily know what the output is going to look like and how it is going to respond to inputs. Every time you change the model, you have to do a tremendous amount of testing to verify that basically you get an output that you can actually do something with. I talked again with the other CTO, founder of a company, and he was talking about the fact that they actually use the Frontier models as they come out, as each new Frontier model comes out, they actually deploy that into their systems. But again, they actually have to assign full-time engineers to that process, because it is such a pain in the ass to do. And this is one of the big things. There's this whole question right now with the value of Frontier models, and why more companies and organizations are not utilizing the new Frontier models as they come out. So I do these interviews every Monday with this guy Isaac Pound over at Tech Report, I guess it's called. And one of the things he always talks about, whenever he brings up these Frontier models, and the utilization isn't as high as the Frontier Labs want it to be, one of the things he always brings up is cost, like cost, like the new OpenAI Astra model, right, it's $10 per million input tokens. Like honestly, it's still a cheat, I mean, depending on what, as long as you architect your system properly, it's still fucking cheap. If you architect your system not worth a damn, you vibe code your architecture, it's gonna get expensive quickly. But one of the big things is the actual uptake of Astra isn't as good as some people expected. And one of the things that I try to point out is like there's a lot of this focus on the cost. The cost of tokens is so high. And one of the things I keep trying to bring people back to is like, no, but remember, as technology professionals, like the model itself is only one component out of your AI stack. So if you're gonna be building AI-powered systems, like the model is important, but you also need record stores, you also need data stores, you need front ends, you need caching systems, you need a whole bunch of stuff. Again, API calls, all kinds of stuff that go into building this overall AI system. And so if I've integrated a model into my AI system and that model works good, then I'm not gonna change it. I'm not gonna change it, right? That's one of the big things in the technology world I think a lot of normies don't understand is the technology world, as soon as you touch anything, touch it.

You fucking own it, right? I have literally walked into fucking server rooms, not touched a good goddamn thing, had a server crash while I was just trying to figure out what was going on. And then they're like, oh, you crashed our server. And now I'm spending 24 hours to rebuild a fucking server that I didn't even touch, right? That's how it is in the technology world, right? So if you have a system, if you have this AI system, right, and you know AI is non-deterministic, which means you can give the exact same input to three different models and get significantly different outputs, you know, right, as soon as you do it, it's like, well, well, here's the thing.