We Tried Jev To See If It's Any Good - This Week In AI

Abhi Aiyer (0:00)

The difference between Jev and a normal LLM, LLMs return text, but Jev returns structured output with a probability. Does not mean that Jev is more factually correct than an LLM, but that probability is within some standard deviation that you care about, you can be very confident that that's the right thing to do. And it's trained differently.

Shane Thomas (0:32)

Welcome to Agents Hour, we're here at September 21st. We are doing the news and there's been a lot of news this week. There's been one topic on everyone's mind and we will talk about it. We'll talk about it very soon. But before we do, first of all, Software Factories are clearly here to stay. There's a lot of people talking about them, a lot of people trying to figure out how to make them work. If you are in San Francisco and you want to talk about Software Factories, you should come hang out with Abhi. I think I'll be in San Francisco on that date as well. I think we'll have a few people from the Mastra team there as well. So if you want to come hang out and talk about Software Factories, come to the first SF Software Factory meetup. It's on October 14th. Speaking of Software Factories, there's this one article I wanted to talk through really quickly. Gergely Orosz, I think that's how you pronounce it. I'm not exactly sure. Kind of shared a deep dive into OpenAI's Agentic Software Factory. So I wrote this article here, talking about how Codex took over at OpenAI. The one thing I wanted to talk about with you, Abhi, was, this is a very long article, but this diagram. You know, we talked about it in Slack. A lot of people were commenting on this when it was shared on X. I think the biggest comment I heard was, that's it? Like they expected OpenAI to be so much further ahead than they are. And there's some cool things here, but it's not beyond what I think most people could imagine. So what was your impression of this when you saw it? Anything that stuck out to you that was cool or interesting, or anything that was like a little disappointing?

Abhi Aiyer (1:58)

Well, I think, I wouldn't say disappointed, but first thing I really liked is we're not that far off in our own factory development. Also, the labs speak with a lot of words when less token do better.

And that's what I expected more, right? Because they like talk about grandiose things. But it's the same kind of vision that most people have when they're describing their own software factories that they build themselves. I do like the Perf Factory. That's something I did not think about, where you're constantly monitoring releases and seeing if anything that the factory produced was a defect, and then automatically trying to fix that. But those things would come out in like reconciliation loops and all that type of stuff. Ultimately, I feel good, because they're the Codex people, so we have to respect them, and now we know that we can all build this ourselves too.

Shane Thomas (2:49)

Yeah, and I think the biggest thing, the only big difference from what I think where we're already at is, I think they have some cool, interesting production monitoring use cases where it feeds back into the loop, which I think is the one missing part that we're working on, but we haven't quite completed yet, where it's like that cycle of continuous information.

And I do know others are focused on trying to do that as well. You want to have this whole kind of feedback loop of, you ship a change, how do we make sure that change is good? You get a code review, of course, you want to write tests, you want to run tests. But sometimes even the best tested thing fails when it interacts with production, right? And so it's like, how do you then understand that it's caused by this issue, feed that back in, immediately fix it or roll it back? So I think there's some interesting things that they did here, but I expect it to be more complex, and so I'm pleasantly surprised that it's kind of within grasp of all of us, within a few short-iteration cycles. I don't think we're that far off.

Abhi Aiyer (3:44)

Yeah, except they have unlimited tokens and infrastructure. So that's the only difference between us and them.

Shane Thomas (3:49)

Yeah, their budget is just a few zeros bigger.

Abhi Aiyer (3:52)

They have like GPT-7 already and we don't, so. And they have no rate limits. There are a lot of advantages on their factory compared to ours.