Is Jev What AI Has Been Missing? I Tested It.

Turing Post (0:00)

I'm in a generic hotel, so the light might not be that good, but it doesn't matter, because I have something to show you that not that many people have access to. If you heard about Jev, the new Model System One Model that has been released just a couple of days ago and still has a wait list, I got access and I'm going to show you what the console is, what Jev can do, what it cannot do, what's so fun about it, and why is it so exciting that this model arrived and how it can change the whole AI world.

Okay, folks, this is the homepage for TypeSafe AI.

Let me show you insights of it, what's happening here. There are cookbooks, there are demos, there's a quick start to set up your own agent. With Jev, we'll get back to that later. Let's go to Playground. This is the Playground and there are two main ingredients. State, right here, is the information you want the model to evaluate and questions define what you want to know about. There are also three different question types. No, that's a yes or no probability of question. Choice, selects from options you define and score evaluates something against an ordered set of descriptions. This model requires predetermined type of question. You can start from scratch, but these built-in examples make this interface much easier to learn. So, how about we start with a super serious question. Is hot dog a sandwich? I add the question, right? Let's add the question and choose now as the type and leave the model on Jeff Latest. Let's run. So, it returned 58% true.

That means the model gives yes a little more probability than no, but we haven't supplied very precise definition. So, let's add a criteria. Criteria would be a sandwich is a food dish where filling such as meat, cheese, vegetables or spread is placed between structural starch.

Let's run now.

Now, it's 77% true. Criteria helped and defined it much more. But what if we define food? For example, food can be ice cream sandwich here. And if we write here food and if we run it again, the result becomes 33%.

The model was not fooled by the word sandwich. It understands that this food, this ice cream sandwich is not sandwich as in this criteria. That's the basic workflow. You supply a context, define a question in its criteria, run it, then inspect the probability. Let's finish here and let's go to check the help desk. Ticket trash. That would be a choice question. You see? Type choice. This is some sort of useful example because let's imagine you have an employee who has been waiting over a week for response about Wi-Fi and this employee wants to understand how's the progress. The example asks four questions at once. You can see them here. What priority does this request express? Which team's domain does the request fall under? The request size, they ask to a specific deadline or scheduled event, and then look at the tools. There are 200 options of possible tools. Let's run this. What we see is that IT helpdesk received 100% of the team probability. Meeting was the leading priority with 83%.

The deadline question came back at just 5%.

For the tool, ticket status led with 73% followed by Wi-Fi troubleshoot at 25%.

It's useful. This is a very useful distinction because this person is asking not only about Wi-Fi, but also what happened to an existing report. We can inspect the alternatives tool. The playground is returning until selection here. It isn't actually operating the helpdesk, of course. So now, let's switch here to overview to JSON. Here's the same response as structured data, and that's what makes Jev different. TypeSafe says Jev produces its outputs in parallel, instead of generating a text response token by token. Its training approach focuses on calibrated decisions. Probabilities intended to reflect how likely an answer is to be right. It also constrains outputs to the types of options you specify. Again, an important distinction worth noticing. Guaranteeing the answer structure does not guarantee that its judgment is correct. You still need good criteria and testing. It's a lot on human to prepare before you run Jev, but then Jev is super fast and much more precise. You can see the appeal. It's decisions, alternatives and uncertainty in the form that ordinary software can work with. This is not for humans. This is specifically for software. Now let's go to the home page and you see quick start here.

I love that option that you can set up an agent and I did it with my Codex. You can copy agent prompt. You can do it yourself. I just went to my Codex and I said, this is the website. Please go connect Jev to Codex. And it did it. And it did it. So you see, how about we go through agent setup and setup of Jev here in Codex. That's what it did. The interesting part starts when Codex and Jev are connected, because now you can ask, how can we check it? Can you build something to check Jev against LLM? And that's what it had built. And Codex, together with Jev in our Model Lab, created a challenge. You can use different examples to choose from, or you can run all eight examples. Let's say, corrected headline. An earlier headline said, Relay is available to everyone with downloadable weights. That headline was incorrect. Relay is invitation only. And the weights are private. Our support bench test scored related 92 versus 86% for keyword router. We have the rules to answer.