Jev AI: How to Automate Model Routing

Julian Goldie (0:00)

Jev AI just changed how model routing works, and this is the part that's going to save you the most. So right now, every job your AI agent does goes to the same big model. The tiny jobs, the huge jobs, all of it, same price. Now, Jev AI actually fixes that. So it sits in front of your agents and picks the right model for every task before the work even starts. Cheap job goes to a cheap model, hard job goes to the strong one, if you set it up in the way that I'm going to show you today. And it makes that calling under a half a second for almost nothing. So today I'll show you exactly how the routing works, what to write so it actually picks correctly every time, and the one setting that most people get wrong, that makes the whole thing fall apart. So stick with me to the end because there's one part of this that turns GEV from a cost saver into something that makes your agents work noticeably better at their own work. Let's get into it. So let's break down how to use GEV AI for model routing. Now to route a task well, you have to understand the task. Understanding the task means reading it. Reading it means calling a model. So the old way of routing a model was, you know, call a big model to ask how hard is this? Then call another model to actually do the work. So you pay twice. And the first call often costs about as much as doing the job with the model in the first place. And so most people give up and pointed everything at one strong model, every task, every time, including the ones a tiny model could have handled in a second. Now GEV actually breaks that loop. TypeSafe released it last week as their first system one model. It's built to make fast structured decisions. The software can use direct leads. Prior to 0.042 per million input tokens, output tokens aren't charged. And there's a 32,000 token context window. Response times land in the low hundreds of milliseconds. At that price and that speed, the routing call stops being a cost you have to justify. It becomes background noise. You can run it in front of every single task and never think about it again. Now, here's what makes Jev different from asking a normal model to route for you. So Jev doesn't generate text at all. You can't chat with it. It won't write you a paragraph explaining its thinking. You give it a state, the situation, and you give it questions. It gives you back answers that fit inside the shape you asked for plus probabilities. For routing, you use the question type called choice. Choice means you hand it a set of options, and it returns a probability for each option plus an overall confidence score. So let's say, for example, here's a job. Here are my four lines, and pick one. And it picks one. It can't invent a fifth lane. It can't wander off and write you an essay about the job. It just returns one of your options with numbers attached, and your code does the rest. And that constraint is the whole point. A routing decision needs to be something software can act on immediately. Paragraph of reasoning is useless to a program. A clean answer of a probability is something you can actually build on. So let's talk about how you can actually define the lanes, because this is the part that most people get wrong. Now, Lang chain actually built middleware for exactly this. You can give it a set of choices. Each choice has a model attached and a criteria line written in plain English. In their example, the fast lane points at a cheap model of the criteria, direct lookups, extraction, and localized changes. The powerful lane points at a stronger model with the criteria, architecture, and high stakes decisions. Then there's one instruction sitting over the top. Choose the least costly model that can complete the task. Read those criteria lines again and notice what they are. They're job descriptions, not code, not keyword lists, not if-then rules. You're describing the kind of work each lane is for the same way you describe it to a new hire on the first day. That's a big deal for anyone who isn't technical. You're not building a decision tree. You're writing three sentences about what each model is good at. And the quality of those three sentences decides how well the whole thing works. Now, vague criteria gives you vague routing. So simple stuff tells Jev nothing. So for example, pulling out one fact of a document, rewriting a sentence, answering a question about content we already have tells it a lot. Write them the way you'd write them for a person. Give examples inside the criteria line. That's your real work here, and it takes about 10 minutes, or you can get your AI to do it for you, which is what we actually did in this example, as you can see. So we've got local, cheap, strong, and human. And let's talk about these four lanes. So local. Local is a model running on your own machine for anything private or repetitive. Cheap is like a small hosted model for volume work. Strong is like your best frontier model for the hard thinking, the human lane for the jobs that should never be automatic. Most people build the first three and skip the fourth. The human lane is the one that changes how you feel about leaving agents running. So when you actually route something to human, your system stops and waits for you instead of guessing.