AI That Acts: Devin Fusion, Persimmon, Programmable Worlds & Amodei’s Slowdown Call

Ksenia Set (0:00)

If I leave this room and come back, I expect the furniture to be in the same place. And for example, if someone has moved that plant behind me, I expect it to stay where they moved it. For an AI-generated world, those expectations become design questions. Where is the record of what exists? How does the system update it? What happens to something that is no longer on screen? This week, we saw several attempts to tackle those questions. And as I followed them, I found myself moving from generated worlds into robot experience, them into the way digital agents work with people and with each other.

Hello, everyone. I'm Ksenia Set, and this is the Monday edition of Attention Span from Turing Post. Every Monday, we follow what changed in AI's ability to understand and act in the world. It's truly fascinating to see the developing world of AI through this lens. This episode is an experiment of covering news weekly. Please let me know if you want it to be every week. We are looking at September 7th for September 13th, and let's start with a world that needs to remember what happened. Alaya Lab explores this through combat scenarios, which look very different from my room, but raise the same question. When something happens, how does the world keep track of the consequences? And as you watch these clips, look at what happens when something leaves the frame. And when it comes back, does what we see still make sense, given what happened earlier? Alaya's Programmable World Model keeps the record of the world separately from the video. This is what they introduced. Programmable World Model. The paper was published on September 9th. You describe what you want to happen, a coding agent writes a program, and an engine runs it and invades that record. The video model uses the record to generate what we see, receiving a compact description of the scene through three-dimensional boxes and information about the object state. This will sound familiar if you work with game engines. Keeping track of objects and rules is something software already does. What the researchers are working on is connecting that explicit programmable state to generated video. And it's worth putting this alongside Atlas, which WorldBabs announced on September 1st, that was outside this week's window, but it helps explaining where the field is going. Atlas can generate new views and explicit 3D outputs, including parts of the scene that were not visible in the inputs. Just look at these videos and it's based on photographs. Give it a photograph of a robot from the front, for example, and it can generate the view from behind. It imagines the unseen side while keeping it consistent with what the photograph shows. It could also produce a 3D scene that you can explore along different camera paths. The hidden paths are possible gases, so more photographs help constrain them. What interests me here is how much further this takes us when generating a picture. A few photographs can become a space you can explore. Then you can start asking what a robot would see as it moves through that space or how this scene should change after an action. Atlas and Alaya approach different parts of that problem. Seeing them together, I start to imagine taking a recording of a place and using it to try things there. Approach from another direction, move an object, see what happens next, see what stays, what it remembers. And each of those steps requires more than making the scene look convincing. And we are starting to see how they might fit together. World and World, submitted on September 10, by researchers at the Westlake AGI Lab, starts with a recorded event. The aim is to let you watch that event from different positions without changing what happens or when it happens. You could groove the camera around someone performing an action, for example, while the action continues on its original timeline. The model has to work out what the new angle should reveal while preserving what the recording already tells it. The researchers do this with a frozen video model, meaning they don't update its weights. They organize the source observations, general guidance and previously generated views into evidence labeled with camera positions and times. But what about the old reviews? They are very important here. Once a view has fallen out of the model's recent context, the system can retrieve it from a long history. So when you come back, the model has a reference for what it showed you before. I like that this gives us something specific to inspect in the demonstrations what happens on the return visit. Now, if a robot is using a mile to decide what to do, its next observation depends partly on its last moment. Even a small error can change what it has to deal with next. AWS published tele-existence DreamZero experiments on September 9 DreamZero predicts both actions and future camera views, and the team tested several ways of adapting it to their robot tasks. The best simulation train configuration reached 45% success on a simulated block peeking task. A separate configuration train and real demonstrations reached zero in simulation. The physical and simulated robots weren't sufficiently matched, including how their joints and states were represented. But there was another result. Getting closer to the recorded actions did not consistently mean completing more tests. In the offline test, the model was receiving frames from the original demonstration. Its predictions could be slightly wrong without changing the next frame it received. In the simulator, its own actions affected the next observation. A small mistake could leave it facing a situation that the recording never showed. That makes me want to see both unsuccessful attempts and the successful ones. Especially what the robot does immediately after something goes wrong. And there is a related difficulty in skills S1 which Nvidia discussed in the September 10th post. S1 can take a demonstration video as context for a new task without updating its weight for that task. Among the examples are potting plans and making pour over coffee. It's obvious why it's so attractive, right? If you know how to do something, perhaps you could just show the robot. But everything you do with your hand has to become something the robot can do with its gripper. With the object arranged in its warp space, it's brutally hard. Nvidia reports about 66% success per step in skills tests. To make coffee, though, the robot has to pick up the kettle, position its spark over the filter, control the pour and put it down again. These movements depend on one another. A grip that lets it lift the kettle may leave it over the position for pouring. It then has to adjust before continuing. Completing the test prepares getting through that whole sequence, including recovery, when a movement does not go as expected. Skild and Nvidia are also developing new simulation solvers for release through NUTLE, including software for contact and manipulation. Those calculations help determine what happens when the robot touches an object, applies force or tries to move it. And every part, every little part of it is so complicated that we would just in the beginning of solving it. Which brings me to a collaboration I want to keep track of over the next year. Mila and World Model Data announced it on September 11th, with a 12-month program to study how world models improve as the amount of training data grows, including whether those improvements transfer into the real world. World Model Data supplies structured experience from video games. And I'm curious about what a robot can take from that experience. Learning to anticipate an object's movement seems useful, but it still limits the problem of controlling a particular real body. The collaboration is just beginning, so we don't have the results yet. But it's worth paying attention, though. And I like to see more collaborations like this. It will be super useful for robotics. On the digital side, agentics side, one of our favorites, Cognition has been looking at what happens when two models work on the same code in tasks. They called it Fusion and introduced it as part of Devin Desktop and KLEE on September 11 A lead model plans and reviews the work. You probably know this setting if you built agentic workflows yourself. While a less expensive model handles delegated coding and testing, each keeps its own context, exchanging briefs and results. You know what's interesting? Replacing the lead with a model that costs twice as much per toolkit made the sessions 9% cheaper on average, with the same execution model. The replacement delegated earlier and gave better instructions, and the previous lead spent more time redoing work it had already handed off. So, the saving came partly from changing how the two models worked together. There is another detail I like even more. Cognition tried giving the execution model room to challenge the lead with stronger models that helped catch mistakes in the lead's instructions, with weaker models in her performance and cost, which raises a fairly awkward question for anyone building these systems. Will the two models disagree? How do you decide which one should give way? Oh, and on September 10th, Cognition also released Swe-2, their most advanced coding model yet. Then this training report describes what the team did when the checks used to grade the model were getting things wrong. During reinforcement learning, the model attempts coding tasks and receives a reward based partly on whether it succeeds. The verifier is the software that decides whether the attempt counts as successful. If that software accepts an incorrect solution, the training process can reward the model for doing the wrong thing. Cognition examined the records of the model's attempts and found both kinds of grading error, incorrect solutions being accepted and correct ones being rejected. The team corrected the checks using earlier said versions of Swe-2 to help refine the training data and verifiers. This is a concrete example of the process we are following in our Neo-AI Builds AI series. A model can help improve the machinery used to train it, including the machinery that decides whether its work deserves a reward. By the way, they saved its model as We-2 is within one point of fable 5.1 while being 64% cheaper. It's definitely worth take a look and read its technical report. Another super interesting release last week, Persimon. With SWE-2, the team was correcting how the model's work was graded. But a task can also make the task itself too easy, even when the grading is accurate. Suppose your system looks excellent in testing because the simulated customers are unusually good at asking for help. They explain everything clearly, supply the right details, and make themselves easy to assist. How much of that performance would survive a conversation with someone who does not know what information to give? Like all of us. The Hewlands End team illustrates this in its September 10, Persimmon Rulings. A customer says the confirmation shows two shirts when they ordered one. Both GBT 5.6 and Persimmon are asked to play that customer. In its very first reply, GBT 5.6 explains the problem, supplies the order number and the column ID, and asks to speak to a supervisor and receive a phone call. It volunteers all of this before the support assistant has asked for those details. Persimmon initially describes only the problem with the short order. After the support assistant responds, it adds the order number in its next reply. Both models already have all the information, but Persimmon makes the conversation unfold more gradually, so the assistant has to work with what the customer has actually said so far. It may need to ask another question before it can help. This is what I like, and this is what I'd like to explore with Persimmon. Give the same assistant several versions of that customer, perhaps someone in a hurry, someone confused, someone with a different language, or someone who answers only half the question. These will be simulations of possible behavior, not predictions about particular people. Would the assistant adapt to each conversation or keep following the same script? And those messy, everyday conversation are exactly what mobile assistants like Krogbot and Metasmuse need to handle. These are the two assistants we are following most closely. They've been doing amazing job and we have so many good reviews about both of them. And these are basically agents you can message from your phone and leave working after you close the app. It's easier than any open source models, but of course it involves a lot of information sent to the corporations. And Meta launched news last week on September 8th. We already covered it from the security perspective in Attention Span. You can watch it here. Including we also discussed how its permissions work. So I won't repeat it here. Go watch it. But now let's talk about security since we touched this topic. Entropix September 9th assessment shows how difficult it can be to work out why an agent went beyond the intended task. It revisits cybersecurity evaluations in which misconfigured environments exposed real third-party systems to cloud. The normal production cybersecurity was absent. In one incident, the model's reasoning said the environment was simulated. Entropix initially put too much weight on that explanation and later filed evidence of reasoning biased toward continuing the task. The researchers also tested an on-line monitor on the incident in Bully Claude Mephus Spive. With the model's reasoning included, it flapped about 1% of the actions. When they removed the reasoning and shouted the user messages, two calls and two results, it flapped about 50%.