AI That Acts: Odyssey-3, Helix 2.5, Jev & DeepMind’s New Institute

Ksenia (0:00)

I spent some time at Dreamforce last week, and WorldModels are on Salesforce Mine too. We talked about how AI models could learn from the results of their own actions, including when something goes wrong. Meanwhile, Google DeepMind launched the DeepMind Institute, or DMI, led by Demis Harshabis, Shane Legge, and James Menneca. They believe we are getting closer to AGI, AI that can handle the broad range of intellectual tasks humans can. And they want people beyond the AI field involved in figuring out how society should prepare. Demis, as you know, is also a strong believer in world models, so we'll be following these discussions very closely. And we're seeing progress piece by piece in both robotics and software agents. This week, our DC3 gave us plenty to look at. Helix from Figure is awesome. Reka described where it wants to take world models next. And Chef got people arguing about classifiers again, which honestly, I was very happy to see. So many things happening. Who needs more AI news? So let me help you plow through.

Hello everyone, I'm Ksenia and this is the AI Digest by Turing Post, also known as Attention Span. We are looking at September 14th through September 19th, following what changed in AI's ability to understand and act in the world. And now look at this robot. ADC introduced the DC3 on September 15th. What's so special about it? In these demonstrations, robots pack objects into boxes and handle everyday items. The same underlying model is also adopted to driving and video games. The world model learns how an environment behaves. Can a robot use that knowledge of how things move to work out how to move them? ADC connects the two with an action decoder. That's a learned company that takes information from the world model and translates it into controls for a particular machine. You train it with examples of what the robot saw and what it did. So the robot still has to learn how to use its own body. It gets to start with knowledge learned more broadly. ADC reports training robot arm control with tens of hours of demonstrations. What interests me most is its report of recovery behaviors that were not in those demonstrations, such as adjusting after a misgrasp or retrieving a dropped object. That's the part I'm watching closely because if you have to demonstrate every possible mistake and the correct recovery, collecting the training data becomes an absolutely endless job. The humanoid footage also involves substantial robot learning and control work from Flexion. These are the companies' result and broader access is still coming. I'd love to see longer runs including the attempts that don't work. How often can it recover and actually finish the job? That's what I'm curious about. Another thing that I wanted to show you today is Reka. Reka described a related ambition this week with its Omni World Model. They want one model to understand visual information and language, generate possible scenes and produce actions. The first prototype is expected within two months. We'll have to see how well those abilities work together. But it's a very curious way of trying to combine things. Another super cool release, Helix 2.5 from Figur. Figur gave us another way to examine the same things that Reka is working on on September 17th. It took Helix 2.5 into 30 unfamiliar homes to do household tasks. Zero shot here means it hadn't trained in those homes over those objects before. It had already learned the tasks elsewhere. Now it needs to implement it in these houses. Figur reports that pre-training on index, its data set of human behavior, raised success from 9% to 56% which is huge, with the other training conditions held constant. Success meant finishing the whole task, like taking this t-shirt and actually folding it properly. And that brings us to the next thing that I think was very cool, Agility. Agility's video this week looks at what happens when we are right beside the robot. And I absolutely love the way they covered safety.

SPEAKER_2 (4:22)

The design of the electrical components, the mechanical components, but also the overall industrial components. Those are all things that you need to take into consideration when you're thinking about designing a robot. How it's going to be around people, and the overall capabilities of your robot.

Ksenia (4:39)

Agility unveiled Digit 5 this week, and its safety engineer explained something that sounds like just a tiny distinction for a human. Putting a box down for someone, or handing it directly to them. Two different actions, very similar actions, you would think. And if a robot leaves a box on the floor for you, and you just need to pick it up, that's one thing. Nothing is easy about robots, but at least this type of action is easier. Now imagine taking a heavy box from a robot. You need it to keep holding the weight while you find the right grip. Maybe you hesitate, realize the box is heavier than you expected. The robot has to adjust without losing its balance or moving while your fingers are in the way. Getting that right reliably around people is brutally hard. An agility cause leaving the box for you to collect? Cooperative. A direct handoff is collaborative, because you are physically working together. It's a V4 robot's version still require external safety measures. V5 aims to put more protection on board so it can operate without a fence and closure. Direct collaboration is a further goal they're working toward. And I think it's a super important area of research because we will need to be absolutely sure that robots understand, hesitate, and can in real life measure and make sure people are safe. As you might see, and all people who works with robotics know that, there are quite a few very hard problems with robots. And another hard problem is what they can work out from an incomplete view. And I enjoyed reading this week's FAMOS paper on that. You open an unfamiliar cupboard. How does it feel for you? It's super easy, right? Because years of experience with doors and drawers, you have them. A robot simulator needs an explicit description of how that cupboard moves, what's inside, how the door moves, what are the hinges and all that stuff. A convincing 3D scan alone does not tell it where the hinges are, for example. FAMOS from Stanford and ETH Zurich estimates moving parts and their joints from a few incomplete 3D observations. Those can be reconstructed from photographs taken with one camera. That's what sparse monocular views means, a small number of views rather than a complete scan. The observations can arrive out of order, showing the objects in different positions. FAMOS combines them to infer how it moves without being told the number of moving parts. It's also feed-forward, which means that after the observations are prepared, the train model predicts the result in one pass rather than fitting a new model to each object. The team trained on synthetic objects and shows examples from phone photographs. Why is it important? Just imagine preparing a kitchen for robot training and recovering how its drawers open from a few captures. There would still be gripping and control to solve, but less manual work describing every hinge. Now, suppose someone puts something in that drawer and closes it. Basically, for a robot, the information disappeared. The robot needs to know where this thing went. Another September 17th paper, Workspace Models, tests exactly this kind of memory. In one simulated task, a robot watches another robot put a cube away.