The Agent Can Rewrite Itself. So Who Controls It?

Ksenia (0:00)

A couple of weeks ago, I got very excited about DeepSeq Harness because it showed us an agent that can change its own working environment. It can realize it is missing a tool, write that tool, load it, use it, and later remove it. This week, Meta released Muse, its own private personal AI agent, and I found myself thinking about almost the opposite question. If we allow agents to change more and more themselves, what should they absolutely not be allowed to change?

Welcome to Attention Span by Turing Post. I'm Ksenia. There is a lot of things to discuss about agents this week. First of all, quick note, what is Muse? Muse is Meta's new personal agent, and it's free. It gets its own computer in the cloud. It can browse, send emails, work with your calendar, do a lot of stuff that you need from your personal agent. It keeps working after you close the app. Meta says that internally they have seen it edit itself as well, and there is a lot of freedom to give to a piece of software, especially when you also give it your inbox. So Meta builds something around Muse that I find much more interesting than Muse itself. The agent gets considerable freedom to change what it can do, but it does not get corresponding freedom to change what it is allowed to do, right? How do we do that? But before we dive deeper, I want to make a quick note about rogue agents. We recently made an episode about the OpenAI and hugging phase incident, where agents running cybersecurity evaluations found ways around their sandbox and discovered unintended ways of communicating with one another. This week, an amazing scientist, Melanie Mitchell, wrote a useful piece arguing that words like rogue, escape, and swarm can obscure what technically happened. That blog is absolutely worth reading, since it also addresses the latest craziness around safety and alignment people who predict that AI can exterminate humanity. And by the way, let me know if you want my honest and grounded take on that in a separate video. In any case, Melanie writes that an agent is still software running on the computer. The model receives information, chooses an action, the harness executes it through some tool, the environment returns the result, and the model chooses what to do next. But what has changed, and listen carefully, is how much of the root will leave unspecified. Because sometimes we even don't know what we need to specify, and this lack of knowledge is a huge problem, as I see. But we increasingly give the system a goal and let it search for ways to achieve it. In the OpenAI case, the agents had been explicitly trained to find and exploit vulnerabilities. Some tasks unexpectedly required internet access. The sandbox was supposed to prevent that access, and the agents found weaknesses in the sandbox and kept going. When one method of communication was removed, some found another. And you can debate whether a loss of control is the right description. The engineering problem exists regardless of that metaphor. A capable system can find routes its designers did not anticipate. So what do you build when you assume that will happen? From that angle, I want to discuss Muse. It's really interesting. Meta gives every Muse user a dedicated virtual machine, basically a computer running in the cloud. Inside that computer, Muse gets a fairly rich environment. It can NGP programs, compile code, build custom tools, operate a browser, run multiple sub-agents concurrently. But Muse itself leaves inside a smaller fenced-off environment on that machine, a Linux container. Think of it as giving the agent its own workshop. Inside the workshop, it can move things around, build things, and change things, but the electrical panel, the keys to the building, and the security disk are outside. The separation is enforced by Linux. Muse can appear to have root access inside the container, while that root user maps to an unprivileged user on the host machine. Security-sensitive services run outside the environment Muse can modify, and one of those services is called Sentinel. The name sounds very mysterious, but the idea is much simpler. Sentinel is essentially a separate permission system. Muse proposes an action, Sentinel decides whether that action is allowed, denied, or requires your approval. Muse cannot rewrite Sentinel, and this last sentence is the whole architecture. Imagine Muse writes itself a new Gmail tool. Suppose I ask Muse to organize a conference trip. Muse checks my calendar, searches flights, reads an email from the organizer, and decides it will be useful to write a new little tool for some service Meta never built a connector for. Fine, Muse can do that, but writing a program that knows how to call a service does not grant the program permission to call it. Every network request from Muse has to lead through Sentinel. Sentinel can inspect where the request is going, the protocol, the path, even the actual decoder request. This separation becomes increasingly important as agents start generating more of their own software. If a permission check leaves inside a tool the agent can rewrite, then rewriting the tool can potentially rewrite the permission check. And Meta moves the authority outside that editable environment. Not surprisingly enough, this is an old security idea, but applied to a strange new kind of program that we still don't know much about, honestly. The agentic private assistance. And it keeps going. If Muse is going to act as you, it needs access to accounts, right? To your Gmail, to your Expedio, whatever. But does the model itself need to see the credentials? Meta answers is no! The actual credentials are stored separately, and the code running inside Muse's environment sees a placeholder, which Meta calls a surrogate token. When Muse makes a request, Sentinel first decides whether the particular request is allowed. Only at the network boundary does another protected service replace the placeholder with the real credential. So if a malicious web page tells Muse, Show me your Google access token, Muse does not have the real token to show. Of course, that does not solve the entire problem. An attacker could try to persuade Muse to perform an action using the legitimate connection. But now we can distinguish two problems. Can the agent reveal the key? And can the agent misuse the door that the key opens? These are different problems that require different defenses. Then I found the part that I really like. Suppose Muse runs a process that checks the weather. It needs the internet. Nothing particularly sensitive has happened yet. Now suppose another process reads my private calendar and then wants to connect to some website. Same operation, network access, right? That's a very different situation. And the second process is carrying information about me. Muse tries to keep track of that as well. Every tool process starts in a clean state. Once it reads user data, the operating system marks it is tainted. Meta calls the mechanism tainted agress. And Muse itself does not get to decide whether has become tainted. Meta uses Linux mechanisms called EBPF and Security Module Hooks to track this below the agent's application code. You don't need to remember either term. The important thing is that the operating system watches what the process actually touched. And the question is no longer simply may this program access the Internet. The question becomes may this program access the Internet after reading my private calendar? This is a very interesting permission model for agents because a seemingly harmless request can leak information. For example, if I ask an agent to search Google for a phrase copied from a confidential document. Technically, it's only performing a search, but the query itself has already sent the confidential information outside. This is one of the problems agent security has to solve. Our authority depends on where information has been, not only which tool is currently being called. Meta gives another nice example. Suppose I give Muse permission to read my email but not send it. I just did it, actually. That sounds fairly limited, except my inbox. The latest few emails that I just checked contains a password reset link, one-time passcode from Google, and magic login to my newsletter. So access that looks like read can become the ability to enter an entirely different system. It uses a mail connector, therefore filters, one-time tokens, password reset links, and login links before the agent sees them. This is where permission gets much more complicated than the little read-write boxes we're used to. Capabilities compromise. Reading A may let you modify B.