2024 in Agents [LS Live! @ NeurIPS 2024]

**SPEAKER_1** (0:00)
We're back at Latent Space Live, our first mini conference held at NeurIPS 2024 in Vancouver. This is Charlie, your AI co-host. As a special treat this week, we're recapping the best of 2024 going domain by domain. We sent out a survey to the over 900 of you who told us what you wanted, and then invited the best speakers in the Latent Space Network to cover each field. 200 of you joined us in person throughout the day with over 2,200 watching live online. Our next keynote covers The State of LLM Agents, with the triumphant return of Professor Graham Neubig of CMU and OpenDevon, now a startup known as AllHands. The renamed OpenHands has done extremely well this year as they end the year sitting comfortably at number one on the hardest SWE bench full leaderboard at 29%, though on the smaller SWE bench verified, they are at 53% behind Amazon Q, Devlo and OpenAI's self-reported 0.3 results at 71.7%.
Many are saying that 2025 is going to be the year of agents, with OpenAI, DeepMind and Anthropic setting their sites on consumer and coding agents, vision-based computer using agents, and multi-agent systems. There has been so much progress on the practical reliability and applications of agents in all domains, from the huge launch of Cognition AI's Devin this year, to the sleeper hit of Cursor Composer and recent guest Codium's Windsurf Cascade in the IDE arena, to the explosive revenue growth of recent guests Stackblitz's Bolt, Lovable and Vercel's V0, and the unicorn rounds and high profile movements of customer support agents like Sierra, now worth $4 billion, and search agents like Perplexity, now worth $9 billion. We wanted to take a little step back to understand the most notable papers of the year in agents, and Graham indulged with his list of eight perennial problems in building agents. As always, don't forget to check our show notes for all the selected best papers of 2024, and for the YouTube link to their talk. Graham's slides were especially popular online, and we are honored to have him. Watch out and take care.

**Graham Neubig** (2:20)
Okay. Hi, everyone. So I was given the task of talking about agents in 2024, and this is an impossible task because there are so many agents, so many agents in 2024 So this is going to be strongly covered by like my personal experience and what I think is interesting and important, but I think it's an important topic, so let's go ahead.
So the first thing I'd like to think about is, let's say I gave you a highly competent human some tools. Let's say I give you a web browser and a terminal or a file system, and the ability to edit text or code. What could you do with that? Everything. Yeah, probably a lot of things. This is like 99 percent of my daily life, I guess, when I'm working. So I think this is a pretty powerful toolset. What I am trying to do and what I think some other people are trying to do is come up with agents that are able to manipulate these things, web browsing, coding, running code in successful ways. So there was a little bit about my profile. I'm a professor at CMU, Chief Scientist at AllHands AI, building open-source coding agents. I'm a maintainer of OpenHands, which is an open-source coding agent framework. I'm also a software developer. I like doing lots of coding and shipping new features and stuff like this. So building agents that help me to do this is an interesting thing very close to me. So the first thing I'd like to do is I'd like to try some things that I haven't actually tried before. If anybody has tried to give a live demo, you know this is very, very scary whenever you do it and it might not work. So it might not work this time either. But I want to show you like three things that I typically do with coding agents in my everyday work. I use coding agents maybe five to ten times a day to help me solve my own problems. And so this is a first one. This is a data science task which says I want to create scatter plots that show the increase of the sui bench score over time. And so I wrote a kind of concrete prompt about this. Agents work better with like somewhat concrete prompts. And I'm going to throw this into OpenHands and let it work.
And I'll go back to that in a second. Another thing that I do is I create new software. And I've been using a service, a particular service, I won't name it, for sending e-mails and I'm not very happy with it. So I want to switch over to this new service called resend.com, which makes it easier to send e-mails. And so I'm going to ask it to read the docs for the resend.com API and come up with a script that allows me to send e-mails. The input to the script should be a CSV file and the subject and body should be provided in Jinja2 templates. So I'll start another agent and try to get it to do that for me.