Flow Engineering and Code Integrity at Scale with Itamar Friedman, CEO of Codium AI

**Itamar Friedman** (0:00)
I say that very soon, even if we're sophisticated programs, we will see higher quality coming from AI. Even in the case we're in a hurry, you don't think a problem, here's my code. Usually, like, here's the problem. Let's think a bit about how I want to design it. Even if it's not too much, you do a few steps. It doesn't make sense that we will ask a model, like prompt, generate for me, even if it's super intelligent. Let's let it sink the same way we do it.

**Nathan Labenz** (0:22)
It doesn't seem intuitive to me that you would ask an AI to do something, then immediately take its solution and ask it to critique its own solution. And yet, it works.

**Itamar Friedman** (0:30)
One day, we would want the AI to be a team member.
A virtual team member cannot just write code without testing it, without understanding the deeper context. A team member needs to be on the entire three parts of pre-build, build, and ship.

**Nathan Labenz** (0:45)
Hello, and welcome to Turpentine AI. What's that you say, Turpentine AI? Well, we've got some news for you. After a year and 100 episodes of The Cognitive Revolution, Turpentine is doubling down on AI. We're creating a new dedicated channel for The Cognitive Revolution. Turpentine will be spinning up new AI-focused feeds as well. And this feed, soon to be known as Turpentine AI, will become a shared platform featuring best-of content from multiple shows and voices. We'll be making this transition gradually over the next month or so, and I'll be posting original content to both the new Cognitive Revolution feed and here on the Turpentine AI feed. So if you want to keep up with all of the latest, definitely make sure to subscribe to the new Cognitive Revolution feed, where I've just posted part two of my recent appearance on the 80,000 Hours podcast as the first full episode.
You can find the new feed at the same URL, cognitiverevolution.ai, and there you'll find links to the new YouTube channel, Apple Podcasts, and Spotify feeds, and for those of you like me who use an old school podcast app, the Bayer RSS feed as well. Please do take a second to make sure you are subscribed to both feeds and stay tuned to both channels for lots more original AI content coming soon.
Today, my guest is Itamar Friedman, co-founder and CEO of Codium AI, a company that's on a mission to make code integrity simple.
Now, for context, at this point, it's broadly understood that large language models make useful coding assistance. Microsoft GitHub's copilot was the first commercially successful implementation of a large language model for developers, and it's already been four months since Microsoft announced that it had become a $100 million business all on its own.
However, code generation and code completion, while they make for amazing and highly inspirational demos, support just one part of the software development lifecycle. Before developers can write code, they must work with teammates to figure out what to build in the first place. And after writing code, they have to test it and integrate it into broader systems and production environments. It turns out, especially as products and teams grow, that the coding itself is often a minority of the total work. And that's where Codium AI comes in.
Focused on code integrity, Codium is meant to support the rest of the software development process, with focus on explaining, testing, improving, and integrating code. These are areas where just about every team could stand to improve. And though the company is only 18 months old, it already serves hundreds of thousands of developers.
In the first half of this conversation, we discuss Codium's product, starting with the value that I found in it, even as a solo prototype developer who doesn't really have to worry about production issues, and then broadening out to discuss how Codium supports larger teams and their correspondingly larger products. Then in the second half, we dive into their recent publication, Alpha Codium, which shows how careful task decomposition, that is the process of breaking down larger projects that language models might struggle with into much smaller tasks that they can do far more reliably, ultimately enables successful workflow automation. The Codium team calls this flow engineering, short for workflow, and while they do use some techniques that are specific to software development, what stuck out to me most about this research is how readily generalizable and applicable it is to many other domains as well. The project is open source, and I definitely encourage anyone building AI-powered workflows to take a close look at their framework, prompts, and insights. Codium's work, as well as that of Google DeepMind, show that large language models are already competitive with human participants in coding competitions. It seems very likely to me, and as you'll hear to Itamar as well, that they will continue to improve and soon achieve elite performance levels. But even as that happens, and in some ways perhaps even more so then than now, it will be critical to ensure that AI-generated code aligns to requirements and performs reliably. And with that in mind, I think Codium is extremely well positioned for success, and definitely encourage you to check it out. As always, if you're finding value in the show, we appreciate it when you take a moment to share it with friends. This one naturally should go to the software developers in your life. Please don't hesitate to send us any feedback, guest suggestions, or questions. My DMs are open on all of the social media platforms. And remember to visit cognitiverevolution.ai to subscribe to the new Cognitive Revolution feed, where you will find exclusive episodes starting this week. Now, I hope you enjoyed this conversation about harnessing large language models for code integrity with Itamar Friedman of Codium AI. Itamar Friedman, founder and CEO of Codium AI, welcome to the Cognitive Revolution.