BONUS: DevCon London: Real Talk on AI ROI, Harnesses & Evals

**Simon Maple** (0:00)
Hello and welcome to AI Native DevCon by Tessl. This is the fourth edition of this conference, the second in person and the first in London. We're at the Brewery in Central London, near the Barbican Center. Let's go in and take a look at some of the Expo Hall, and we'll give you a flavor of what AI Native DevCons like. Follow me.
The AI Native Dev is a podcast for developers and engineering leads at the cutting edge of AI and agentic coding. Join your hosts, Guy Podjarny and me, Simon Maple, every week as we chat with the most exciting voices in AI and tackle the biggest questions facing developers today. This is the AI Native Dev.
Okay. So here we are at the Expo Center of AI Native DevCon. We're going to walk in. It's a break. It's busy. Let's try and talk to some attendees. So follow me. You can see the Expo Hall is packed, full of people, full of attendees, and full of sponsors. There's areas to get some free shirts and the sponsors going right round. So let's find someone to talk to. Follow me. Tell us a little bit about yourself. What's your name and who do you work for?

**Sergo Vashikvatsi** (1:06)
Sergo Vashikvatsi. I have a couple companies, but the reason I'm here, I have a fintech startup I'm building alone.

**Simon Maple** (1:11)
Oh, nice.

**Sergo Vashikvatsi** (1:12)
Combining AI and blockchain, public and private markets. So I'm here to learn what are the tools available to make my life easier, to explain the process and to make the production process output more reliable.

**Simon Maple** (1:25)
So I look for tools and practices and processes. What are some of the sessions that you've really enjoyed so far?

**Sergo Vashikvatsi** (1:32)
Max, the guy from...

**Simon Maple** (1:34)
Max Furman?

**Sergo Vashikvatsi** (1:35)
No, no, the Max Maximilian from Argentina.

**Simon Maple** (1:38)
Oh, yeah.

**Sergo Vashikvatsi** (1:40)
Web MCP.

**Simon Maple** (1:41)
Yeah, Max Furman, yeah, yeah.

**Sergo Vashikvatsi** (1:42)
Max, I don't remember his last name.
He was very interesting. Then Dave Farley.

**Simon Maple** (1:48)
Yeah.

**Sergo Vashikvatsi** (1:48)
Dave Farley with N-Wave. So I'm planning to try all those tools in my stack and see what it can...

**Simon Maple** (1:57)
Yeah, very interesting. One of the things I learned with Dave Farley is, everyone has a different approach to doing agentic development and agentic coding. And so Dave's, some of the things that Dave was saying works obviously really well for him. And then some other people kind of like saying, actually I agree with this bit, but not that. So I think it's really important for us to find our own way, but learn from others who are doing this at scale as well.

**Sergo Vashikvatsi** (2:16)
One more session that caught my attention yesterday was Matthias Lübeck from Pi.
The company is different, I forgot the name, but he's using Pi. So I want to try that one as well.

**Simon Maple** (2:28)
Amazing. We'll enjoy the rest of the conference. Thank you.

**Chris Baty** (2:31)
Awesome.

**Simon Maple** (2:32)
Thank you. So Chris, this is Chris Baty, one of our speakers. Chris, tell us a little bit about yourself and what you presented this week.

**Chris Baty** (2:41)
Yeah, so I've been an engineer for 20 years or so. And what I've done over the last couple of years is I've been building a product team within a consultancy. So we've hired new people, we've tried to build products. And so my talk was about our lessons learned of adopting AI in that team, and what we're able to achieve, which we wouldn't have been able to achieve, I think, if we tried to do this five years ago. Yeah.

**Simon Maple** (3:00)
And what are some of the key learnings that you would say if you were to give advice to someone who is on this journey as well? What are the key things that you wish you could have told yourself?

**Chris Baty** (3:09)
Avoid vanity metrics at all costs, like code commits or how many tokens people are using, have a look at your flow of delivery, from idea to someone using your piece of software. Don't stop at commit or outputs. Think about the actual outcomes you want to achieve, and find the friction, because it's not normally in code production. For our experience, it's not even idea production, like product management. We've been pretty good at that now.
The bottleneck for us is actually getting real people to use our software and quickly feeding back on those.

**Simon Maple** (3:44)
How well are you finding, I guess, execs or senior folks within the engineering teams to accept those types of metrics over and above token usage?