Inverted Podcast #30 - Evidence Guided Product Management with Itamar Gilad

Dario (0:05)

Welcome, everybody, to another episode of The Inverted Podcast, where we talk about things that products and services that have the weird characteristics that success is when nothing happens. Security, quality, infrastructure, when nothing happens, that's success. And that makes it weird for us to work on them, to sell them, to communicate about them. And every time we unpack a little bit more of it. Today, as always, my dear friends, Dario and Dana, welcome.

And welcome Itamar. We're so excited to have you here.

Itamar Gilad (0:38)

Welcome. Thank you very much, guys. Happy to be here.

Dario (0:41)

Yeah. And for those of you who don't know Itamar, obviously, you can find him on LinkedIn and all the other social media that you use.

But you do know him if you use Gmail because there is something called promotions, updates, social. That's something he built a long time ago into Gmail, which is super exciting. And he tells the story of how this came to be in this amazing book called Evidence-Guided Product Management. And that's why we're so excited to have him here today to talk about how this works for Inverted products. So welcome, Itamar. Why don't you start by just unpacking this framework for us?

Itamar Gilad (1:19)

Yeah, for sure.

Just for your audience, apart from Gmail, I used to be an engineer in the past.

Before I turned to the dark side, I used to work at Microsoft on enterprise security back in the 2000s. So I know some of your sufferings. And since 2017, I've been consulting and working with many different types of technology companies to evolve product management, and especially the practices of evidence-guided development. And what that basically means, and I'm sure the audience is familiar with the core concept, is that instead of building our products solely based on opinions, or themes, or logic, or whoever the highest-paid person's opinion, we try to inject evidence into every step of the way, every step of the process, and kind of supercharge our brains, in a sense. If you want an example of this in law, in medicine, in science, people don't make decisions solely based on what seems right. A judge will never pass judgment without seeing the evidence, for example. A doctor will never prescribe a medication or an operation without seeing the evidence. So I kind of try to push companies, tech companies to start thinking along these lines. And that's basically the topic of the book.

Dario (2:47)

And so how, so this is a really great start, right? We are all building a bunch of different products in security, passkey, sign in, all those things. And there's with us a lot of people that keep basically the world safe without nobody noticing. But imagine them being in a big company, right? And I can imagine when you're building like a social platform or a classic product where you have a funnel, like you have increasing revenue or increasing daily active users, like you can basically do this evidence. In your book, you have this really great example of how to use this framework. And you have two competing features. And as you gather evidence, slowly one wins and then you launch it, right?

But in security, that's very different because let's say we have a flow of criminals that come in and attack our products or abusers, right? Fake reviews, fake comments, fake installs, attacks, whatever.

Like this entire thing goes upside down, right? Like I can't really know how much badness there is out there. So how would you approach it? How would you tell this to a group of security product managers or quality product managers? How does this evidence framework work for them?

Itamar Gilad (4:03)

So let me introduce the framework a little bit. It's called GIST, Goals, Ideas, Steps, and Tasks. And those are basically the four areas where I see product organizations needing to change or to evolve in order to do evidence guided development. So at the goals layer, which I think is exactly what you're aiming for, one of the key concept is the North Star metric, which many of you are using. But I will explain just in case the North Star metric is one metric that measures the total amount of value we are creating for all users.

There might be other top level metrics that measure how much value we're creating for ourselves, revenue, market share, paying customers, data.

These are all important. I usually recommend picking one, by the way, just to be more focused. But we have to counterbalance this with user-facing or customer-facing metrics. And there's a lot of overlap, by the way. It's not one is competing with the other.