The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon)

**Ronny Kohavi** (0:00)
I'm very clear that I'm a big fan of Test Everything, which is any code change that you make, any feature that you introduce has to be in some experiment, because again, I've observed this sort of surprising result that even small bug fixes, even small changes can sometimes have surprising unexpected impact.
And so I don't think it's possible to experiment too much. You have to allocate sometimes to these high-risk, high-reward ideas. We're going to try something that's most likely to fail, but if it does win, it's going to be a home run. And you have to be ready to understand and agree that most will fail. And it's amazing how many times I've seen people come up with new designs or a radical new idea, and they believe in it and that's okay. I'm just cautioning them all the time to say, if you go for something big, try it out, but be ready to fail 80% of the time.

**Lenny Rachitsky** (1:05)
Welcome to Lenny's Podcast, where I interview world-class product leaders and growth experts to learn from their hard-won experiences building and growing today's most successful products. Today, my guest is Ronny Kohavi. Ronny is seen by many as the world expert on A-B testing and experimentation. Most recently, he was VP and Technical Fellow of Relevance at Airbnb, where he led their search experience team. Prior to that, he was Corporate Vice President at Microsoft, where he led Microsoft's experimentation platform team. Before that, he was Director of Data Mining and Personalization at Amazon. He's currently a full-time advisor and instructor. He's also the author of the go-to book on experimentation called Trustworthy Online Controlled Experiments. And in our show notes, you'll find a code to get a discount on taking his live cohort-based course on Maven. In our conversation, we get super tactical about A-B testing. Ronny shares his advice for when you should start considering running experiments at your company, how to change your company's culture to be more experiment-driven, what are signs your experiments are potentially invalid, why trust is the most important element of a successful experiment, culture, and platform, how to get started if you want to start running experiments at your company. He also explains what actually is a p-value and something called Twyman's Law plus some hot takes about Airbnb and experiments in general. This episode is for anyone who is interested in either creating an experiment-driven culture at their company or just fine-tuning one that already exists. Enjoy this episode with Ronny Kohavi after a short word from our sponsors.
This episode is brought to you by Mixpanel. Get deep insights into what your users are doing at every stage of the funnel at a fair price that scales as you grow. Mixpanel gives you quick answers about your users from awareness to acquisition through retention. And by capturing website activity, ad data, and multi-touch attribution right in Mixpanel, you can improve every aspect of the full user funnel. Powered by first-party behavioral data instead of third-party cookies, Mixpanel is built to be more powerful and easier to use than Google Analytics. Explore plans for teams of every size and see what Mixpanel can do for you at mixpanel.com/friends/lenny. And while you're at it, they're also hiring. So check it out at mixpanel.com/friends/lenny.
This episode is brought to you by Round. Round is the private network built by tech leaders for tech leaders. Round combines the best of coaching, learning, and authentic relationships to help you identify where you want to go and accelerate your path to get there. Which is why their waitlist tops thousands of tech execs. Round is on a mission to shape the future of technology and its impact on society. Leading in tech is uniquely challenging, and doing it well is easiest when surrounded by leaders who understand your day-to-day experiences.
When we're meeting and building relationships with the right people, we're more likely to learn, find new opportunities, be dynamic in our thinking, and achieve our goals.
Building and managing your network doesn't have to feel like networking. Join Round to surround yourself with leaders from tech's most innovative companies. Build relationships, be inspired, take action, visit round.tech slash apply, and use promo code Lenny to skip the waitlist. That's round.tech slash apply.
Ronny, welcome to the podcast.

**Ronny Kohavi** (4:32)
Thank you for having me.

**Lenny Rachitsky** (4:34)
So, you're known by many as maybe the leading expert on A-B testing and experimentation, which I think is something every product company eventually ends up trying to do, often badly.
And so, I'm excited to dig quite deep into the world of experimentation and A-B testing to help people run better experiments. So, thank you again for being here.