How Incrementality Cuts Waste + What and Where to Test in Ecommerce

**Sean Frank** (0:00)
We're here with Northbeam, one of our favorite sponsors, and we're talking incrementality, and Austin promises it's not going to be boring. So Austin, what's the spicy take, my brother?

**Austin Harrison** (0:09)
Spicy take is, a lot of people have been saying recently on social, hey, you need to run longer tests, right? You need to run longer tests to get signal.
And that's honestly a nonsensical thing to say. Maybe you need to run a longer test, maybe you can run a shorter test, and that's why Northbeam is different, right? Because we've worked with both of you for years on MTA, right? Multi-touch attribution on hourly transactional data. And so, yeah, maybe one channel, you need to run a three months test, maybe one, a three week, maybe one, a six week, right? But just to blanketly say, which people are saying, oh, you need to run longer tests. That's just not, that's not a smart thing to say.

**Katy Mimari** (0:52)
So you don't have to run it long, and isn't it have to do with like statistical difference? So it's kind of like the volume that you have.

**Austin Harrison** (0:59)
It's conversion lag sometimes, right? It's conversion lag, it could be statistical volume, it could be a multitude of things, right? But because Northbeam has their MTA data, we can not only do a better holdout that's more precise because people in the valley where Sean is may purchase slightly differently on an MTA hourly basis and they do in like downtown LA, right? Where Jason and Hex Cloud folks are. So we could do a better holdout, statistically speaking, but also understand the right lag. So should it be three months, six, you know, four months, two weeks, six weeks? So people are just saying the wrong things.

**Sean Frank** (1:32)
Austin Habanero Harrison coming in with a...

**Austin Harrison** (1:35)
Coming in hot, saying people are saying the wrong stuff, wrong thinking. Yeah.

**Sean Frank** (1:39)
I think for this episode, what we should do is put yourself in the shoes of the average listener.
Cody is a guy I know who listens to the show, and not Cody from Drone to Be A Different Cody. Doing $5 million, $10 million, $20 million a year. It's an e-commerce brand. They're just starting to figure some stuff out. First, what the hell are we measuring? Why is incrementality important?
Yeah, like bare bones, day one. What are we talking about here?

**Austin Harrison** (2:08)
Yeah. So when you think about multi-touch attribution or just basic analytics, we've worked with you all for years, but when I think about people that signed up for us early, like Groons or Comfort, we all know some of the fastest growing companies in the history of our industry, they signed up early because they wanted to understand every hour, how are things performing, whether it's their influencers, whether it's Meta, Snapchat, TikTok, whatever it is. So that's like, did someone click and buy something? Did it take them 60 days? Did it take them a week?
Northbeam launched our clicks into hermistic views where we're splitting credit on a view basis. So Katy views an ad on TikTok, views something on Snap, views something on Meta and buys a widget, we divide it a third, a third, a third. So that's MTA. For the listeners out there, when do you need incrementality? Well, I talked to Connor about this, the CMO of the Ridge about calibrating your MTA numbers. So if you're looking at like, hey, on Meta and Northbeam, I'm looking at like a $50 CAC.
But you think maybe people convert in retail or on Amazon, then you want to calibrate that number to say, oh, well, it's a $50 CAC, but if I include Amazon or maybe conversions I don't see, maybe it's a $40 CAC, right? So it's that calibration layer. So I think when you're small, Sean, you talk about it, like, yeah, you can keep it pretty simple. Like you look at your MTA numbers, you see your bank account moving, you try to see what's what. But eventually when you're omni-channel, right? So you're in retail, you're on Amazon, you're really diversifying your revenue where it lands. You need to understand your spend better, right? Because if you're going to spend more, you've got to calibrate it.

**Sean Frank** (3:54)
Yeah. And let's talk about those early days. You're a $5 million brand listen to this and you're just running Facebook ads, right? There's, you're probably just running conversion optimized Facebook campaigns, right? ASC, Big Cap, something like that, right? You're doing like the most basic stuff possible.
You don't need incrementality and you do not need MTA tools, right? Austin said you're being like, hold on, hold on.