Incrementality Testing vs SegmentStream: What’s the difference?

Knowing the true value of your marketing activities is what every digital marketer ultimately wants. Wouldn’t you want to better understand which of your ads are actually bringing you sales? 

Yet it’s not an easy task to measure the true value. There are multiple ways to approach it. The first solution that comes to mind is measuring the incremental impact of the activities you’re running through Incrementality Testing. However, with SegmentStream we also aim to find the real value of the marketing channels, campaigns or even ads. Which one should I go for then?

Let’s have a closer look at how Incrementality Testing operates, what are its pros and cons, and how it compares to SegmentStream.

What is Incrementality Testing?

Incrementality Testing is basically an A/B testing intended to measure the incremental value of a marketing strategy or a tactic. Its main difference from the A/B testing is that Incremental Testing would typically divide your target audience into a test and a control group, where the test group is exposed to the marketing activities the impact of which you’d like to measure while the control group is completely unexposed. Then you carry out an experiment and compare the performance of these two groups. The typical way to calculate the incrementality would be to calculate the conversion or any other desired outcome difference between the test group and the control group.

The incrementality shows the percentage of conversions that happened due to the tested marketing activities beyond the ones that would have happened anyway.

Ex.: Your CMO wonders whether Facebook ads are worth running at all, as he sees a very low performance based on the last non-direct attribution. The drastic way to test it would be to switch off Facebook ads for some time at all and observe whether there will be any change to the amount of conversions through other channels. However, that’s a pretty big risk for business. To play it safe, they launched Incrementality Testing, having a test group exposed to the Facebook ads among all other channels and a control group that is not.

When they compared the results of these two groups, they saw that the test group that was exposed to Facebook ads generated $190K sales revenue, while the control one generated only $100K. That means that those sales for $100K would have happened anyway. So Facebook brought an additional $90K or 47% incremental sales revenue. The CMO decides to continue running Facebook ads.

All Incrementality Testing tools can be divided into two core groups: cookie-based and cookieless. In digital ad space most of the Incrementality Testing tools would still rely on cookies to split the audience and assess the results. They are typically called auction- and audience-split tests, meaning that the audience is split either at the moment of the auction or the test and control groups are targeting different cookie-based audiences. However, over the recent years many cookieless solutions appeared, they would split the audience by geo, time, product or demographics. They also tend to look into higher-level business metrics such as revenue, new customer and site visitors, compared to CTR, attributed channel conversion rate or CPA that a traditional A/B test would typically look into. Let’s review what are the pros and cons of each group.

Cookie-based Incrementality Testing Pros and Cons

Pros:

• Auction- and audience-based tools allow to have a proper random split of the audience which makes the test more fair.
• Can be used for media planning on strategic and tactical levels and make manual budget reallocation decisions down to ad group level.
• Does not require any historical data collected prior to the launch of the test itself, you simply set up the experiment and wait for the data to accumulate. 

Cons:

• Due to the essence of the split, these methods require the conversions to happen within the same cookie, otherwise they won’t be reported as the results of the test and control groups. Since capturing conversions within a single cookie across different devices and browsers is close to impossible, such tools are lacking quite a lot of valuable conversion data and are biased towards lower-funnel activities. 
• Similar to Multi-Touch Attribution, you would need to capture a significant amount of conversions to assess the Incrementality Testing results. It typically takes around 1-30 days, depending on the attribution window of your business. Since ITP limits the life of a cookie down to 1-7 days, audience- and auction-based Incrementality testing would work for you only if your customers convert using the same device or browser within 1-7 days after an ad click.
• You can make manual budget adjustments based on Incrementality Testing output, but you can’t use the data for any automatic ad optimisation. 

Cookieless Incrementality Testing Pros and Cons

Pros:

• Does not depend on cookies as the audience is split based on geo, product, demographics, etc. That means that conversion can be captured even if customers converted through a different device or browser, as long as they are a part of a control or a test group based on the selected criteria.
• Not impacted by ITP limitations and short attribution windows, allowing to account for longer sales cycles.
• Same as for Cookie-based Incrementality Testing solutions, no historical data collected prior to the launch of the experiment is needed.

Cons:

• Such audience splits are less random compared to audience- or auction-based ones. Test and control groups can be impacted by many additional factors influencing their performance like specific sale periods, local holidays, offline advertising or simply other online advertising channels. 
• To compare different channels’ incrementality you need to run several tests each time keeping a control group aside of the tested channel, which limits the ability to attract new users and scale the business. 
• It is extremely hard to run such experiments for specific channels in isolation and almost impossible to pinpoint specific campaigns, not speaking about ad groups. That means that you can barely use them for tactical planning and optimisation if you want to go beyond the channel level. Even if you do, it’ll be hard to observe the impact of the optimisation actions.

What about SegmentStream?

SegmentStream is an AI-powered platform that enhances marketing budget decisions, offering a reliable alternative to incrementality testing.

SegmentStream finds a correlation between ad spend, clicks, website visits (incl. user behaviour) and total sales in the same region/country.

This way, SegmentStream measures true contribution of each channel and campaign on total revenue, even when conversions cannot be directly attributed using cookies. 

It provides 3 solutions:

1. Automated Marketing Reporting to measure key metrics in one place.

2. Cookieless Attribution. It analyzes ad clicks, impressions, costs and user behaviour to measure the incremental impact of ad campaigns.

3. Optimization suit that offers specific budget recommendations.

To sum things up,

Incrementality Testing is a good testing mechanism but due to the fact that you would always have to keep a control group totally unexposed to the tested channel or campaign, it significantly limits your ability to scale and can lead to lots of missed opportunities and lost revenue. 

It is also much less actionable than SegmentStream as it’s not compatible with any automatic ad optimisation and it doesn’t always solve the cookie limitations challenge (depending on the type of a split you’re running). SegmentStream not only helps identify the true incremental ROAS of your marketing campaigns, but also provides AI-generated budget allocation recommendations.

Book a demo here.