Incrementality Testing vs Conversion Modelling: What’s the difference?
Knowing the true value of your marketing activities is what every digital marketer ultimately wants. Wouldn’t you want to better understand which of your ads are actually bringing you sales?
Yet it’s not an easy task to measure the true value. There are multiple ways to approach it. The first solution that comes to mind is measuring the incremental impact of the activities you’re running through Incrementality Testing. However, with Conversion Modelling we also aim to find the real value of the marketing channels, campaigns or even ads. Which one should I go for then?
Let’s have a closer look at how Incrementality Testing operates, what are its pros and cons, and how it compares to Conversion Modelling.
What is Incrementality Testing?
Incrementality Testing is basically an A/B testing intended to measure the incremental value of a marketing strategy or a tactic. Its main difference from the A/B testing is that Incremental Testing would typically divide your target audience into a test and a control group, where the control group is exposed to the marketing activities the impact of which you’d like to measure while the test group is completely unexposed. Then you carry out an experiment and compare the performance of these two groups. The typical way to calculate the incrementality would be to calculate the conversion or any other desired outcome difference between the test group and the control group.
The incrementality shows the percentage of conversions that happened due to the tested marketing activities beyond the ones that would have happened anyway.
Ex.: Your CMO wonders whether Facebook ads are worth running at all, as he sees a very low performance based on the last non-direct attribution. The drastic way to test it would be to switch off Facebook ads for some time at all and observe whether there will be any change to the amount of conversions through other channels. However, that’s a pretty big risk for business. To play it safe, he launched Incrementality Testing, having a test group exposed to the Facebook ads among all other channels and a control group that is not.
When he compared the results of these two groups, he saw that the control group that was exposed to Facebook ads generated $190K sales revenue, while the test one only $100K. That means that those sales for $100K would have happened anyway. So Facebook brought additional $90K or 47% incremental sales revenue. The CMO decides to continue running Facebook ads.
All Incrementality Testing tools can be divided into two core groups: cookie-based and cookieless. In digital ad space most of the Incrementality Testing tools would still rely on cookies to split the audience and assess the results. They are typically called auction- and audience-split tests, meaning that the audience is split either at the moment of the auction or the test and control groups are targeting different cookie-based audiences. However, over the recent years many cookieless solutions appeared, they would split the audience by geo, time, product or demographics. They also tend to look into higher-level business metrics such as revenue, new customer and site visitors, compared to CTR, attributed channel conversion rate or CPA that a traditional A/B test would typically look into. Let’s review what are the pros and cons of each group.
Cookie-based Incrementality Testing Pros and Cons
- Auction- and audience-based tools allow to have a proper random split of the audience which makes the test more fair.
- Can be used for media planning on strategic and tactical levels and make manual budget reallocation decisions down to ad group level.
- Does not require any historical data collected prior to the launch of the test itself, you simply set up the experiment and wait for the data to accumulate.
- Due to the essence of the split, these methods require the conversions to happen within the same cookie, otherwise they won’t be reported as the results of the test and control groups. Since capturing conversions within a single cookie across different devices and browsers is close to impossible, such tools are lacking quite a lot of valuable conversion data and are biased towards lower-funnel activities.
- Similar to Multi-Touch Attribution, you would need to capture a significant amount of conversions to assess the Incrementality Testing results. It typically takes around 1-30 days, depending on the attribution window of your business. Since ITP limits the life of a cookie down to 1-7 days, audience- and auction-based Incrementality testing would work for you only if your customers convert using the same device or browser within 1-7 days after an ad click.
- You can make manual budget adjustments based on Incrementality Testing output, but you can’t use the data for any automatic ad optimisation.
Cookieless Incrementality Testing Pros and Cons
- Does not depend on cookies as the audience is split based on geo, product, demographics, etc. That means that conversion can be captured even if customers converted through a different device or browser, as long as they are a part of a control or a test group based on the selected criteria.
- Not impacted by ITP limitations and short attribution windows, allowing to account for longer sales cycles.
- Same as for Cookie-based Incrementality Testing solutions, no historical data collected prior to the launch of the experiment is needed.
- Such audience splits are less random compared to audience- or auction-based ones. Test and control groups can be impacted by many additional factors influencing their performance like specific sale periods, local holidays, offline advertising or simply other online advertising channels.
- To compare different channels’ incrementality you need to run several tests each time keeping a control group aside of the tested channel, which limits the ability to attract new users and scale the business.
- It is extremely hard to run such experiments for specific channels in isolation and almost impossible to pinpoint specific campaigns, not speaking about ad groups. That means that you can barely use them for tactical planning and optimisation if you want to go beyond the channel level. Even if you do, it’ll be hard to observe the impact of the optimisation actions.
What about Conversion Modelling?
Conversion Modelling tackles the real value assessment from a different angle. Conversion Modelling uses Machine Learning to assess the impact of each visit and predict visitors probability to convert in the future based on the behaviour data. So instead of waiting for the final conversion to happen within the same cookie, which can be unavailable due to all the cookie limitations we face these days, it analyses the quality of the website traffic in close to real-time and shows the value of the marketing activity based on the visitors probability to convert in the future.
Based on the Conversion Modelling output you can make proper optimisation of all your channels, including your upper-funnel activities, and send more valuable signals to ad platforms for smart bidding.
Ex.: Let’s take the same Facebook example. Your digital marketing director wants to have a fair assessment of your Marketing Mix, they’re using Conversion Modelling on a day-to-day basis. Looking at the results they can see that although Facebook has a very low contribution to overall last non-direct click conversions amount, i.e., 1.2% out of all other channels, it actually attracts lots of users with a high probability to convert, i.e., 16% of users across all channels are actively engaged on your website and will likely convert later thought this or another channel. You can also export these modelled conversions data to Facebook and use it as a conversion objective on the ad set level to make sure Facebook optimises towards all valuable clicks.
What are Conversion Modelling Pros and Cons?
- Conversion Modelling provides you with a holistic cross-channel overview that can be further broken down to campaign, ad group, specific ad and even keyword levels, making it easy to compare different channels and tactics performance. The output can be used to make budget allocation decisions on strategic and tactical levels.
- Modelled conversions can be exported to ad platforms in exactly the same way as normal conversions which makes Conversion Modelling actionable as it’s possible to use modelled conversions for ad campaigns optimisation and smart-bidding strategies.
- Once you have a stable model you can start taking actions based on the results already after 1 day. This makes it work even for the recently launched campaigns.
- Conversion Modelling tackles the cookies and ITP limitations as it predicts probability right after the visit happens without waiting for the real conversion to occur. It makes this approach much more aligned with cross-device and cross-browser customer behaviour as well as long sales cycles.
- You would need at least 3 weeks of historical data so that the ML prediction model can be trained.
- To have a good prediction model you need to have at least 300 online conversions on a monthly basis.
To sum things up,
Incrementality Testing is a good testing mechanism but due to the fact that you would always have to keep a control group totally unexposed to the tested channel or campaign, it significantly limits your ability to scale and can lead to lots of missed opportunities and lost revenue.
It is also much less actionable than Conversions Modelling as it’s not compatible with any automatic ad optimisation and it doesn’t always solve the cookie limitations challenge (depending on the type of a split you’re running). Conversions Modelling helps not only with manual Media Planning and Budget Allocation but also enhances Ads Optimisation and Smart Bidding with more valuable signals without limiting your marketing activities.
To see Conversion Modelling in action, reach out to us and request a trial!
You might also be interested in
Watch the recording: SegmentStream at DigiMarCon about marketing in the cookieless worldLearn more
Cross-device marketing attribution: is there a solution?Learn more
How does ITP in Safari affect performance marketing and what can we do about it?Learn more
Never miss an article
Get the latest articles, event invitations and product updates delivered straight to your inbox.
Thank you! You’ve been signed up for our newsletter.
Get started with SegmentStream
Request a personalised demo with our team!