A Case Study of Lowering your Cost of Sale with Causiq

Causiq did a controlled experiment together with a large Swedish online brand (with some physical stores) to measure the effect of applying our marketing mix model.

🎯 Goal: lower the cost-of-sale (CoS)
👌 Method: pairwise pick countries into test/control and apply Causiq’s insights.
📈 Results per week, adjusted for the control market:
Week 1️⃣: reduced CoS by 12.5%
Week 2️⃣: reduced CoS by 5.2%
Week 3️⃣: reduced CoS by 18.4%

where

Cost of Sale = marketing expenditure / revenue

➡ Average adjusted CoS reduction: 12% The customer has a marketing spend of 500 kEUR/month, the cost structure of an online brand. The reference price of Causiq is €7500/month (spreadsheet).

If revenue is maximised this increases profits by ~€995 000/year, which provides a return on investment [ROI] of using Causiq of 11.1x.

If costs are minimised this increases profits by ~€612 000/year in more in profits, equalling a ROI of 6.8x.

How does Causiq work?

Causiq is on a mission to empower marketers with the best kind of end-to-end marketing analytics, optimisation and budgeting. We offer a real time marketing mix model that constantly learns from marketing spend or impressions, and predicts conversions or revenue. Our model works per-country, per-channel and per-funnel-level. For example, this means that you can see how effective your branding marketing is in Germany on Google Ads.

For the purposes of understanding marketing effectiveness, we don’t track individuals, giving people a much-needed respite from the monetisation of personal data. Instead, we track populations and use machine learning/artificial intelligence to understand channel effectiveness. Another way of seeing it, is that once we can attribute revenue to channels we can optimise those same channels, either by:

• moving money between countries (markets), or
• reducing budget in the marketing by reducing spend in channels that have a bad return on ad spend (ROAS), or
• optimising the media mix in the market by reducing some channel spend and increasing others, keeping the total marketing budget in that market constant.

It takes about two hours to onboard onto Causiq; one hour for connecting the channels and another hour to go through the model outputs once all data has been synchronised.

Our daily marketing mix model models a number of variables, including:

• channel coefficients (that assume decreasing marginal returns)
• funnel-level coefficients (e.g. what’s the ROAS on branding in general?)
• cross-channel effects (e.g. what’s the effect of brand marketing on TV on other channels than TV?)
• time-dependent effects: day of week, seasonality
• baseline, brand strength and holidays
• product discounts, etc…

On top of this, we’ve built an intuitive user interface, so that you don’t have to be scientist to incorporate our AI into your daily workflow.

Background

To determine what effect size can be expected when applying Causiq to markets that have already been optimised according to the best knowledge of an expert marketer. To see whether a significant effect size can discerned, and finally, to understand if a similar effect could be induced just by reducing marketing spend, because there would be "purchase momentum".

Hypothesis

If we apply the recommendations of Causiq, we can improve CoS by at least 5%.

Previous to this study, we’ve spent about 20 months constructing and iterating on a multi-variate auto-regressive timeseries model, trying out different features, testing 15 different machine learning models and performed backtesting for multiple customer datasets, week by week, 2-3 years back. Our model differs from what you commonly see, because it:

• optimises daily, not weekly
• re-trains multiple times per day as data becomes available
• supports manual offline channels and dummy/mechanism variables such as discounts
• re-attributes brand search back to top/mid/low funnel marketing

By validating the model extensively across datasets, we were confident our customer would see an improvement when we applied the model with a real budget. The reasoning goes, that if our model accurately models the dependent variable: aggregated revenue (or number of conversions), that same model can also be used to accurately predict how changes in spend in a country × channel × funnel-level affects the revenue.

Method

We decided to experiment three weeks, with a test market and a control market.

Because each ad-platform/channel optimises itself, it’s nearly impossible to force a certain budget; instead a target budget is selected for the channel × funnel-level.

During these weeks, no particular new campaigns were added or removed, there were no new discounts and these weeks did not coincide with holidays.

The model used in this case study is identical to the model we apply to every customer who onboards onto our platform, meaning it has not been configured specifically for this experiment.

This case study was done on channels: paid social and newsletters and with discounts uploaded via our user interface. This customer also has always-on brand-search marketing (i.e. searching for the brand-name on Google or Microsoft Bing will trigger advertising for that brand); this is back-attributed to other marketing channels to the greatest extent possible given the evidence in the data.

The test market budgeting for the upcoming week was performed in Causiq's scenario modeller, together with the customer. The previous week's budgeting was analysed and changes implemented based on each channel × funnel-level ROAS as presented and modelled by the system.

The scenario modeller, the channel-view, and the channel × funnel-level view are all built with explainable AI (XAI) in mind, i.e. Causiq visualises both the actual and predicted revenue, the decreasing marginal utility and the saturation in this channel × funnel-level, as well as error rates for the channel × funnel-level in question.

All experiments were performed in October 2022.

Week 1: reduce both markets’ budget

The first week, we decided to reduce the budget in the test and the control market. In the test market, we reduced the channel × funnel-level budget according to what Causiq recommended. In the control market we used the customer’s existing business intelligence/analytics to guide which channel × funnel-level to reduce the budget in.

We decided to configure the channels to reduce the budget by 12.5% in both markets.

This was an important experiment, because it is a counter-argument to the hypothesis “if I just reduce budget, customers will still purchase approximately as much, in the short run”. However, since we reduce the budget both markets, if there’s signal in Causiq’s model outputs, the test market’s cost of sale ought to reduce more.

Control market: 🇫🇮 Finland, regular budget reduction

Test market: 🇸🇪 Sweden, Causiq-informed budget reduction

Week 2: keep both markets’ budgets constant

The second week, we decided to reallocate budget from one channel × funnel-level, only in the test market, while keeping the control constant.

Control market: 🇫🇮 Finland, no change performed

Test market: 🇸🇪Sweden, directed budget reallocation from Meta low-funnel to Meta top-funnel

Week 3: keep both markets' budgets constant

The third week, we decided to reallocate budget from one channel × funnel-level, only in the test market, while keeping the control constant.

Control market: 🇩🇰 Denmark, no change performed

Test market: 🇳🇴 Norway, directed budget reallocation from Snapchat low-funnel to Meta top-funnel and increase budget by 10%

Results

Our aim, each week was to reduce the cost of sale in the test market with a "neutral" control market. As such, a reduction in the cost of sale in a market is good thing, as it means better marketing effectiveness.

Week 1: reduce both markets’ budget

This week a 12.5% budget decrease was configured for both test and control channel × funnel-level for each market. The actual budget decrease across the whole market did not sum to 12.5%, due to other channel × funnel-level pairs utilising more budget.

Control market: 🇫🇮 Finland. Actual budget reduction 9%. CoS decreased by 5.4%.

Test market: 🇸🇪 Sweden. Actual budget reduction 7%. CoS decreased by 17.9%.

control - test = effect size = -12.5%

Week 2: keep both markets’ budgets constant

This week, budget was kept constant in the control market and reallocated between channel × funnel-levels in the test market.

Control market: 🇫🇮 Finland. Actual budget identical. CoS increased by 0.4%.

Test market: 🇸🇪 Sweden. Actual budget reduction 3%. CoS decreased by 4.8%.

control - test = effect size = -5.2%

Week 3: keep both markets' budgets constant

This week, we picked new countries for test and control, to ensure that the results were not only due to peculiarities in Finland/Sweden. Budget was kept constant in the control market and reallocated between channel × funnel-levels in the test market.

Control market: 🇩🇰 Denmark. Actual budget reduction 4%. CoS decreased by 3%.

Test market: 🇳🇴 Norway. Actual budget increase of 10.4%. CoS decreased by 21.8%.

control - test = effect size = -18.4%

Results summary

Diagram 1: Cost of Sale, one series per week for control, test, and effect size

Discussion

In this case study, we've shown how Causiq typically can be used to improve the cost of sale for an online brand.

During the first week a reduction of marketing spend led to a reduction in cost of sale across both test and control, but with more than 3x as large effect in the test market than in the control market.

During the second week, the reallocation of budget for the test market led to a large reduction in the cost of sale, while the control market didn't change.

During the third week, the reallocation of budget in the test market seems to have coincided with in-channel variance in both test and contorl. Even so, an actual improvement of 21.8% in the test market: a controlled effect size of 18.4% is a very large effect size.

Conclusion

The controlled market experiments were very successful and fully met both our own and the customer's expectations.

Would you like to learn more about Causiq? Ping us at hello@causiq.com!