The MMM barrier didn’t disappear. It moved

The MMM barrier did not vanish, but merely shifted. Open-source MMM packages, such as Robyn from Meta, have made it possible for anyone to prompt an AI agent, load data, and estimate a model with minimal modelling expertise. However, while the process has become cheaper, the accuracy of the resulting model remains a challenge. A study published by Julian Runge and Koen Pauwels highlights that the actual cost of producing a correct model remains unchanged.

The affordable aspect of the process is the creation of the model itself, but the interpretation of the results remains as difficult as before. This is because the interpretation differences between different packages, despite their similar setup, have not been flattened. As a result, even though the model is produced quickly, it does not provide clear answers to crucial questions, such as whether the estimated ROAS is accurate.

The discrepancy between the observed and estimated lift in campaigns can be as high as three times, leading to inaccurate budget allocation decisions. Furthermore, the report points out that models often fail not due to the modelling process itself, but due to data failures. For instance, channels that the model cannot see, such as AI-generated traffic, may not be accounted for in the model, leading to specification errors.

Similarly, spend lines that do not exist, like founder publishing, can also affect the model's accuracy, as they are not reflected in the dataset and are absorbed into trends. Additionally, input quality plays a significant role in the model's output, as thin competitive and audience context can lead to confidently allocated figures based on outdated market information.

To address these issues, the report suggests running two packages on the same data and comparing their outputs. If the results disagree by an order of magnitude on one channel, it indicates a real problem in the identification of that channel in the data. Moreover, the report emphasizes the need for supervision and ownership of the model's results before they are handed off to the budget team.

This oversight is currently lacking in most teams, as there is no dedicated analytics bench to monitor the model's performance.