Measurement Strategy in the Digital Era

MEASUREMENT STRATEGY IN THE DIGITAL ERA:A GREEN PAPERMeasurement Strategy in the Digital Era: a green paper produced by the IPA and Gain TheoryContents

Appendix:

Thanks for reading. Want to get involved? Have Your Say

MEASUREMENT STRATEGY IN THE DIGITAL ERA : A GREEN PAPERThis Green Paper and accompanying attachments have been produced as a provocative discussion piece on measurement strategy as part of the 2017 EffWeek Thought Leadership Conference organised by the IPA. Co-produced with Gain Theory, the IPA and a selection of leading industry figures, the green paper and the topic was launched and debated at an interactive think tank at EffWeek and is set to help develop the agenda for a bigger cross-industry EffWorks research project in 2018. For more information on this paper's contributors please visit: www.ipa.org.uk www.gaintheory.com

IntroductionThe Green Paper has been written by Gain Theory with input from a cross-industry working group including:

Alan Bloodworth & Jon Webb - Gain Theory

Kathy Dykeman - Facebook

Jonny Protheroe - Google

Matt Hill - Thinkbox

Janet Hull & Joyce Kelso - IPA

Chris Hutchings - BBC (representing EffWeek Client Advisory Board)

Vasileios Kourakis - L’Oreal (representing ISBA)

We have also consulted with a wider circle of clients, agencies, content owners and industry bodies: including Dixons, Direct Line, Mondelez, Macmillan and Asda.ScopeWe have asked a number of leading marketeers to define what they meant by marketing effectiveness. Jan Gooding from Aviva helps set the challenge:

"Did it work?
Every organisation makes choices about how it invests money. For Marketing to get its share it needs to prove value."

Jan Gooding, Global Inclusion Director, Aviva

Our working definitions, for the purposes of this paper are:MarketingWe are focusing on the narrow definition of marketing, mainly surrounding direct communications with current and potential consumers. This covers the paid (P), owned (O) and earned (E) media space.Wider considerations are covered where P, O, E communications has a clear impact on some other element of the marketing mix.We are aware that, in time, we have the opportunity to stretch our discussion much wider, to encompass the 7Ps of classical marketing theory.EffectivenessThere are many ways to demonstrate effectiveness. The debate here centres around marketing activity where the primary aim is either to build for the long term or to see sharp increases now.This was summarised nicely by Billy Ryan from Direct Line in the quote:

"I’m often asked how our media budget is split between digital and non-digital. This misses the point – they should really be asking how much is split between demand generation and demand activation."

Billy Ryan, Marketing Effectiveness Manager, Direct Line

In the course of our discussions we have also touched upon the following themes of relevance to the wider topic:

Effectiveness versus efficiency

Return on Investment (ROI)

Manifold effects

Short-term and long-term performance

Correlation and causation

Incrementality

Comparability

Key brand health measures

Mapping tools and techniques

What is new and different?

Pain points

METRICS THAT MATTEROne of the key challenges we have identified in addressing marketing measurement is metrics overload: the increase in the number of channels available to marketers has led to an increase in the number of metrics used to report on activity. As one marketer commented in recent independent CMO research commissioned by Gain Theory:

"Too many data points = analysis paralysis"

CMO, Retail

Newer metrics are often not comparable with old ones, measured in silos and insufficiently linked to business outcomes. There can be a mismatch between real-time metrics and reporting processes (marketing teams’ weekly reports etc.).In the same research, one senior marketer said ‘We can’t agree on which few metrics to measure or focus on’. Another one said: ‘We need one source of truth that will drive our metrics and resulting insights’.So, what are the key metrics that marketing leaders could and should focus on?

Our diagnostics or KPIs are based on a number of sources:

Research conducted on behalf of the IPA and CIMA by Fran Cassidy in the new IPA EffWorks publication ‘Culture First’

IPA Databank evidence contained within Media in Focus, the latest IPA EffWorks publication in the series authored by Les Binet and Peter Field

Feedback from the various contributors to this report

However, it is worth making five points at the outset that resonated with virtually all contributors.

There is not a single metric that on its own captures all relevant factors, though some come close.

A metric without context has very little use. For example, an increase in like-for-like sales on the back of a competitor exiting the market is not really indicative of the success of your marketing, at least not in any straightforward way.

Unless there is buy-in from all stakeholder groups, metrics are meaningless. Integration of metrics is key to move beyond a ‘reporting culture’ to a ‘learning culture’.

Broadly speaking, in current practice, metrics fall into three main buckets:

Financial performance
Brand health
Customer service/engagement

The volume of data available varies by market sector. Service businesses tend to have a plethora of customer experience data from their many touchpoints, particularly those in retail, and have many more complex primary lever analyses.

Setting prioritiesBinet and Field’s most recent research of the IPA Databank ranks the shared characteristics of campaigns that successfully generated profitable growth. Their table is shown below:

Source: Media in Focus: Les Binet and Peter Field. (IPA EffWorks 2017)

The sorts of questions that came up in our working meetings were:

What should guide the metrics we look at?

Should they vary by department or should we all use

the same ones?

How often should they be reviewed?

Who agrees them?

Who owns them?

MEASUREMENT STRATEGIES: WHAT DOES GOOD LOOK LIKE?Listing the key metrics to focus on is the relatively easy part. Much harder is implementing a measurement strategy that:

Provides a framework within which KPIs make sense

Results in the framework not changing every time a campaign changes

Is consistent in approach i.e. each KPI is defined and measured in the same way

Is consistent in activation e.g. we know that staff move around frequently, so we should plan with this in mind

MMM and Digital Attribution are among the most popular methodologies deployed, but both have shortcomings.MMM is a top down approach that is great for measuring channel level impacts and maybe one or two levels down but struggles to go any further. For example, on TV, it is possible to look at the differential impact by network, by daypart and to split the impact between demand generating and demand gathering creatives. But for advertisers running more than two or three creatives it struggles to differentiate. Moreover, the focus of interest is often sales volume or value which means upper funnel activity that does not directly affect sales will be discounted.Digital Attribution by way of contrast, is a bottom up approach and is great at providing detailed and granular results within the realms of digital touchpoints. But it struggles to incorporate both off-line media and events in the “real world” – the weather, for example. Or the actions of competitors. Other issues surround cross-device usage and of-course the well-known debate around viewability may skew results. While the technique is great at determining relative effectiveness of different activities, it has little to say on the absolute impact.Classically the divide between the two techniques can be described below.

Econometric analysis has a good record when it comes to identifying the overall impact of an activity on a dependent variable or KPI.

But Digital Attribution is required to dig further – yes, Display had this impact, as measured through econometrics, but this combination of publisher, tactic, creative and positioning was most impactful, as measured through attribution.

Often, MMM would provide a boundary for the absolute impact on incremental sales of an activity and then Digital Attribution would be used to fractionally allocate between tactics. This is fine as far as it goes, but MMM analytics invariably lags behind Digital Attribution. Assuming the results of an MMM study from 3 months ago are still holding good can be a stretch. As a result, the industry has been moving towards Multi-Touch Attribution as well as towards more sophisticated unified measurement approaches.Other models also have important contributions to make. For example, other methodologies allow base-line sales to evolve, reflecting changing tastes and preferences – a key development. Path-to-purchase approaches allow us to explicitly evaluate the impact of upper funnel activity. What we need is a system that brings all these approaches together into a single framework – combining the holistic nature of econometric analysis with the granularity of digital attribution.Unified Measurement – The Future

"Today’s increasingly complex media environment means once reliable marketing performance measurement techniques, such as marketing mix and attribution models, fail to properly credit marketing tactics with a customer action"

Forrester

Coined by Forrester in ‘The Marketing Measurement And Insights Playbook For 2017’, the label Unified Marketing Impact Analysis (UMIA) describes perfectly what many marketers are striving to achieve:

Evaluate all marketing touchpoints, upper/ lower funnel and on and off-line, in as much granular detail as required

While taking account of all other factors that may have an impact on sales, from competitor activity to the weather. Quickly enough for action to be taken in-campaign.

Based on the correspondents who have contributed to this report, we expect these developments to play a leading role in the 2018 research.

So, what would be the 6-step plan towards a perfect marketing effectiveness measurement strategy? Our next stage of market research is designed to provide best practice cases to illustrate this. Based on current knowledge the straw man which we will be using as a framework for discussion is as follows:

From green paper to whiteGain Theory is a global marketing foresight consultancy that helps brands make faster, smarter business decisions via data, predictive analytics, technology and consumer-insight capabilities. The consultancy leverages WPP’s intellectual capital in media and marketing helping brands move away from insight into actionable foresight in order to positively impact the bottom line.As a partner sponsor of the IPA EffWorks initiative, Gain Theory has agreed to lead on a 2018 research programme which will consolidate on, and take forward, this green paper, with best practice case examples. We will be conducting our research between November 2017 and June 2018 across the UK, the USA and China.