One honest number

Every company says it is data driven. Fewer can say what a campaign brought in last quarter, in francs, and fewer still have changed a decision because a number told them to. The gap between collecting data and deciding with it is where most of the value sits, and where most of it is lost.

This piece looks at what the research says about companies that decide on data, what happens when marketing is measured all the way to the money, and the traps that turn numbers into decoration.

What the research found

The best known evidence comes from MIT. In 2011 Erik Brynjolfsson, Lorin Hitt and Heekyung Kim studied 179 large listed companies and found that those which emphasised decisions based on data and analytics had output and productivity 5 to 6 percent higher than their other investments and their use of IT would predict. The link also showed in asset utilisation, return on equity and market value, and the authors found no sign that it simply ran the other way, with successful firms buying more analytics.

Later work with US Census Bureau data widened the picture. Brynjolfsson and Kristina McElheran found that data driven decision making in American manufacturing plants nearly tripled between 2005 and 2010, from 11 to 30 percent. In a sample of more than 18'000 plants, later adopters showed value added about 3 percent higher than comparable plants that had not adopted, and performance improved after adoption, not before. Smaller firms adopted later, but their link to performance was stronger.

The size of the effect matters less than its shape. The researchers found adoption went together with information technology and educated workers. Data pays when it changes how decisions are made, and when the people and systems around it change too.

Measured to the money

Marketing is where the gap is widest. In a Gartner survey of 377 users of marketing analytics in May and June 2022, analytics influenced just over half of marketing decisions, 53 percent. A third of the respondents said decision makers cherry pick data to tell the story they had already decided on.

Part of the reason is structural. The numbers live in many tools: analytics in one, the CRM in another, the newsletter, the social channels and the ads each in their own, and the finances somewhere else again. Each tool reports success in its own currency, clicks here, opens there, leads somewhere else. None of them reports the number a company runs on: how much money came back for the money that went out.

Measuring to the money means following a lead from the first visit through the form, the call and the offer to the paid invoice. Once that chain exists, the conversation changes. A channel that brings many cheap leads and no customers looks different. So does a slow channel that brings few leads and most of the revenue.

When one number turns a company

Sometimes a single honest measurement is enough to change a direction. In a large field experiment published in 2014, the economists Tom Blake, Chris Nosko and Steven Tadelis switched off eBay’s paid search ads in parts of the United States and compared sales. For searches on eBay’s own name, almost all of the paid traffic simply arrived through free results instead, and the ads had no measurable short term benefit. For other keywords, the ads did help new and occasional buyers, but most of the money went on frequent buyers who would have bought anyway, so the average return was negative. The real returns were a fraction of what the usual estimates had shown.

Yahoo learned a similar lesson, described by Ron Kohavi and Stefan Thomke in the Harvard Business Review. An observational study estimated that display ads for a brand increased searches for it by 871 to 1'198 percent. A controlled experiment measured 5.4 percent.

Airbnb changed course at company scale. In 2020, the year the pandemic stopped travel, it cut its brand and performance marketing from 1.14 billion dollars to 479 million, a drop of 58 percent. In its first annual report as a listed company it set out a strategy to grow brand marketing, attract more guests through direct and unpaid channels, and spend less on performance marketing than in 2019. In May 2021 its chief executive, Brian Chesky, told analysts that 90 percent of its traffic was unpaid or direct.

The traps

Numbers can mislead as well as guide. Vanity metrics are the first trap. Followers, impressions, page views and open rates are easy to grow and easy to report, and they say little about revenue. They belong on a dashboard only when they are linked to something that pays.

Missing data is the second. Visitors decline cookies, forms are not tagged, calls never reach the CRM, and the finance system talks to nothing. A report that quietly fills the gaps with estimates looks complete and is not. A missing number should be shown as missing.

Correlation dressed as cause is the third. Kohavi and Thomke describe teams at Microsoft Office concluding that advanced features reduced churn, when heavy users simply try more features and also leave less often. By the same logic, users who saw error messages also left less often. Only a controlled test separates cause from coincidence.

The quietest trap is deciding nothing. A dashboard nobody acts on is a cost.

What stays with people

Data does not decide. People do, and they should. The numbers say what happened and, with a good test, what caused it. They do not say what a company wants to be, which customers it would rather not have, or when to take a risk the history cannot support. That judgement, and the responsibility for it, stays with the people who run the company. Honest numbers give that judgement better material. They do not replace it.

What a company can do

Start with one question that costs money when you get it wrong, for example which channel brings paying customers. Connect the few sources that answer it, from the first visit to the invoice, and mark every gap instead of filling it. Then decide one thing on that number, and test the next.

This is the job of North. It brings analytics, forms, CRM, email, social media, ads and the real finances into one view, follows every funnel to the money, and shows every figure as measured or as missing. An assistant answers questions with the source of each number, and a person decides. Where a decision needs a cause rather than a correlation, Proof runs the test.