Chapter 8 of 11 · 8 min read
Every Metric Lies
From The Market for Your Mind by Wrotebook
Monday, ten o'clock, a meeting room on the third floor of a digital publisher in London. The room is a composite, assembled from a great many rooms like it, and so are the people in it and the figures on the screen, but anyone who has worked in media in the past fifteen years will recognise the chairs. There are glass walls, a whiteboard nobody has cleaned since Thursday, eight people with laptops and coffee, and at the end of the table a large screen showing a dashboard. The dashboard does not speak. It is nonetheless the most senior person present.
"Last week," says the head of audience, and clicks. A line appears. It rises on Monday, sags on Tuesday, recovers on Wednesday and falls off a cliff on Friday. "Sessions down eleven per cent week on week. Time on page down. Returning visitors flat. Newsletter opens down two points."
"Friday was a slow news day," says the news editor.
"Friday's always a slow news day. This Friday was slower."
The features editor says, carefully, that Friday was the day they published the investigation into care homes, which took four months. The head of audience clicks again and a table appears, and the investigation is near the bottom of it: nine thousand views, an average of more than four minutes on the page, which is very good, and almost no traffic from social media or search, which is not. At the top of the same table is a short piece about a celebrity's kitchen, written in forty minutes by a junior reporter from three photographs and an estate agent's listing, which has been read nearly four hundred thousand times.
"We tested four headlines," says the head of audience. "'Inside her new kitchen' did nothing. 'You won't believe what she did to her kitchen' doubled it. 'Neighbours furious over' did six times as well. We went with neighbours."
"Were the neighbours furious?"
"One of them posted something. Mildly put out."
The laugh around the table is not a comfortable one. The editor-in-chief asks what, precisely, is being proposed, and the head of audience says that the numbers are the numbers, that sessions are what the advertising team sells, and that if Friday looks like that again the month's revenue will look like that too. The investigation should carry on, she says, though perhaps not on Fridays, and perhaps with a stronger headline. The features editor says it was a strong headline. She says it was an accurate one. They move on to the newsletter, whose best-performing subject line last week was "Quick question", attached to an email that did not contain a question, and which raised the open rate by a satisfying margin and the number of people unsubscribing by a margin described as within tolerance. The video team reports that completion rates are up, because the first three seconds of each clip now show the end of the story, and nobody asks what proportion of viewers stay for the middle. Someone proposes a regular slot for celebrity homes. The editor-in-chief asks whether anyone knows how many of the four hundred thousand people who read about the kitchen came back the next day. The head of audience says that they can find out, and that it is on a different dashboard. The features editor writes something in a notebook and says nothing else.
The trouble in that room lies less in any of the people than in the relationship between the screen and the work, and the relationship has a name. In 1975 the British economist Charles Goodhart, writing about monetary policy, observed that a statistic on which a government relies tends to stop behaving once the government starts trying to control it, because people change what they do in order to move the number. The anthropologist Marilyn Strathern later put the idea in the form most people now quote, which is that a measure which becomes a target ceases to be a good measure. The insight has travelled a long way from central banking. It describes, with uncomfortable accuracy, almost every number in the attention market.
The numbers on the dashboard began life as reasonable proxies. A publisher wants to know whether people are reading and valuing what it publishes. It cannot see inside their heads, so it counts what it can see: visits, minutes, return trips, emails opened. Proxies are perfectly serviceable as long as nobody is trying to move them. They become dangerous the moment pay, promotion and survival depend on them, because then people learn, entirely rationally, to move the proxy instead of the thing it stands for. If the target is sessions, a headline that provokes a click will beat a headline that describes the article, whether or not the click leaves the reader satisfied. If the target is time on page, articles grow longer and slower to reach the point. If the target is opens, subject lines learn to sound personal, urgent or vaguely worrying. If the target is completion, videos front-load their payoff or promise one that never arrives. Each change works, in the sense that the number rises, and each moves the number a little further from what it was supposed to measure.
The pattern is not confined to news. On Spotify a play counts as a stream for royalty purposes only once it has lasted at least thirty seconds, and the people who write and produce songs have noticed. Long instrumental introductions became a commercial liability, and a great many songs began to arrive at their chorus, or at their most arresting sound, almost immediately, so that a listener skimming a playlist would not skip before the counter ticked over. On video platforms, where success came to be measured by watch time and retention, an entire grammar grew up around the measurement: the frantic opening, the promise of something extraordinary later in the video, the cut every few seconds to stop the eye from wandering. Nobody decreed any of it. Each was a reasonable person's reasonable response to a number that decided whether they were paid.
The publisher's own dashboard is less its own than it looks. A good share of the traffic on that Monday screen arrived from search engines and social platforms, and the figures describing it, how many people saw a link, how many clicked, how long the videos were watched, came from those platforms, calculated on their definitions and reported on their schedules. The analytics software that drew the line chart belonged to someone else again. A newsroom that believes it is steering by its own instruments is, much of the time, steering by readings supplied by the companies whose rankings decide where the traffic goes, and when those companies change a definition the newsroom's week can improve or collapse without a single reader having changed their habits. The head of audience is not reading the audience. She is reading the meters other people own, and doing it diligently, because it is the only reading on offer.
Here is the Monday dashboard again, rendered as a receipt, with what each line shows and what it hides:
Line on the screen — What it shows — What it hides
Sessions: 1.2 million, down 11% — Visits counted — Whether anyone was glad they came
Time on page: 52 seconds — How long a tab was open — Whether anybody was reading it
Top story: 380,000 views — The kitchen headline worked — One mildly put-out neighbour, and readers who will believe the next headline a little less
Care homes: 9,000 views — It did not travel — Four minutes of real attention from each reader, and the only story that week anyone might remember next year
Newsletter opens: up — "Quick question" moved the number — Forty thousand people told there was a question, and no question
The third column never appears on the screen. Nobody has hidden it. Nobody knows how to count it.
The industry's numbers have never been quite what they claimed, and the oldest of them turned out to be no exception. Since the days of the Audimeter, American television advertising had run on Nielsen's ratings, a figure whose value lay in its being agreed rather than in its being true. During the pandemic the agreement broke. The networks complained that Nielsen was undercounting their audiences at a time when viewing habits had changed abruptly, and in September 2021 the Media Rating Council, the industry body that accredits audience measurement, suspended Nielsen's accreditation for national television. It was restored on 17 April 2023. In the meantime the networks did something that would have been unthinkable a generation earlier. In January 2023 the big American television companies formed a Joint Industry Committee to certify rival measurement firms, so that advertising could be bought and sold on numbers other than Nielsen's. Paramount went further, letting its Nielsen contract expire at the end of September 2024 in a dispute over cost, trading on a rival's figures for about four months, and signing a new multiyear deal with Nielsen on 3 February 2025.
The episode is the clearest demonstration in the whole history of what a rating actually is. For seventy years the networks had accepted Nielsen's number because everyone else did, and the moment enough of them stopped accepting it, it stopped working as money, whatever its statistical merits. Nothing about the audiences had changed. What had changed was the agreement. A banknote that half the shops refuse is still a perfectly accurate piece of paper, and it is no longer a banknote.
Who pays for a market run on proxies? Three parties, of whom only one appears on the dashboard. The audience pays in trust: every headline that over-promises teaches readers to discount the next, every padded article teaches them to skim, and over the years the habit of distrust spreads from the worst offenders to everyone, including the publishers who never played the game. The workers pay in craft. Journalists, designers, video producers and songwriters find their work reshaped by numbers that measure something smaller than what they are trying to do, and some adapt, and get good at it, and some leave. And sometimes the business itself pays. A publisher that chases sessions with cheap, provocative pieces can grow its traffic for years while quietly losing the loyal readers who would have paid for a subscription, defended its reputation or stayed when the platforms changed their rankings and the traffic went elsewhere. The dashboard reports that the business is healthy until the morning it reports that the business has gone.
Nobody sensible proposes to stop counting, since a business that counts nothing is merely guessing with more confidence. The useful discipline is to remember, every time, what a count stands for, and to choose counts that sit as close as possible to the thing one actually cares about. Some publishers that moved from selling advertising to selling subscriptions found their dashboards changing shape almost by themselves. A subscriber who cancels is a much blunter signal than a session and much harder to game, and the questions in the meeting room became whether readers came back, whether they stayed for years and whether they told their friends. On those terms an investigation read by nine thousand people for four minutes each begins to look less like a failure and more like the reason anybody pays. Even those measures can be chased in the wrong way. They lie less, because they sit closer to what they count.
The Monday dashboard, meanwhile, still has no line for the thing the meeting was really arguing about. If it did, it would sit at the bottom of the screen beneath the sessions and the opens and the completion rates, labelled something like readers who will trust us next time, with a cell beside it that the software leaves blank every week, not because the figure is zero but because there is no way of entering it, and a meeting that has learned, through long practice, to treat an empty cell as a number it does not need to discuss.