Weak Signals of Success: The Lesser-Known Metrics That Determine an Article’s Worth

By Chris Sutcliffe May 15, 2024

By: Chris Sutcliffe

For subscription-based publications, the primary role of an article is to convert fly-by-night audiences to paid-up members—but even articles that don’t directly convert audiences contribute to that success.

Identifying the articles that convert anonymous readers to paying members was paramount for the staff at Argentinian newspaper Clarín—as it is for all subscription-based titles.

Media professor Ismael Nafria recently wrote for Revista de Innovación en Periodismo about how the staff at Clarín went about establishing which of its content was most likely to convert users. They determined there were four key criteria an article required in order to contribute to that goal, which they helpfully outlined in an internal memo entitled ‘Decisive Notes’.

These ‘notes’ or criteria are exclusivity, depth and quality, providing a great digital experience, and utility for the user. They will be familiar to any news organization whose success is predicated on converting users—they are practically the north stars that newspapers aim for with the majority of their ‘hero’ content. 

However, given the nature of digital publishing, it is extremely improbable that the majority of a publisher’s content will fit all those criteria. Live sports reporting, for instance, is only ‘exclusive’ in that any given newspaper will report with slightly different language, while it is unclear what would make an opinion piece a ‘great digital experience’. As such, there are many articles that do not contribute directly to the mission of converting a user. 

In fact, a study from Poool, which examined 75 digital publishers, found that, while there is a correlation between the amount of “premium” content consumed by a reader and the likelihood that they will actually convert, that correlation was only true up to 40% of “premium” content. In other words, fully 60% of the articles to which users were exposed were not considered likely to convert users directly.

However, as Nafria makes clear, while the user’s decision to subscribe is frequently “impulsive,” the overall “convincing process can be long.” As such, the overall body of a publication’s output contributes to the mission.

Weak signals of success

Measuring the success of those ‘hero’ articles is easy: the key metric is how many users were converted directly from that page. But how do publications go about measuring the contribution of non-hero articles to that mission? What are the ‘weak’ signals of success that, in totality, end up making their production worth the while of publications’ journalists?

Harry Slater is the engagement editor for Dazed Media, a global publisher behind lifestyle titles including Dazed, Hunger, and AnOther. He explains that the team takes a wider view of what makes an article ‘successful’: “Like a lot of publishers these days, we’re less focused on pure page views and reach for reach’s sake. Instead, most conversations about our audience and journalism are grounded in a couple of key engagement metrics.”

He says that the team at Dazed has made Chartbeat’s “quality views” metric one of the key ways by which they understand how a piece has performed. Unlike a page view, a quality view is a view that’s active for at least 15 seconds: “By cutting out bounce and sugar rush traffic, we get a clearer understanding of the pieces that resonate with our audience and can make better-informed decisions about what we cover.”

By eschewing scale for its own sake, the ‘weak’ signal of time spent is therefore used by the team to work out which type of content is resonating with audiences and keeping them on site for longer. 

As the team at Clarín also found, longer time spent with a publication—even passively consuming its content—increases the propensity to sign up for a subscription in the long-term.

That is backed up by research from paywall and membership tech provider Piano. Its SVP of media strategy Michel Silberman told me: “The biggest predictor we’ve seen so far is active days. The more active you are actually both before you become a paying member or subscriber and after—the more active you are, the more likely you are to stay.”

So the ‘weak’ signal that nevertheless suggests an article or series is contributing to the bottom line is that it attracts readers back time and again. Slater says: “It definitely takes time. Your audience needs time to discover a series and realize it’s published every day, week, month, however often, and then make reading it a habit, like the Guardian’s [weekly] Blind Date column, for example.”

That’s the nature of audience development. It takes time and works best when multiple teams are involved—it’s not just the responsibility of audience specialists; it needs developers, product managers, designs, editors—every one – to contribute.

Aggregated metrics

For other publications, ‘weak’ signals like time spent and recurring visits are used in the creation of an RFV model (Recency, Frequency, Value), which informs the overall engagement strategy. The team at Der Spiegel, acknowledging the value of ‘weak’ signals of loyalty, added a new metric to the analysis—the variety of content accessed: “A further fundamental shift from the classic RFV model is the way volume is measured. Instead of focusing solely on clicks on articles, Der Spiegel considers usage time across the entire site. 

“This means that user engagement is evaluated by how deeply they immerse themselves in editorial content, spanning various formats such as text, video, stories, and audio consumption.” 

One thing that publications need to bear in mind is that while conversions are a measure of success in and of itself, ‘weak’ signals like time spent and content variety are best used in concert with one another: on their own, they do not add up to a complete picture of the worth of a piece of content. Nicholas Thompson, CEO of The Atlantic, says:

My favorite perverse incentive of the time spent metric is that actually if you’re fully optimizing, to maximize the time someone spends on a story, you would edit the piece so that it’s crystal clear until the very end—and then you would make it extremely confusing. 

[However] time spent is a really good one. When I was at the New Yorker and Condé Nast, I pushed really hard to make it a metric that was more integral to the way our performance was assessed.”

‘Weak’ signals, then, are only weak in isolation. They should be used together: the value of an article, piece of multimedia content, or series of articles becomes readily apparent in totality. So even if an article does not contain any ‘decisive notes’ or the shock factor that leads to a user converting, it is often a cobble in the road that ultimately brings a fly-by-night user into the paid ecosystem of a newspaper.