Is the tech industry measuring the right things?
Within BBC R&D, the Human Values team has been thinking about how we define success in the digital era in ways that complement measures of audience numbers or time spent online. We have created a range of measures that look at the impact of digital services on people, assessing how they might support their personal values and achieve goals in their lives. We believe that these values-based measures have the potential to add to existing audience metrics to provide a richer understanding of digital value, especially as online services are becoming increasingly personalised.
At present, most audience measurements for online services are based on measures of attention, typically encompassing aspects such as number of visitors, time spent, and return visits, among others. They have been used both within and between companies to communicate and celebrate worthiness, value, and success, and are generally tied to advertising revenue. However, they have been brought over from traditional broadcasting, where they are used to show how many people have engaged with a piece of TV or audio content.
In the broadcast world, these numbers can be used to sell advertising slots, define concepts like peak viewing times, justify programme investment, and translate engagement into sales. Now, these advertisers are operating in the digital space, and broadcast-oriented metrics have followed them, with adverts supporting ‘free’ products and data about users treated as currency in a world where the online media landscape trades in the attention of ‘users’.
Measuring attention as a data point is incredibly easy for companies, and with proficiency in data science growing the potential to capture more data is becoming easier and cheaper. This then feeds into company reward schemes and product Key Performance Indicators (KPIs), resulting in a focus on increasing the things that are measurable, and product developers end up focused on the goal of increasing user attention.
Doing this might seem an obvious way to increase the impact of digital services, with the underlying assumption that increasing the number and time people spend with a product equals more value, growth and success. However, this is not the case: a range of other factors are at play that go beyond visitor numbers and time spent. The success of a product depends on the intention behind it, the desired impact, and even whether it fulfils some deeper need in people. Perhaps an online service helped someone achieve a goal, learn something new, or feel fulfilled in their life. Or perhaps time spent online meant they missed out on something happening in the ‘real’ (as opposed to online) world.
When research suggests that the average adult spends 6 hours on their phone in an average day it begs the question about how much attention a person can realistically expend online without it eating into other aspects of their lives. We are seeing more and more research showing some of the harmful effects that excessive screen time has on wellbeing, mental health and sleep, so any model that assumes that more time spent online is good for people should be challenged.
This is why we are reflecting on what success and value truly mean for digital services, not just for the BBC, but for the whole online world.
Our starting point is the acknowledgment that attention metrics are proxies for something harder to capture: value and impact. While they can be indicative, and a lot of work has gone into refining them, they can never be an entirely accurate way to determine how people truly engage with content or a product because they do not adequately capture the full picture.
This does not mean that these measures should not be used, but that they are not enough, as all they really tell us is how many people were exposed to something for a length of time. Even the advertising industry acknowledges that the number of eyeballs and amount of time a person expends on a given product does not translate to sales. So we should not expect it to be a useful proxy for public value or other factors that we consider important at the BBC.
We also need to be aware of how the use of attention metrics can change the culture of companies; as the demand for higher figures leads designers and developers to adopt design patterns to grab and keep attention, often turning to persuasion technology and psychological techniques from the gambling industry to ‘’trigger and hook’’ people through what are termed ‘dark patterns’. For e.g., making it hard for people to opt-out of services, using infinite scroll, or manipulating or misdirecting attention are some of the most widespread dark patterns.
According to Goodhart’s law ”when a measure becomes a target, it ceases to be a good measure’’, largely because people’s behaviour changes when they are trying to meet a target. If you measure the number of total pins made, you’ll get lots of small pins; if you measure the weight of pins manufactured, you’ll get fewer, bigger pins. Neither is wrong, but the decisions made are created to serve the specific measure.
Whilst this might seem a slight and subtle shift in operation, bigger problems can arise. If we look at mass consumerism as an example, over the twentieth-century marketing was aimed at increasing the consumption of goods and did not account for the negative consequences produced on the environment. Unbeknownst to us at the time, we were in danger of optimising for the wrong outcomes and now with that knowledge in hand companies have introduced measures alongside consumption, to account for social corporate responsibility and environmental impact.
The pitfalls of measuring attention operate in the same way; practitioners become ultra focused on creating attention grabbing and addictive products with the end goal of increasing the number of users and the time they spend with the product. These metrics can cause businesses to lose sight of what is truly important, factors such as quality of experience and impact on people and societies.
Are digital services measuring the right thing?
As we move away from an era marked by industrial, homogenised and non-sustainable practice, we see sustainable approaches to business coming into play across many sectors. For example, the triple bottom line aims to measure outcomes for people and the planet in addition to profit. Recognising that a focus on consumption or profit alone misses the bigger picture, and adding in measures that take into account a wider view of success offers a broader perspective on meeting people’s needs. Similarly, measuring the success of a film has moved away from relying purely on tickets sold to incorporate measures that capture quality, impact and enjoyment.
What is the bigger picture in terms of digital services? Thinking about the triple bottom line we believe that factors relating to the quality of experience, personal value and impact need to be taken into account, but for that to happen we need ways to think about and measure these elements. That is exactly what our work on human values is doing; giving providers of digital services new ways to measure value, to complement their current practice.