Psychometrics for Human Values: a new way to measure success and value

We embarked on an empirical investigation to understand what people value in the digital age. In doing so, we are able to understand more about what is important to our audience as well as begin to measure how well we are delivering services and products to enable people to fulfil those important values.

We identified a set of 14 human values that are underpinned by deeper psychological needs, as shown in existing psychological literature such as everything from the need for social connection to the need to be safe and well, described in dozens of different research papers within the white paper.

Copy taken from BBC R&D blog – Copyright all rights reserved

Kristie To, an undergraduate placement student from the University of Bath, has been helping the Digital Wellbeing team to conduct research and joins Lianne on this blogpost.

https://www.bbc.co.uk/rd/blog/2019-11-human-values-digital-wellbeing-audience-behaviour

How should we quantify the extent to which a service, a digital product, even an individual TV or radio programme, contributes to the fulfilment of a person’s value system? We believe that psychometrics – a way of specifically measuring or calculating performance – is the most important and direct way of doing this. Psychometric tests are used by researchers, clinicians and sometimes the general public, to record, track and compare the behavioural and emotional reactions associated with experiences.

The psychometric directory that we are developing aims to measure how well a BBC product or service is helping audience members fulfil their values. For example, to measure the value of growing myself, we assess the different underlying psychological needs of why people value personal growth, such as learning new things, trying things differently, challenging perceptions and feeling as though one is developing skills and themselves.

Using our fourteen empirically derived, scientifically evidenced human values, we are embarking on a process of transforming these values into a robust psychometric measurement tool, or metric.

A woman in a diner, wearing headphones, sipping on a milkshake.

How are we getting there?

To develop the psychometric directory, we established certain standards for the questions to meet:

  • The language of our metric will be taken from the way that people described the 14 human values.
  • Several questions per value will be included to fully explore the underlying psychological needs that relate to each value.
  • The questions will respect the underlying theory that formed each of the human values and will either be closely aligned with previous academic research or exist within validated, reliable and scientifically grounded metrics that exist already.

What have we done so far?

As a pilot exploration to test the language of the psychometric, we asked 468 people to evaluate how much their use of the BBC helped them achieve what they value in life, asking questions such as “To what extent does the BBC… enable you to learn new information?” We also asked whether any words in our survey were confusing or unclear, avoiding ambiguous and vague terms like ‘good’, ‘easy’ or ‘happy.’

It is crucial to use the correct language in our psychometric as it is important that people interpret the words we use in the way we intend for them to be understood. We want the language and expressions of values to meaningfully capture people’s feelings, as well as accurately reflect their underlying psychological needs in a way that can be consistently used and understood by everyone. The latter is important to ensure that the results of the survey are comparable.

A man at home, holding his dog.

The feedback from the survey showed us what participants thought was unclear and highlighted that although asking questions about personal values was understandable, people found it confusing in the context of the BBC. Certain words like “togetherness” or phrases like “I feel as though I matter” were some of the problematic terms that caused confusion. This was valuable insight, as it identified areas that we should invest time to review and refine. We conducted further desk research and in-depth focus group studies with participants to specifically address words that were problematic, misunderstood or not relevant.

Using focus groups enabled us to have a back and forth discussion with participants about the language we used. Unlike surveys, focus groups allow us to ask participants to elaborate on their responses in real-time. We have conducted six focus groups with people from varying professions and backgrounds, to gain a diverse range of views on what the values mean to them.

We see this as a balancing act between combining original empirical data, scientific items and participant derived terms to get to where we are. It’s intended that what participants are asked would be adapted case by case to apply to the relevant BBC product or service.

What’s next?

As we wrap up a series of focus groups, we are adapting our survey to several different situations to discover how it performs when applied to specific products or services.  By trialling this across various separate case studies, we can test for reliability and validity. This will help us embark on validation studies, to enable the psychometric measurement to become more accurate.

A woman sat on her sofa with her dog, reading a book.