We interviewed Benjamin Fields, Lead Data Scientist with BBC News, about the growing role and impact of content recommendation systems.
EBU: Why should public service media (PSM) use recommendation algorithms?
Benjamin Fields (BF): There’s always this kind of struggle between serving the public interest but also being relevant. This is where I see the need for automatic decision-making processes to come to PSM, because ultimately they represent some competitive advantage and competitive potential. If we don’t use these tools, we’re going to eventually lose the battle for eyeballs. This isn’t to say we should all become Netflix – I don’t think we’d win that battle anyway – but we do have to acknowledge that getting people quickly and easily to content that interests them is something we need to do as well as some of our commercial competition does. If we don’t, we’re going to slowly become irrelevant and lose our public remit, and that’s to society’s loss.
EBU: Why should PSM have their own recommendation systems rather than using commercial solutions?
BF: The incentives in commercial recommendation systems tend to be sufficiently different that they don’t really serve the needs, at least as I see them, of PSM. They tend to build for something that looks and feels like an e-commerce site and because of that they’re very focused on short-term, either click-through or purchase-rate, optimization, with no real facility to do anything broader or more subtle. PSM is about more than just that: it’s about making sure that people have a shared understanding of the world around them, that people have a common set of facts. You can imagine recommenders fulfilling that remit through measuring and optimizing things like diversity, novelty, social cohesion.
Serendipity, novelty, diversity are classic, fairly well-understood metrics that sit in that space, as opposed to just optimizing for click-through rates. There’s also a new block of work about this idea of self-actualization: not just serving content that meets someone’s current interests but serving media that helps people work towards the person they want to be. You might actually lose a few clicks compared to the best recommender, but you might end up with a better service in the medium-to-long term and a better society that you’re serving. PSM are uniquely positioned to do that and to work in that fashion with recommender systems.
EBU: How should organizations choose which values and metrics to use?
BF: This is difficult and probably the most important question, more than a particular algorithm. There’s a big role for traditional news-gatherers, editors and producers to feed into that process of understanding what will best serve your audience and how that can be best measured and built into the way that recommendation systems behave. That should happen in dialogue between data scientists, technologists and editorial teams, and the more traditional decision-makers.
EBU: How do you see this field evolving?
BF: The common understanding of the world around us and our reliance on truth is a really tricky future problem. There’s this emerging technology that is popularly called “deep fakes”: it won’t take much resource to be able to create pretty good-looking fraudulent video and audio. The way we think about facts as verified is going to change a bit and that is going to create a big systemic challenge for PSM in general. For the kind of automatic tooling that I work on, it significantly increases the burden of making sure that everyone has some kind of common understanding. I think that’s one of the big emerging problems that that may significantly change the way organize ourselves as an industry and the tools for being successful.
There’s also an interesting question about how data science becomes formalized as a practice. The role of data science has existed for 7-8 years – if we look a generation down the road, does data science as a profession look more like a lawyer or more like a software developer? As a practice it’s a fast-moving field and not as mature as a lot of other skills.