publications

2024

1. Recommending Art Online: Investigating Engagement and Interactions with a National Public Collection

   Lukas Hughes-Noehrer, Jonathan Carlton, and Caroline Jay

   2024

   Abs

   Museum online collections now contain millions of objects, making developing tools for supporting users in navigating these a priority. We present a user-centric study of a recommendation system created to browse Art UK’s digital collection according to personal preference. Three forms of recommendations were explored: image-only, metadata-only, and a combination of the two. Recommendations were evaluated against a baseline of artworks chosen at random. To create a holistic picture of user engagement, both subjective ratings and interaction data were collected. Whilst ratings data showed a uniformly positive perception of the system, the interaction data showed significantly deeper and longer engagement with the artworks that were recommended. Results indicate 1) that recommendations can be a useful tool to help users explore online art collections, and 2) that when evaluating recommendation systems, interaction data can capture patterns of increased engagement that are not evident in subjective ratings.

2022

1. Using Data to Understand How Audiences Engage with Interactive Media

   Jonathan Carlton

   The University of Manchester (United Kingdom), 2022

   Abs

   Media is evolving from traditional linear narratives to personalised experiences, where control over information (or how it is presented) is given to individual audience members. Measuring and understanding audience engagement with this media is important: a post-hoc understanding of how engaged audiences are with the content will help production teams learn from experiences and improve future productions. Engagement is typically measured by asking samples of users to self-report, which is time consuming and expensive. In some domains, however, interaction data have been used to infer engagement. The nature of interactive media facilitates a much richer set of interaction data than traditional media; this thesis aims to understand if these data can be used to understand and infer audience engagement and, by extension, the abandonment of content. This thesis reports studies, run in collaboration with the BBC, of engagement and abandonment using data captured from audience interactions with an interactive TV show and an adaptive tutorial. It was found that engagement can be modelled and predicted in the interactive TV show, and that users appear to behave differently based on their level of engagement. For example, high engagement is associated with consumption-type behaviours, while low engagement is associated with skipping-type behaviours. When investigating the data collected from the adaptive tutorial, the results revealed that user context, rather than user interactions, affects the engagement of users. Abandonment was investigated using a wider dataset collected from the national release of the interactive TV show; it was demonstrated that abandonment could be accurately predicted from the interactions of users. An increase in moving backwards and forwards in the show were indicative of an increase in abandonment, suggesting an exploratory-type behaviour. When exploring the link between abandonment and engagement, it was found that low engagement users were predicted to drop out further from the end, suggesting a relationship between the two. The results demonstrate that interaction data is a viable method for the evaluation of media is this evolving domain. To move towards consistency in the interaction data analysis field, the thesis proposes a framework to provide methodological support for researchers. Through an analysis of the literature, meta-issues were identified in the communication of research which create barriers in reproducibility and reduces transparency. The framework provides structure for those undertaking research on understanding users through their interactions and a terminology that can be applied consistently across the different disciplines in this area. It is conjectured that using such a framework should improve both the quality of science and science communication in the area, with more reproducible and transparent research being enabled.

2021

1. Machine learning and museum collections: A data conundrum

   Lukas Noehrer, Jonathan Carlton, and Caroline Jay

   In International Conference on Emerging Technologies and the Digital Transformation of Museums and Heritage Sites, 2021

   Abs

   Museums contain vast amounts of information and knowledge, providing a vital source of engagement for diverse audiences. As society becomes ever more digital, museums are moving towards making their collections available online to the public. However, just providing a searchable interface to the entirety of the collection could be a barrier to successful engagement. Tremendous craftsmanship is put into creating interesting and informative in-person curations of selected items, and a challenge exists in replicating this online. One solution could be the application of recommender systems, which personalise information to the individual based on their previous interactions and tastes. These systems power many popular online services, but cannot be applied without considerations and decisions being made about the data that is given to the engine. As museum collections vary in their nature and content, particular care should be taken when handling the data – standard methods may not apply. In this paper, we present the challenges of data curation in the context of using machine learning techniques with museum collections, supported by two case studies.

2. Using Interaction Data to Predict Engagement with Interactive Media

   Jonathan Carlton, Andy Brown, Caroline Jay, and 1 more author

   In Proceedings of the 29th ACM International Conference on Multimedia, 2021

   Abs

   Media is evolving from traditional linear narratives to personalised experiences, where control over information (or how it is presented) is given to individual audience members. Measuring and understanding audience engagement with this media is important in at least two ways: (1) a post-hoc understanding of how engaged audiences are with the content will help production teams learn from experience and improve future productions; (2), this type of media has potential for real-time measures of engagement to be used to enhance the user experience by adapting content on-the-fly. Engagement is typically measured by asking samples of users to self-report, which is time consuming and expensive. In some domains, however, interaction data have been used to infer engagement. Fortuitously, the nature of interactive media facilitates a much richer set of interaction data than traditional media; our research aims to understand if these data can be used to infer audience engagement. In this paper, we report a study using data captured from audience interactions with an interactive TV show to model and predict engagement. We find that temporal metrics, including overall time spent on the experience and the interval between events, are predictive of engagement. The results demonstrate that interaction data can be used to infer users’ engagement during and after an experience, and the proposed techniques are relevant to better understand audience preference and responses.

2019

1. Inferring user engagement from interaction data

   Jonathan Carlton, Andy Brown, Caroline Jay, and 1 more author

   In Extended abstracts of the 2019 CHI conference on Human Factors in Computing Systems, 2019

   Abs

   This paper presents preliminary results of a study designed to quantify users’ engagement levels with interactive media content, through self-reported measures and interaction data. The broad hypothesis of the study is that interaction data can be used to predict the level of engagement felt by the user. The challenge addressed in this work is to explore the effectiveness of interaction data to act as a proxy for engagement levels and reveal what that data shows about engagement with media content. Preliminary results suggest several interesting insights about participants engagement and behaviour. Crucially, temporal statistics support the hypothesis that the participant making use of the controls in the interactive, video-based experience positively correlates with higher engagement.

2018

1. Using Low-Level Interaction Data to Explore User Behaviour in Interactive-Media Experiences

   Jonathan Carlton, Andy Brown, John Keane, and 1 more author

   In 11th International Conference on Methods and Techniques in Behavioral Research: Measuring Behavior 2018, 2018
2. Identifying Latent Indicators of Technical Difficulties from Interaction Data

   Jonathan Carlton, Joshua Woodcock, Andy Brown, and 2 more authors

   In ACM KDD Workshop on Data Science, Journalism, and Media (DSJM 2018), 2018

   Abs

   A significant amount of resource is spent in maximising the retention levels of consumers of online media content. Moreover, it is considered a success if the consumer stays engaged throughout and consumes the media in its entirety. Despite this, unforeseen circumstances may hinder the ability of the consumer to enjoy the produced content and lead to low overall retention rates. In this paper, we explore interaction data, collected from an interactive media experience, to discover user behaviours that serve as indicators for technical difficulties. Detecting technical faults, such as video buffering, from the analysis of interaction data could offer the ability to provide corrective suggestions. It also, crucially, helps us to determine when dropout is caused by factors other than the content itself. We report that users who experienced and reported videorelated faults share similar descriptive statistics to those who did not report faults; however, analysis of discrete sequences of events demonstrates that there are, in actuality, fundamental differences between the two groups.

2017

1. Recruiting from the network: Discovering twitter users who can help combat zika epidemics

   Paolo Missier, Callum McClean, Jonathan Carlton, and 5 more authors

   In Web Engineering: 17th International Conference, ICWE 2017, Rome, Italy, June 5-8, 2017, Proceedings 17, 2017

   Abs

   Tropical diseases like Chikungunya and Zika have come to prominence in recent years as the cause of serious health problems. We explore the hypothesis that monitoring and analysis of social media content streams may effectively complement institutional disease prevention efforts. Specifically, we aim to identify selected members of the public who are likely to be sensitive to virus combat initiatives. Focusing on Twitter and on the topic of Zika, our approach involves (i) training a classifier to select topic-relevant tweets from the Twitter feed, and (ii) discovering the top users who are actively posting relevant content about the topic. In this short paper we describe our analytical approach and prototype architecture, discuss the challenges of dealing with noisy and sparse signal, and present encouraging preliminary results.