AttentiveVideo: A Multimodal Approach to Quantify Emotional Responses to Mobile Advertisements

Auteur-es

Pham, Phuong; Wang, Jingtao

Nombre Auteurs

2

Titre

AttentiveVideo: A Multimodal Approach to Quantify Emotional Responses to Mobile Advertisements

Année de publication

2019

Référence (APA)

Pham, P., & Wang, J. (2019). AttentiveVideo : A Multimodal Approach to Quantify Emotional Responses to Mobile Advertisements. ACM Transactions on Interactive Intelligent Systems, 9(2‑3), 8:1-8:30. https://doi.org/10.1145/3232233

Mots-clés

Affective Computing, Signal Processing, User Modeling, Computational advertising, heart rate, facial expression, mobile interfaces

URL

https://dl.acm.org/doi/10.1145/3232233

doi

https://doi.org/10.1145/3232233

Accessibilité de l'article

Open access

Champ

Human-Computer Interaction and Visualization

Machine Intelligence

Type contenu (théorique Applicative méthodologique)

Applicative

Méthode

AttentiveVideo employs a combination of implicit photoplethysmography (PPG) sensing and facial expression analysis (FEA) to detect the attention, engagement,and sentiment of viewers as they watch video advertisements on unmodified smartphones.

The viewer’s photoplethysmography (PPG) signals are extracted implicitly by continuously monitoring the changes in transparency of the covering finger through the back camera.
AttentiveVideo simultaneously captures and analyzes the viewer’s facial expressions via the front camera.
From the collected PPG signals and facial expressions, the prediction module of AttentiveVideo leverages machine-learning algorithms to infer viewers’ emotional responses.

Cas d'usage

Mobile video advertising

Objectifs de l'article

Understanding a target audience’s emotional responses to a video advertisement is crucial to evaluate the advertisement’s effectiveness. However, traditional methods for collecting such information are slow, expensive, and coarse grained.
We propose AttentiveVideo, a scalable intelligent mobile interface with corresponding inference algorithms to monitor and quantify the effects of mobile video advertising in real time.

Question(s) de recherche/Hypothèses/conclusion

Research question(s) : Understanding a target audience’s emotional responses to a video advertisement is crucial to evaluate the advertisement’s effectiveness. However, traditional methods for collecting such information are slow, expensive, and coarse grained.

Hypothesis(es) : AttentiveVideo achieves implicit PPG sensing and FEA on unmodified smartphones. Our approach eliminates the additional sensor requirement for the large-scale deployment of affect sensing systems. We explore three new ideas in this work: (1) a new PPG-based feature extraction (LocalDiff), (2) reducing lens covering time, and (3) model fusion (combining different machine-learning algorithms).

Conclusion(s) : AttentiveVideo achieved good accuracy on a wide range of emotional measures (the best average accuracy = 82.6% across nine measures). While feature fusion alone did not improve prediction accuracy with a single model, it significantly improved the accuracy when working together with model fusion. We also found that the PPG sensing channel and the FEA technique have different strength in data availability, latency detection, accuracy, and usage environment.

Cadre théorique/Auteur.es

Advertising Effectiveness (Linden, Smith, et York, 2003 ; Lohtia, Donthu, et Hershberger, 2003 ; Mei, Hua, et Li, 2009 ; Broder et al, 2008)

Emotions as indirect measures of the effectiveness of branding advertising (Aaker, Stayman, et Hagerty, 1986 ; Micu et Plummer, 2010 ; Stout et Leckenby, 1986) Informatique affective (Picard, 1997 ; Lang, 1990)

Facial expressions (Calvo et D'Mello, 2010 ; McDuff et al., 2015 ; McDuff et al., 2014)

Physiological signals (Blanchard et al., 2014 ; Calvo et D'Mello, 2010 ; Kemp et Quintana, 2013 ; Lang, 1990)

Combining different modalities (Bailenson et al., 2008 ; D'Mello et Graesser, 2010 ; Hussain, Monkaresi, et Calvo, 2012)

Collecting Physiological Signals without Additional Sensors (Pham et Wand 2015 ; Pham et Wang, 2016 ; Pham et Wang, 2017 ; Xiao et Wang, 2015 ; Xiao et Wang, 2016)

Mobile Video Interfaces (Ganhör, 2012 ; Li, Zhang, et Yuan, 2011 ; Li et al., 2015 ; Wu et al., 2012 ; Xiao et Wang, 2015 ; Zhang et al., 2013)

Concepts clés

Affective computing

Sentiment analysis

Données collectées (type source)

In the formal study, the participant watched an episode of “The Big Bang Theory” with three embedded advertising slots, each slot containing four ads. Similarly to the training session, the participant gave an emotional self-report after each advertising slot and continued with the episode. At the end of the study, the participant rated the usability of AttentiveVideo and ranked the six most liked ads

The viewer’s photoplethysmography (PPG) signals are extracted implicitly by continuously monitoring the changes in transparency of the covering finger through the back camera. Furthermore, AttentiveVideo simultaneously captures and analyzes the viewer’s facial expressions via the front camera.

In this study, we used two types of self-reporting: verbal selfreporting for discrete emotions and visual self-reporting for dimensional emotions. Participants responded to six discrete emotions related questions on the effectiveness of each advertisement, i.e., Attention, Share, Touching, Rewatch, Recall, and Amusing. We used Touching as a warmth emotion in advertising, i.e., the viewer feels moved by the ad. We also used the Self-Assessment Manikin (SAM) [34] to collect responses for two dimensional emotions, Valence and Arousal. These ratings are in a 7-point Likert scale format (1: highly disagree;7:highly agree). In addition, participants rated the Like measure by ranking the 6 most liked ads at the end of the study. In total, we collected measures of nine emotional states that can be grouped into three categories: attention, engagement, and sentiment (Table 3).

Définition des émotions

No definition

Description of facial expressions and emotions observed

Ampleur expérimentation (volume de comptes)

24 participants (13 females) from a local university. The average age was 25.58 (σ = 3.01); the ages ranged from 21 to 33.

Technologies associées

AttentiveVideo collects a viewer’s PPG signals through the back camera (AttentiveVideo extended the LivePulse algorithm [15] to identify the interbeat intervals (NN) from PPG signals).

AttentiveVideo utilizes the front camera to capture the user’s facial expressions as she watches advertisements. AttentiveVideo used Affdex SDK [27] to analyze facial expressions from the recorded clips.

Use of machine-learning algorithms to infer the user’s emotional responses to the advertisement

Mention de l'éthique

ND

Finalité communicationnelle

AttentiveVideo can help advertisers gain a richer and more fine-grained understanding of users’ emotional responses toward video advertisements. AttentiveVideo can also help viewers to enjoy more high-quality video materials for free via subsidized video ads.
AttentiveVideo can additionally be applied to personalized advertising.

Résumé

Understanding a target audience's emotional responses to a video advertisement is crucial to evaluate the advertisement's effectiveness. However, traditional methods for collecting such information are slow, expensive, and coarse grained. We propose AttentiveVideo, a scalable intelligent mobile interface with corresponding inference algorithms to monitor and quantify the effects of mobile video advertising in real time. Without requiring additional sensors, AttentiveVideo employs a combination of implicit photoplethysmography (PPG) sensing and facial expression analysis (FEA) to detect the attention, engagement, and sentiment of viewers as they watch video advertisements on unmodified smartphones. In a 24-participant study, AttentiveVideo achieved good accuracy on a wide range of emotional measures (the best average accuracy = 82.6% across nine measures). While feature fusion alone did not improve prediction accuracy with a single model, it significantly improved the accuracy when working together with model fusion. We also found that the PPG sensing channel and the FEA technique have different strength in data availability, latency detection, accuracy, and usage environment. These findings show the potential for both low-cost collection and deep understanding of emotional responses to mobile video advertisements.