Publications · EMOTICONES

Meta

Auteur-es: Adam D. I. Kramer
Nombre Auteurs: 1
Titre: The spread of emotion via Facebook
Année de publication: 2012
Référence (APA): Kramer, A. D. I. (2012). The spread of emotion via facebook. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 767‑770. https://doi.org/10.1145/2207676.2207787
résumé: In this paper we study large-scale emotional contagion through an examination of Facebook status updates. After a user makes a status update with emotional content, their friends are significantly more likely to make a valence-consistent post.

This effect is significant even three days later, and even after controlling for prior emotion expressions by both users and their friends. This indicates not only that emotional contagion is possible via text-only communication and that emotions flow through social networks, but also that emotion spreads via indirect communications media.
Mots-clés: Computer-mediated communication; emotional contagion;
emotion; Facebook; social networks
URL: https://research.facebook.com/file/2916163195290181/the-spread-of-emotion-via-facebook.pdf
doi: https://doi.org/10.1145/2207676.2207787
Accessibilité de l'article: Libre
Champ: Data science
Type contenu (théorique Applicative méthodologique): Théorique, méthodologique
Méthode: Computationnelle, Predictive, hierarchical logistic regression, multi-day lagged regression design
Cas d'usage: Facebook
Objectifs de l'article: This article examine the implications of emotional contagion theory for social networks, to determine whether emotions spread through SNSs, and to contribute to a growing body of evidence suggesting that interaction with "friends" on SNSs mirrors offline interactions.

"We examine an important implication of emotional contagion theory: whether and how emotions “spread” via SNSs."
Question(s) de recherche/Hypothèses/conclusion: The research question was whether and how emotions spread via SNSs.; The hypothesis was that emotional contagion occurs through indirect communication on social networks.

"We predict that when users use positive or negative words (i.e., express positive or negative affect [13]), their friends' emotional states will be affected: Those friends will express more of the corresponding emotion (i.e., use positive or negative words in subsequent updates)"; The conclusions were that emotional contagion does occur through Facebook status updates, and that this has important implications for our understanding of psychological processes and social interactions.
Cadre théorique/Auteur.es: The theoretical framework of the article includes psychology, emotional contagion theory and the broaden-and-build theory of positive emotions, with main authors cited including Murray, Pennebaker, and Fredrickson.
Concepts clés: Emotional suppression, Emotional contagion, Positive emotions
Données collectées (type source): The data collected was a large dataset of Facebook status updates
Définition des émotions: LIWC
Ampleur expérimentation (volume de comptes): 61,289 users
Technologies associées: Hive Data Warehousing
Mention de l'éthique: "no researchers ever saw any user-generated text in an identified form, in compliance with Facebook’s Terms of Service"
Finalité communicationnelle: Exploring the spread of emotions through social networks

Collections: Facebook

Auteur-es: Felix Kreuk, Adam Polyak, Jade Copet, Eugene Kharitonov, Tu-Anh Nguyen, Morgane Rivière, Wei-Ning Hsu, Emmanuel Dupoux, Yossi Adi
Nombre Auteurs: 9
Titre: Textless Speech Emotion Conversion using Discrete & Decomposed Representations
Année de publication: 2022
Référence (APA): Kreuk, F., Polyak, A., Copet, J., Kharitonov, E., Nguyen, T.-A., Rivière, M., Hsu, W.-N., Mohamed, A., Dupoux, E., & Adi, Y. (2022). Textless Speech Emotion Conversion using Discrete & Decomposed Representations.
résumé: Speech emotion conversion is the task of modifying the perceived emotion of a speech utterance while preserving the lexical content and speaker identity. In this study, we cast the problem of emotion conversion as a spoken language translation task. We use a decomposition of the speech signal into discrete learned representations, consisting of phonetic-content units, prosodic features, speaker, and emotion. First, we modify the speech content by translating the phonetic-content units to a target emotion, and then predict the prosodic features based on these units. Finally, the speech waveform is generated by feeding the predicted representations into a neural vocoder. Such a paradigm allows us to go beyond spectral and parametric changes of the signal, and model non-verbal vocalizations, such as laughter insertion, yawning removal, etc. We demonstrate objectively and subjectively that the proposed method is vastly superior to current approaches and even beats text-based systems in terms of perceived emotion and audio quality. We rigorously evaluate all components of such a complex system and conclude with an extensive model analysis and ablation study to better emphasize the architectural choices, strengths and weaknesses of the proposed method. Samples are available under the following link here.
URL: https://research.facebook.com/file/509033718007989/Textless-Speech-Emotion-Conversion-using-Discrete-and-Decomposed-Representations.pdf
doi: https://doi.org/10.48550/arXiv.2111.07402
Accessibilité de l'article: Libre
Champ: Natural Language Processing & Speech
Type contenu (théorique Applicative méthodologique): Méthodologique, applicatif
Méthode: The method allows for the modification of non-verbal vocalizations such as laughter insertion and yawning removal while preserving the lexical content and speaker identity.
Cas d'usage: N/A
Objectifs de l'article: The study presents a novel approach to modifying the perceived emotion of a speech utterance while preserving the lexical content and speaker identity. The study also includes a rigorous evaluation of the proposed method and compares it against current approaches. The proposed method has been rigorously evaluated and has been found to be superior to current approaches in terms of perceived emotion and audio quality.
Question(s) de recherche/Hypothèses/conclusion: Research question is How to modify the perceived emotion of a speech utterance while preserving the lexical content and speaker identity ?; The hypothesis is that "modifying the learned representations of a speech signal can effectively change the perceived emotion of the speech while preserving the lexical content and speaker identity."; The concusions are "The researchers found a way to break down the voice into different parts and change them to make the voice sound happy, sad, angry, or other emotions. They tested this new way and found that it works better than other ways that people have tried before. They also found that changing the non-verbal parts of the voice, like laughter or sighs, can make a big difference in how the voice sounds. Overall, this new way of changing someone's voice is really cool and could be useful in things like movies or video games where characters need to sound different emotions."
Cadre théorique/Auteur.es: Speech processing, natural language processing : Vaswani et al. (2017); Polyak et al. (2021); Kharitonov et al. (2021a); Schuller et al. (2013); Fan et al. (2014); Wang et al. (2018)
Concepts clés: Speech emotion conversion, Non-verbal communication cues, Spoken language translation
Données collectées (type source): Dataset of emotional speech recordings, which were labeled with six different emotions (Neutral, Happy, Sad, Angry, Disgusted, and Sleepy)
Définition des émotions: Categorical emotions
Ampleur expérimentation (volume de comptes): 7000 speech utterances based on transcripts from the CMU Arctic Database
Technologies associées: HuBERT: A pre-trained neural network model for speech processing, which is used as a feature extractor in the emotion conversion model.

Transformer: A type of neural network architecture that uses self-attention mechanisms to process sequential data, which is used as the main architecture for the emotion conversion model.

PyTorch: A popular open-source machine learning framework, which is used to implement the emotion conversion model and train it on the emotional speech dataset.
Mention de l'éthique: Non
Finalité communicationnelle: "We demonstrated how the proposed system is able to model expressive non-verbal vocalizations as well as generate high- quality expressive speech. We conclude with an ablation study and analysis of the different components composing our system. This study serves as the foundation for improving speech emotion conversion and building general textless expressive speech generation models."

Collections: Facebook

Auteur-es: Yi-Chia Wang, Moira Burke, Robert Kraut
Nombre Auteurs: 3
Titre: Modeling Self-Disclosure in Social Networking Sites
Année de publication: 2016
Référence (APA): Wang, Y.-C., Burke, M., & Kraut, R. (2016). Modeling Self-Disclosure in Social Networking Sites. https://doi.org/10.1145/2818048.2820010
résumé: Social networking sites (SNSs) offer users a platform to build and maintain social connections. Understanding when people feel comfortable sharing information about themselves on SNSs is critical to a good user experience, because self-disclosure helps maintain friendships and increase relationship closeness. This observational research develops a machine learning model to measure self-disclosure in SNSs and uses it to understand the contexts where it is higher or lower. Features include emotional valence, social distance between the poster and people mentioned in the post, the language similarity between the post and the community and post topic. To validate the model and advance our understanding about online self- disclosure, we applied it to de-identified, aggregated status updates from Facebook users. Results show that women self-disclose more than men. People with a stronger desire to manage impressions self-disclose less. Network size is negatively associated with self-disclosure, while tie strength and network density are positively associated.
Mots-clés: Social networking sites; Facebook; computer-mediated
communication; self-disclosure; personality; audience;
context collapse; natural language analysis; applied
machine learning
URL: https://research.facebook.com/file/271555647830591/modeling_self-disclosure_in_social_networking_sites.pdf
doi: https://doi.org/10.1145/2818048.2820010
Accessibilité de l'article: Libre
Champ: Data Science, Human Computer Interaction & UX
Type contenu (théorique Applicative méthodologique): Méthodologique
Méthode: The authors developed a machine learning model to measure self-disclosure and applied it to de-identified, aggregated status updates from Facebook users to validate the model and advance our understanding about online self-disclosure. They also used Latent Dirichlet Allocation (LDA) to identify the topics common in status updates.

Status updates were annotated by users and by trained judges.
Cas d'usage: Facebook
Objectifs de l'article: The objectives of the article are to develop a machine learning model to measure self-disclosure on social networking sites, to validate the model using deidentified, aggregated status updates from Facebook users, and to understand the contexts where self-disclosure is higher or lower.
Question(s) de recherche/Hypothèses/conclusion: The research question is how can we measure self-disclosure on social networking sites using machine learning?; The hypothesis are :
H1: Individuals with a stronger desire for impression management will self-disclose less.
H2: Women will self-disclose more than men.
H3: Network size will be negatively correlated with selfdisclosure.
H4: Average tie strength will be positively correlated with self-disclosure.
H5: Network density will be positively correlated with selfdisclosure; The authors conclude that their machine learning model was able to accurately measure self-disclosure on Facebook and that self-disclosure was higher for certain topics, such as politics and memorial, and lower for others, such as Christianity and deep thoughts.
Cadre théorique/Auteur.es: The theoretical framework of the article includes social penetration theory and communication privacy management theory. The main authors cited include Altman and Taylor, Derlega and Chaikin, and Petronio.
Concepts clés: Self-disclosure, Social networking sites, Machine learning
Données collectées (type source): The authors collected de-identified, aggregated status updates from Facebook users (recruited on AMT). They used dataset from the myPersonality project. The dataset contains users’ status updates as well as their demographic information and self-report impression management scores
Définition des émotions: LIWC
Ampleur expérimentation (volume de comptes): 2000 Facebook status updates

Then a random sample of 412,470 English language Facebook active users for approximately one month in late 2014
Technologies associées: ML et Latent Dirichlet Allocation (LDA)
Mention de l'éthique: Non
Commentaires: Financed by the grants from National Science Foundation (IIS-0968485) and National Institute of Mental Health (R21 MH106880-01).

Collections: Facebook

Auteur-es: Felix Kreuk, Adam Polyak, Jade Copet, Eugene Kharitonov, Tu-Anh Nguyen, Morgane Rivière, Wei-Ning Hsu, Emmanuel Dupoux, Yossi Adi
Nombre Auteurs: 9
Titre: Textless Speech Emotion Conversion using Discrete & Decomposed Representations
Année de publication: 2022
Référence (APA): Kreuk, F., Polyak, A., Copet, J., Kharitonov, E., Nguyen, T.-A., Rivière, M., Hsu, W.-N., Mohamed, A., Dupoux, E., & Adi, Y. (2022). Textless Speech Emotion Conversion using Discrete & Decomposed Representations.
résumé: Speech emotion conversion is the task of modifying the perceived emotion of a speech utterance while preserving the lexical content and speaker identity. In this study, we cast the problem of emotion conversion as a spoken language translation task. We use a decomposition of the speech signal into discrete learned representations, consisting of phonetic-content units, prosodic features, speaker, and emotion. First, we modify the speech content by translating the phonetic-content units to a target emotion, and then predict the prosodic features based on these units. Finally, the speech waveform is generated by feeding the predicted representations into a neural vocoder. Such a paradigm allows us to go beyond spectral and parametric changes of the signal, and model non-verbal vocalizations, such as laughter insertion, yawning removal, etc. We demonstrate objectively and subjectively that the proposed method is vastly superior to current approaches and even beats text-based systems in terms of perceived emotion and audio quality. We rigorously evaluate all components of such a complex system and conclude with an extensive model analysis and ablation study to better emphasize the architectural choices, strengths and weaknesses of the proposed method. Samples are available under the following link here.
URL: https://research.facebook.com/file/509033718007989/Textless-Speech-Emotion-Conversion-using-Discrete-and-Decomposed-Representations.pdf
doi: https://doi.org/10.48550/arXiv.2111.07402
Accessibilité de l'article: Libre
Champ: Natural Language Processing & Speech
Type contenu (théorique Applicative méthodologique): Méthodologique, applicatif
Méthode: The method allows for the modification of non-verbal vocalizations such as laughter insertion and yawning removal while preserving the lexical content and speaker identity.
Cas d'usage: N/A
Objectifs de l'article: The study presents a novel approach to modifying the perceived emotion of a speech utterance while preserving the lexical content and speaker identity. The study also includes a rigorous evaluation of the proposed method and compares it against current approaches. The proposed method has been rigorously evaluated and has been found to be superior to current approaches in terms of perceived emotion and audio quality.
Question(s) de recherche/Hypothèses/conclusion: Research question is How to modify the perceived emotion of a speech utterance while preserving the lexical content and speaker identity ?; The hypothesis is that "modifying the learned representations of a speech signal can effectively change the perceived emotion of the speech while preserving the lexical content and speaker identity."; The concusions are "The researchers found a way to break down the voice into different parts and change them to make the voice sound happy, sad, angry, or other emotions. They tested this new way and found that it works better than other ways that people have tried before. They also found that changing the non-verbal parts of the voice, like laughter or sighs, can make a big difference in how the voice sounds. Overall, this new way of changing someone's voice is really cool and could be useful in things like movies or video games where characters need to sound different emotions."
Cadre théorique/Auteur.es: Speech processing, natural language processing : Vaswani et al. (2017); Polyak et al. (2021); Kharitonov et al. (2021a); Schuller et al. (2013); Fan et al. (2014); Wang et al. (2018)
Concepts clés: Speech emotion conversion, Non-verbal communication cues, Spoken language translation
Données collectées (type source): Dataset of emotional speech recordings, which were labeled with six different emotions (Neutral, Happy, Sad, Angry, Disgusted, and Sleepy)
Définition des émotions: Categorical emotions
Ampleur expérimentation (volume de comptes): 7000 speech utterances based on transcripts from the CMU Arctic Database
Technologies associées: HuBERT: A pre-trained neural network model for speech processing, which is used as a feature extractor in the emotion conversion model.

Transformer: A type of neural network architecture that uses self-attention mechanisms to process sequential data, which is used as the main architecture for the emotion conversion model.

PyTorch: A popular open-source machine learning framework, which is used to implement the emotion conversion model and train it on the emotional speech dataset.
Mention de l'éthique: Non
Finalité communicationnelle: "We demonstrated how the proposed system is able to model expressive non-verbal vocalizations as well as generate high- quality expressive speech. We conclude with an ablation study and analysis of the different components composing our system. This study serves as the foundation for improving speech emotion conversion and building general textless expressive speech generation models."

Collections: Facebook

Auteur-es: Jane Dwivedi-Yu, Alon Y. Halevy
Nombre Auteurs: 2
Titre: “That’s so cute!”: The CARE Dataset for Affective Response Detection
Année de publication: 2022
Référence (APA): Dwivedi-Yu, J., & Halevy, A. Y. (2022). “That’s so cute!” : The CARE Dataset for Affective Response Detection.
résumé: Social media plays an increasing role in our communication with friends and family, and in our consumption of entertainment and information. Hence, to design effective ranking functions for posts on social media, it would be useful to predict the affective responses of a post (e.g., whether it is likely to elicit feelings of entertainment, inspiration, or anger). Similar to work on emotion detection (which focuses on the affect of the publisher of the post), the traditional approach to recognizing affective response would involve an expensive investment in human annotation of training data.

We create and publicly release CAREdb, a dataset of 230k social media post annotations according to seven affective responses using the Common Affective Response Expression (CARE) method. The CARE method is a means of leveraging the signal that is present in comments that are posted in response to a post, providing high-precision evidence about the affective response to the post without human annotation. Unlike human annotation, the annotation process we describe here can be iterated upon to expand the coverage of the method, particularly for new affective responses. We present experiments that demonstrate that the CARE annotations compare favorably with crowdsourced annotations. Finally, we use CAREdb to train competitive BERT-based models for predicting affective response as well as emotion detection, demonstrating the utility of the dataset for related tasks.
Mots-clés: affective response, social media, emotion classification, CARE dataset, human annotation, machine learning
URL: https://research.facebook.com/file/393453116239728/Thats-so-cute--The-CARE-Dataset-for-Affective-Response-Detection.pdf
doi: https://doi.org/10.48550/arXiv.2201.11895
Accessibilité de l'article: Libre
Champ: Artificial Intelligence
Type contenu (théorique Applicative méthodologique): Méthodologique, applicatif
Méthode: The method used in this article is the Common Affective Response Expression (CARE) method, which is a cost-effective alternative to human annotation for predicting the emotional impact of social media posts. The CARE method involves two steps: (1) identifying the CARE patterns in the text, and (2) using a lexicon to label the publisher affect of the comments. The authors also compare the performance of the CARE method to a state-of-the-art emotion classifier.

Annotations according to the CARE method, as well as a human annotated dataset for comparison.
Cas d'usage: Reddit
Objectifs de l'article: The objectives of the article are to introduce the CARE dataset, evaluate its accuracy compared to crowdsourced annotations, and explore its potential applications in social media ranking functions.
Question(s) de recherche/Hypothèses/conclusion: The research question is whether the CARE dataset can accurately predict affective response in social media posts.; The hypothesis is that the CARE dataset will perform comparably to crowdsourced annotations in predicting affective response in social media posts.; The authors conclude that the CARE dataset is a valuable resource for predicting affective response in social media posts, and that it performs comparably to crowdsourced annotations. They also suggest that the CARE dataset could be used to improve social media ranking functions.
Cadre théorique/Auteur.es: The theoretical framework of the article is based on previous work in affective computing and emotion classification. The main authors cited include Stark and Hoey (2021), Demszky et al. (2020), Lin et al. (2008), Ma et al. (2005), and Mohammad and Kiritchenko (2015).
Concepts clés: Affective response, Social Media, Emotion classification, Machine learning
Données collectées (type source): Our experiments use a dataset that is created from Reddit posts and comments in the pushshift.io database that were created between 2011 and 2019.
Définition des émotions: Non
Ampleur expérimentation (volume de comptes): 34 million comments from
24 million distinct posts

TOTAL in CARE : 230k
Technologies associées: CARE method, machine learning algorithms, Emotion classifier
Mention de l'éthique: Yes, there is mention of ethics in this article. The authors note that the CARE method should not be used for ill-intended purposes such as purposefully recommending particular content to manipulate a user's perception or preferences.
Finalité communicationnelle: "We described a method for extracting training data for models that predict the affective responses of a post on social media"
Commentaires: Voir la bibliographie de l'article, plusieurs références qui pourraient entrer dans la RL.

Collections: Facebook

Auteur-es: Moira Burke and Mike Develin
Nombre Auteurs: 2
Titre: Once More with Feeling: Supportive Responses to Social Sharing on Facebook
Année de publication: 2016
Référence (APA): Burke, M., & Develin, M. (2016). Once More with Feeling : Supportive Responses to Social Sharing on Facebook. Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing, 1462‑1474. https://doi.org/10.1145/2818048.2835199
résumé: Life is more than cat pictures. There are tough days,heartbreak, and hugs. Under what contexts do people share these feelings online, and how do their friends respond? Using millions of de-identified Facebook status updates with poster-annotated feelings (e.g., feeling thankful or feeling worried), we examine the magnitude and circumstances in which people share positive or negative feelings and characterize the nature of the responses they receive. We find that people share greater proportions of both positive and negative emotions when their friend networks are smaller and denser. Consistent with social sharing theory, hearing about a friend’s troubles on Facebook causes friends to reply with more emotional and supportive comments. Friends’ comments are also more numerous and longer. Posts with positive feelings, on the other hand, receive more likes, and their comments have more positive language. Feelings that relate to the poster’s self worth, such as feeling defeated, feeling unloved, or feeling accomplished amplify these effects.
Mots-clés: Emotional support; social network sites; social sharing
URL: https://research.facebook.com/file/373780360921794/once-more-with-feeling-supportive-responses-to-social-sharing-on-facebook.pdf
doi: https://doi.org/10.1145/2818048.2835199
Accessibilité de l'article: Libre
Champ: Data Science, Human Computer Interaction, & UX
Type contenu (théorique Applicative méthodologique): Théorique
Méthode: S1 : Two judges independently coded the top 200
feelings (which comprised 90% of feelings usage) on a 3-point valence scale (positive, ambiguous, or negative). Inter-
rater reliability was good (Cohen’s kappa = 0.77).

S2 : As in Study 1, the 14.2 million posts with feeling annotations were categorized according to the valence and self-relevance of the emotion.
Cas d'usage: Facebook
Objectifs de l'article: The objectives of the article are to explore the ways in which people share their emotions on Facebook and how their friends respond, and to examine the relationship between characteristics of the emotions shared and characteristics of the responses.

"The present research is the first very large-scale quantitative
study of social sharing of feelings in Facebook status
updates, examining (1) how network properties (size and
density) are associated with the valence of emotions shared, and (2) how characteristics of the emotions shared (valence and self-relevance) relate to how the audience responds (quantity of responses, emotional and supportive content in comments, and whether the responses come in network-visible or private channels)."
Question(s) de recherche/Hypothèses/conclusion: Research question : Do people share different kinds of emotions online depending on whom they think is listening?
Does social sharing of emotion online elicit emotion in viewers, and what form do their responses take?; The hypothesis is that people share more positive and negative emotions when their friend networks are smaller and denser, and that hearing about a friend's troubles on Facebook leads to more emotional and supportive comments.; The conclusion are that "the research demonstrates that when people have smaller, denser networks, they share more positive and negative emotions, and friends respond in greater volume to posts with emotion,
especially negative emotion. Responses to posts with negative emotion are more emotional themselves and contain more supportive language; more extreme expressions of negative self-worth (e.g., feeling unloved)
often elicit private responses from friends."
Cadre théorique/Auteur.es: The theoretical framework of the article is based on psychology (Pennebaker), on previous research on social sharing and emotional support, including studies by Rimé, Reis, and Shaver. The authors also draw on theories of social capital and social network analysis.
Concepts clés: Social sharing, Emotional valence, Social capital, Social network analysis
Données collectées (type source): Large dataset of de-identified text-based status updates on Facebook's servers. One annotated and the other one without feeling annotations (as a baseline).

"We analyzed network properties and status updates by a random sample of people who used the feeling annotation tool. N=1,399,921 English speakers in the U.S.
(Mean age = 32.2, SD=10.9, 79% female) who posted at least one status update in June 2015 that included an emotion (using the annotation widget in Figure 1), had been Facebook users for at least six months, and had at least 20 friends, were randomly selected. Their status updates for one month (approximately 30 million) and friend networks were included in the analysis."

S2 : N=31.7 million de-identified text-based status updates, comprised of a random sample of 14.2 million posts with attached feeling annotations and a random sample of 17.5 million without feeling annotation.
Définition des émotions: Non
Ampleur expérimentation (volume de comptes): 14,5 millions posts annotated

17,5 millions posts not annotated

The sample represented 19.3 million unique posters aged 13-64 (M=29.9)
Technologies associées: Facebook and statistical analysis software.
Mention de l'éthique: Oui?
Finalité communicationnelle: The study found that people share more positive and negative emotions when their friend networks are smaller and denser, and that hearing about a friend's troubles on Facebook leads to more emotional and supportive comments. Posts with positive feelings receive more likes and positive language in comments.

Collections: Facebook

Auteur-es: Gregoire Mesnil, Tomas Mikolov, Marc'Aurelio Ranzato, Yoshua Bengio
Nombre Auteurs: 4
Titre: Ensemble of Generative and Discriminative Techniques for Sentiment Analysis of Movie Reviews
Année de publication: 2015
Référence (APA): Mesnil, G., Mikolov, T., Ranzato, M., & Bengio, Y. (2015). Ensemble of Generative and Discriminative Techniques for Sentiment Analysis of Movie Reviews (arXiv:1412.5335). arXiv. http://arxiv.org/abs/1412.5335
résumé: Sentiment analysis is a common task in natural language processing that aims to detect polarity of a text document (typically a consumer review). In the simplest settings, we discriminate only between positive and negative sentiment, turning the task into a standard binary classification problem. We compare several machine learning approaches to this problem, and combine them to achieve a new state of the art. We show how to use for this task the standard generative language models, which are slightly complementary to the state of the art techniques. We achieve strong results on a well-known dataset of IMDB movie reviews. Our results are easily reproducible, as we publish also the code needed to repeat the experiments. This should simplify further advance of the state of the art, as other researchers can combine their techniques with ours with little effort.
URL: https://research.facebook.com/file/167078822167214/ensemble-of-generative-and-discriminative-techniques-for-sentiment-analysis-of-movie-reviews.pdf
doi: https://doi.org/10.48550/arXiv.1412.5335
Accessibilité de l'article: Libre
Champ: Artificial Intelligence
Type contenu (théorique Applicative méthodologique): Applicatif
Méthode: The method involves comparing several machine learning approaches to sentiment analysis, including generative language models, and combining them to achieve a new state of the art.
Cas d'usage: IMDB
Objectifs de l'article: The objectives of the article are to improve the accuracy of sentiment analysis by combining several machine learning approaches. It aims to provide a reproducible method for achieving state-of-the-art result
Question(s) de recherche/Hypothèses/conclusion: The research question is how to improve the accuracy of sentiment analysis using machine learning approaches.; The hypothesis is that combining different machine learning approaches, including generative language models, will lead to improved accuracy in sentiment analysis.; The conclusions are that the proposed ensemble method achieves state-of-the-art results on a well-known dataset of movie reviews and that the code provided by the authors allows for easy reproducibility of their results.
Cadre théorique/Auteur.es: The theoretical framework of the article includes machine learning and natural language processing, and the main authors cited include Pang and Lee, Pascanu, Mikolov, Bengio, Socher, Pennington, Huang, Ng, Manning, and Stolcke.
Concepts clés: Sentiment analysis, Machine learning, Generative models, Discriminative models, Ensemble methods, Natural language processing
Données collectées (type source): The article uses the Stanford IMDB dataset of movie reviews.
Définition des émotions: Non
Ampleur expérimentation (volume de comptes): 25 000 IMDB reviews for training

25 000 reviews for testing
Technologies associées: Machine learning algorithms, Natural language processing techniques, Generative language models.
Mention de l'éthique: Non
Finalité communicationnelle: "combining both generative and discriminative models together for sentiment prediction [...] one based on a generative approach (language models), one based on continuous representations of sentences and one based on a clever reweighing of tf-idf bag-of-word representation of the document." give better results.

Collections: Facebook

Auteur-es: Yi-Chia Wang, Moira Burke, Robert Kraut
Nombre Auteurs: 3
Titre: Modeling Self-Disclosure in Social Networking Sites
Année de publication: 2016
Référence (APA): Wang, Y.-C., Burke, M., & Kraut, R. (2016). Modeling Self-Disclosure in Social Networking Sites. https://doi.org/10.1145/2818048.2820010
résumé: Social networking sites (SNSs) offer users a platform to build and maintain social connections. Understanding when people feel comfortable sharing information about themselves on SNSs is critical to a good user experience, because self-disclosure helps maintain friendships and increase relationship closeness. This observational research develops a machine learning model to measure self-disclosure in SNSs and uses it to understand the contexts where it is higher or lower. Features include emotional valence, social distance between the poster and people mentioned in the post, the language similarity between the post and the community and post topic. To validate the model and advance our understanding about online self- disclosure, we applied it to de-identified, aggregated status updates from Facebook users. Results show that women self-disclose more than men. People with a stronger desire to manage impressions self-disclose less. Network size is negatively associated with self-disclosure, while tie strength and network density are positively associated.
Mots-clés: Social networking sites; Facebook; computer-mediated
communication; self-disclosure; personality; audience;
context collapse; natural language analysis; applied
machine learning
URL: https://research.facebook.com/file/271555647830591/modeling_self-disclosure_in_social_networking_sites.pdf
doi: https://doi.org/10.1145/2818048.2820010
Accessibilité de l'article: Libre
Champ: Data Science, Human Computer Interaction & UX
Type contenu (théorique Applicative méthodologique): Méthodologique
Méthode: The authors developed a machine learning model to measure self-disclosure and applied it to de-identified, aggregated status updates from Facebook users to validate the model and advance our understanding about online self-disclosure. They also used Latent Dirichlet Allocation (LDA) to identify the topics common in status updates.

Status updates were annotated by users and by trained judges.
Cas d'usage: Facebook
Objectifs de l'article: The objectives of the article are to develop a machine learning model to measure self-disclosure on social networking sites, to validate the model using deidentified, aggregated status updates from Facebook users, and to understand the contexts where self-disclosure is higher or lower.
Question(s) de recherche/Hypothèses/conclusion: The research question is how can we measure self-disclosure on social networking sites using machine learning?; The hypothesis are :
H1: Individuals with a stronger desire for impression management will self-disclose less.
H2: Women will self-disclose more than men.
H3: Network size will be negatively correlated with selfdisclosure.
H4: Average tie strength will be positively correlated with self-disclosure.
H5: Network density will be positively correlated with selfdisclosure; The authors conclude that their machine learning model was able to accurately measure self-disclosure on Facebook and that self-disclosure was higher for certain topics, such as politics and memorial, and lower for others, such as Christianity and deep thoughts.
Cadre théorique/Auteur.es: The theoretical framework of the article includes social penetration theory and communication privacy management theory. The main authors cited include Altman and Taylor, Derlega and Chaikin, and Petronio.
Concepts clés: Self-disclosure, Social networking sites, Machine learning
Données collectées (type source): The authors collected de-identified, aggregated status updates from Facebook users (recruited on AMT). They used dataset from the myPersonality project. The dataset contains users’ status updates as well as their demographic information and self-report impression management scores
Définition des émotions: LIWC
Ampleur expérimentation (volume de comptes): 2000 Facebook status updates

Then a random sample of 412,470 English language Facebook active users for approximately one month in late 2014
Technologies associées: ML et Latent Dirichlet Allocation (LDA)
Mention de l'éthique: Non
Commentaires: Financed by the grants from National Science Foundation (IIS-0968485) and National Institute of Mental Health (R21 MH106880-01).

Collections: Facebook

Auteur-es: Santhosh Rajamanickam, Pushkar Mishra, Helen Yannakoudakis, Ekaterina Shutova
Nombre Auteurs: 4
Titre: Joint Modelling of Emotion and Abusive Language Detection
Année de publication: 2020
Référence (APA): Rajamanickam, S., Mishra, P., Yannakoudakis, H., & Shutova, E. (2020). Joint Modelling of Emotion and Abusive Language Detection. https://doi.org/10.18653/v1/2020.acl-main.394
résumé: The rise of online communication platforms has been accompanied by some undesirable effects, such as the proliferation of aggressive and abusive behaviour online. Aiming to tackle this problem, the natural language processing (NLP) community has experimented with a range of techniques for abuse detection. While achieving substantial success, these methods have so far only focused on modelling the linguistic properties of the comments and the online communities of users, disregarding the emotional state of the users and how this might affect their language. The latter is, however, inextricably linked to abusive behaviour. In this paper, we present the first joint model of emotion and abusive language detection, experimenting in a multi-task learning framework that allows one task to inform the other. Our results demonstrate that incorporating affective features leads to significant improvements in abuse detection performance across datasets.
Mots-clés: emotion detection, abusive language detection, natural language processing, multi-task learning, affective features
URL: https://research.facebook.com/file/763339231062964/Joint-Modelling-of-Emotion-and-Abusive-Language-Detection.pdf
doi: https://doi.org/10.18653/v1/2020.acl-main.394
Accessibilité de l'article: Libre
Champ: Artificial Intelligence, Natural Language Processing & Speech
Type contenu (théorique Applicative méthodologique): Méthodologique
Méthode: They propose a new approach that jointly learns to detect emotion and abuse, using a multi-task learning (MTL) framework that allows one task to inform the other.
Cas d'usage: Twitter
Objectifs de l'article: The main objectives of the article are to propose a new approach to abusive language detection that incorporates emotional features into the model via a multi-task learning framework, and to evaluate the effectiveness of this approach in improving abuse detection performance across datasets. The authors aim to address the limitations of existing methods, which have focused solely on modeling the linguistic properties of comments and online communities, and have not taken into account the emotional state of users and how this might affect their language. By developing a joint model of emotion and abusive language detection, the authors hope to provide a more comprehensive and accurate approach to identifying abusive behavior online.
Question(s) de recherche/Hypothèses/conclusion: Research question is whether incorporating affective features into a joint model of emotion and abusive language detection can improve abuse detection performance across datasets.; Hypothesis : Multi-task learning framework that allows one task to inform the other will lead to better detection of abusive language, as the model will be able to leverage the complementary information provided by the two tasks.; Conclusions : incorporating affective features into a joint model of emotion and abusive language detection leads to significant improvements in abuse detection performance across datasets. the emotional state of users is indeed linked to abusive behavior, and that jointly modeling emotion and abusive language detection can provide a more comprehensive and accurate approach to identifying abusive behavior online. The authors suggest that their approach could be extended to other complex semantic tasks, such as figurative language processing and inference, to further improve abuse detection performance.
Cadre théorique/Auteur.es: Emotion with Ekman, Natural language processing (NLP) and machine learning techniques, specifically deep learning models such as recurrent neural networks (RNNs) and convolutional neural networks (CNNs). Matthew E Peters et al. ; Sara Owsley Sood et al.

Theories of emotion and abusive behavior. Pushkar Mishra, Marco Del Tredici, Helen Yannakoudakis, and Ekaterina Shutova
Concepts clés: Abusive language detection, Affective features
Données collectées (type source): datasets related to abusive language and emotion detection from Twitter.

Abuse detection task : annotated tweets from OffensEval 2019 and Waseem and Hovy 2016.

Emotion detection task : SemEval-2018
Définition des émotions: Ekman
Ampleur expérimentation (volume de comptes): OffensEval : 13240 tweets

Waseem and Hovy : 16202 tweets

SemEval18 : 11000 tweets
Technologies associées: NLP, MTL, Classifiers, Convolutional neural networks (RNNs and CNNs), character-based models, and graph-based learning methods
Mention de l'éthique: Non
Finalité communicationnelle: "Overall, our results also suggest the superiority of MTL over STL for abuse detection. With this new approach, one can build more complex models introducing new auxiliary tasks for abuse detection. For instance, we expect that abuse detection may also benefit from joint learning with complex semantic tasks, such as figurative language processing and inference."

Make digital spaces safer ?

Collections: Facebook

Auteur-es: Ali Israr, Freddy Abnousi
Nombre Auteurs: 2
Titre: Towards Pleasant Touch: Vibrotactile Grids for Social Touch Interactions
Année de publication: 2018
Référence (APA): Israr, A., & Abnousi, F. (2018). Towards Pleasant Touch : Vibrotactile Grids for Social Touch Interactions. https://doi.org/10.1145/3170427.3188546
résumé: In this paper, we realize a wearable tactile device that delivers smooth pleasant strokes, those resemble of caressing and calming sensations, on the forearm. In Study 1, we develop a psychophysical model of continuous illusory motion on a discrete vibrotactile array. We use this model to generate a variety of tactile strokes that vary in frequency (quality), amplitude (strength) and duration (speed), and test them on a hedonic scale of pleasant-unpleasant in Study 2. Our results show that low frequency (<40 Hz) strokes at low amplitude (light touch) are felt pleasant, while high frequency strokes are unpleasant. High amplitude strokes are biased towards unpleasantness. Our results are useful for artificial means to enhance social presence among individuals in virtual and augmented settings.
Mots-clés: Social touch, vibrotactile illusions, pleasant touch,
vibrotactile grid displays, wearable
URL: https://research.facebook.com/file/1147760666043554/towards-pleasant-touch_vibrotactile-grids-for-social-touch-interactions.pdf
doi: https://doi.org/10.1145/3170427.3188546
Accessibilité de l'article: Libre
Champ: AR/VR, Human Computer Interaction & UX
Type contenu (théorique Applicative méthodologique): Applicatif
Méthode: The method involves developing a wearable haptic device that delivers smooth and pleasant strokes on the forearm, resembling caressing and calming sensations. The authors develop a psychophysical model of continuous illusory motion on a discrete vibrotactile array and use it to generate a variety of tactile strokes that vary in frequency, amplitude, and duration.
Cas d'usage: N/A
Objectifs de l'article: The objectives of the article are to develop a wearable haptic device that can enhance social presence among individuals in virtual and augmented settings, and to investigate the psychophysical properties of vibrotactile grids for social touch interactions.
Question(s) de recherche/Hypothèses/conclusion: The research question is how can vibrotactile grids be used to enhance social presence among individuals in virtual and augmented settings?; The hypothesis is that low-frequency strokes at low amplitude are felt pleasant, while high-frequency and high-amplitude strokes are biased towards unpleasantness.; The conclusions are that vibrotactile grids can be used to enhance social presence among individuals in virtual and augmented settings, and that low-frequency strokes at low amplitude are felt pleasant, while high-frequency and high-amplitude strokes are biased towards unpleasantness.
Cadre théorique/Auteur.es: The theoretical framework of the article is based on recent research on successful communication of emotions through touch, particularly the work of Hertenstein et al. (2006) and Eid and Al Osman (2016).
Concepts clés: Vibrotactile grids, Wearable haptic devices, Social touch interactions
Données collectées (type source): All participants were employees. The data collected was psychophysical data from human participants using the vibrotactile grid mounted on the dorsal forearm.
Définition des émotions: Non
Ampleur expérimentation (volume de comptes): 6 participants
Technologies associées: Wearable haptic devices, Vibrotactile grids
Mention de l'éthique: Non
Finalité communicationnelle: "The present work can be used to generate a variety of tactile strokes and synchronize them with visuals and other contextual cues in human-to-human interactions. Current work is a first step to deliver predefined tactile patterns from a device to a human. Our future work includes exploration of real time human-to-human communication system, that senses real time interactions of one user and delivers them to the second user, who are physically apart."

Pour expériences VR/AR

Collections: Facebook

Auteur-es: Lauren Scissors, Moira Burke, Steve Wengrovitz
Nombre Auteurs: 3
Titre: What’s in a Like? Attitudes and Behaviors Around Receiving Likes on Facebook
Année de publication: 2016
Référence (APA): Scissors, M. B. L. (2016). What’s in a Like ? Attitudes and behaviors around receiving Likes on Facebook. CSCW ’.
résumé: What social value do Likes on Facebook hold? This research examines people’s attitudes and behaviors related to receiving one-click feedback in social media. Likes and other kinds of lightweight affirmation serve as social cues of acceptance and maintain interpersonal relationships, but may mean different things to different people. Through surveys and de-identified, aggregated behavioral Facebook data, we find that in general, people care more about who Likes their posts than how many Likes they receive, desiring feedback most from close friends, romantic partners, and family members other than their parents. While most people do not feel strongly that receiving “enough” Likes is important, roughly two-thirds of posters regularly receive more than “enough.” We also note a “Like paradox,” a phenomenon in which people’s friends receive more Likes because their friends have more friends to provide those Likes. Individuals with lower levels of self- esteem and higher levels of self-monitoring are more likely to think that Likes are important and to feel bad if they do not receive “enough” Likes. The results inform product design and our understanding of how lightweight interactions shape our experiences online.
Mots-clés: Social network sites; Facebook; Likes; Like paradox; selfesteem; self-monitoring
URL: https://research.facebook.com/file/814820045881507/what-s-in-a-like-attitudes-and-behaviors-around-receiving-likes-on-facebook.pdf
doi: http://dx.doi.org/10.1145/2818048.2820066
Accessibilité de l'article: Libre
Champ: Data Science, Human Computer Interaction & UX
Type contenu (théorique Applicative méthodologique): Théorique, méthodologique
Méthode: The method used in this article is a combination of surveys and behavioral Facebook data analysis.
Cas d'usage: Facebook
Objectifs de l'article: The objectives of the article are to explore the social value of Likes and other forms of lightweight affirmation on social media, to uncover a "Like paradox," and to examine how self-esteem and self-monitoring play a role in one's perception of Likes.
Question(s) de recherche/Hypothèses/conclusion: The research question is: What social value do Likes on Facebook hold?; The conclusions of the article are that Likes on Facebook are important to people, but the value of Likes is influenced by factors such as self-esteem and self-monitoring. The study also found a "Like paradox" where people who receive more Likes may actually feel worse about themselves
Cadre théorique/Auteur.es: The theoretical framework of the article includes concepts from social psychology and communication studies. The main authors cited include D. W. Johnson, R. T. Johnson, and E. J. W. Johnson.
Concepts clés: Likes, Feedback, Self-esteem, Self-monitoring, "Like paradox."
Données collectées (type source): "Participants filled out an online survey with two sections: (1) attitudes and behaviors about Likes on Facebook and (2) personal characteristics"
All data was de-identified and analyzed in aggregate by Facebook employees. Behavioral data included the number of posts, number of Likes and comments given and received, and the average number of Likes and comments received per post across participants’ 970,135 friends.
Définition des émotions: Non
Ampleur expérimentation (volume de comptes): Survey participants : 2109
Technologies associées: Survey software
Mention de l'éthique: Non
Finalité communicationnelle: "This work demonstrates that lightweight feedback, despite requiring little effort to produce, is important to social media users. We found that many people care more about whom they get feedback from, rather than the exact amount of feedback received. In addition, individual traits like selfesteem and self-monitoring influence people’s attitudes toward lightweight feedback."

Collections: Facebook

Auteur-es: Moira Burke and Mike Develin
Nombre Auteurs: 2
Titre: Once More with Feeling: Supportive Responses to Social Sharing on Facebook
Année de publication: 2016
Référence (APA): Burke, M., & Develin, M. (2016). Once More with Feeling : Supportive Responses to Social Sharing on Facebook. Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing, 1462‑1474. https://doi.org/10.1145/2818048.2835199
résumé: Life is more than cat pictures. There are tough days,heartbreak, and hugs. Under what contexts do people share these feelings online, and how do their friends respond? Using millions of de-identified Facebook status updates with poster-annotated feelings (e.g., feeling thankful or feeling worried), we examine the magnitude and circumstances in which people share positive or negative feelings and characterize the nature of the responses they receive. We find that people share greater proportions of both positive and negative emotions when their friend networks are smaller and denser. Consistent with social sharing theory, hearing about a friend’s troubles on Facebook causes friends to reply with more emotional and supportive comments. Friends’ comments are also more numerous and longer. Posts with positive feelings, on the other hand, receive more likes, and their comments have more positive language. Feelings that relate to the poster’s self worth, such as feeling defeated, feeling unloved, or feeling accomplished amplify these effects.
Mots-clés: Emotional support; social network sites; social sharing
URL: https://research.facebook.com/file/373780360921794/once-more-with-feeling-supportive-responses-to-social-sharing-on-facebook.pdf
doi: https://doi.org/10.1145/2818048.2835199
Accessibilité de l'article: Libre
Champ: Data Science, Human Computer Interaction, & UX
Type contenu (théorique Applicative méthodologique): Théorique
Méthode: S1 : Two judges independently coded the top 200
feelings (which comprised 90% of feelings usage) on a 3-point valence scale (positive, ambiguous, or negative). Inter-
rater reliability was good (Cohen’s kappa = 0.77).

S2 : As in Study 1, the 14.2 million posts with feeling annotations were categorized according to the valence and self-relevance of the emotion.
Cas d'usage: Facebook
Objectifs de l'article: The objectives of the article are to explore the ways in which people share their emotions on Facebook and how their friends respond, and to examine the relationship between characteristics of the emotions shared and characteristics of the responses.

"The present research is the first very large-scale quantitative
study of social sharing of feelings in Facebook status
updates, examining (1) how network properties (size and
density) are associated with the valence of emotions shared, and (2) how characteristics of the emotions shared (valence and self-relevance) relate to how the audience responds (quantity of responses, emotional and supportive content in comments, and whether the responses come in network-visible or private channels)."
Question(s) de recherche/Hypothèses/conclusion: Research question : Do people share different kinds of emotions online depending on whom they think is listening?
Does social sharing of emotion online elicit emotion in viewers, and what form do their responses take?; The hypothesis is that people share more positive and negative emotions when their friend networks are smaller and denser, and that hearing about a friend's troubles on Facebook leads to more emotional and supportive comments.; The conclusion are that "the research demonstrates that when people have smaller, denser networks, they share more positive and negative emotions, and friends respond in greater volume to posts with emotion,
especially negative emotion. Responses to posts with negative emotion are more emotional themselves and contain more supportive language; more extreme expressions of negative self-worth (e.g., feeling unloved)
often elicit private responses from friends."
Cadre théorique/Auteur.es: The theoretical framework of the article is based on psychology (Pennebaker), on previous research on social sharing and emotional support, including studies by Rimé, Reis, and Shaver. The authors also draw on theories of social capital and social network analysis.
Concepts clés: Social sharing, Emotional valence, Social capital, Social network analysis
Données collectées (type source): Large dataset of de-identified text-based status updates on Facebook's servers. One annotated and the other one without feeling annotations (as a baseline).

"We analyzed network properties and status updates by a random sample of people who used the feeling annotation tool. N=1,399,921 English speakers in the U.S.
(Mean age = 32.2, SD=10.9, 79% female) who posted at least one status update in June 2015 that included an emotion (using the annotation widget in Figure 1), had been Facebook users for at least six months, and had at least 20 friends, were randomly selected. Their status updates for one month (approximately 30 million) and friend networks were included in the analysis."

S2 : N=31.7 million de-identified text-based status updates, comprised of a random sample of 14.2 million posts with attached feeling annotations and a random sample of 17.5 million without feeling annotation.
Définition des émotions: Non
Ampleur expérimentation (volume de comptes): 14,5 millions posts annotated

17,5 millions posts not annotated

The sample represented 19.3 million unique posters aged 13-64 (M=29.9)
Technologies associées: Facebook and statistical analysis software.
Mention de l'éthique: Oui?
Finalité communicationnelle: The study found that people share more positive and negative emotions when their friend networks are smaller and denser, and that hearing about a friend's troubles on Facebook leads to more emotional and supportive comments. Posts with positive feelings receive more likes and positive language in comments.

Collections: Facebook

Auteur-es: Felix Kreuk, Adam Polyak, Jade Copet, Eugene Kharitonov, Tu-Anh Nguyen, Morgane Rivière, Wei-Ning Hsu, Emmanuel Dupoux, Yossi Adi
Nombre Auteurs: 9
Titre: Textless Speech Emotion Conversion using Discrete & Decomposed Representations
Année de publication: 2022
Référence (APA): Kreuk, F., Polyak, A., Copet, J., Kharitonov, E., Nguyen, T.-A., Rivière, M., Hsu, W.-N., Mohamed, A., Dupoux, E., & Adi, Y. (2022). Textless Speech Emotion Conversion using Discrete & Decomposed Representations.
résumé: Speech emotion conversion is the task of modifying the perceived emotion of a speech utterance while preserving the lexical content and speaker identity. In this study, we cast the problem of emotion conversion as a spoken language translation task. We use a decomposition of the speech signal into discrete learned representations, consisting of phonetic-content units, prosodic features, speaker, and emotion. First, we modify the speech content by translating the phonetic-content units to a target emotion, and then predict the prosodic features based on these units. Finally, the speech waveform is generated by feeding the predicted representations into a neural vocoder. Such a paradigm allows us to go beyond spectral and parametric changes of the signal, and model non-verbal vocalizations, such as laughter insertion, yawning removal, etc. We demonstrate objectively and subjectively that the proposed method is vastly superior to current approaches and even beats text-based systems in terms of perceived emotion and audio quality. We rigorously evaluate all components of such a complex system and conclude with an extensive model analysis and ablation study to better emphasize the architectural choices, strengths and weaknesses of the proposed method. Samples are available under the following link here.
URL: https://research.facebook.com/file/509033718007989/Textless-Speech-Emotion-Conversion-using-Discrete-and-Decomposed-Representations.pdf
doi: https://doi.org/10.48550/arXiv.2111.07402
Accessibilité de l'article: Libre
Champ: Natural Language Processing & Speech
Type contenu (théorique Applicative méthodologique): Méthodologique, applicatif
Méthode: The method allows for the modification of non-verbal vocalizations such as laughter insertion and yawning removal while preserving the lexical content and speaker identity.
Cas d'usage: N/A
Objectifs de l'article: The study presents a novel approach to modifying the perceived emotion of a speech utterance while preserving the lexical content and speaker identity. The study also includes a rigorous evaluation of the proposed method and compares it against current approaches. The proposed method has been rigorously evaluated and has been found to be superior to current approaches in terms of perceived emotion and audio quality.
Question(s) de recherche/Hypothèses/conclusion: Research question is How to modify the perceived emotion of a speech utterance while preserving the lexical content and speaker identity ?; The hypothesis is that "modifying the learned representations of a speech signal can effectively change the perceived emotion of the speech while preserving the lexical content and speaker identity."; The concusions are "The researchers found a way to break down the voice into different parts and change them to make the voice sound happy, sad, angry, or other emotions. They tested this new way and found that it works better than other ways that people have tried before. They also found that changing the non-verbal parts of the voice, like laughter or sighs, can make a big difference in how the voice sounds. Overall, this new way of changing someone's voice is really cool and could be useful in things like movies or video games where characters need to sound different emotions."
Cadre théorique/Auteur.es: Speech processing, natural language processing : Vaswani et al. (2017); Polyak et al. (2021); Kharitonov et al. (2021a); Schuller et al. (2013); Fan et al. (2014); Wang et al. (2018)
Concepts clés: Speech emotion conversion, Non-verbal communication cues, Spoken language translation
Données collectées (type source): Dataset of emotional speech recordings, which were labeled with six different emotions (Neutral, Happy, Sad, Angry, Disgusted, and Sleepy)
Définition des émotions: Categorical emotions
Ampleur expérimentation (volume de comptes): 7000 speech utterances based on transcripts from the CMU Arctic Database
Technologies associées: HuBERT: A pre-trained neural network model for speech processing, which is used as a feature extractor in the emotion conversion model.

Transformer: A type of neural network architecture that uses self-attention mechanisms to process sequential data, which is used as the main architecture for the emotion conversion model.

PyTorch: A popular open-source machine learning framework, which is used to implement the emotion conversion model and train it on the emotional speech dataset.
Mention de l'éthique: Non
Finalité communicationnelle: "We demonstrated how the proposed system is able to model expressive non-verbal vocalizations as well as generate high- quality expressive speech. We conclude with an ablation study and analysis of the different components composing our system. This study serves as the foundation for improving speech emotion conversion and building general textless expressive speech generation models."

Collections: Facebook

Auteur-es: Abhinav Shukla, Stavros Petridis, Maja Pantic
Nombre Auteurs: 3
Titre: Does Visual Self-Supervision Improve Learning
of Speech Representations for Emotion
Recognition?
Année de publication: 2021
Référence (APA): Shukla, A., Petridis, S., & Pantic, M. (2021). Does Visual Self-Supervision Improve Learning of Speech Representations for Emotion Recognition? IEEE Transactions on Affective Computing, 14(1), 406‑420. https://doi.org/10.1109/TAFFC.2021.3062406
résumé: Self-supervised learning has attracted plenty of recent research interest. However, most works for self-supervision in speech are typically unimodal and there has been limited work that studies the interaction between audio and visual modalities for cross-modal self-supervision. This work (1) investigates visual self-supervision via face reconstruction to guide the learning of audio representations; (2) proposes an audio-only self-supervision approach for speech representation learning; (3) shows that a multi-task combination of the proposed visual and audio self-supervision is beneficial for learning richer features that are more robust in noisy conditions; (4) shows that self-supervised pretraining can outperform fully supervised training and is especially useful to prevent overfitting on smaller sized datasets. We evaluate our learned audio representations for discrete emotion recognition, continuous affect recognition and automatic speech recognition. We outperform existing self-supervised methods for all tested downstream tasks. Our results demonstrate the potential of visual self-supervision for audio feature learning and suggest that joint visual and audio self-supervision leads to more informative audio representations for speech and emotion recognition.
Mots-clés: Self-supervised learning, Representation learning, Generative modeling, Audiovisual speech, Emotion recognition,
Speech recognition, Cross-modal self-supervision.
URL: https://research.facebook.com/file/4493784684073249/Does-Visual-Self-Supervision-Improve-Learning-of-Speech-Representations-for-Emotion-Recognition.pdf
doi: https://doi.org/10.1109/TAFFC.2021.3062406
Accessibilité de l'article: Libre
Champ: Artificial Intelligence, Machine Learning
Type contenu (théorique Applicative méthodologique): Méthodologique, applicatif
Méthode: self-supervised learning (SSL).
The method proposed in this article is a multi-task approach that combines audio and visual modalities for self-supervised learning in speech representation. The approach involves using visual self-supervision via face reconstruction to guide the learning of audio representations. The authors evaluate the effectiveness of their approach on several tasks, including discrete emotion recognition, continuous affect recognition, and automatic speech recognition.
Cas d'usage: The three self-supervised methods we
compare against are CPC, APC, and PASE
Objectifs de l'article: The objectives of the article are to investigate the benefits of combining audio and visual modalities for self-supervised learning in speech representation, and to propose a multi-task approach that outperforms existing self-supervised methods for several tasks.

"we investigate self-supervised learning for audio representations."

"we examine the state-of-the-art in self-supervised audio feature learning which we use as baselines. We then propose a novel visual self-supervised method and a novel audio-only self-supervised method for learning audio features. We also show how visual self-supervision helps encode emotional information into the audio features."
Question(s) de recherche/Hypothèses/conclusion: The research question is "Can combining audio and visual modalities for self-supervised learning in speech representation improve performance on tasks such as discrete emotion recognition, continuous affect recognition, and automatic speech recognition?"; The hypothesis is that the proposed multi-task approach that combines audio and visual modalities for self-supervised learning in speech representation will outperform existing self-supervised methods for several tasks.; The authors conclude that their proposed multi-task approach that combines audio and visual modalities for self-supervised learning in speech representation outperforms existing self-supervised methods for several tasks, including discrete emotion recognition, continuous affect recognition, and automatic speech recognition.
"proposed visual self-supervision is superior when compared to the proposed audio-only self-supervision. The results on both discrete and continuous affect recognition offer evidence that the learned representation is good for emotion."

"The models trained using a combination of audio and visual self-supervision are able to encode complementary information from each modality to yield the best possible representations among all tested methods in this work."
Cadre théorique/Auteur.es: The theoretical framework of the article is based on previous work in self-supervised learning and multimodal learning. The main authors cited include Hjelm et al., 2018; Owens et al., 2018; and Arandjelovic and Zisserman, 2017.
Concepts clés: Discrete Emotion Recognition, Continuous Affect Recognition, Automatic Speech Recognition.
Données collectées (type source): CREMA-D dataset : 91 actors who utter 12 sentences multiples times each with a
different level of intensity for each of 6 basic emotional
labels (anger, fear, disgust, neutral, happy, sad).

The Ravdess dataset contains 1440 samples of 24 different actors who acted out two sentences with 8 different basic emotions (anger, calm, sad, neutral, happy, disgusted, surprised, fear) and two different intensity levels.

The RECOLA dataset contains dyadic conversations in French between a pair of participants working to solve a collaborative task over video conference. The annotated part of the dataset consists of 5-minute long clips (with audio from one speaker only) with continuous valence and arousal annotations from 6 annotators.

The SEWA dataset [58] contains dyadic conversations over video conference between a pair of participants discussing about an advertisement that they have just watched. The audio clips are typically 3-minute long and have continuous valence and arousal annotations.

The IEMOCAP dataset [59] contains dyadic conversations between 10 speakers for a total of 12 hours of audiovisual data. The discrete emotion labels comprise of 8
categories (anger, happiness, sadness, neutral, excitement, frustration, fear, surprise), however we only consider the first 4 categories for our experiments (anger, happiness,
sadness, neutral

The SPC (Speech Commands) dataset contains 64,727 total utterances of 30 different words by 1,881 speakers. We use SPC as a speech recognition evaluation dataset.
The LRW dataset is a large, in-the-wild dataset of 500 different isolated words primarily from BBC recordings. It is an audiovisual speech dataset and is thus appropriate for training our methods.
Définition des émotions: Categorical emotions
Ampleur expérimentation (volume de comptes): 1440 samples of 24 different actors

12 hours of au-
diovisual data

64,727 total utterances of 30 different words by 1,881 speakers

500 different isolated words
Technologies associées: Self-supervised learning, Audio-visual modalities, Speech representation, Emotion recognition, Affect recognition, Automatic speech recognition.
Mention de l'éthique: Non
Finalité communicationnelle: To propose "a multi-task approach that outperforms existing self-supervised methods for discrete emotion recognition, continuous affect recognition, and automatic speech recognition."

Collections: Facebook

Auteur-es: Fan Yang, Xiaochang Peng, Gargi Ghosh, Eider Moore, Goran Predovic
Nombre Auteurs: 5
Titre: Exploring Deep Multimodal Fusion of Text and Photo for Hate Speech Classification
Année de publication: 2019
résumé: Interactions among users on social network platforms are usually positive, constructive and insightful. However, sometimes people also get exposed to objectionable content such as hate speech, bullying, and verbal abuse etc. Most social platforms have explicit policy against hate speech because it creates an environment of intimidation and exclusion, and in some cases may promote real-world violence. As users’ interactions on today’s social networks involve multiple modalities, such as texts, images and videos, in this paper we explore the challenge of automatically identifying hate speech with deep multimodal technologies, extending previous research which mostly focuses on the text signal alone. We present a number of fusion approaches to integrate text and photo signals. We show that augmenting text with image embedding information immediately leads to a boost in performance, while applying additional attention fusion methods brings further improvement.
URL: https://research.facebook.com/file/2400404176758343/Exploring-Deep-Multimodal-Fusion-of-Text-and-Photo-for-Hate-Speech-Classification.pdf
doi: https://doi.org/10.18653/v1/W19-3502
Accessibilité de l'article: Libre
Champ: Artificial Intelligence, Natural Language Processing & Speech
Méthode: The method discussed in the provided text involves deep multimodal fusion of text and photo signals for the task of hate speech classification on social networks. The goal is to improve the identification of hate speech by integrating information from both text and image modalities. Various fusion techniques are explored, including simple concatenation, bilinear transformation, gated summation, and attention mechanisms.
Cas d'usage: N/A
Objectifs de l'article: The objectives of the article are to explore the challenge of automatically identifying hate speech using deep multimodal technologies, extend previous research focused on text signals alone, and provide insights into improving the detection of hate speech content on social platforms.
Question(s) de recherche/Hypothèses/conclusion: The research question revolves around how deep multimodal fusion of text and photo signals can enhance the automatic identification of hate speech on social networks.; The hypothesis is that by combining information from text and photo modalities through deep multimodal fusion techniques, the performance of hate speech classification can be significantly improved.; The conclusions highlight the effectiveness of augmenting text with image embedding information and applying attention fusion methods in boosting the performance of hate speech classification. The study demonstrates that fusion techniques like attention with deep cloning, sparsemax, and symmetric gate offer the best results in identifying hate speech content on social networks.
Cadre théorique/Auteur.es: The theoretical framework of the article involves deep multimodal fusion techniques for hate speech classification. The main authors cited in the text include Tong et al. (2017), Mishra et al. (2018), Gunasekara and Nejadgholi (2018), Kshirsagar et al. (2018), Magu and Luo (2018), and Sahlgren et al. (2018).
Concepts clés: Hate speech, Multimodal fusion, Text and photo signals
Données collectées (type source): Photos from social media platforms
Définition des émotions: Non
Ampleur expérimentation (volume de comptes): Not available
Technologies associées: Deep learning, Multimodal fusion techniques
Mention de l'éthique: Non
Finalité communicationnelle: The ultimate objective is to develop more effective methods for detecting hate speech.

Collections: Facebook

Auteur-es: Hadar Averbuch-Elor, Daniel Cohen-Or, Johannes Kopf, Michael Cohen
Nombre Auteurs: 4
Titre: Bringing Portraits to Life
Année de publication: 2017
Référence (APA): Averbuch-Elor, H., Cohen-Or, D., Kopf, J., & Cohen, M. F. (2017). Bringing portraits to life. ACM Transactions on Graphics, 36(6), 1‑13. https://doi.org/10.1145/3130800.3130818
résumé: We present a technique to automatically animate a still portrait, making it possible for the subject in the photo to come to life and express various emotions. We use a driving video (of a different subject) and develop means to transfer the expressiveness of the subject in the driving video to the target portrait. In contrast to previous work that requires an input video of the target face to reenact a facial performance, our technique uses only a single target image. We animate the target image through 2D warps that imitate the facial transformations in the driving video. As warps alone do not carry the full expressiveness of the face, we add fine-scale dynamic details which are commonly associated with facial expressions such as creases and wrinkles. Furthermore, we hallucinate regions that are hidden in the input target face, most notably in the inner mouth. Our technique gives rise to reactive profiles, where people in still images can automatically interact with their viewers. We demonstrate our technique operating on numerous still portraits from the internet.
Mots-clés: face animation, facial reenactment
URL: https://research.facebook.com/file/265001855227077/elor2017_bringingportraits-1.pdf
doi: https://doi.org/10.1145/3130800.3130818
Accessibilité de l'article: Libre
Champ: Computer Vision, Computational Photography & Intelligent Cameras
Type contenu (théorique Applicative méthodologique): Applicatif
Méthode: The method involves using a driving video and 2D warps to transfer the expressiveness of the driving subject to the target portrait, adding fine-scale dynamic details and hallucinating hidden regions. The method is designed to work in real-time and provide an interactive experience for the viewers.

"Our method takes a single target image of a neutral face in frontal pose and generates a video that expresses various emotions."

"We extract and track facial and non-facial features in the driving video (colored in red and yellow, respectively) and compute correspondences to the target image. To generate the animated target frames, we perform a 2D warping to generate a coarse target frame, followed by a transfer of hidden regions (i.e., the mouth interior) and fine-scale details."
Cas d'usage: N/A
Objectifs de l'article: The objective of the article is to "present a technique to automatically animate a still portrait, making it possible for the subject in the photo to come to life and express various
emotions and to demonstrate its effectiveness through a user study, and discuss its potential applications in various domains."
Question(s) de recherche/Hypothèses/conclusion: Research question is whether it is possible to animate still portraits and transfer the expressiveness of a driving subject to the target portrait in real-time and make it look real.; The hypothesis is that by using a driving video and 2D warps, it is possible to transfer the expressiveness of the driving subject to the target portrait and create a realistic and interactive animation.; The conclusions are that the proposed technique is effective in animating still portraits and transferring the expressiveness of a driving subject to the target portrait in real-time. The user study shows that the animations are perceived as realistic and engaging, and the technique has potential applications in various domains, such as entertainment, education, and communication.
Cadre théorique/Auteur.es: The theoretical framework of the article includes computer vision, facial expression analysis, and machine learning. The main authors cited include Kai Li, Feng Xu, Jue Wang, Qionghai Dai, Yebin Liu, Zicheng Liu, Ying Shan, Zhengyou Zhang, Iacopo Masi, Anh Tuan Tran, Jatuporn Toy Leksut, Tal Hassner, Gérard G. Medioni, and Maja Pantic
Concepts clés: Portrait animation, Facial expression synthesis, 2D warps, Emotion transfer
Données collectées (type source): "The participants were presented with 24 randomly selected videos,
eight of which are real. They were asked to rate them based on how
real the animation looks." - from very likely fake, likely fake, could equally be
real or fake, likely real to very likely real.
Définition des émotions: Categorical emotions
Ampleur expérimentation (volume de comptes): 24 videos rated by 30 users
Technologies associées: Computer vision, Facial expression analysis, Machine learning
Mention de l'éthique: Non
Finalité communicationnelle: The proposed technique has potential applications in various domains, such as entertainment, education, and communication.

Collections: Facebook

Auteur-es: Felix Kreuk, Adam Polyak, Jade Copet, Eugene Kharitonov, Tu-Anh Nguyen, Morgane Rivière, Wei-Ning Hsu, Emmanuel Dupoux, Yossi Adi
Nombre Auteurs: 9
Titre: Textless Speech Emotion Conversion using Discrete & Decomposed Representations
Année de publication: 2022
Référence (APA): Kreuk, F., Polyak, A., Copet, J., Kharitonov, E., Nguyen, T.-A., Rivière, M., Hsu, W.-N., Mohamed, A., Dupoux, E., & Adi, Y. (2022). Textless Speech Emotion Conversion using Discrete & Decomposed Representations.
résumé: Speech emotion conversion is the task of modifying the perceived emotion of a speech utterance while preserving the lexical content and speaker identity. In this study, we cast the problem of emotion conversion as a spoken language translation task. We use a decomposition of the speech signal into discrete learned representations, consisting of phonetic-content units, prosodic features, speaker, and emotion. First, we modify the speech content by translating the phonetic-content units to a target emotion, and then predict the prosodic features based on these units. Finally, the speech waveform is generated by feeding the predicted representations into a neural vocoder. Such a paradigm allows us to go beyond spectral and parametric changes of the signal, and model non-verbal vocalizations, such as laughter insertion, yawning removal, etc. We demonstrate objectively and subjectively that the proposed method is vastly superior to current approaches and even beats text-based systems in terms of perceived emotion and audio quality. We rigorously evaluate all components of such a complex system and conclude with an extensive model analysis and ablation study to better emphasize the architectural choices, strengths and weaknesses of the proposed method. Samples are available under the following link here.
URL: https://research.facebook.com/file/509033718007989/Textless-Speech-Emotion-Conversion-using-Discrete-and-Decomposed-Representations.pdf
doi: https://doi.org/10.48550/arXiv.2111.07402
Accessibilité de l'article: Libre
Champ: Natural Language Processing & Speech
Type contenu (théorique Applicative méthodologique): Méthodologique, applicatif
Méthode: The method allows for the modification of non-verbal vocalizations such as laughter insertion and yawning removal while preserving the lexical content and speaker identity.
Cas d'usage: N/A
Objectifs de l'article: The study presents a novel approach to modifying the perceived emotion of a speech utterance while preserving the lexical content and speaker identity. The study also includes a rigorous evaluation of the proposed method and compares it against current approaches. The proposed method has been rigorously evaluated and has been found to be superior to current approaches in terms of perceived emotion and audio quality.
Question(s) de recherche/Hypothèses/conclusion: Research question is How to modify the perceived emotion of a speech utterance while preserving the lexical content and speaker identity ?; The hypothesis is that "modifying the learned representations of a speech signal can effectively change the perceived emotion of the speech while preserving the lexical content and speaker identity."; The concusions are "The researchers found a way to break down the voice into different parts and change them to make the voice sound happy, sad, angry, or other emotions. They tested this new way and found that it works better than other ways that people have tried before. They also found that changing the non-verbal parts of the voice, like laughter or sighs, can make a big difference in how the voice sounds. Overall, this new way of changing someone's voice is really cool and could be useful in things like movies or video games where characters need to sound different emotions."
Cadre théorique/Auteur.es: Speech processing, natural language processing : Vaswani et al. (2017); Polyak et al. (2021); Kharitonov et al. (2021a); Schuller et al. (2013); Fan et al. (2014); Wang et al. (2018)
Concepts clés: Speech emotion conversion, Non-verbal communication cues, Spoken language translation
Données collectées (type source): Dataset of emotional speech recordings, which were labeled with six different emotions (Neutral, Happy, Sad, Angry, Disgusted, and Sleepy)
Définition des émotions: Categorical emotions
Ampleur expérimentation (volume de comptes): 7000 speech utterances based on transcripts from the CMU Arctic Database
Technologies associées: HuBERT: A pre-trained neural network model for speech processing, which is used as a feature extractor in the emotion conversion model.

Transformer: A type of neural network architecture that uses self-attention mechanisms to process sequential data, which is used as the main architecture for the emotion conversion model.

PyTorch: A popular open-source machine learning framework, which is used to implement the emotion conversion model and train it on the emotional speech dataset.
Mention de l'éthique: Non
Finalité communicationnelle: "We demonstrated how the proposed system is able to model expressive non-verbal vocalizations as well as generate high- quality expressive speech. We conclude with an ablation study and analysis of the different components composing our system. This study serves as the foundation for improving speech emotion conversion and building general textless expressive speech generation models."

Collections: Facebook

Auteur-es: Abhinav Shukla, Stavros Petridis, Maja Pantic
Nombre Auteurs: 3
Titre: Does Visual Self-Supervision Improve Learning
of Speech Representations for Emotion
Recognition?
Année de publication: 2021
Référence (APA): Shukla, A., Petridis, S., & Pantic, M. (2021). Does Visual Self-Supervision Improve Learning of Speech Representations for Emotion Recognition? IEEE Transactions on Affective Computing, 14(1), 406‑420. https://doi.org/10.1109/TAFFC.2021.3062406
résumé: Self-supervised learning has attracted plenty of recent research interest. However, most works for self-supervision in speech are typically unimodal and there has been limited work that studies the interaction between audio and visual modalities for cross-modal self-supervision. This work (1) investigates visual self-supervision via face reconstruction to guide the learning of audio representations; (2) proposes an audio-only self-supervision approach for speech representation learning; (3) shows that a multi-task combination of the proposed visual and audio self-supervision is beneficial for learning richer features that are more robust in noisy conditions; (4) shows that self-supervised pretraining can outperform fully supervised training and is especially useful to prevent overfitting on smaller sized datasets. We evaluate our learned audio representations for discrete emotion recognition, continuous affect recognition and automatic speech recognition. We outperform existing self-supervised methods for all tested downstream tasks. Our results demonstrate the potential of visual self-supervision for audio feature learning and suggest that joint visual and audio self-supervision leads to more informative audio representations for speech and emotion recognition.
Mots-clés: Self-supervised learning, Representation learning, Generative modeling, Audiovisual speech, Emotion recognition,
Speech recognition, Cross-modal self-supervision.
URL: https://research.facebook.com/file/4493784684073249/Does-Visual-Self-Supervision-Improve-Learning-of-Speech-Representations-for-Emotion-Recognition.pdf
doi: https://doi.org/10.1109/TAFFC.2021.3062406
Accessibilité de l'article: Libre
Champ: Artificial Intelligence, Machine Learning
Type contenu (théorique Applicative méthodologique): Méthodologique, applicatif
Méthode: self-supervised learning (SSL).
The method proposed in this article is a multi-task approach that combines audio and visual modalities for self-supervised learning in speech representation. The approach involves using visual self-supervision via face reconstruction to guide the learning of audio representations. The authors evaluate the effectiveness of their approach on several tasks, including discrete emotion recognition, continuous affect recognition, and automatic speech recognition.
Cas d'usage: The three self-supervised methods we
compare against are CPC, APC, and PASE
Objectifs de l'article: The objectives of the article are to investigate the benefits of combining audio and visual modalities for self-supervised learning in speech representation, and to propose a multi-task approach that outperforms existing self-supervised methods for several tasks.

"we investigate self-supervised learning for audio representations."

"we examine the state-of-the-art in self-supervised audio feature learning which we use as baselines. We then propose a novel visual self-supervised method and a novel audio-only self-supervised method for learning audio features. We also show how visual self-supervision helps encode emotional information into the audio features."
Question(s) de recherche/Hypothèses/conclusion: The research question is "Can combining audio and visual modalities for self-supervised learning in speech representation improve performance on tasks such as discrete emotion recognition, continuous affect recognition, and automatic speech recognition?"; The hypothesis is that the proposed multi-task approach that combines audio and visual modalities for self-supervised learning in speech representation will outperform existing self-supervised methods for several tasks.; The authors conclude that their proposed multi-task approach that combines audio and visual modalities for self-supervised learning in speech representation outperforms existing self-supervised methods for several tasks, including discrete emotion recognition, continuous affect recognition, and automatic speech recognition.
"proposed visual self-supervision is superior when compared to the proposed audio-only self-supervision. The results on both discrete and continuous affect recognition offer evidence that the learned representation is good for emotion."

"The models trained using a combination of audio and visual self-supervision are able to encode complementary information from each modality to yield the best possible representations among all tested methods in this work."
Cadre théorique/Auteur.es: The theoretical framework of the article is based on previous work in self-supervised learning and multimodal learning. The main authors cited include Hjelm et al., 2018; Owens et al., 2018; and Arandjelovic and Zisserman, 2017.
Concepts clés: Discrete Emotion Recognition, Continuous Affect Recognition, Automatic Speech Recognition.
Données collectées (type source): CREMA-D dataset : 91 actors who utter 12 sentences multiples times each with a
different level of intensity for each of 6 basic emotional
labels (anger, fear, disgust, neutral, happy, sad).

The Ravdess dataset contains 1440 samples of 24 different actors who acted out two sentences with 8 different basic emotions (anger, calm, sad, neutral, happy, disgusted, surprised, fear) and two different intensity levels.

The RECOLA dataset contains dyadic conversations in French between a pair of participants working to solve a collaborative task over video conference. The annotated part of the dataset consists of 5-minute long clips (with audio from one speaker only) with continuous valence and arousal annotations from 6 annotators.

The SEWA dataset [58] contains dyadic conversations over video conference between a pair of participants discussing about an advertisement that they have just watched. The audio clips are typically 3-minute long and have continuous valence and arousal annotations.

The IEMOCAP dataset [59] contains dyadic conversations between 10 speakers for a total of 12 hours of audiovisual data. The discrete emotion labels comprise of 8
categories (anger, happiness, sadness, neutral, excitement, frustration, fear, surprise), however we only consider the first 4 categories for our experiments (anger, happiness,
sadness, neutral

The SPC (Speech Commands) dataset contains 64,727 total utterances of 30 different words by 1,881 speakers. We use SPC as a speech recognition evaluation dataset.
The LRW dataset is a large, in-the-wild dataset of 500 different isolated words primarily from BBC recordings. It is an audiovisual speech dataset and is thus appropriate for training our methods.
Définition des émotions: Categorical emotions
Ampleur expérimentation (volume de comptes): 1440 samples of 24 different actors

12 hours of au-
diovisual data

64,727 total utterances of 30 different words by 1,881 speakers

500 different isolated words
Technologies associées: Self-supervised learning, Audio-visual modalities, Speech representation, Emotion recognition, Affect recognition, Automatic speech recognition.
Mention de l'éthique: Non
Finalité communicationnelle: To propose "a multi-task approach that outperforms existing self-supervised methods for discrete emotion recognition, continuous affect recognition, and automatic speech recognition."

Collections: Facebook

Auteur-es: Santhosh Rajamanickam, Pushkar Mishra, Helen Yannakoudakis, Ekaterina Shutova
Nombre Auteurs: 4
Titre: Joint Modelling of Emotion and Abusive Language Detection
Année de publication: 2020
Référence (APA): Rajamanickam, S., Mishra, P., Yannakoudakis, H., & Shutova, E. (2020). Joint Modelling of Emotion and Abusive Language Detection. https://doi.org/10.18653/v1/2020.acl-main.394
résumé: The rise of online communication platforms has been accompanied by some undesirable effects, such as the proliferation of aggressive and abusive behaviour online. Aiming to tackle this problem, the natural language processing (NLP) community has experimented with a range of techniques for abuse detection. While achieving substantial success, these methods have so far only focused on modelling the linguistic properties of the comments and the online communities of users, disregarding the emotional state of the users and how this might affect their language. The latter is, however, inextricably linked to abusive behaviour. In this paper, we present the first joint model of emotion and abusive language detection, experimenting in a multi-task learning framework that allows one task to inform the other. Our results demonstrate that incorporating affective features leads to significant improvements in abuse detection performance across datasets.
Mots-clés: emotion detection, abusive language detection, natural language processing, multi-task learning, affective features
URL: https://research.facebook.com/file/763339231062964/Joint-Modelling-of-Emotion-and-Abusive-Language-Detection.pdf
doi: https://doi.org/10.18653/v1/2020.acl-main.394
Accessibilité de l'article: Libre
Champ: Artificial Intelligence, Natural Language Processing & Speech
Type contenu (théorique Applicative méthodologique): Méthodologique
Méthode: They propose a new approach that jointly learns to detect emotion and abuse, using a multi-task learning (MTL) framework that allows one task to inform the other.
Cas d'usage: Twitter
Objectifs de l'article: The main objectives of the article are to propose a new approach to abusive language detection that incorporates emotional features into the model via a multi-task learning framework, and to evaluate the effectiveness of this approach in improving abuse detection performance across datasets. The authors aim to address the limitations of existing methods, which have focused solely on modeling the linguistic properties of comments and online communities, and have not taken into account the emotional state of users and how this might affect their language. By developing a joint model of emotion and abusive language detection, the authors hope to provide a more comprehensive and accurate approach to identifying abusive behavior online.
Question(s) de recherche/Hypothèses/conclusion: Research question is whether incorporating affective features into a joint model of emotion and abusive language detection can improve abuse detection performance across datasets.; Hypothesis : Multi-task learning framework that allows one task to inform the other will lead to better detection of abusive language, as the model will be able to leverage the complementary information provided by the two tasks.; Conclusions : incorporating affective features into a joint model of emotion and abusive language detection leads to significant improvements in abuse detection performance across datasets. the emotional state of users is indeed linked to abusive behavior, and that jointly modeling emotion and abusive language detection can provide a more comprehensive and accurate approach to identifying abusive behavior online. The authors suggest that their approach could be extended to other complex semantic tasks, such as figurative language processing and inference, to further improve abuse detection performance.
Cadre théorique/Auteur.es: Emotion with Ekman, Natural language processing (NLP) and machine learning techniques, specifically deep learning models such as recurrent neural networks (RNNs) and convolutional neural networks (CNNs). Matthew E Peters et al. ; Sara Owsley Sood et al.

Theories of emotion and abusive behavior. Pushkar Mishra, Marco Del Tredici, Helen Yannakoudakis, and Ekaterina Shutova
Concepts clés: Abusive language detection, Affective features
Données collectées (type source): datasets related to abusive language and emotion detection from Twitter.

Abuse detection task : annotated tweets from OffensEval 2019 and Waseem and Hovy 2016.

Emotion detection task : SemEval-2018
Définition des émotions: Ekman
Ampleur expérimentation (volume de comptes): OffensEval : 13240 tweets

Waseem and Hovy : 16202 tweets

SemEval18 : 11000 tweets
Technologies associées: NLP, MTL, Classifiers, Convolutional neural networks (RNNs and CNNs), character-based models, and graph-based learning methods
Mention de l'éthique: Non
Finalité communicationnelle: "Overall, our results also suggest the superiority of MTL over STL for abuse detection. With this new approach, one can build more complex models introducing new auxiliary tasks for abuse detection. For instance, we expect that abuse detection may also benefit from joint learning with complex semantic tasks, such as figurative language processing and inference."

Make digital spaces safer ?

Collections: Facebook

Auteur-es: Fan Yang, Xiaochang Peng, Gargi Ghosh, Eider Moore, Goran Predovic
Nombre Auteurs: 5
Titre: Exploring Deep Multimodal Fusion of Text and Photo for Hate Speech Classification
Année de publication: 2019
résumé: Interactions among users on social network platforms are usually positive, constructive and insightful. However, sometimes people also get exposed to objectionable content such as hate speech, bullying, and verbal abuse etc. Most social platforms have explicit policy against hate speech because it creates an environment of intimidation and exclusion, and in some cases may promote real-world violence. As users’ interactions on today’s social networks involve multiple modalities, such as texts, images and videos, in this paper we explore the challenge of automatically identifying hate speech with deep multimodal technologies, extending previous research which mostly focuses on the text signal alone. We present a number of fusion approaches to integrate text and photo signals. We show that augmenting text with image embedding information immediately leads to a boost in performance, while applying additional attention fusion methods brings further improvement.
URL: https://research.facebook.com/file/2400404176758343/Exploring-Deep-Multimodal-Fusion-of-Text-and-Photo-for-Hate-Speech-Classification.pdf
doi: https://doi.org/10.18653/v1/W19-3502
Accessibilité de l'article: Libre
Champ: Artificial Intelligence, Natural Language Processing & Speech
Méthode: The method discussed in the provided text involves deep multimodal fusion of text and photo signals for the task of hate speech classification on social networks. The goal is to improve the identification of hate speech by integrating information from both text and image modalities. Various fusion techniques are explored, including simple concatenation, bilinear transformation, gated summation, and attention mechanisms.
Cas d'usage: N/A
Objectifs de l'article: The objectives of the article are to explore the challenge of automatically identifying hate speech using deep multimodal technologies, extend previous research focused on text signals alone, and provide insights into improving the detection of hate speech content on social platforms.
Question(s) de recherche/Hypothèses/conclusion: The research question revolves around how deep multimodal fusion of text and photo signals can enhance the automatic identification of hate speech on social networks.; The hypothesis is that by combining information from text and photo modalities through deep multimodal fusion techniques, the performance of hate speech classification can be significantly improved.; The conclusions highlight the effectiveness of augmenting text with image embedding information and applying attention fusion methods in boosting the performance of hate speech classification. The study demonstrates that fusion techniques like attention with deep cloning, sparsemax, and symmetric gate offer the best results in identifying hate speech content on social networks.
Cadre théorique/Auteur.es: The theoretical framework of the article involves deep multimodal fusion techniques for hate speech classification. The main authors cited in the text include Tong et al. (2017), Mishra et al. (2018), Gunasekara and Nejadgholi (2018), Kshirsagar et al. (2018), Magu and Luo (2018), and Sahlgren et al. (2018).
Concepts clés: Hate speech, Multimodal fusion, Text and photo signals
Données collectées (type source): Photos from social media platforms
Définition des émotions: Non
Ampleur expérimentation (volume de comptes): Not available
Technologies associées: Deep learning, Multimodal fusion techniques
Mention de l'éthique: Non
Finalité communicationnelle: The ultimate objective is to develop more effective methods for detecting hate speech.

Collections: Facebook

Auteur-es: Ali Israr, Freddy Abnousi
Nombre Auteurs: 2
Titre: Towards Pleasant Touch: Vibrotactile Grids for Social Touch Interactions
Année de publication: 2018
Référence (APA): Israr, A., & Abnousi, F. (2018). Towards Pleasant Touch : Vibrotactile Grids for Social Touch Interactions. https://doi.org/10.1145/3170427.3188546
résumé: In this paper, we realize a wearable tactile device that delivers smooth pleasant strokes, those resemble of caressing and calming sensations, on the forearm. In Study 1, we develop a psychophysical model of continuous illusory motion on a discrete vibrotactile array. We use this model to generate a variety of tactile strokes that vary in frequency (quality), amplitude (strength) and duration (speed), and test them on a hedonic scale of pleasant-unpleasant in Study 2. Our results show that low frequency (<40 Hz) strokes at low amplitude (light touch) are felt pleasant, while high frequency strokes are unpleasant. High amplitude strokes are biased towards unpleasantness. Our results are useful for artificial means to enhance social presence among individuals in virtual and augmented settings.
Mots-clés: Social touch, vibrotactile illusions, pleasant touch,
vibrotactile grid displays, wearable
URL: https://research.facebook.com/file/1147760666043554/towards-pleasant-touch_vibrotactile-grids-for-social-touch-interactions.pdf
doi: https://doi.org/10.1145/3170427.3188546
Accessibilité de l'article: Libre
Champ: AR/VR, Human Computer Interaction & UX
Type contenu (théorique Applicative méthodologique): Applicatif
Méthode: The method involves developing a wearable haptic device that delivers smooth and pleasant strokes on the forearm, resembling caressing and calming sensations. The authors develop a psychophysical model of continuous illusory motion on a discrete vibrotactile array and use it to generate a variety of tactile strokes that vary in frequency, amplitude, and duration.
Cas d'usage: N/A
Objectifs de l'article: The objectives of the article are to develop a wearable haptic device that can enhance social presence among individuals in virtual and augmented settings, and to investigate the psychophysical properties of vibrotactile grids for social touch interactions.
Question(s) de recherche/Hypothèses/conclusion: The research question is how can vibrotactile grids be used to enhance social presence among individuals in virtual and augmented settings?; The hypothesis is that low-frequency strokes at low amplitude are felt pleasant, while high-frequency and high-amplitude strokes are biased towards unpleasantness.; The conclusions are that vibrotactile grids can be used to enhance social presence among individuals in virtual and augmented settings, and that low-frequency strokes at low amplitude are felt pleasant, while high-frequency and high-amplitude strokes are biased towards unpleasantness.
Cadre théorique/Auteur.es: The theoretical framework of the article is based on recent research on successful communication of emotions through touch, particularly the work of Hertenstein et al. (2006) and Eid and Al Osman (2016).
Concepts clés: Vibrotactile grids, Wearable haptic devices, Social touch interactions
Données collectées (type source): All participants were employees. The data collected was psychophysical data from human participants using the vibrotactile grid mounted on the dorsal forearm.
Définition des émotions: Non
Ampleur expérimentation (volume de comptes): 6 participants
Technologies associées: Wearable haptic devices, Vibrotactile grids
Mention de l'éthique: Non
Finalité communicationnelle: "The present work can be used to generate a variety of tactile strokes and synchronize them with visuals and other contextual cues in human-to-human interactions. Current work is a first step to deliver predefined tactile patterns from a device to a human. Our future work includes exploration of real time human-to-human communication system, that senses real time interactions of one user and delivers them to the second user, who are physically apart."

Pour expériences VR/AR

Collections: Facebook

Auteur-es: Jane Dwivedi-Yu, Alon Y. Halevy
Nombre Auteurs: 2
Titre: “That’s so cute!”: The CARE Dataset for Affective Response Detection
Année de publication: 2022
Référence (APA): Dwivedi-Yu, J., & Halevy, A. Y. (2022). “That’s so cute!” : The CARE Dataset for Affective Response Detection.
résumé: Social media plays an increasing role in our communication with friends and family, and in our consumption of entertainment and information. Hence, to design effective ranking functions for posts on social media, it would be useful to predict the affective responses of a post (e.g., whether it is likely to elicit feelings of entertainment, inspiration, or anger). Similar to work on emotion detection (which focuses on the affect of the publisher of the post), the traditional approach to recognizing affective response would involve an expensive investment in human annotation of training data.

We create and publicly release CAREdb, a dataset of 230k social media post annotations according to seven affective responses using the Common Affective Response Expression (CARE) method. The CARE method is a means of leveraging the signal that is present in comments that are posted in response to a post, providing high-precision evidence about the affective response to the post without human annotation. Unlike human annotation, the annotation process we describe here can be iterated upon to expand the coverage of the method, particularly for new affective responses. We present experiments that demonstrate that the CARE annotations compare favorably with crowdsourced annotations. Finally, we use CAREdb to train competitive BERT-based models for predicting affective response as well as emotion detection, demonstrating the utility of the dataset for related tasks.
Mots-clés: affective response, social media, emotion classification, CARE dataset, human annotation, machine learning
URL: https://research.facebook.com/file/393453116239728/Thats-so-cute--The-CARE-Dataset-for-Affective-Response-Detection.pdf
doi: https://doi.org/10.48550/arXiv.2201.11895
Accessibilité de l'article: Libre
Champ: Artificial Intelligence
Type contenu (théorique Applicative méthodologique): Méthodologique, applicatif
Méthode: The method used in this article is the Common Affective Response Expression (CARE) method, which is a cost-effective alternative to human annotation for predicting the emotional impact of social media posts. The CARE method involves two steps: (1) identifying the CARE patterns in the text, and (2) using a lexicon to label the publisher affect of the comments. The authors also compare the performance of the CARE method to a state-of-the-art emotion classifier.

Annotations according to the CARE method, as well as a human annotated dataset for comparison.
Cas d'usage: Reddit
Objectifs de l'article: The objectives of the article are to introduce the CARE dataset, evaluate its accuracy compared to crowdsourced annotations, and explore its potential applications in social media ranking functions.
Question(s) de recherche/Hypothèses/conclusion: The research question is whether the CARE dataset can accurately predict affective response in social media posts.; The hypothesis is that the CARE dataset will perform comparably to crowdsourced annotations in predicting affective response in social media posts.; The authors conclude that the CARE dataset is a valuable resource for predicting affective response in social media posts, and that it performs comparably to crowdsourced annotations. They also suggest that the CARE dataset could be used to improve social media ranking functions.
Cadre théorique/Auteur.es: The theoretical framework of the article is based on previous work in affective computing and emotion classification. The main authors cited include Stark and Hoey (2021), Demszky et al. (2020), Lin et al. (2008), Ma et al. (2005), and Mohammad and Kiritchenko (2015).
Concepts clés: Affective response, Social Media, Emotion classification, Machine learning
Données collectées (type source): Our experiments use a dataset that is created from Reddit posts and comments in the pushshift.io database that were created between 2011 and 2019.
Définition des émotions: Non
Ampleur expérimentation (volume de comptes): 34 million comments from
24 million distinct posts

TOTAL in CARE : 230k
Technologies associées: CARE method, machine learning algorithms, Emotion classifier
Mention de l'éthique: Yes, there is mention of ethics in this article. The authors note that the CARE method should not be used for ill-intended purposes such as purposefully recommending particular content to manipulate a user's perception or preferences.
Finalité communicationnelle: "We described a method for extracting training data for models that predict the affective responses of a post on social media"
Commentaires: Voir la bibliographie de l'article, plusieurs références qui pourraient entrer dans la RL.

Collections: Facebook

Auteur-es: Kiela Douwe, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, manpreet Singh, Pratik Ringshia, Davide Testuggine
Nombre Auteurs: 7
Titre: The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes
Année de publication: 2020
Référence (APA): Kiela, D., Firooz, H., Mohan, A., Goswami, V., Singh, A., Ringshia, P., & Testuggine, D. (2020). The Hateful Memes Challenge : Detecting Hate Speech in Multimodal Memes.
résumé: This work proposes a new challenge set for multimodal classification, focusing on detecting hate speech in multimodal memes. It is constructed such that unimodal models struggle and only multimodal models can succeed: difficult examples (“benign confounders”) are added to the dataset to make it hard to rely on unimodal signals. The task requires subtle reasoning, yet is straightforward to evaluate as a binary classification problem. We provide baseline performance numbers for unimodal models, as well as for multimodal models with various degrees of sophistication. We find that state-of-the-art methods perform poorly compared to humans, illustrating the difficulty of the task and highlighting the challenge that this important problem poses to the community.
URL: https://research.facebook.com/file/4239227162870648/The-Hateful-Memes-Challenge-Detecting-Hate-Speech-in-Multimodal-Memes.pdf
doi: https://doi.org/10.48550/arXiv.2005.04790
Accessibilité de l'article: Libre
Champ: Artificial Intelligence, Computer Vision, Machine Learning, Natural Language Processing & Speech
Type contenu (théorique Applicative méthodologique): Applicatif
Méthode: The method involves evaluating a variety of models, belonging to one of three classes: unimodal models, multimodal models that were unimodally pretrained, and multimodal models that were multimodally pretrained. Baseline scores are established for these models on the task of detecting hate speech in multimodal memes.
Cas d'usage: N/A
Objectifs de l'article: The objectives of the article are to propose a new challenge set for multimodal classification, to evaluate the performance of various models on this task, and to highlight the challenges and difficulties associated with detecting hate speech in multimodal memes.
Question(s) de recherche/Hypothèses/conclusion: The research question is how well can unimodal and multimodal models detect hate speech in multimodal memes?; The hypothesis is that performance relative to humans is still poor, indicating that there is a lot of room for improvement.; The conclusions are that the task of detecting hate speech in multimodal memes presents a challenge for both unimodal and multimodal models, and that there is a lot of room for improvement in this area.
Cadre théorique/Auteur.es: The theoretical framework of the article is not explicitly stated, but the main authors cited include Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Danna Gurari.
Concepts clés: Multimodal classification, Hate speech detection, Visual modality Natural language processing.
Données collectées (type source): Hateful memes
Définition des émotions: Non
Ampleur expérimentation (volume de comptes): Non renseigné
Technologies associées: Natural language processing, Computer vision, Machine learning.
Mention de l'éthique: Non

Collections: Facebook

Auteur-es: Moira Burke, Justin Cheng, Bethany de Gant
Nombre Auteurs: 3
Titre: Social Comparison and Facebook: Feedback, Positivity, and Opportunities for Comparison
Année de publication: 2020
Référence (APA): Burke, M., & Cheng, J. (2020). Social Comparison and Facebook:Feedback, Positivity, and Opportunities for Comparison.
résumé: People compare themselves to one another both offline and online. The specific online activities that worsen social comparison are partly understood, though much existing research relies on people recalling their own online activities post hoc and is situated in only a few countries. To better understand social comparison worldwide and the range of associated behaviors on social media, a survey of 38,000 people from 18 countries was paired with logged activity on Facebook for the prior month. People who reported more frequent social comparison spent more time on Facebook, had more friends, and saw proportionally more social content on the site. They also saw greater amounts of feedback on friends’ posts and proportionally more positivity. There was no evidence that social comparison happened more with acquaintances than close friends. One in five respondents recalled recently seeing a post that made them feel worse about themselves but reported conflicting views: half wished they hadn’t seen the post, while a third felt very happy for the poster. Design opportunities are discussed, including hiding feedback counts, filters for topics and people, and supporting meaningful interactions, so that when comparisons do occur, people are less affected by them.
Mots-clés: Social comparison; Facebook; envy; social media; well-being
URL: https://research.facebook.com/file/3068383903380990/Social-Comparison-and-Facebook-Feedback-Positivity-and-Opportunities-for-Comparison.pdf
doi: https://doi.org/10.1145/3313831.3376482
Accessibilité de l'article: Libre
Champ: Data Science, Human Computer Interaction & UX, Core Data Science
Type contenu (théorique Applicative méthodologique): Méthodologique
Méthode: The method involved surveying 37,729 people from 18 countries and analyzing their Facebook activity data from the prior month to understand the kinds of activities most strongly associated with social comparison.
Cas d'usage: Facebook
Objectifs de l'article: The objectives of the article were to gain a better understanding of how people compare themselves to others on social media and the impact it has on their well-being, and to provide design opportunities to help mitigate the negative effects of social comparison on Facebook users.
Question(s) de recherche/Hypothèses/conclusion: The research question was how do people compare themselves to others on Facebook and what are the consequences of this behavior on their well-being?; The hypothesis was that social comparison on Facebook would have a negative impact on users' well-being.; The conclusions were that social comparison on Facebook is associated with negative outcomes such as decreased life satisfaction and increased depressive symptoms, and that certain online activities such as viewing more feedback on others' posts are strongly associated with social comparison.
Cadre théorique/Auteur.es: The theoretical framework of the article is social comparison theory, and the main authors cited include Leon Festinger and Abraham Tesser.
Concepts clés: Social comparison, Well-being
Données collectées (type source): The type of data collected was survey responses and Facebook activity data
Définition des émotions: Non
Ampleur expérimentation (volume de comptes): 37,729 people from 18 countries
Technologies associées: Server log data.
Mention de l'éthique: Non

Collections: Facebook

Auteur-es: Shuster, K., Ju, D., Roller, S., Dinan, E., Boureau, Y.-L., & Weston, J.
Nombre Auteurs: 6
Titre: The Dialogue Dodecathlon: Open-Domain Knowledge and Image Grounded Conversational Agents
Année de publication: 2020
Référence (APA): Shuster, K., Ju, D., Roller, S., Dinan, E., Boureau, Y.-L., & Weston, J. (2020). The Dialogue Dodecathlon : Open-Domain Knowledge and Image Grounded Conversational Agents. https://doi.org/10.18653/v1/2020.acl-main.222
résumé: We introduce dodecaDialogue: a set of 12 tasks that measures if a conversational agent can communicate engagingly with personality and empathy, ask questions, answer questions by utilizing knowledge resources, discuss topics and situations, and perceive and converse about images. By multi-tasking on such a broad large-scale set of data, we hope to both move towards and measure progress in producing a single unified agent that can perceive, reason and converse with humans in an open-domain setting. We show that such multi-tasking improves over a BERT pretrained baseline, largely due to multi-tasking with very large dialogue datasets in a similar domain, and that the multi-tasking in general provides gains to both text and image-based tasks using several metrics in both the finetune and task transfer settings. We obtain stateof-the-art results on many of the tasks, providing a strong baseline for this challenge.
URL: https://research.facebook.com/file/556482462439112/The-Dialogue-Dodecathlon-Open-Domain-Knowledge-and-Image-Grounded-Conversational-Agents.pdf
doi: https://doi.org/10.18653/v1/2020.acl-main.222
Accessibilité de l'article: Libre
Champ: Artificial Intelligence, Natural Language Processing & Speech
Type contenu (théorique Applicative méthodologique): Méthodologique, applicatif
Méthode: Dialogue Dodecathlon, which is a set of 12 tasks that measure a conversational agent's ability to communicate engagingly with personality and empathy, utilize knowledge resources, discuss topics and situations, and converse about images.
Using metrics, multi-tasking, single task fine-tuning, zero-shot Transfer and human evaluation.
Cas d'usage: N/A
Objectifs de l'article: The objectives of the article are to introduce the Dialogue Dodecathlon as a way to measure the performance of conversational agents and to produce a single unified agent that can perceive, reason, and converse with humans in an open-domain setting.
Question(s) de recherche/Hypothèses/conclusion: The conclusion are that "The goal of introducing this task is not just as another challenge dataset, but to further motivate building and evaluating conversational agents capable of multiple skills – one of the core goals of AI. We believe current systems are closer to that goal than ever before – but we also still have a long way to go."
Cadre théorique/Auteur.es: The theoretical framework of the article includes natural language processing, machine learning, and conversational agents. The main authors cited include Dinan et al., Rashkin et al., and See et al.
Concepts clés: Conversational agents, Empathy, Knowledge grounding, Situation grounding, Image grounding.
Données collectées (type source): Cornell Movie: 309,987 training, 38,974 validation, 38,636 test utterances
LIGHT: 110,877 training, 6,623 validation, 13,272 test utterances
ELI5: 231,410 training, 9,828 validation, 24,560 test utterances
Ubuntu: 1,000,000 training, 19,560 validation, 18,920 test utterances
Twitter: 2,580,428 training, 10,405 validation, 10,405 test utterances
pushshift.io Reddit: approximately 2,200 million training, 10,000 validation, 10,000 test utterances
Image Chat: 355,862 training, 15,000 validation, 29,991 test utterances
IGC: 4,353 training, 486 validation, 7,773 test utterances
Définition des émotions: Non
Technologies associées: Natural language processing, Machine learning, Conversational agents, BERT, Image+Seq2Seq
Mention de l'éthique: Non
Finalité communicationnelle: "Such an agent should be able to get to know you when you first talk to it (ConvAI2), discuss everyday topics (DailyDialog, pushshift.io Reddit, Twitter, Cornell Movie), speak knowledgeably at depth (Wizard of Wikipedia, Ubuntu) and answer questions on such topics (ELI5)."

Collections: Facebook

Auteur-es: Santhosh Rajamanickam, Pushkar Mishra, Helen Yannakoudakis, Ekaterina Shutova
Nombre Auteurs: 4
Titre: Joint Modelling of Emotion and Abusive Language Detection
Année de publication: 2020
Référence (APA): Rajamanickam, S., Mishra, P., Yannakoudakis, H., & Shutova, E. (2020). Joint Modelling of Emotion and Abusive Language Detection. https://doi.org/10.18653/v1/2020.acl-main.394
résumé: The rise of online communication platforms has been accompanied by some undesirable effects, such as the proliferation of aggressive and abusive behaviour online. Aiming to tackle this problem, the natural language processing (NLP) community has experimented with a range of techniques for abuse detection. While achieving substantial success, these methods have so far only focused on modelling the linguistic properties of the comments and the online communities of users, disregarding the emotional state of the users and how this might affect their language. The latter is, however, inextricably linked to abusive behaviour. In this paper, we present the first joint model of emotion and abusive language detection, experimenting in a multi-task learning framework that allows one task to inform the other. Our results demonstrate that incorporating affective features leads to significant improvements in abuse detection performance across datasets.
Mots-clés: emotion detection, abusive language detection, natural language processing, multi-task learning, affective features
URL: https://research.facebook.com/file/763339231062964/Joint-Modelling-of-Emotion-and-Abusive-Language-Detection.pdf
doi: https://doi.org/10.18653/v1/2020.acl-main.394
Accessibilité de l'article: Libre
Champ: Artificial Intelligence, Natural Language Processing & Speech
Type contenu (théorique Applicative méthodologique): Méthodologique
Méthode: They propose a new approach that jointly learns to detect emotion and abuse, using a multi-task learning (MTL) framework that allows one task to inform the other.
Cas d'usage: Twitter
Objectifs de l'article: The main objectives of the article are to propose a new approach to abusive language detection that incorporates emotional features into the model via a multi-task learning framework, and to evaluate the effectiveness of this approach in improving abuse detection performance across datasets. The authors aim to address the limitations of existing methods, which have focused solely on modeling the linguistic properties of comments and online communities, and have not taken into account the emotional state of users and how this might affect their language. By developing a joint model of emotion and abusive language detection, the authors hope to provide a more comprehensive and accurate approach to identifying abusive behavior online.
Question(s) de recherche/Hypothèses/conclusion: Research question is whether incorporating affective features into a joint model of emotion and abusive language detection can improve abuse detection performance across datasets.; Hypothesis : Multi-task learning framework that allows one task to inform the other will lead to better detection of abusive language, as the model will be able to leverage the complementary information provided by the two tasks.; Conclusions : incorporating affective features into a joint model of emotion and abusive language detection leads to significant improvements in abuse detection performance across datasets. the emotional state of users is indeed linked to abusive behavior, and that jointly modeling emotion and abusive language detection can provide a more comprehensive and accurate approach to identifying abusive behavior online. The authors suggest that their approach could be extended to other complex semantic tasks, such as figurative language processing and inference, to further improve abuse detection performance.
Cadre théorique/Auteur.es: Emotion with Ekman, Natural language processing (NLP) and machine learning techniques, specifically deep learning models such as recurrent neural networks (RNNs) and convolutional neural networks (CNNs). Matthew E Peters et al. ; Sara Owsley Sood et al.

Theories of emotion and abusive behavior. Pushkar Mishra, Marco Del Tredici, Helen Yannakoudakis, and Ekaterina Shutova
Concepts clés: Abusive language detection, Affective features
Données collectées (type source): datasets related to abusive language and emotion detection from Twitter.

Abuse detection task : annotated tweets from OffensEval 2019 and Waseem and Hovy 2016.

Emotion detection task : SemEval-2018
Définition des émotions: Ekman
Ampleur expérimentation (volume de comptes): OffensEval : 13240 tweets

Waseem and Hovy : 16202 tweets

SemEval18 : 11000 tweets
Technologies associées: NLP, MTL, Classifiers, Convolutional neural networks (RNNs and CNNs), character-based models, and graph-based learning methods
Mention de l'éthique: Non
Finalité communicationnelle: "Overall, our results also suggest the superiority of MTL over STL for abuse detection. With this new approach, one can build more complex models introducing new auxiliary tasks for abuse detection. For instance, we expect that abuse detection may also benefit from joint learning with complex semantic tasks, such as figurative language processing and inference."

Make digital spaces safer ?

Collections: Facebook

Auteur-es: Xi Laura Cang, Ali Israr
Nombre Auteurs: 2
Titre: Communicating Socio-Emotional Sentiment Through Haptic Messages
Année de publication: 2020
Référence (APA): Cang, X. L., & Israr, A. (2020). Communicating Socio-Emotional Sentiment Through Haptic Messages.
résumé: Touch plays an important social role in fostering and maintaining emotional communication in vital human relationships, but when close relations live apart for extended periods of time, this nonverbal channel is lost. We explore how a custom-built, low degree-of-freedom, wearable haptic display may mediate the encoding and decoding of a set of complex socio-emotional messages that is sent and received by strangers, intimate partners, and even the same individual a week later.
URL: https://research.facebook.com/file/985449752308807/Communicating-Socio-Emotional-Sentiment-Through-Haptic-Messages.pdf
Accessibilité de l'article: Libre
Champ: AR/VR, Human Computer Interaction & UX
Type contenu (théorique Applicative méthodologique): Méthodologique
Méthode: The method involved building a custom wearable haptic display and conducting a small pilot study with dyad pairs to ensure that haptic patterns were discernible. Participants were given a set of 10 emotion-centric scenarios and one that they created, and used a touchscreen control interface to convert it into a haptic sensation. They then decoded the designs to test interpretation accuracy.
Cas d'usage: N/A
Objectifs de l'article: The objectives of the article are to explore the importance of touch in fostering emotional communication, to demonstrate how a wearable haptic display can mediate the encoding and decoding of complex socio-emotional messages, and to investigate the accuracy of interpretation of haptic messages.
Question(s) de recherche/Hypothèses/conclusion: The research questions are : 1) What messages do people wish to convey in an LDR? (Message Definition and Selection) 2) How might people design haptic sensations to communicate these messages? (Message Generation) 3) Can recipients of these haptic messages decode the complex root emotions? (Message Interpretation); The hypothesis is that haptic messages designed to convey anger will be more consistently recognized than those designed to convey calmness, and that messages designed by a close partner will be more accurately interpreted than those designed by a stranger or the participant themselves.; The conclusions are that haptic messages can be accurately interpreted based on the embedded emotion and relation-distance of the message designer, and that a wearable haptic display has potential for facilitating emotional communication in long-distance relationships.
Cadre théorique/Auteur.es: The theoretical framework of the article includes research on the importance of touch in emotional communication, as well as studies on haptic technology and its potential for communication. The main authors cited include Kuchenbecker, Culbertson, and Okamura.
Concepts clés: Haptic communication, Relationships, Emotion, Touch Wearable technology.
Données collectées (type source): The type of data collected was qualitative, and the sources were the participants' designs of haptic messages, descriptions of emotions "evoked when experiencing the message", and their interpretation of the unprompted message designed by the partner.
Définition des émotions: Non
Ampleur expérimentation (volume de comptes): 20 participants in 10 pairs
Technologies associées: Custom-built wearable haptic display
Mention de l'éthique: Non
Finalité communicationnelle: We are excited to demonstrate the viability of haptics in machine-mediated communication as a vehicle for conveying the important but subtly nuanced emotional content that we project so naturally in person.

Collections: Facebook

Auteur-es: X. Alice Li, Devi Parikh
Nombre Auteurs: 2
Titre: Lemotif: An Affective Visual Journal Using Deep Neural Networks
Année de publication: 2020
Référence (APA): Li, X. A., & Parikh, D. (2020). Lemotif : An Affective Visual Journal Using Deep Neural Networks.
résumé: We present Lemotif, an integrated natural language processing and image generation system that uses machine learning to (1) parse a text-based input journal entry describing the user’s day for salient themes and emotions and (2) visualize the detected themes and emotions in creative and appealing image motifs. Synthesizing approaches from artificial intelligence and psychology, Lemotif acts as an affective visual journal, encouraging users to regularly write and reflect on their daily experiences through visual reinforcement. By making patterns in emotions and their sources more apparent, Lemotif aims to help users better understand their emotional lives, identify opportunities for action, and track the effectiveness of behavioral changes over time. We verify via human studies that prospective users prefer motifs generated by Lemotif over corresponding baselines, find the motifs representative of their journal entries, and think they would be more likely to journal regularly using a Lemotif-based app.
URL: https://research.facebook.com/file/804280910247449/Lemotif-An-Affective-Visual-Journal-Using-Deep-Neural-Networks.pdf
doi: https://doi.org/10.48550/arXiv.1903.07766
Accessibilité de l'article: Libre
Champ: Artificial Intelligence, Machine Learning, Human Computer Interaction & UXNatural Language Processing & Speech
Type contenu (théorique Applicative méthodologique): Applicatif
Méthode: The method involves using natural language processing and machine learning to analyze journal entries and generate visual motifs that represent salient themes and emotions. The effectiveness of these mappings and the overall system is evaluated through studies with AMT users
Cas d'usage: Lemotif
Objectifs de l'article: The objective of the article are : to develop and evaluate a system that can visually represent journal entries, with the aim of making associations between feelings and life events more apparent to users and
to present a system that helps users better understand their emotional lives and track the effectiveness of behavioral changes over time
to evaluate the effectiveness of the system through human studies.
Question(s) de recherche/Hypothèses/conclusion: The research question is whether Lemotif, the system presented in the article, can effectively encourage users to regularly write and reflect on their daily experiences.; The hypothesis is that the creative and appealing image motifs generated by Lemotif will make journaling more actionable and engaging for users.; The conclusions are that Lemotif is effective in encouraging users to journal regularly and that the generated motifs are meaningful and representative of users' journal entries.
Cadre théorique/Auteur.es: The theoretical framework of the article includes psychology, NLP, affective computing and machine learning, and the main authors cited include Pennebaker, Cowen and Keltner.
Concepts clés: Image generation, Color-emotion mapping, Journaling
Données collectées (type source): The data collected are text/journal entries from 500 respondents on AMT. These entries were used for training and analysis.
Définition des émotions: Categorical emotions
Ampleur expérimentation (volume de comptes): 500 respondents
Technologies associées: Natural language processing, Machine learning, Image generation
Mention de l'éthique: Non
Finalité communicationnelle: "Lemotif aims to make associations between feelings and parts of a user’s life more apparent, presenting opportunities to take actions towards improved emotional well being.

We also find that subjects are interested in using an app like Lemotif and consider the generated motifs representative of their journal entries."

Collections: Facebook

Auteur-es: Jane Dwivedi-Yu, Alon Y. Halevy
Nombre Auteurs: 2
Titre: “That’s so cute!”: The CARE Dataset for Affective Response Detection
Année de publication: 2022
Référence (APA): Dwivedi-Yu, J., & Halevy, A. Y. (2022). “That’s so cute!” : The CARE Dataset for Affective Response Detection.
résumé: Social media plays an increasing role in our communication with friends and family, and in our consumption of entertainment and information. Hence, to design effective ranking functions for posts on social media, it would be useful to predict the affective responses of a post (e.g., whether it is likely to elicit feelings of entertainment, inspiration, or anger). Similar to work on emotion detection (which focuses on the affect of the publisher of the post), the traditional approach to recognizing affective response would involve an expensive investment in human annotation of training data.

We create and publicly release CAREdb, a dataset of 230k social media post annotations according to seven affective responses using the Common Affective Response Expression (CARE) method. The CARE method is a means of leveraging the signal that is present in comments that are posted in response to a post, providing high-precision evidence about the affective response to the post without human annotation. Unlike human annotation, the annotation process we describe here can be iterated upon to expand the coverage of the method, particularly for new affective responses. We present experiments that demonstrate that the CARE annotations compare favorably with crowdsourced annotations. Finally, we use CAREdb to train competitive BERT-based models for predicting affective response as well as emotion detection, demonstrating the utility of the dataset for related tasks.
Mots-clés: affective response, social media, emotion classification, CARE dataset, human annotation, machine learning
URL: https://research.facebook.com/file/393453116239728/Thats-so-cute--The-CARE-Dataset-for-Affective-Response-Detection.pdf
doi: https://doi.org/10.48550/arXiv.2201.11895
Accessibilité de l'article: Libre
Champ: Artificial Intelligence
Type contenu (théorique Applicative méthodologique): Méthodologique, applicatif
Méthode: The method used in this article is the Common Affective Response Expression (CARE) method, which is a cost-effective alternative to human annotation for predicting the emotional impact of social media posts. The CARE method involves two steps: (1) identifying the CARE patterns in the text, and (2) using a lexicon to label the publisher affect of the comments. The authors also compare the performance of the CARE method to a state-of-the-art emotion classifier.

Annotations according to the CARE method, as well as a human annotated dataset for comparison.
Cas d'usage: Reddit
Objectifs de l'article: The objectives of the article are to introduce the CARE dataset, evaluate its accuracy compared to crowdsourced annotations, and explore its potential applications in social media ranking functions.
Question(s) de recherche/Hypothèses/conclusion: The research question is whether the CARE dataset can accurately predict affective response in social media posts.; The hypothesis is that the CARE dataset will perform comparably to crowdsourced annotations in predicting affective response in social media posts.; The authors conclude that the CARE dataset is a valuable resource for predicting affective response in social media posts, and that it performs comparably to crowdsourced annotations. They also suggest that the CARE dataset could be used to improve social media ranking functions.
Cadre théorique/Auteur.es: The theoretical framework of the article is based on previous work in affective computing and emotion classification. The main authors cited include Stark and Hoey (2021), Demszky et al. (2020), Lin et al. (2008), Ma et al. (2005), and Mohammad and Kiritchenko (2015).
Concepts clés: Affective response, Social Media, Emotion classification, Machine learning
Données collectées (type source): Our experiments use a dataset that is created from Reddit posts and comments in the pushshift.io database that were created between 2011 and 2019.
Définition des émotions: Non
Ampleur expérimentation (volume de comptes): 34 million comments from
24 million distinct posts

TOTAL in CARE : 230k
Technologies associées: CARE method, machine learning algorithms, Emotion classifier
Mention de l'éthique: Yes, there is mention of ethics in this article. The authors note that the CARE method should not be used for ill-intended purposes such as purposefully recommending particular content to manipulate a user's perception or preferences.
Finalité communicationnelle: "We described a method for extracting training data for models that predict the affective responses of a post on social media"
Commentaires: Voir la bibliographie de l'article, plusieurs références qui pourraient entrer dans la RL.

Collections: Facebook

Auteur-es: Felix Kreuk, Adam Polyak, Jade Copet, Eugene Kharitonov, Tu-Anh Nguyen, Morgane Rivière, Wei-Ning Hsu, Emmanuel Dupoux, Yossi Adi
Nombre Auteurs: 9
Titre: Textless Speech Emotion Conversion using Discrete & Decomposed Representations
Année de publication: 2022
Référence (APA): Kreuk, F., Polyak, A., Copet, J., Kharitonov, E., Nguyen, T.-A., Rivière, M., Hsu, W.-N., Mohamed, A., Dupoux, E., & Adi, Y. (2022). Textless Speech Emotion Conversion using Discrete & Decomposed Representations.
résumé: Speech emotion conversion is the task of modifying the perceived emotion of a speech utterance while preserving the lexical content and speaker identity. In this study, we cast the problem of emotion conversion as a spoken language translation task. We use a decomposition of the speech signal into discrete learned representations, consisting of phonetic-content units, prosodic features, speaker, and emotion. First, we modify the speech content by translating the phonetic-content units to a target emotion, and then predict the prosodic features based on these units. Finally, the speech waveform is generated by feeding the predicted representations into a neural vocoder. Such a paradigm allows us to go beyond spectral and parametric changes of the signal, and model non-verbal vocalizations, such as laughter insertion, yawning removal, etc. We demonstrate objectively and subjectively that the proposed method is vastly superior to current approaches and even beats text-based systems in terms of perceived emotion and audio quality. We rigorously evaluate all components of such a complex system and conclude with an extensive model analysis and ablation study to better emphasize the architectural choices, strengths and weaknesses of the proposed method. Samples are available under the following link here.
URL: https://research.facebook.com/file/509033718007989/Textless-Speech-Emotion-Conversion-using-Discrete-and-Decomposed-Representations.pdf
doi: https://doi.org/10.48550/arXiv.2111.07402
Accessibilité de l'article: Libre
Champ: Natural Language Processing & Speech
Type contenu (théorique Applicative méthodologique): Méthodologique, applicatif
Méthode: The method allows for the modification of non-verbal vocalizations such as laughter insertion and yawning removal while preserving the lexical content and speaker identity.
Cas d'usage: N/A
Objectifs de l'article: The study presents a novel approach to modifying the perceived emotion of a speech utterance while preserving the lexical content and speaker identity. The study also includes a rigorous evaluation of the proposed method and compares it against current approaches. The proposed method has been rigorously evaluated and has been found to be superior to current approaches in terms of perceived emotion and audio quality.
Question(s) de recherche/Hypothèses/conclusion: Research question is How to modify the perceived emotion of a speech utterance while preserving the lexical content and speaker identity ?; The hypothesis is that "modifying the learned representations of a speech signal can effectively change the perceived emotion of the speech while preserving the lexical content and speaker identity."; The concusions are "The researchers found a way to break down the voice into different parts and change them to make the voice sound happy, sad, angry, or other emotions. They tested this new way and found that it works better than other ways that people have tried before. They also found that changing the non-verbal parts of the voice, like laughter or sighs, can make a big difference in how the voice sounds. Overall, this new way of changing someone's voice is really cool and could be useful in things like movies or video games where characters need to sound different emotions."
Cadre théorique/Auteur.es: Speech processing, natural language processing : Vaswani et al. (2017); Polyak et al. (2021); Kharitonov et al. (2021a); Schuller et al. (2013); Fan et al. (2014); Wang et al. (2018)
Concepts clés: Speech emotion conversion, Non-verbal communication cues, Spoken language translation
Données collectées (type source): Dataset of emotional speech recordings, which were labeled with six different emotions (Neutral, Happy, Sad, Angry, Disgusted, and Sleepy)
Définition des émotions: Categorical emotions
Ampleur expérimentation (volume de comptes): 7000 speech utterances based on transcripts from the CMU Arctic Database
Technologies associées: HuBERT: A pre-trained neural network model for speech processing, which is used as a feature extractor in the emotion conversion model.

Transformer: A type of neural network architecture that uses self-attention mechanisms to process sequential data, which is used as the main architecture for the emotion conversion model.

PyTorch: A popular open-source machine learning framework, which is used to implement the emotion conversion model and train it on the emotional speech dataset.
Mention de l'éthique: Non
Finalité communicationnelle: "We demonstrated how the proposed system is able to model expressive non-verbal vocalizations as well as generate high- quality expressive speech. We conclude with an ablation study and analysis of the different components composing our system. This study serves as the foundation for improving speech emotion conversion and building general textless expressive speech generation models."

Collections: Facebook

Auteur-es: Abhinav Shukla, Stavros Petridis, Maja Pantic
Nombre Auteurs: 3
Titre: Does Visual Self-Supervision Improve Learning
of Speech Representations for Emotion
Recognition?
Année de publication: 2021
Référence (APA): Shukla, A., Petridis, S., & Pantic, M. (2021). Does Visual Self-Supervision Improve Learning of Speech Representations for Emotion Recognition? IEEE Transactions on Affective Computing, 14(1), 406‑420. https://doi.org/10.1109/TAFFC.2021.3062406
résumé: Self-supervised learning has attracted plenty of recent research interest. However, most works for self-supervision in speech are typically unimodal and there has been limited work that studies the interaction between audio and visual modalities for cross-modal self-supervision. This work (1) investigates visual self-supervision via face reconstruction to guide the learning of audio representations; (2) proposes an audio-only self-supervision approach for speech representation learning; (3) shows that a multi-task combination of the proposed visual and audio self-supervision is beneficial for learning richer features that are more robust in noisy conditions; (4) shows that self-supervised pretraining can outperform fully supervised training and is especially useful to prevent overfitting on smaller sized datasets. We evaluate our learned audio representations for discrete emotion recognition, continuous affect recognition and automatic speech recognition. We outperform existing self-supervised methods for all tested downstream tasks. Our results demonstrate the potential of visual self-supervision for audio feature learning and suggest that joint visual and audio self-supervision leads to more informative audio representations for speech and emotion recognition.
Mots-clés: Self-supervised learning, Representation learning, Generative modeling, Audiovisual speech, Emotion recognition,
Speech recognition, Cross-modal self-supervision.
URL: https://research.facebook.com/file/4493784684073249/Does-Visual-Self-Supervision-Improve-Learning-of-Speech-Representations-for-Emotion-Recognition.pdf
doi: https://doi.org/10.1109/TAFFC.2021.3062406
Accessibilité de l'article: Libre
Champ: Artificial Intelligence, Machine Learning
Type contenu (théorique Applicative méthodologique): Méthodologique, applicatif
Méthode: self-supervised learning (SSL).
The method proposed in this article is a multi-task approach that combines audio and visual modalities for self-supervised learning in speech representation. The approach involves using visual self-supervision via face reconstruction to guide the learning of audio representations. The authors evaluate the effectiveness of their approach on several tasks, including discrete emotion recognition, continuous affect recognition, and automatic speech recognition.
Cas d'usage: The three self-supervised methods we
compare against are CPC, APC, and PASE
Objectifs de l'article: The objectives of the article are to investigate the benefits of combining audio and visual modalities for self-supervised learning in speech representation, and to propose a multi-task approach that outperforms existing self-supervised methods for several tasks.

"we investigate self-supervised learning for audio representations."

"we examine the state-of-the-art in self-supervised audio feature learning which we use as baselines. We then propose a novel visual self-supervised method and a novel audio-only self-supervised method for learning audio features. We also show how visual self-supervision helps encode emotional information into the audio features."
Question(s) de recherche/Hypothèses/conclusion: The research question is "Can combining audio and visual modalities for self-supervised learning in speech representation improve performance on tasks such as discrete emotion recognition, continuous affect recognition, and automatic speech recognition?"; The hypothesis is that the proposed multi-task approach that combines audio and visual modalities for self-supervised learning in speech representation will outperform existing self-supervised methods for several tasks.; The authors conclude that their proposed multi-task approach that combines audio and visual modalities for self-supervised learning in speech representation outperforms existing self-supervised methods for several tasks, including discrete emotion recognition, continuous affect recognition, and automatic speech recognition.
"proposed visual self-supervision is superior when compared to the proposed audio-only self-supervision. The results on both discrete and continuous affect recognition offer evidence that the learned representation is good for emotion."

"The models trained using a combination of audio and visual self-supervision are able to encode complementary information from each modality to yield the best possible representations among all tested methods in this work."
Cadre théorique/Auteur.es: The theoretical framework of the article is based on previous work in self-supervised learning and multimodal learning. The main authors cited include Hjelm et al., 2018; Owens et al., 2018; and Arandjelovic and Zisserman, 2017.
Concepts clés: Discrete Emotion Recognition, Continuous Affect Recognition, Automatic Speech Recognition.
Données collectées (type source): CREMA-D dataset : 91 actors who utter 12 sentences multiples times each with a
different level of intensity for each of 6 basic emotional
labels (anger, fear, disgust, neutral, happy, sad).

The Ravdess dataset contains 1440 samples of 24 different actors who acted out two sentences with 8 different basic emotions (anger, calm, sad, neutral, happy, disgusted, surprised, fear) and two different intensity levels.

The RECOLA dataset contains dyadic conversations in French between a pair of participants working to solve a collaborative task over video conference. The annotated part of the dataset consists of 5-minute long clips (with audio from one speaker only) with continuous valence and arousal annotations from 6 annotators.

The SEWA dataset [58] contains dyadic conversations over video conference between a pair of participants discussing about an advertisement that they have just watched. The audio clips are typically 3-minute long and have continuous valence and arousal annotations.

The IEMOCAP dataset [59] contains dyadic conversations between 10 speakers for a total of 12 hours of audiovisual data. The discrete emotion labels comprise of 8
categories (anger, happiness, sadness, neutral, excitement, frustration, fear, surprise), however we only consider the first 4 categories for our experiments (anger, happiness,
sadness, neutral

The SPC (Speech Commands) dataset contains 64,727 total utterances of 30 different words by 1,881 speakers. We use SPC as a speech recognition evaluation dataset.
The LRW dataset is a large, in-the-wild dataset of 500 different isolated words primarily from BBC recordings. It is an audiovisual speech dataset and is thus appropriate for training our methods.
Définition des émotions: Categorical emotions
Ampleur expérimentation (volume de comptes): 1440 samples of 24 different actors

12 hours of au-
diovisual data

64,727 total utterances of 30 different words by 1,881 speakers

500 different isolated words
Technologies associées: Self-supervised learning, Audio-visual modalities, Speech representation, Emotion recognition, Affect recognition, Automatic speech recognition.
Mention de l'éthique: Non
Finalité communicationnelle: To propose "a multi-task approach that outperforms existing self-supervised methods for discrete emotion recognition, continuous affect recognition, and automatic speech recognition."

Collections: Facebook

Auteur-es: Badr AlKhamissi, Faisal Ladhak, Srini Iyer, Zornitsa Kozareva, Xian Li, Pascale Fung, Lambert Mathias, Asli Celikyilmaz, Mona Diab
Nombre Auteurs: 9
Titre: TOKEN: Task Decomposition and Knowledge Infusion for Few-Shot Hate Speech Detection
Année de publication: 2022
Référence (APA): AlKhamissi, B., Ladhak, F., Iyer, S., Stoyanov, V., Kozareva, Z., Li, X., Fung, P., Mathias, L., Celikyilmaz, A., & Diab, M. (2022). TOKEN: Task Decomposition and Knowledge Infusion for Few-Shot Hate Speech Detection.
résumé: Hate speech detection is complex; it relies on commonsense reasoning, knowledge of stereotypes, and an understanding of social nuance that differs from one culture to the next. It is also difficult to collect a large-scale hate speech annotated dataset. In this work, we frame this problem as a few-shot learning task, and show significant gains with decomposing the task into its "constituent" parts. In addition, we see that infusing knowledge from reasoning datasets (e.g. ATOMIC 20/20 ) improves the performance even further. Moreover, we observe that the trained models generalize to out-of-distribution datasets, showing the superiority of task decomposition and knowledge infusion compared to previously used methods. Concretely, our method outperforms the baseline by 17.83% absolute gain in the 16-shot case.
URL: https://research.facebook.com/file/8394145417323280/TOKEN--Task-Decomposition-and-Knowledge-Infusion-for-Few-Shot-Hate-Speech-Detection.pdf
doi: https://doi.org/10.48550/arXiv.2205.12495
Accessibilité de l'article: Libre
Champ: Artificial Intelligence, Natural Language Processing & Speech
Type contenu (théorique Applicative méthodologique): Méthodologique
Méthode: The method involves breaking down the task of hate speech detection into smaller subtasks, using a few-shot learning approach to train the model on a small amount of labeled data, and infusing knowledge from reasoning datasets to improve performance.
Cas d'usage: N/A
Objectifs de l'article: The objectives of the article are to improve the performance of hate speech detection models, particularly in low-resource settings, and to explore the use of task decomposition and knowledge infusion in natural language processing tasks.
Question(s) de recherche/Hypothèses/conclusion: The research question is how can task decomposition and knowledge infusion be used to improve the performance of hate speech detection models?; The hypothesis is that breaking down the task of hate speech detection into smaller subtasks and infusing knowledge from reasoning datasets will improve the performance of the model.; The conclusions are that the proposed method outperforms previous approaches to hate speech detection and shows promising results in detecting harmful content on social media platforms.
Cadre théorique/Auteur.es: The theoretical framework of the article includes natural language processing and machine learning, with references to authors such as Sanguinetti, Assimakopoulos, and de Gibert.
Concepts clés: Few-shot learning, Task decomposition, Hate speech detection, Social media
Données collectées (type source): datasets for evaluation, including HateXplain, HS18, and Ethos
Définition des émotions: Non
Ampleur expérimentation (volume de comptes): 12,838 examples
Technologies associées: Natural language processing, Machine learning
Mention de l'éthique: The article does not mention ethics explicitly, but it does discuss the potential impact of hate speech on individuals and society.

Collections: Facebook

Auteur-es: Carine Rognon, Taylor Bunge, Meiyuzi Gao, Chip Conor, Benjamin Stephens-Fripp, Casey Brown, Ali Israr
Nombre Auteurs: 7
Titre: An Online Survey on the Perception of Mediated Social Touch Interaction and Device Design
Année de publication: 2022
Référence (APA): Rognon, C., Bunge, T., Gao, M., Conor, C., Stephens, B., Brown, C., & Israr, A. (2022). An Online Survey on the Perception of Mediated Social Touch Interaction and Device Design. J. ACM, 11(11).
résumé: Social touch is essential for our social interactions, communication, and well-being. It has been shown to reduce anxiety and loneliness; and is a key channel to transmit emotions for which words are not sufficient, such as love, sympathy, reassurance. However, direct physical contact is not always possible due to being remotely located, interacting in a virtual environment, or as a result of a health issue. Mediated social touch enables physical interactions, despite the distance, by transmitting the haptic cues that constitute social touch through devices. As this technology is fairly new, the users’ needs and their expectations on a device design and its features are unclear, as well as who would use this technology, and in which conditions. To better understand these aspects of mediated interaction, we conducted an online survey on 258 respondents located in the USA. Results give insights on the type of interactions and device features that the US population would like to use.
Mots-clés: Mediated Social Touch, Hardware and Software that Enable Social Touch
Interactions, Haptic Technology
URL: https://research.facebook.com/file/589149892136476/An-Online-Survey-on-the-Perception-of-Mediated-Social-Touch-Interaction-and-Device-Design.pdf
doi: https://doi.org/10.1145/1122445.1122456
Accessibilité de l'article: Libre
Champ: Human Computer Interaction & UX
Type contenu (théorique Applicative méthodologique): Théorique, méthodologique
Méthode: The authors conducted an online Qualtrics survey consisting of 18 questions divided into 6 themes.
Cas d'usage: N/A
Objectifs de l'article: The objectives of the article were to identify the needs and expectation of the population for mediated social touch technology/devices by identifying : "what type of social touch people would like to communicate and in what context. [...] emotions that people feel unable to communicate effectively with current technologies. [...] types of devices respondents would like to use as well as types of messages they would like to transmit [...] the scenarios in which respondents would be likely to use an MST device."
Question(s) de recherche/Hypothèses/conclusion: The conclusions of the article were that there is a need for mediated social touch technology to improve communication and well-being, and that users have specific needs and expectations for such technology.; "In conclusion, the main take home messages from this research are that social touch can vary regarding the interlocutors and it can have various meanings depending on the context of the interaction. Concerning the device design, respondents felt more comfortable to communicate with loved ones using MST devices that have small and familiar form factors. The MST device should allow to transmit various types of social touch that are personalizable regarding the emotions they want to convey to their interlocutor. The focus should primarily be put on conveying positive emotions. In addition, the multisensorial aspect of the social touch interaction is essential to be preserved."
Cadre théorique/Auteur.es: The theoretical framework of the article includes haptic technologies and social touch. The main authors cited include Stanley E Jones and A Elaine Yarbrough, Eva-Lotta Sallnäs.
Concepts clés: Social touch, Human-centered computing, Haptic devices
Données collectées (type source): The online Qualtrics survey consisted of 18 questions divided into 6 themes: the respondent’s background (5 questions), the type of social touch missed and in which context (4), the limitations of current technologies to communicate emotions (1), the type of device(s) people would like to use and to communicate what (6), in which scenarios would people use an MST device (1), and a free comment section (1).

Respondants were recruited on AMT.
Définition des émotions: Non
Ampleur expérimentation (volume de comptes): 258 respondants
Technologies associées: Phone calls, Video calls, Texting, Virtual reality, Wearable devices.
Mention de l'éthique: Non
Finalité communicationnelle: Results should help the design of MST devices

Collections: Facebook

Auteur-es: Jane Dwivedi-Yu, Yi-Chia Wang, Lijing Qin, Cristian Canton-Ferrer, Alon Y. Halevy
Nombre Auteurs: 5
Titre: Affective Signals in a Social Media Recommender System
Année de publication: 2022
Référence (APA): Dwivedi-Yu, J., Wang, Y.-C., Qin, L., Canton-Ferrer, C., & Halevy, A. Y. (2022). Affective Signals in a Social Media Recommender System. https://doi.org/10.1145/3534678.3539054
résumé: People come to social media to satisfy a variety of needs, such as being informed, entertained and inspired, or connected to their friends and community. Hence, to design a ranking function that gives useful and personalized post recommendations, it would be helpful to be able to predict the affective response a user may have to a post (e.g., entertained, informed, angered). This paper describes the challenges and solutions we developed to apply Affective Computing to social media recommendation systems.

We address several types of challenges. First, we devise a taxonomy of affects that was small (for practical purposes) yet covers the important nuances needed for the application. Second, to collect training data for our models, we balance between signals that are already available to us (namely, different types of user engagement) and data we collected through a carefully crafted human annotation effort on 800k posts. We demonstrate that affective response information learned from this dataset improves a module in the recommendation system by more than 8%. Online experimentation also demonstrates statistically significant decreases in surfaced violating content and increases in surfaced content that users find valuable.
Mots-clés: affective computing, recommendation systems, social media
URL: https://research.facebook.com/file/1116060675655243/Affective-Signals-in-a-Social-Media-Recommender-System.pdf
doi: https://doi.org/10.1145/3534678.3539054
Accessibilité de l'article: Libre
Champ: Machine Learning
Type contenu (théorique Applicative méthodologique): Applicatif
Méthode: The authors collected training data for their models through a combination of engagement data on the platform and human labeling of posts. They used a two-tower architecture multi-class classifier to train their models.
Cas d'usage: N/A
Objectifs de l'article: The objectives of the article are to explore the challenges and solutions of applying Affective Computing to social media recommendation systems, and to demonstrate the effectiveness of incorporating affective response information in improving recommendation systems.
Question(s) de recherche/Hypothèses/conclusion: The research question is how can affective response information be leveraged to improve social media recommendation systems?; The hypothesis is that incorporating affective response information in recommendation systems will lead to improvements in user engagement and content quality.; The conclusions are that incorporating affective response information in recommendation systems can lead to significant improvements in user engagement and content quality, and that this approach can be applied to a variety of social media platforms.
Cadre théorique/Auteur.es: The theoretical framework of the article is Affective Computing, and the main authors cited include Jan Mizgajski, Mikołaj Morzy, Saif M Mohammad, Svetlana Kiritchenko, and Jessica Gall Myrick.
Concepts clés: Affective computing, Recommendation systems, Social media, Engagement data, human labeling, and Multi-class classification
Données collectées (type source): de-identified Facebook posts, using a combination of engagement data and human labeling.
Définition des émotions: The authors do not provide a specific definition of emotions, but they do provide a taxonomy of affects that covers the important nuances needed for the application.
Ampleur expérimentation (volume de comptes): 800k posts
Technologies associées: Affective Computing, Recommendation systems
Mention de l'éthique: The authors do not explicitly mention ethics, but they do note that explicitly asking users their affective states regarding content has the drawbacks of being intrusive and potentially unreliable.

Collections: Facebook

Auteur-es: Carine Rognon, Benjamin Stephens-Fripp, Jess Hartcher-O’Brien, Bob Rost, Ali Israr
Nombre Auteurs: 5
Titre: Linking Haptic Parameters to the Emotional Space for Mediated Social Touch
Année de publication: 2022
Référence (APA): Rognon, C., Stephens-Fripp, B., Hartcher-O’Brien, J., & Israr, A. (2022). Linking Haptic Parameters to the Emotional Space for Mediated Social Touch. https://doi.org/10.3389/fcomp.2022.826545
résumé: Social touch is essential for creating and maintaining strong interpersonal bonds amongst humans. However, when distance separates users, they often rely on voice and video communication technologies to stay connected with each other, and the lack of tactile interactions between users lowers the quality of the social interactions. In this research, we investigated haptic patterns to communicate five tactile messages comprising of four types of social touch (high five, handshake, caress, and asking for attention) and one physiological signal (the pulse of a heartbeat), delivered on the hand through a haptic glove. Since social interactions are highly dependent on their context, we conceived two interaction scenarios for each of the five tactile messages, conveying distinct emotions being spread across the circumplex model of emotions. We conducted two user studies: in the first one participants tuned the parameters of haptic patterns to convey tactile messages in each scenario, and a follow up study tested naïve participants to assess the validity of these patterns. Our results show that all haptic patterns were recognized above chance level, and the well-defined parameter clusters had a higher recognition rate, reinforcing the hypothesis that some social touches have more universal patterns than others. We also observed parallels between the parameters’ levels and the type of emotions they conveyed based on their mapping in the circumplex model of emotions.
Mots-clés: Social Touch, Mediated Social Touch, Affective Touch, Haptics, Human Computer Interaction, Emotional Space
URL: https://research.facebook.com/file/565647498393044/Linking-Haptic-Parameters-to-the-Emotional-Space-for-Mediated-Social-Touch.pdf
doi: https://doi.org/10.3389/fcomp.2022.826545
Accessibilité de l'article: Libre
Champ: AR/VR
Type contenu (théorique Applicative méthodologique): Méthodologique
Méthode: The first study had participants tune parameters for ten different interaction scenarios. The second study had participants attempt to recognize the correct interaction scenario using the tuned parameters from the first study
Cas d'usage: N/A
Objectifs de l'article: The objectives of the article were to explore how haptic patterns can be used to communicate tactile messages and physiological signals, such as high fives, handshakes, caresses, and heartbeats, through a haptic glove, and to investigate how these patterns can convey distinct emotions across the circumplex model of emotions.

"we aim to look at building blocks of social touch and how users can tune them to haptically represent emotional content. In addition, we aim to determine how well these parameters can be generalized across participants."
Question(s) de recherche/Hypothèses/conclusion: The research question was how can haptic patterns be used to convey different tactile messages and emotions in mediated social touch?; The hypothesis is that some social touches have more universal patterns thanothers. The hypothesis was that haptic patterns can be used to convey distinct emotions across the circumplex model of emotions and that these patterns can be recognized above chance level, demonstrating their potential for enhancing social interactions in distance communication.; The conclusions are that the patterns derived from the first user study are generalizable to naïve users and that specific emotions belong to a specific area of the parametric space
Cadre théorique/Auteur.es: The theoretical framework of the article is based on the circumplex model of emotions, which is a two-dimensional model that organizes emotions based on their valence (positive or negative) and arousal (high or low). The main authors cited include James A. Russell and Lisa Feldman Barrett.
Concepts clés: Haptic messages, Emotional content
Données collectées (type source): Demographics from the participants (questionnaire), participants rated the haptic signal depending on the corresponding social touch and scenario.
Définition des émotions: Non
Valence and arousal
Ampleur expérimentation (volume de comptes): 14 participants
Technologies associées: Haptic glove, Oculus head mounted display, VR environment
Mention de l'éthique: Non
Finalité communicationnelle: "These results demonstrate the potential of creating haptic building blocks to map a social touch to the emotional spaces."

Collections: Facebook

Auteur-es: Jane Dwivedi-Yu, Alon Y. Halevy
Nombre Auteurs: 2
Titre: “That’s so cute!”: The CARE Dataset for Affective Response Detection
Année de publication: 2022
Référence (APA): Dwivedi-Yu, J., & Halevy, A. Y. (2022). “That’s so cute!” : The CARE Dataset for Affective Response Detection.
résumé: Social media plays an increasing role in our communication with friends and family, and in our consumption of entertainment and information. Hence, to design effective ranking functions for posts on social media, it would be useful to predict the affective responses of a post (e.g., whether it is likely to elicit feelings of entertainment, inspiration, or anger). Similar to work on emotion detection (which focuses on the affect of the publisher of the post), the traditional approach to recognizing affective response would involve an expensive investment in human annotation of training data.

We create and publicly release CAREdb, a dataset of 230k social media post annotations according to seven affective responses using the Common Affective Response Expression (CARE) method. The CARE method is a means of leveraging the signal that is present in comments that are posted in response to a post, providing high-precision evidence about the affective response to the post without human annotation. Unlike human annotation, the annotation process we describe here can be iterated upon to expand the coverage of the method, particularly for new affective responses. We present experiments that demonstrate that the CARE annotations compare favorably with crowdsourced annotations. Finally, we use CAREdb to train competitive BERT-based models for predicting affective response as well as emotion detection, demonstrating the utility of the dataset for related tasks.
Mots-clés: affective response, social media, emotion classification, CARE dataset, human annotation, machine learning
URL: https://research.facebook.com/file/393453116239728/Thats-so-cute--The-CARE-Dataset-for-Affective-Response-Detection.pdf
doi: https://doi.org/10.48550/arXiv.2201.11895
Accessibilité de l'article: Libre
Champ: Artificial Intelligence
Type contenu (théorique Applicative méthodologique): Méthodologique, applicatif
Méthode: The method used in this article is the Common Affective Response Expression (CARE) method, which is a cost-effective alternative to human annotation for predicting the emotional impact of social media posts. The CARE method involves two steps: (1) identifying the CARE patterns in the text, and (2) using a lexicon to label the publisher affect of the comments. The authors also compare the performance of the CARE method to a state-of-the-art emotion classifier.

Annotations according to the CARE method, as well as a human annotated dataset for comparison.
Cas d'usage: Reddit
Objectifs de l'article: The objectives of the article are to introduce the CARE dataset, evaluate its accuracy compared to crowdsourced annotations, and explore its potential applications in social media ranking functions.
Question(s) de recherche/Hypothèses/conclusion: The research question is whether the CARE dataset can accurately predict affective response in social media posts.; The hypothesis is that the CARE dataset will perform comparably to crowdsourced annotations in predicting affective response in social media posts.; The authors conclude that the CARE dataset is a valuable resource for predicting affective response in social media posts, and that it performs comparably to crowdsourced annotations. They also suggest that the CARE dataset could be used to improve social media ranking functions.
Cadre théorique/Auteur.es: The theoretical framework of the article is based on previous work in affective computing and emotion classification. The main authors cited include Stark and Hoey (2021), Demszky et al. (2020), Lin et al. (2008), Ma et al. (2005), and Mohammad and Kiritchenko (2015).
Concepts clés: Affective response, Social Media, Emotion classification, Machine learning
Données collectées (type source): Our experiments use a dataset that is created from Reddit posts and comments in the pushshift.io database that were created between 2011 and 2019.
Définition des émotions: Non
Ampleur expérimentation (volume de comptes): 34 million comments from
24 million distinct posts

TOTAL in CARE : 230k
Technologies associées: CARE method, machine learning algorithms, Emotion classifier
Mention de l'éthique: Yes, there is mention of ethics in this article. The authors note that the CARE method should not be used for ill-intended purposes such as purposefully recommending particular content to manipulate a user's perception or preferences.
Finalité communicationnelle: "We described a method for extracting training data for models that predict the affective responses of a post on social media"
Commentaires: Voir la bibliographie de l'article, plusieurs références qui pourraient entrer dans la RL.

Collections: Facebook

Auteur-es: Felix Kreuk, Adam Polyak, Jade Copet, Eugene Kharitonov, Tu-Anh Nguyen, Morgane Rivière, Wei-Ning Hsu, Emmanuel Dupoux, Yossi Adi
Nombre Auteurs: 9
Titre: Textless Speech Emotion Conversion using Discrete & Decomposed Representations
Année de publication: 2022
Référence (APA): Kreuk, F., Polyak, A., Copet, J., Kharitonov, E., Nguyen, T.-A., Rivière, M., Hsu, W.-N., Mohamed, A., Dupoux, E., & Adi, Y. (2022). Textless Speech Emotion Conversion using Discrete & Decomposed Representations.
résumé: Speech emotion conversion is the task of modifying the perceived emotion of a speech utterance while preserving the lexical content and speaker identity. In this study, we cast the problem of emotion conversion as a spoken language translation task. We use a decomposition of the speech signal into discrete learned representations, consisting of phonetic-content units, prosodic features, speaker, and emotion. First, we modify the speech content by translating the phonetic-content units to a target emotion, and then predict the prosodic features based on these units. Finally, the speech waveform is generated by feeding the predicted representations into a neural vocoder. Such a paradigm allows us to go beyond spectral and parametric changes of the signal, and model non-verbal vocalizations, such as laughter insertion, yawning removal, etc. We demonstrate objectively and subjectively that the proposed method is vastly superior to current approaches and even beats text-based systems in terms of perceived emotion and audio quality. We rigorously evaluate all components of such a complex system and conclude with an extensive model analysis and ablation study to better emphasize the architectural choices, strengths and weaknesses of the proposed method. Samples are available under the following link here.
URL: https://research.facebook.com/file/509033718007989/Textless-Speech-Emotion-Conversion-using-Discrete-and-Decomposed-Representations.pdf
doi: https://doi.org/10.48550/arXiv.2111.07402
Accessibilité de l'article: Libre
Champ: Natural Language Processing & Speech
Type contenu (théorique Applicative méthodologique): Méthodologique, applicatif
Méthode: The method allows for the modification of non-verbal vocalizations such as laughter insertion and yawning removal while preserving the lexical content and speaker identity.
Cas d'usage: N/A
Objectifs de l'article: The study presents a novel approach to modifying the perceived emotion of a speech utterance while preserving the lexical content and speaker identity. The study also includes a rigorous evaluation of the proposed method and compares it against current approaches. The proposed method has been rigorously evaluated and has been found to be superior to current approaches in terms of perceived emotion and audio quality.
Question(s) de recherche/Hypothèses/conclusion: Research question is How to modify the perceived emotion of a speech utterance while preserving the lexical content and speaker identity ?; The hypothesis is that "modifying the learned representations of a speech signal can effectively change the perceived emotion of the speech while preserving the lexical content and speaker identity."; The concusions are "The researchers found a way to break down the voice into different parts and change them to make the voice sound happy, sad, angry, or other emotions. They tested this new way and found that it works better than other ways that people have tried before. They also found that changing the non-verbal parts of the voice, like laughter or sighs, can make a big difference in how the voice sounds. Overall, this new way of changing someone's voice is really cool and could be useful in things like movies or video games where characters need to sound different emotions."
Cadre théorique/Auteur.es: Speech processing, natural language processing : Vaswani et al. (2017); Polyak et al. (2021); Kharitonov et al. (2021a); Schuller et al. (2013); Fan et al. (2014); Wang et al. (2018)
Concepts clés: Speech emotion conversion, Non-verbal communication cues, Spoken language translation
Données collectées (type source): Dataset of emotional speech recordings, which were labeled with six different emotions (Neutral, Happy, Sad, Angry, Disgusted, and Sleepy)
Définition des émotions: Categorical emotions
Ampleur expérimentation (volume de comptes): 7000 speech utterances based on transcripts from the CMU Arctic Database
Technologies associées: HuBERT: A pre-trained neural network model for speech processing, which is used as a feature extractor in the emotion conversion model.

Transformer: A type of neural network architecture that uses self-attention mechanisms to process sequential data, which is used as the main architecture for the emotion conversion model.

PyTorch: A popular open-source machine learning framework, which is used to implement the emotion conversion model and train it on the emotional speech dataset.
Mention de l'éthique: Non
Finalité communicationnelle: "We demonstrated how the proposed system is able to model expressive non-verbal vocalizations as well as generate high- quality expressive speech. We conclude with an ablation study and analysis of the different components composing our system. This study serves as the foundation for improving speech emotion conversion and building general textless expressive speech generation models."

Collections: Facebook

Auteur-es: Marija Jegorova, Stavros Petridis, Maja Pantic
Nombre Auteurs: 3
Titre: SS-VAERR: Self-Supervised Apparent Emotional Reaction Recognition from Video
Année de publication: 2023
Référence (APA): Jegorova, M., Petridis, S., & Pantic, M. (2023). SS-VAERR : Self-Supervised Apparent Emotional Reaction Recognition from Video. 2023 IEEE 17th International Conference on Automatic Face and Gesture Recognition (FG), 1‑8. https://doi.org/10.1109/FG57933.2023.10042638
résumé: This work focuses on the apparent emotional reaction recognition (AERR) from the video-only input, conducted in a self-supervised fashion. The network is first pre-trained on different self-supervised pretext tasks and later fine-tuned on the downstream target task. Self-supervised learning facilitates the use of pre-trained architectures and larger datasets that might be deemed unfit for the target task and yet might be useful to learn informative representations and hence provide useful initializations for further fine-tuning on smaller more suitable data. Our presented contribution is two-fold: (1) an analysis of different state-of-the-art (SOTA) pretext tasks for the video-only apparent emotional reaction recognition architecture, and (2) an analysis of various combinations of the regression and classification losses that are likely to improve the performance further. Together these two contributions result in the current state-of-the-art performance for the video-only spontaneous apparent emotional reaction recognition with continuous annotations.
URL: https://research.facebook.com/file/1055065362555485/SS-VAERR-Self-Supervised-Apparent-Emotional-Reaction-Recognition-from-Video.pdf
doi: https://doi.org/10.1109/FG57933.2023.10042638
Accessibilité de l'article: Libre
Champ: Computer vision, machine learning
Type contenu (théorique Applicative méthodologique): Théorique, méthodologique
Méthode: self-supervised learning (SSL), We examine three suitable pretext methods: LiRA [36], BYOL[13], and DINO [14].

Datasets are converted into gray-scale videos and cropped around the face to 96×96 based on the landmark detection. More specifically we use RetinaFace face detector [43] and the Face Alignment Network (FAN) [44] to detect 68 facial landmarks and crop the face based on these.
Cas d'usage: N/A
Objectifs de l'article: "(1) a review of several pretext tasks for apparent emotional
reaction recognition from video for their downstream per-
formance across several spontaneous emotion datasets; (2)
analysis of the impact of the combined regression and classi-
fication losses, data augmentations, and downstream learning parameters; (3) adding up to the first to our knowledge Self-Supervised Visual Apparent Emotional Reaction Recognition method for spontaneous emotions with continuous annotations, SS-VAERR."
Question(s) de recherche/Hypothèses/conclusion: The research question is how to improve video-only spontaneous AERR with continuous annotations using self-supervised learning and fine-tuning on downstream tasks.
"we have presented the first to our knowledge a self-supervised technique for the video-only natural apparent emotional reactions recognition, yielding the current state-of-the-art (or closely comparable) results for
video-only natural AERR."; The hypothesis is that self-supervised learning and fine-tuning on downstream tasks can improve video-only spontaneous AERR with continuous annotations.
"we argue that the facial apparent emotional reactions recognition is highly data-specific. [...] video tends to be a better indicator for the video-aided recognition, and arousal tends to be better detected from audio modality. This makes the video-only AERR particularly challenging in terms of identifying the correct levels of arousal, and explains valence-arousal discrepancy for several results in this paper."; The conclusions are that the proposed method achieves state-of-the-art performance for video-only spontaneous AERR with continuous annotations, and that the choice of pretext task and combination of losses can impact downstream performance.
"The self- supervised setting alone helps beating (or at least reaching
comparable results with) the current state-of-the-art without
even touching upon the loss function design."
Cadre théorique/Auteur.es: The theoretical framework of the article includes self-supervised learning and apparent emotional reaction recognition. The main authors cited include E. Sanchez, M. K. Tellamekala, M. Hu, Q. Chu, J. Kossaifi, and R. Walecki.
Concepts clés: Emotional reaction recognition
Données collectées (type source): Lip Reading Sentences 3 dataset
(LRS3) [42], containing thousands of spoken sentences from TED and TEDx videos.

SEWA database consists of the videos of volunteers watching adverts chosen to elicit apparent emotional
reactions, and later discussing what they have seen

RECOLA is a database of multi-domain data recordings of
native French-speaking participants completing a collaborative task in pairs during a video conference call, collected in France.
Définition des émotions: Categorical emotions
Ampleur expérimentation (volume de comptes): Thousands
Technologies associées: Retina Face Detector, Face Alignment Net-
work (FAN), Machine learning, Deep learning, Computer vision
Mention de l'éthique: Non
Finalité communicationnelle: As a result, we achieve the current state-of-the-art performance for video-only spontaneous AERR with continuous annotations.
Commentaires: Contenu supplémentaire vidéo : https://www.facebook.com/watch/?v=3040865399547468

Collections: Facebook

Google

Auteur-es: Ryokai, Kimiko; Raffle, Hayes; Kowalski, Robert
Nombre Auteurs: 3
Titre: StoryFaces: pretend-play with ebooks to support social-emotional storytelling
Année de publication: 2012
Référence (APA): Ryokai, K., Raffle, H., & Kowalski, R. (2012). StoryFaces : Pretend-play with ebooks to support social-emotional storytelling. Proceedings of the 11th International Conference on Interaction Design and Children, 125‑133. https://doi.org/10.1145/2307096.2307111
Mots-clés: storytelling, social-emotional development, video recording, children, communication tools, facial expressions
URL: https://people.ischool.berkeley.edu/~kimiko/papers/IDC.2012.Ryokai.StoryFaces.pdf
doi: https://doi.org/10.1145/2307096.2307111
Accessibilité de l'article: Open access
Champ: Human-Computer Interaction and Visualization
Type contenu (théorique Applicative méthodologique): Applicative
Méthode: StoryFaces invites children to record emotional expressions that become part of storybook illustrations.
As children watch the story being told, they see their faces bring the story to life; then they can "go backstage" to play with the story by rearranging the videos and altering the story text. More experienced children can build an interactive ebook from the ground up, creating scenes, characters and expressive faces to craft personalized narratives.
Our design rationale focuses on supporting children's exploration of emotional expression through their narrative play.
Cas d'usage: StoryFaces, digital tool for children
Objectifs de l'article: We introduce StoryFaces, a new composition and storytelling tool for children to explore the role of emotional expressions in children's narrative. Our goal is to provoke new ideas about how pretend play with digital tools can empower young children in social-emotional narrative activities.
We are interested in studying which kinds of expressions and stories children at various stages of development can successfully engage in with StoryFaces. [...] The primary goal of our study was not to quantify children's performance with StoryFaces in an empirical setting, but rather to observe the kinds of expressive activities children were able to engage in with the tool.
Question(s) de recherche/Hypothèses/conclusion: Research question(s) : Facial expression is one of the strong non-verbal communication cues we humans use in understanding one another. For young children, understanding facial expressions seems to play a role in their cognitive, social, and language development [3]. While it is assumed that children naturally develop these capabilities in interaction with other people in their environment, research suggests benefits in fostering these skills, and these skills seem to be teachable [9]. [...] There is a clear opportunity for open-ended play with digital tools today. [...] While games can teach explicit knowledge like math or spelling, they do not usually encourage children's creativity or social play and learning. But games keep children engaged; can open-ended digital play engage children as well?; Hypothesis(es) : In this paper, we argue that children's natural interest in pretend play and emotional expression can provide an appealing point of entry for children to engage in self-guided play and learning with digital tools.; Conclusion(s) : Results indicate that emotional expressions are inviting and motivating for children across this broad age range to engage in both reading and creating narrative. StoryFaces gave children's ephemeral facial expressions concrete forms with which they could manipulate, discuss, and think about the role of emotion in narratives.
Cadre théorique/Auteur.es: Developmental psychology; Facial expressions and development of social skills in children (Bergen, 2002; Declerck and Bogaert, 2008; Marsh, Kozak and Ambady, 2007; Walden and Field, 1982; Grinspan, Hemphill and Nowicki, 2003; Profyt and Whissell, 1991; Widen and Russell, 2003); Role-playing and storytelling (Goodman, 1996; Singer, 1998; Teale and Sulzby, 1986; Vygotsky, 1986); Theory of multiple intelligences (Gardner, 1982); Theory of children's co-evolution and creative tools (Raffle, 2008); Digital tools developed in Human-Computer Interaction to develop children's knowledge (Hourcade et al., 2002 ; Humphries, and McDonald, 2011 ; Kapoor, Mota and Picard, 2001 ; Raffle et al., 2010 ; Raffle et al., 2011 ; Raffle et al., 2007 ; Tseng and Ellen, 2010 ; Ryokai, Kowalski, and Raffle, 2011)
Concepts clés: Development of social skills
Données collectées (type source): In our study, we focused on this age group [preschool (age 4-5), and elementary school (age 6-10)] to observe how StoryFaces supported these emergent expressive skills in pretend play and storytelling.
All children played with StoryFaces in a pair with another playmate within the same age group. One of the premade stories, "Walking in the Woods" was used as a practice story with some assistance from a researcher. After the practice story, children were invited to play with StoryFaces by themselves. The children were told that they were free to play with it as long as they wished. Children's sessions were video recorded for later analyses.
Définition des émotions: No definition; Evokes children's understanding of facial emotions
Ampleur expérimentation (volume de comptes): 18 children participated in the study: 8 children were enrolled in a preschool (age 4-5), and 10 children were enrolled in an elementary school (age 6-10).
Each session lasted approximately 45 minutes.
Technologies associées: StoryFaces is written in ActionScript 3, which provides access to video and audio hardware as well as an animation framework
Mention de l'éthique: ND
Finalité communicationnelle: Looking ahead, we would also like to see how the findings of this research combining narrative and emotion may help children with autism spectrum disorder.
Further, as more children are using tablets in their everyday learning, we hope this research can inspire a new generation of software and content designers to use media in ways that engage children's creativity and social- emotional learning.
Résumé: We introduce StoryFaces, a new composition and storytelling tool for children to explore the role of emotional expressions in children's narrative. StoryFaces invites children to record emotional expressions that become part of storybook illustrations. As children watch the story being told, they see their faces bring the story to life; then they can "go backstage" to play with the story by rearranging the videos and altering the story text. More experienced children can build an interactive ebook from the ground up, creating scenes, characters and expressive faces to craft personalized narratives. Our design rationale focuses on supporting children's exploration of emotional expression through their narrative play. Results with eighteen children ages 4-10 indicate that emotional expressions are inviting and motivating for children across this broad age range to engage in both reading and creating narrative. StoryFaces gave children's ephemeral facial expressions concrete forms with which they could manipulate, discuss, and think about the role of emotion in narratives. Our goal is to provoke new ideas about how pretend play with digital tools can empower young children in social-emotional narrative activities.

Collections: Google

Auteur-es: Shafran, I.; Mohri, M.
Nombre Auteurs: 2
Titre: A comparison of classifiers for detecting emotion from speech
Année de publication: 2005
Référence (APA): Shafran, I., & Mohri, M. (2005). A comparison of classifiers for detecting emotion from speech. Proceedings. (ICASSP ’05). IEEE International Conference on Acoustics, Speech, and Signal Processing, 2005., 1, I/341-I/344 Vol. 1. https://doi.org/10.1109/ICASSP.2005.1415120
Mots-clés: ND
URL: https://ieeexplore.ieee.org/document/1415120
doi: https://doi.org/10.1109/ICASSP.2005.1415120
Accessibilité de l'article: Open access
Champ: Machine Intelligence
Type contenu (théorique Applicative méthodologique): Applicative
Méthode: This paper compares several techniques for detecting emotion by evaluating their performance on a common corpus of speech data collected from a deployed customer-care application (HMIHY 0300).
Emotion detection classifiers can use diverse information sources, e.g., acoustic or lexical information. To use a common set of input features, we compared classifiers using spoken words as the input.
We present a comparison of three classification algorithms that we have implemented: two popular classifiers from the literature modeling the word content via n-gram sequences, one based on an interpolated language model [6], another on a mutual information-based (MI-based) feature-selection approach [9, 8], and compare them with a discriminant kernelbased technique that we recently adopted [2, 13].
Cas d'usage: Data extracted from a customer service system (AT&T "How May I Help You")
Objectifs de l'article: This paper compares several techniques for detecting emotion by evaluating their performance on a common corpus of speech data collected from a deployed customer-care application (HMIHY 0300).
This paper presents three classifiers, two popular classifiers from the literature modeling the word content via n-gram sequences, one based on an interpolated language model, another on a mutual information-based feature-selection approach, and compares them with a discriminant kernel-based technique that we recently adopted.
Question(s) de recherche/Hypothèses/conclusion: Research question(s) : Several techniques for detecting emotion from speech have been recently described. But the relative performance of these techniques has not been measured since the experiments reported by the authors were carried out on distinct corpora. The results reported are not always indicative of the performance of these techniques in real-world applications. Some are based on the unrealistic assumption that the word transcription of the spoken utterance is given in advance. Others are derived from experiments with speech data produced by professional actors expressing distinct emotion categories.; Hypothesis(es) : This paper compares several techniques for detecting emotion by evaluating their performance on a common corpus of speech data collected from a deployed customer-care application (HMIHY 0300). [...] To use a common set of input features, we compared classifiers using spoken words as the input. We present a comparison of three classification algorithms that we have implemented.; Conclusion(s) : The results show that our kernelbased classifier achieves an accuracy of 80.6%, and outperforms both the interpolated language model classifier, which achieved a classification accuracy of 70.1%, and the classifier using mutual information-based feature selection (78.8%).
[...]
The results reflect the performance of these classifiers in a real-word task since the data used in our experiments was extracted from a deployed customer-care system, (HMIHY 0300). They demonstrate that the discriminant classifier based on rational kernels outperforms the two other popular classification techniques.
Cadre théorique/Auteur.es: Detecting emotion from speech (Batliner et al., 2000 ; Dellaert, Polzin, et Waibel, 1996 ; Devillers et Vasilescu, 2003 ; Devillers, Vasilescu, et Lamel, 2003 ; Lee et Narayanan, 2004 ; Lee, Narayanan, et Pieraccini, 2002 ; Petrushin, 2000 ; Polzin et Waibel, 1998 ; Shafran, Riley, et Mohri, 2003); Classifiers using spoken words as the input (Cortes, Haffner, et Mohri, 2004 ; Devillers, Vasilescu, et Lamel, 2003 ; Lee et Narayanan, 2004 ; Lee, Narayanan, et Pieraccini, 2002 ; Shafran, Riley, et Mohri, 2003)
Concepts clés: Sentiment analysis
Données collectées (type source): We evaluated their performance on data extracted from a deployed customercare system, the AT&T “How May I Help You” system (HMIHY 0300).

The corpus used consisted of utterances from speakers. The emotion category of the speaker for each utterance was originally tagged into one of seven emotion categories [Shafran, Riley, and Mohri, 2003]. For this study, they were grouped into only two categories – negative and non-negative [Lee and Narayanan, 2004]. The utterances were presented to human annotators in the order of occurrence, thus they had the advantage of knowing the context beyond the utterance being labeled.
Définition des émotions: No definition; Use of sentiment categories/groups; Negative, neutral, positive labeling
Ampleur expérimentation (volume de comptes): The corpus used consisted of 5147 utterances from 1854 speakers.
On the average, the utterances were about 15 words long. A subset of 448 utterance was used for testing on which two human labelers were in full agreement.
Technologies associées: N-gram sequences; Interpolated Language Model Classifier; Mutual Information-based Feature-Selection Classifier; Kernel-Based Discriminant Classifier
Mention de l'éthique: ND
Finalité communicationnelle: Accurate detection of emotion from speech has clear benefits for the design of more natural human-machine speech interfaces or for the extraction of useful information from large quantities of speech data. It can help design more natural spoken-dialog systems than those currently deployed in call centers or used in tutoring systems. The speaker’s emotion can be exploited by the system’s dialog manager to provide more suitable responses, thereby achieving better task completion rates. Emotion detection can also be used to rapidly identify relevant speech or multimedia documents from a large data set.
Résumé: Accurate detection of emotion from speech has clear benefits for the design of more natural human-machine speech interfaces or for the extraction of useful information from large quantities of speech data. The task consists of assigning, out of a fixed set, an emotion category, e.g., anger, fear, or satisfaction, to a speech utterance. In recent work, several classifiers have been proposed for automatic detection of a speaker's emotion using spoken words as the input. These classifiers were designed independently and tested on separate corpora, making it difficult to compare their performance. This paper presents three classifiers, two popular classifiers from the literature modeling the word content via n-gram sequences, one based on an interpolated language model, another on a mutual information-based feature-selection approach, and compares them with a discriminant kernel-based technique that we recently adopted. We have implemented these three classification algorithms and evaluated their performance by applying them to a corpus collected from a spoken-dialog system that was widely deployed across the USA. The results show that our kernel-based classifier achieves an accuracy of 80.6%, and outperforms both the interpolated language model classifier, which achieved a classification accuracy of 70.1%, and the classifier using mutual information-based feature selection (78.8%).

Collections: Google

Auteur-es: Cui, Hang; Mittal, Vibhu; Datar, Mayur
Nombre Auteurs: 3
Titre: Comparative experiments on sentiment classification for online product reviews
Année de publication: 2006
Référence (APA): Cui, H., Mittal, V., & Datar, M. (2006). Comparative experiments on sentiment classification for online product reviews. Proceedings of the 21st national conference on Artificial intelligence - Volume 2, 1265‑1270. https://dl.acm.org/doi/10.5555/1597348.1597389
Mots-clés: ND
URL: https://dl.acm.org/doi/10.5555/1597348.1597389
Accessibilité de l'article: Open access
Champ: Natural Language Processing
Type contenu (théorique Applicative méthodologique): Applicative
Méthode: We conduct experiments on a corpus of online reviews with an average length of over 800 bytes crawled from the Web. Such a large-scale data set allows us not only to train language models and cull high order n-grams as features, but also to study the effectiveness and robustness of classifiers in a simulating context of the Web.

We study multiple classification algorithms for processing large-scale data. We employ three algorithms: (i) Winnow (Nigam & Hurst 2004), (ii) a generative model based on language modeling, and (iii) a discriminative classifier that employs projection based learning (Shalev-Shwartz et al. 2004).
Cas d'usage: Online customer reviews
Objectifs de l'article: This paper looks at a simplified version of the problem: classifying online product reviews into positive and negative classes. We discuss a series of experiments with different machine learning algorithms in order to experimentally evaluate various trade-offs.
Question(s) de recherche/Hypothèses/conclusion: Research question(s) : Evaluating text fragments for positive and negative subjective expressions and their strength can be important in applications such as single - or multi- document summarization, document ranking, data mining, etc. This paper looks at a simplified version of the problem : classifying online product reviews into positive and negative classes.; Hypothesis(es) : We conjecture that those experiments were hindered by the small training corpora, and thus were not able to show the effectiveness of high order n-grams (n > 3) indiscerning subtleties in expressing sentiments.
[...]
One of the main difficulties is that people typically use both positive and negative words in the same review ,regardless of the rating score. As such, we hypothesize that a discriminative classifier could gain more strength in differentiating the mixed sentiments. We experimentally compare a discriminative model with a generative model by language modeling to verify this hypothesis.; Conclusion(s) : In this paper, we presented the experiments we have done on sentiment classification using large-scale data set. Our experimental results show that a discriminating classifier combined with high order n-grams as features can achieve comparable, or better performance than that reported in academic papers. More importantly, this paper shows that sentiment classification is possible to be learned from online product reviews, even with very disparate products and authors. In addition, we have shown that high order n-grams do help indiscriminating the articles’ polarity in the mixture context. This observation based on large-scale data set has never been testified before.
Cadre théorique/Auteur.es: Classification of words according to their semantic orientation (Hatzivassiloglou et McKeown, 1997 ; Turney et Littman, 2003); Sentiment classification on the article level (Pang et al., 2002 ; Pang & Lee, 2004); Classifying polarity of documents (Nigam and Hurst, 2004); Online product reviews (Dave et al., 2003 ; Hu and Liu, 2004 ; Popescu and Etzioni, 2005 ; Wilson et al., 2005); Classifier (Shalev-Shwartz et al., 2004 ; Manning & Schütze, 1999 ; Nigam and Hurst, 2004)
Concepts clés: Sentiment analysis
Données collectées (type source): We accumulate reviews about electronic products like digital cameras, laptops, PDAs, MP3 players, etc. from Froogle [Google Shopping].

Each review comes with the full text and the rating score by the reviewer. These reviews are crawled from prominent sites, such as cnet.com, ciao.co.uk and shopping.yahoo.com.
Définition des émotions: No definition; Negative, neutral, positive labeling
Ampleur expérimentation (volume de comptes): The size of the whole corpus is around 0.4GB, including a total of over 320k product reviews about over 80k unique products. The average length of the reviews is 875 bytes.
Technologies associées: Passive-Aggressive (PA) Algorithm Based Classifier; Language Modeling (LM) Based Classifier; Winnow Classifier; N-gram models
Mention de l'éthique: ND
Finalité communicationnelle: A large amount of Web content is subjective and reflects peoples’ opinions. With the rapid growth of the Web, more and more people write reviews for all types of products and services and place them online. It is becoming a common practice for a consumer to learn how others like or dislike a product before buying, or for a manufacturer to keep track of customer opinions on its products to improve the user satisfaction. However, as the number of reviews available for any given product grows, it becomes harder and harder for people to understand and evaluate what the prevailing/majority opinion about the product is.
Sentiment classification, also known as affect or polarity classification, attempts to address this problem by (i) presenting the user with an aggregate view of the entire data set, summarized by a label or a score, and (ii) segmenting the articles/text-fragments into two classes that can be further explored as desired. While many review sites, such as Epinions, CNet and Amazon, help reviewers quantify the positivity of their comments, sentiment classification can still play an important role in classifying documents that do not have explicit ratings.
Often, web sites, such as personal blogs, have user reviews with personal experiences in using a particular product without giving any score. The review comments from these sites are valuable because they cover a lot more products than those formal review sites.
Résumé: Evaluating text fragments for positive and negative subjective expressions and their strength can be important in applications such as single- or multi- document summarization, document ranking, data mining, etc. This paper looks at a simplified version of the problem: classifying online product reviews into positive and negative classes. We discuss a series of experiments with different machine learning algorithms in order to experimentally evaluate various trade-offs, using approximately 100K product reviews from the web.

Collections: Google

Auteur-es: McDonald, Ryan; Hannan, Kerry; Neylon, Tyler; Wells, Mike; Reynar, Jeff
Nombre Auteurs: 5
Titre: Structured Models for Fine-to-Coarse Sentiment Analysis
Année de publication: 2007
Référence (APA): McDonald, R., Hannan, K., Neylon, T., Wells, M., & Reynar, J. (2007). Structured Models for Fine-to-Coarse Sentiment Analysis. Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, 432‑439. https://aclanthology.org/P07-1055
Mots-clés: ND
URL: https://aclanthology.org/P07-1055
Accessibilité de l'article: Open access
Champ: Natural Language Processing
Type contenu (théorique Applicative méthodologique): Applicative
Méthode: We describe a simple structured model that jointly learns and infers sentiment on different levels of granularity
Cas d'usage: Online customer reviews
Objectifs de l'article: In this paper we investigate a structured model for jointly classifying the sentiment of text at varying levels of granularity.
In this paper we have investigated the use of a global structured model that learns to predict sentiment on different levels of granularity for a text.
Question(s) de recherche/Hypothèses/conclusion: Research question(s) : Extracting sentiment from text is a challenging problem with applications throughout Natural Language Processing and Information Retrieval. [...] The ability to classify sentiment on multiple levels is important since different applications have different needs. [...] We call the problem of identifying the sentiment of the document and of all its subcomponents, whether at the paragraph, sentence, phrase or word level, fine-to-coarse sentiment analysis.; Hypothesis(es) : The simplest approach to fine-to-coarse sentiment analysis would be to create a separate system for each level of granularity. There are, however, obvious advantages to building a single model that classifies each level in tandem. [...]
This work focuses on models that jointly classify sentiment on multiple levels of granularity.; Conclusion(s) : We described a simple model for sentence-document analysis and showed that inference in it is tractable. Experiments show that this model obtains higher accuracy than classifiers trained in isolation as well as cascaded systems that pass information from one level to another at test time. Furthermore, extensions to the sentence-document model were discussed and it was argued that a nested hierarchical structure would be beneficial since it would allow for efficient inference algorithms.
Cadre théorique/Auteur.es: Sentiment analysis (Pang et al., 2002 ; Turney, 2002 ; Pang et Lee, 2004 ; Choi et al., 2005 ; Choi et al., 2006 ; Mao et Lebanon, 2006 ; Thomas et al., 2006); Structured learning algorithms (Rabiner, 1989 ; Lafferty et al., 2001 ; Sutton et McCallum, 2006 ; Taskar et al., 2003 ; Tsochantaridis et al., 2004 ; McDonald et al., 2005 ; Daumé III et al., 2006 ; Taskar et al., 2004 ; Liang et al, 2006); Structured models used for sentiment analysis (Choi et al., 2005 ; Choi et al., 2006 ; Mao et Lebanon, 2006); Cascaded models for fine-to-coarse sentiment analysis (Pang et Lee, 2004); Learning and/or predicting multiple outputs jointly (Miller et al., 2000 ; Roth et Yih, 2004 ; Sutton et al., 2004 ; Popescu et Etzioni, 2005)
Concepts clés: Sentiment analysis
Données collectées (type source): To test the model we compiled a corpus of online product reviews from three domains: car seats for children, fitness equipment, and Mp3 players.

We discarded duplicate reviews, reviews with insufficient text, and spam. All reviews were labeled by online customers as having a positive or negative polarity on the document level, i.e., Y(d) = {pos, neg}. Each review was then split into sentences and every sentence annotated by a single annotator as either being positive, negative or neutral, i.e., Y(s) = {pos, neg, neu}. All sentences were annotated based on their context within the document.
Définition des émotions: No definition; Negative, neutral, positive labeling
Ampleur expérimentation (volume de comptes): Corpus of 600 online product reviews, 554 kept

Car seats : 178 reviews, 1179 sentences
Fitness equipment : 189 reviews, 1574 sentences
Mp3 players : 187 reviews, 1163 sentences
TOTAL : 554 reviews, 3916 sentences
Technologies associées: Standard sequence classification techniques using constrained Viterbi
Structured models
Mention de l'éthique: ND
Finalité communicationnelle: One interesting application of sentence level sentiment analysis is summarizing product reviews on retail websites like Amazon.com or review aggregators like Yelp.com.

Extracting sentiment from text is a challenging problem with applications throughout Natural Language Processing and Information Retrieval.
Résumé: In this paper we investigate a structured model for jointly classifying the sentiment of text at varying levels of granularity. Inference in the model is based on standard sequence classification techniques using constrained Viterbi to ensure consistent solutions. The primary advantage of such a model is that it allows classification decisions from one level in the text to influence decisions at another. Experiments show that this method can significantly reduce classification error relative to models trained in isolation.

Collections: Google

Auteur-es: Blair-Goldensohn, Sasha; Hannan, Kerry; McDonald, Ryan; Neylon, Tyler; Reis, George; Reynar, Jeff
Nombre Auteurs: 5
Titre: Building a Sentiment Summarizer for Local Service Reviews
Année de publication: 2008
Référence (APA): Blair-Goldensohn, S., Hannan, K., McDonald, R., Neylon, T., Reis, G., & Reynar, J. (2008). Building a Sentiment Summarizer for Local Service Reviews. WWW Workshop on NLP Challenges in the Information Explosion Era (NLPIX). https://storage.googleapis.com/pub-tools-public-publication-data/pdf/34368.pdf
Mots-clés: ND
URL: https://storage.googleapis.com/pub-tools-public-publication-data/pdf/34368.pdf
Accessibilité de l'article: Open access
Champ: Natural Language Processing
Type contenu (théorique Applicative méthodologique): Applicative
Méthode: In this paper, we present a system that summarizes the sentiment of reviews for a local service such as a restaurant or hotel.
In particular we focus on aspect-based summarization models [8], where a summary is built by extracting relevant aspects of a service, such as service or value, aggregating the sentiment per aspect, and selecting aspect-relevant text.
The model we employ for sentiment classification is a hybrid that uses both lexicon-based and machine learning algorithms.
The model we employ for aspect extraction is a hybrid but this time we combine a dynamic aspect extractor, where aspects are determined from the text of the review alone, and a static extractor, where aspects are pre-defined and extraction classifiers trained on a set of labeled data.
Cas d'usage: Online customer reviews
Objectifs de l'article: Our goal is to create a general system that can handle all services with sufficient accuracy to be of utility to users.
In this paper, we present a system that summarizes the sentiment of reviews for a local service such as a restaurant or hotel. In particular we focus on aspect-based summarization models [8], where a summary is built by extracting relevant aspects of a service, such as service or value, aggregating the sentiment per aspect, and selecting aspect-relevant text.
Question(s) de recherche/Hypothèses/conclusion: Research question(s) : The ability to analyze a set of online reviews and produce an easy to digest summary is a major challenge for online merchants, review aggregators and local search services. In this study, we look at the problem of aspect-based sentiment summarization.; Hypothesis(es) : Central to our system is the ability to exploit different sources of information when available. In particular, we show how user provided document level sentiment can aid in the prediction of sentiment on the phrase/sentence level through a variety of models. Furthermore, we argue that the service domain has specific characteristics that can be exploited in order to improve both quality and coverage of generated summaries. This includes the observation that nearly all services share basic aspects with one another and that a large number of queries for online reviews pertain only to a small number of service types.; Conclusion(s) : The resulting system is highly precise for frequently queried services, yet also sufficiently general to produce quality summaries for all service types. The main technical contributions include new sentiment models that leverage context and user-provided labels to improve sentence level classification as well as a hybrid aspect extractor and summarizer that combines supervised and unsupervised methods to improve accuracy.
Cadre théorique/Auteur.es: Aspect-based sentiment summarization (Carenini, Ng, et Pauls, 2006 ; Gamon et al, 2005 ; Hu et Liu, 2004 ; Popescu et Etzioni, 2005 ; Zhuang, Jing, et Zhu, 2006); Lexicon-based sentiment analysis (Hu et Liu, 2004 ; Turney, 2002 ; Wiebe, 2000); Machine-learning for sentiment analysis (Choi et al., 2005 ; Dredze, Blitzer, et Pereira, 2007 ; Mao et Lebanon, 2006 ; McDonald et al., 2007 ; B. Pang, Lee, et Vaithyanathan, 2002 ; Snyder et Barzilay, 2007); Summarizing sentiment (Beineke et al., 2003); Summarizing sentiment by extracting and aggregating sentiment over ratable aspects (Gamon et al., 2005 ; Hu et Liu, 2004 ; Hu et Liu, 2004 ; Carenini, Ng, et Paul, 2006 ; Carenini, Ng, et Zwart, 2005 ; Popescu et Etzioni, 2005)
Concepts clés: Sentiment analysis; Aspect Ratings
Données collectées (type source): Dataset of local business reviews collected on Google Maps (maps.google.com)
Définition des émotions: No definition; Negative, neutral, positive labeling
Ampleur expérimentation (volume de comptes): The amount of review input data varies, but each example has a minimum of 16 input reviews (56 total sentences) of input text.

Department Store (43 Reviews)
Greek Restaurant (85 Reviews)
Children’s Barber Shop (16 Reviews)
Hotel/Casino (46 Reviews)
Technologies associées: Aspect-based summarization models
Mention de l'éthique: ND
Finalité communicationnelle: In the future we plan to adapt the system to products, which is a domain that has been well studied in the past. Just as in services, we believe that hybrid models can improve system performance since there again exists a pattern that a few products account for most review queries (e.g., electronics). Additionally, there is a set of aspects that is common across most products, such as customer service, warranty, and value, which can be utilized to improve the performance for less queried products.
Résumé: Online user reviews are increasingly becoming the de-facto standard for measuring the quality of electronics, restau- rants, merchants, etc. The sheer volume of online reviews makes it dicult for a human to process and extract all meaningful information in order to make an educated pur- chase. As a result, there has been a trend toward systems that can automatically summarize opinions from a set of re- views and display them in an easy to process manner (1, 9). In this paper, we present a system that summarizes the sen- timent of reviews for a local service such as a restaurant or hotel. In particular we focus on aspect-based summarization models (8), where a summary is built by extracting relevant aspects of a service, such as service or value, aggregating the sentiment per aspect, and selecting aspect-relevant text. We describe the details of both the aspect extraction and sentiment detection modules of our system. A novel aspect of these models is that they exploit user provided labels and domain specic characteristics of service reviews to increase quality.

Collections: Google

Auteur-es: Titov, Ivan; McDonald, Ryan
Nombre Auteurs: 2
Titre: A Joint Model of Text and Aspect Ratings for Sentiment Summarization
Année de publication: 2008
Référence (APA): Titov, I., & McDonald, R. (2008). A Joint Model of Text and Aspect Ratings for Sentiment Summarization. Proceedings of ACL-08: HLT, 308‑316. https://aclanthology.org/P08-1036
Mots-clés: ND
URL: https://aclanthology.org/P08-1036
Accessibilité de l'article: Open access
Champ: Natural Language Processing
Type contenu (théorique Applicative méthodologique): Applicative
Méthode: In this paper we presented a joint model of text and aspect ratings for extracting text to be displayed in sentiment summaries.
The model uses aspect ratings
We propose a statistical model which is able to discover corresponding topics in text and extract textual evidence from reviews.
We propose an unsupervised model that leverages aspect ratings that frequently accompany an online review.
Cas d'usage: Online customer reviews
Objectifs de l'article: We propose a statistical model which is able to discover corresponding topics in text and extract textual evidence from reviews supporting each of these aspect ratings – a fundamental problem in aspect-based sentiment summarization (Hu and Liu, 2004a).
Question(s) de recherche/Hypothèses/conclusion: Research question(s) : The most pressing challenge in an aspect-based summarization system is to extract all relevant mentions for each aspect [...]. When labeled data exists, this problem can be solved effectively using a wide variety of methods available for text classification and information extraction. However, labeled data is often hard to come by, especially when one considers all possible domains of products and services.; Hypothesis(es) : Instead, we propose an unsupervised model that leverages aspect ratings that frequently accompany an online review. In order to construct such model, we make two assumptions. First, ratable aspects normally represent coherent topics which can be potentially discovered from co-occurrence information in the text. Second, we hypothesize that the most predictive features of an aspect rating are features derived from the text segments discussing the corresponding aspect. Motivated by these observations, we construct a joint statistical model of text and sentiment ratings.; Conclusion(s) : Our model achieves high accuracy, without any explicitly labeled data except the user provided opinion ratings.
We demonstrated that the model indeed discovers corresponding coherent topics and achieves accuracy in sentence labeling comparable to a standard supervised model.
Cadre théorique/Auteur.es: Aspect-based sentiment summarization (Hu et Liu, 2004a ; Popescu et Etzioni, 2005 ; Gamon et al., 2005 ; Carenini et al., 2006 ; Zhuang et al., 2006); Sentiment classification (Wiebe, 2000 ; Pang et al., 2002 ; Turney, 2002); Aspect identification (Hu et Liu, 2004b ; Gamon et al, 2005 ; Titov et McDonald, 2008); Text classification and information extraction (Manning et Schutze, 1999); Summarizing sentiment by extracting and aggregating sentiment over ratable aspects (Hu et Liu, 2004a ; Popescu et Etzioni, 2005 ; Gamon et al, 2005 ; Mei et al., 2007 ; Titov et McDonald, 2008); Joint sentiment and topic modeling (Blei et McAuliffe, 2008 ; Branavan et al., 2008)
Concepts clés: Sentiment analysis; Aspect Ratings
Données collectées (type source): Qualitative evaluation : To perform qualitative experiments we used a set of reviews of hotels taken from TripAdvisor.com. Every review was rated with at least three aspects: service, location and rooms. Each rating is an integer from 1 to 5. The dataset was tokenized and sentence split automatically.

Quantitative evaluation : [...] We hand labeled random sentences from the dataset considered in the previous set of experiments. The sentences were labeled with one or more aspects, related to aspects service, location and rooms.
Définition des émotions: No definition
Ampleur expérimentation (volume de comptes): Qualitative evaluation : 10,000 reviews of hotels (109,024 sentences, 2,145,313 words in total)

Quantitative evaluation : We hand labeled 779 random sentences with one or more aspects. Among them, 164, 176 and 263 sentences were labeled as related to aspects service, location and rooms, respectively. The remaining sentences were not relevant to any of the rated aspects.
Technologies associées: Multi-Aspect Sentiment model (MAS); Multi-Grain Latent Dirichlet Allocation model (MG-LDA)
Mention de l'éthique: ND
Finalité communicationnelle: The proposed approach is general and can be used for segmentation in other applications where sequential data is accompanied with correlated signals.

The primary area of future work is to incorporate the model into an end-to-end sentiment summarization system in order to evaluate it at that level.
Résumé: Online reviews are often accompanied with numerical ratings provided by users for a set of service or product aspects. We propose a statistical model which is able to discover corresponding topics in text and extract textual evidence from reviews supporting each of these aspect ratings – a fundamental problem in aspect-based sentiment summarization (Hu and Liu, 2004a). Our model achieves high accuracy, without any explicitly labeled data except the user provided opinion ratings. The proposed approach is general and can be used for segmentation in other applications where sequential data is accompanied with correlated signals.

Collections: Google

Auteur-es: Francis, Anthony G. Jr.; Mehta, Manish; Ram, Ashwin
Nombre Auteurs: 3
Titre: Emotional Memory and Adaptive Personalities
Année de publication: 2009
Référence (APA): Francis, A. G. Jr., Mehta, M., & Ram, A. (2009). Emotional Memory and Adaptive Personalities. In Handbook of Research on Synthetic Emotions and Sociable Robotics : New Applications in Affective Computing and Artificial Intelligence (p. 391‑412). IGI Global. https://doi.org/10.4018/978-1-60566-354-8.ch020
Mots-clés: ND
URL: https://web.archive.org/web/20150905184906id_/http://www.cc.gatech.edu/faculty/ashwin/papers/er-08-10.pdf
Accessibilité de l'article: Open access
Champ: Machine Intelligence
Type contenu (théorique Applicative méthodologique): Applicative
Méthode: We propose to create believable, engaging artificial characters capable of long-term interaction with a human user by explicitly modeling the emotional adaptation that goes on in humans and animals.
We developed two implementations of the PEPE architecture (Projet "Personal Pet", un animal de compagnie artificiel capable d'interagir avec plusieurs utilisateurs humains & Jack et Jill, deux personnages incarnés créés à la main et conçus pour jouer à un jeu de Tag).
The first was used for testing reactive control and facial recognition [Stoytchev & Tanawongsuwan 1998]. The second implementation was focused on testing the emotional long term memory system.
Cas d'usage: Project "Personal Pet" (PEPE), an artificial pet capable of interacting with several human users.
Two embodied agents (characters in a game) involved in a game of tag (wolf game).
Objectifs de l'article: To aid the authoring of adaptive agents, we present an artificial intelligence model inspired by these psychological results in which an emotion model triggers case-based emotional preference learning and behavioral adaptation guided by personality models.
Question(s) de recherche/Hypothèses/conclusion: Research question(s) : Believable agents designed for long-term interaction with human users need to adapt to them in a way which appears emotionally plausible while maintaining a consistent personality. For short-term interactions in restricted environments, scripting and state machine techniques can create agents with emotion and personality, but these methods are labor intensive, hard to extend, and brittle in new environments.; Hypothesis(es) : Fortunately, research in memory, emotion and personality in humans and animals points to a solution to this problem. [...]
We argue that robots and synthetic characters should have the same ability to interpret their interactions with us, to remember these interactions, and to recall them appropriately as a guide for future behaviors, and we present a working model of how this can be achieved.
[...] It may seem a tall order make robots have this kind of flexibility – but we argue it is possible by using emotion to trigger behavior revision guided by a personality model, and we present a working model of how it can be achieved.
[...] We argue that using explicit emotion models integrated into an agent’s memory but guided by the agent’s personality model can aid the development of agents for long term interaction; Conclusion(s) : Our tests of this model on robot pets and embodied characters show that emotional adaptation can extend the range and increase the behavioral sophistication of an agent without the need for authoring additional hand-crafted behaviors.
Cadre théorique/Auteur.es: Believable agents (Loyall, 1997); Techniques for Creating Agent Personalities (Johnston et Thomas, 1995 ; Paiva, 2005 ; Standife,r 1995 ; Saltzman 1999 ; Schwab 2004 ; Millington 2006 ; Huebner, 2000 ; Spector, 2000 ; Ohlen et al., 2001); Challenges in Creating Agent Personalities (Mateas et Stern, 2003 ; Reilly, 1996 ; Maxis, 2000); Nature of Emotion (LeDoux, 1996 ; Damasio, 2000 ; Minsky, 2007 ; Simon, 1983 ; Frijda, 1987 ; Ohman et al., 2000 ; Winkielman et Berridge, 2004 ; Ruys & Stapel, 2008); Memory and Learning (Anderson, 2000 ; Tulving & Craik, 2000 ; Purdy et al., 2001 ; Gluck, 2008 ; McGaugh, 2003 ; McGaugh, 2007 ; Haist et al., 2001 ; Bartlett, 1932); Relationship of Memory and Emotion (McGaugh, 2003 ; Gluck et al., 2008 ; Benjamin et al., 1981); Personality and Self-Regulation (Caprara & Cervone, 2000 ; Minsky, 2007); Modeling Emotion in Intelligent Systems (Simon, 1983 ; Frijda, 1987 ; Velasquez, 1997 ; Velasquez, 1998 ; Ekman, 1992, Izard, 1991 ; Johnson-Laird et Oately, 1992 ; Ortony, Clore et Collins, 1988 ; Reilly, 1996 ; Elliott, 1992 ; Studdard, 1995 ; Koda, 1996 ; Karunaratne & Yan, 2001 ; Bartneck, 2002 ; Li et al., 2007); Memory Retrieval and Machine Learning (Mitchell, 1997 ; Alpaydin, 2004 ; Kolodner, 1993 ; Kolodner, 1984 ; Sutton & Barto, 1998 ; Santamaria, 1997); Personality Regulation and Behavior Transformation (Peot et Smith, 1992 ; Weld et. al., 1998)
Concepts clés: Emotional memory; Adaptive personality; Emotional Long Term Memory
Données collectées (type source): Behavioral data for artificial characters.
To test the PEPE architecture and our ELTM model, we implemented a simple library of behaviors, such as wandering, approach and avoidance, which could in turn be composed into higher level behaviors such as “playing” (alternately wandering and approaching an object) and “fleeing” (backing up, executing a fast 180, and running away). The emotion model extended this with a simple set of concerns, including avoiding pain, which we derived from “kicks” to the rear sensor, and socialization, which we derived from a combination of proximity to people objects and “petting” the head sensor. The robot had several emotional states, including a neutral state, a “happy” state associated with socialization, and a “fearful” state associated with pain.
Définition des émotions: Definition of emotions
Ampleur expérimentation (volume de comptes): ND
Technologies associées: Pet robots and embodied characters; Artificial Intelligence
Mention de l'éthique: ND
Finalité communicationnelle: Our work used emotion as a trigger for learning about the environment and agents within it, and as a trigger for behavioral change. This model made it possible for us to develop sophisticated and sometimes surprising agent behaviors in less time and with less effort than we could have done otherwise. Therefore, we conclude that emotion-driven learning and emotiondriven behavioral updates are a useful method for developing believable agents that adapt to their environments and users in a way which appears emotionally plausible while maintaining a consistent personality.
Résumé: Believable agents designed for long-term interaction with human users need to adapt to them in a way which appears emotionally plausible while maintaining a consistent personality. For short-term interactions in restricted environments, scripting and state machine techniques can create agents with emotion and personality, but these methods are labor intensive, hard to extend, and brittle in new environments. Fortunately, research in memory, emotion and personality in humans and animals points to a solution to this problem. Emotions focus an animal’s attention on things it needs to care about, and strong emotions trigger enhanced formation of memory, enabling the animal to adapt its emotional response to the objects and situations in its environment. In humans this process becomes reflective: emotional stress or frustration can trigger re-evaluating past behavior with respect to personal standards, which in turn can lead to setting new strategies or goals. To aid the authoring of adaptive agents, we present an artificial intelligence model inspired by these psychological results in which an emotion model triggers case-based emotional preference learning and behavioral adaptation guided by personality models. Our tests of this model on robot pets and embodied characters show that emotional adaptation can extend the range and increase the behavioral sophistication of an agent without the need for authoring additional hand-crafted behaviors.

Collections: Google

Auteur-es: Lerman, Kevin; Blair-Goldensohn, Sasha; McDonald, Ryan
Nombre Auteurs: 3
Titre: Sentiment summarization: evaluating and learning user preferences
Année de publication: 2009
Référence (APA): Lerman, K., Blair-Goldensohn, S., & McDonald, R. (2009). Sentiment summarization : Evaluating and learning user preferences. Proceedings of the 12th Conference of the European Chapter of the Association for Computational Linguistics, 514‑522. https://aclanthology.org/E09-1059
Mots-clés: ND
URL: https://aclanthology.org/E09-1059
Accessibilité de l'article: Open access
Champ: Natural Language Processing
Type contenu (théorique Applicative méthodologique): Applicative
Méthode: Our initial set of experiments were over the three opinion-based summarization systems: SM, SMAC, and SAM.
We evaluated summary performance for reviews of consumer electronics.
Cas d'usage: ND
Objectifs de l'article: We present the results of a large-scale, end-toend human evaluation of three sentiment summarization models applied to user reviews of consumer products.
Question(s) de recherche/Hypothèses/conclusion: Research question(s) : While this means that users have plenty of information on which to base their purchasing decisions, in practice this is often too much information for a user to absorb. To alleviate this information overload, research on systems that automatically aggregate and summarize opinions have been gaining interest. Evaluating these systems has been a challenge, however, due to the number of human judgments required to draw meaningful conclusions.; Hypothesis(es) : The study presented here differs from Carenini et al. in many respects: First, our evaluation is over different extractive summarization systems in an attempt to understand what model properties are correlated with human preference irrespective of presentation; Secondly, our evaluation is on a larger scale including hundreds of judgments by hundreds of raters; Finally, we take a major next step and show that it is possible to automatically learn significantly improved models by leveraging data collected in a large-scale evaluation.; Conclusion(s) : Our results indicated that humans prefer sentiment informed summaries over a simple baseline. This shows the usefulness of modeling sentiment and aspects when summarizing opinions. However, the evaluations also show no strong preference between different sentiment summarizers. A detailed analysis of the results led us to take the next step in this line of research – leveraging preference data gathered in human evaluations to automatically learn new summarization models. These new learned models show large improvements in preference prediction accuracy over the previous single best model.
Cadre théorique/Auteur.es: Systems that automatically aggregate and summarize opinions (Hu and Liu, 2004a ; Hu and Liu, 2004b ; Gamon et al., 2005 ; Popescu and Etzioni, 2005 ; Carenini et al., 2005 ; Carenini et al., 2006 ; Zhuang et al., 2006 ; Blair-Goldensohn et al, 2008); Human evaluation of these systems (McKeown et al., 2005 ; Carenini et Cheung, 2008 ; Carenini et al., 2006); Sentiment Summarization (Jindal et Liu, 2006 ; Stoyanov et Cardie, 2008 ; Hu et Liu, 2004a ; Carenini et al., 2006 ; Choi et al., 2005)
Concepts clés: Sentiment analysis
Données collectées (type source): We evaluated summary performance for reviews of consumer electronics. [...] We gathered reviews for electronics products from several online review aggregators. The products covered a variety of electronics, such as MP3 players, digital cameras, printers, wireless routers, and video game systems. All summaries were roughly equal length to avoid length-based rater bias.
In each experiment two summaries of the same product were placed side-by-side in a random order. Raters were also shown an overall rating, R, for each product. Raters were asked to express their preference for one summary over the other. Raters were free to choose any rating, but were specifically instructed that their rating should account for a summaries representativeness of the overall set of reviews. Raters were also asked to provide a brief comment justifying their rating.
Définition des émotions: Methodological explanation of their classification (quantitative, mathematical); No definition; Positive and negative labeling
Ampleur expérimentation (volume de comptes): Reviews for 165 electronics products from several online review aggregators (each product had a minimum of four reviews and up to a maximum of nearly 3000).
The mean number of reviews per product was 148, and the median was 70. We ran each of our algorithms over the review corpus and generated summaries for each product with K = 650.
In total we ran four experiments for a combined number of 1980 rater judgments.
Technologies associées: 3 extractive sentiment summarization systems :; Sentiment Match (SM); Sentiment Match + Aspect Coverage (SMAC); Sentiment-Aspect Match (SAM)
Mention de l'éthique: ND
Finalité communicationnelle: The growth of the Internet as a commerce medium, and particularly the Web 2.0 phenomenon of user-generated content, have resulted in the proliferation of massive numbers of product, service and merchant reviews. While this means that users have plenty of information on which to base their purchasing decisions, in practice this is often too much information for a user to absorb. To alleviate this information overload, research on systems that automatically aggregate and summarize opinions have been gaining interest.

We can thus conclude that the data gathered in human preference evaluation experiments, such as the one presented here, have a beneficial secondary use as training data for constructing a new and more accurate summarizer.
Résumé: We present the results of a large-scale, end-to-end human evaluation of various sentiment summarization models. The evaluation shows that users have a strong preference for summarizers that model sentiment over non-sentiment baselines, but have no broad overall preference between any of the sentiment-based models. However, an analysis of the human judgments suggests that there are identifiable situations where one summarizer is generally preferred over the others. We exploit this fact to build a new summarizer by training a ranking SVM model over the set of human preference judgments that were collected during the evaluation, which results in a 30% relative reduction in error over the previous best summarizer.

Collections: Google

Auteur-es: Councill, Isaac; McDonald, Ryan; Velikovich, Leonid
Nombre Auteurs: 3
Titre: What's great and what's not: learning to classify the scope of negation for improved sentiment analysis
Année de publication: 2010
Référence (APA): Councill, I., McDonald, R., & Velikovich, L. (2010). What’s great and what’s not : Learning to classify the scope of negation for improved sentiment analysis. Proceedings of the Workshop on Negation and Speculation in Natural Language Processing, 51‑59. https://aclanthology.org/W10-3110
Mots-clés: ND
URL: https://aclanthology.org/W10-3110
Accessibilité de l'article: Open access
Champ: Natural Language Processing
Type contenu (théorique Applicative méthodologique): Applicative
Méthode: Creation of a detection system capable of correctly identifying the presence or absence of negation in portions of text that are expressions of feelings.
Towards this end in we describe both an annotated negation span corpus as well as a negation span detector that is trained on the corpus. The span detector is based on conditional random fields (CRFs) (Lafferty, McCallum, and Pereira, 2001), which is a structured prediction learning framework common in sub-sentential natural language processing tasks, including sentiment analysis (Choi and Cardie, 2007; McDonald et al., 2007)..
Cas d'usage: ND
Objectifs de l'article: The goal of the present work is to develop a system that is robust to differences in the intended scope of negation introduced by the syntactic and lexical features in each negation category. In particular, as the larger context of this research involves sentiment analysis, it is desirable to construct a negation system that can correctly identify the presence or absence of negation in spans of text that are expressions of sentiment.
Question(s) de recherche/Hypothèses/conclusion: Research question(s) : This paper describes an approach to negation scope detection in the context of sentiment analysis, particularly with respect to sentiment expressed in online reviews. The canonical need for proper negation detection in sentiment analysis can be expressed as the fundamental difference in semantics inherent in the phrases, “this is great,” versus, “this is not great.” Unfortunately, expressions of negation are not always so syntactically simple.; Hypothesis(es) : This paper presents a system for identifying the scope of negation using shallow parsing, by means of a conditional random field model informed by a dependency parser.; Conclusion(s) : Results were presented on the standard BioScope corpus that compare favorably to the best results reported to date, using a software stack that is significantly simpler than the best-performing approach. Cross-training by learning a model on one corpus and testing on another suggests that scope boundary detection in the product reviews corpus may be a more difficult learning problem, although the method used to annotate the reviews corpus may result in a more consistent representation of the problem. Finally, the negation system was built into a state-of-the-art sentiment analysis system in order to measure the practical impact of accurate negation scope detection, with dramatic results. The negation system improved the precision of positive sentiment polarity detection by 35.9% and negative sentiment polarity detection by 46.8%. Error reduction on the recall measure was less dramatic, but still significant, showing improved recall for positive polarity of 20.0% and improved recall for negative polarity of 6.6%
Cadre théorique/Auteur.es: Linguistic negation (Givon, 1993, Tottie, 1991); Negation and its scope in the context of sentiment analysis (Moilanen et Pulman, 2007 ; Choi et Cardie, 2008 ; Danescu-Niculescu-Mizil et al., 2009 ; Wilson et al., 2005 ; Nakagawa et al., 2010); Conditional random field (Lafferty, McCallum, and Pereira, 2001); CRF in the context of sentiment analysis (Choi and Cardie, 2007 ; McDonald et al., 2007); CRF and negation in the context of sentiment analysis (Morante and Daelemans, 2009)
Concepts clés: Sentiment analysis
Données collectées (type source): BioScope corpus (Vincze et al., 2008) : annotated clinical radiology reports, biological full papers, and biological abstracts. Annotations in BioScope consist of labeled negation and speculation cues along with the boundary of their associated text scopes. Each cue is associated with exactly one scope, and the cue itself is considered to be part of its own scope.

Product Reviews corpus : a novel corpus was developed containing the text of entire reviews, annotated according to spans of negated text. A sample of product reviews were obtained by randomly sampling reviews from Google Product Search and checking for the presence of negation. Each review was manually annotated with the scope of negation by a single person, after achieving inter-annotator agreement of 91% with a second person on a smaller subset of 20 reviews containing negation.
Définition des émotions: No definition; Negative, neutral, positive labeling
Ampleur expérimentation (volume de comptes): BioScope corpus (Vincze et al., 2008) : 9 papers and a total of 2670 sentences

Product Reviews corpus : A sample of 268 product reviews were obtained by randomly sampling reviews from Google Product Search and checking for the presence of negation. The annotated corpus contains 2111 sentences in total, with 679 sentences determined to contain negation.
Technologies associées: Conditional random field; Structured prediction learning framework
Mention de l'éthique: ND
Finalité communicationnelle: The automatic detection of the scope of linguistic negation is a problem encountered in wide variety of document understanding tasks, including but not limited to medical data mining, general fact or relation extraction, question answering, and sentiment analysis.
Résumé: Automatic detection of linguistic negation in free text is a critical need for many text processing applications, including sentiment analysis. This paper presents a nega- tion detection system based on a conditional random field modeled using features from an English dependency parser. The scope of negation detection is limited to explicit rather than implied negations within single sentences. A new negation corpus is presented that was constructed for the domain of English product reviews obtained from the open web, and the proposed negation extraction system is evaluated against the reviews corpus as well as the standard BioScope negation corpus, achieving 80.0% and 75.5% F1 scores, respectively. The impact of accurate negation detection on a state-of-the-art sentiment analysis system is also reported.

Collections: Google

Auteur-es: Täckström, Oscar; McDonald, Ryan
Nombre Auteurs: 2
Titre: Discovering Fine-Grained Sentiment with Latent Variable Structured Prediction Models
Année de publication: 2011
Référence (APA): Täckström, O., & McDonald, R. (2011). Discovering Fine-Grained Sentiment with Latent Variable Structured Prediction Models. Advances in Information Retrieval, 368‑374. https://doi.org/10.1007/978-3-642-20161-5_37
Mots-clés: ND
URL: https://link.springer.com/chapter/10.1007/978-3-642-20161-5_37
doi: https://doi.org/10.1007/978-3-642-20161-5_37
Accessibilité de l'article: Restricted
Champ: Natural Language Processing
Type contenu (théorique Applicative méthodologique): Applicative
Méthode: We propose a model that learns to analyze fine-grained sentiment strictly from coarse annotations. Such a model can leverage the plethora of labeled documents from multiple domains available on the web. In particular, we focus on sentence level sentiment (or polarity) analysis.
The model we present is based on hidden conditional random fields (HCRFs) [10], a well-studied latent variable structured learning model that has been used previously in speech and vision.
Cas d'usage: ND
Objectifs de l'article: In this paper we investigate the use of latent variable structured prediction models for fine-grained sentiment analysis in the common situation where only coarse-grained supervision is available. Specifically, we show how sentencelevel sentiment labels can be effectively learned from document-level supervision using hidden conditional random fields (HCRFs)
Question(s) de recherche/Hypothèses/conclusion: Research question(s) : This study inverts the evaluation and attempts to assess the accuracy of the latent structure induced from the observed coarse signal; Hypothesis(es) : In fact, one could argue that learning fine-grained sentiment from document level labels is the more relevant question for multiple reasons as 1) document level annotations are the most common naturally observed sentiment signal, e.g., star-rated consumer reviews, and 2) document level sentiment analysis is too coarse for most applications, especially those that rely on aggregation and summarization across fine-grained topics [3]; Conclusion(s) : In this paper we showed that latent variable structured prediction models can effectively learn fine-grained sentiment from coarse-grained supervision. Empirically, reductions in error of up to 20% were observed relative to both lexicon-based and machine-learning baselines. In the common case when document labels are available at test time as well, we observed error reductions close to 30% and over 20%, respectively, relative to the same baselines. In the latter case, our model reduces errors relative to the strongest baseline with 8%
Cadre théorique/Auteur.es: Sentiment analysis (Pang, Lee, 2008); Lexicon-based sentiment analysis (Hatzivassiloglou et McKeown, 1997 ; Kim et Hovy, 2004 ; Turney, 2002); Machine-learning for sentiment analysis (Pang, Lee, Vaithyanathan, 2002); Limits of both models (Hu et Liu, 2004 ; Wilson, Wiebe et Hoffmann, 2005); Hidden conditional random fields (Quattoni et al., 2007); Latent-variable structured learning models (Nakagawa, Inui, Kurohashi, 2010 ; Yessenalina, Yue, Cardie, 2010)
Concepts clés: Supervised learning; Sentiment analysis
Données collectées (type source): For our experiments we constructed a large balanced corpus of consumer reviews from a range of domains.
A training set was created by sampling reviews from five different domains: books, dvds, electronics, music and videogames. Document sentiment labels were obtained by labeling one and two star reviews as negative (NEG), three star reviews as neutral (NEU), and four and five star reviews as positive (POS).
Définition des émotions: No definition; Negative, neutral, positive labeling
Ampleur expérimentation (volume de comptes): A training set was created by sampling a total of 143,580 positive, negative and neutral reviews (the total number of sentences is roughly 1.5 million).

To study the impact of the training set size, additional training sets, denoted S and M,were created by sampling 1,500 and 15,000 documents from the full training set, denoted L.

A smaller separate test set of 294 reviews was constructed by the same procedure. This set consists of 97 positive, 98 neutral and 99 negative reviews
Technologies associées: Latent variable structured prediction models; Hidden conditional random fields
Mention de l'éthique: ND
Finalité communicationnelle: The model we employed, a hidden conditional random field, leaves open a number of further avenues for investigating weak prior knowledge in fine-grained sentiment analysis, most notably semi-supervised learning when small samples of data annotated with fine-grained information are available.
Résumé: In this paper we investigate the use of latent variable structured prediction models for fine-grained sentiment analysis in the common situation where only coarse-grained supervision is available. Specifically, we show how sentence-level sentiment labels can be effectively learned from document-level supervision using hidden conditional random fields (HCRFs) [10]. Experiments show that this technique reduces sentence classification errors by 22% relative to using a lexicon and 13% relative to machine-learning baselines.

Collections: Google

Auteur-es: Täckström, Oscar; McDonald, Ryan
Nombre Auteurs: 2
Titre: Semi-supervised latent variable models for sentence-level sentiment analysis
Année de publication: 2011
Référence (APA): Täckström, O., & McDonald, R. (2011). Semi-supervised latent variable models for sentence-level sentiment analysis. Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers - Volume 2, 569‑574. https://aclanthology.org/P11-2100
Mots-clés: ND
URL: https://aclanthology.org/P11-2100
Accessibilité de l'article: Open access
Champ: Natural Language Processing
Type contenu (théorique Applicative méthodologique): Applicative
Méthode: Creation of two variants of a semi-supervised model for fine sentiment analysis.
For the following experiments, we used the same data set and a comparable experimental setup to that of Tackström and McDonald (2011).3 We compare the two proposed hybrid models (Cascaded and Interpolated) to the fully supervised model of McDonald et al. (2007) (FineToCoarse) as well as to the soft variant of the coarsely supervised model of Tackström and McDonald (2011) (Coarse).
Cas d'usage: ND
Objectifs de l'article: In this paper, we demonstrate how combining coarse-grained and fine-grained supervision benefits sentence-level sentiment analysis. coarsely supervised model with a fully supervised model.
The proposed model is a fusion of a fully supervised structured conditional model and its partially supervised counterpart. This allows for highly efficient estimation and inference algorithms with rich feature definitions.
Question(s) de recherche/Hypothèses/conclusion: Research question(s) : These approaches all rely on the availability of fine-grained annotations, but T ̈ackström and McDonald (2011) showed that latent variables can be used to learn fine-grained sentiment using only coarse-grained supervision. While this model was shown to beat a set of natural baselines with quite a wide margin, it has its shortcomings. Most notably, due to the loose constraints provided by the coarse supervision, it tends to only predict the two dominant fine-grained sentiment categories well for each document sentiment category, so that almost all sentences in positive documents are deemed positive or neutral, and vice versa for negative documents.; Hypothesis(es) : As a way of overcoming these shortcomings, we propose to fuse a coarsely supervised model with a fully supervised model.
The proposed model is a fusion of a fully supervised structured conditional model and its partially supervised counterpart. This allows for highly efficient estimation and inference algorithms with rich feature definitions. We describe the two variants as well as their component models and verify experimentally that both variants give significantly improved results for sentence-level sentiment analysis compared to all baselines; Conclusion(s) : We introduced two simple, yet effective, methods of combining fully labeled and coarsely labeled data for sentence-level sentiment analysis. First, a cascaded approach where a coarsely supervised model is used to generate features for a fully supervised model. Second, an interpolated model that directly optimizes a combination of joint and marginal likelihood functions. Both proposed models are structured conditional models that allow for rich overlapping features, while maintaining highly efficient exact inference and robust estimation properties. Empirically, the interpolated model is superior to the other investigated models, but with sufficient amounts of coarsely labeled and fully labeled data, the cascaded approach is competitive.
Cadre théorique/Auteur.es: Sentence-level sentiment analysis (Pang et Lee, 2008); Fine-grained supervision (Wiebe et al., 2005); Exploiting document structure for sentiment analysis (Pang et Lee, 2004 ; McDonald et al., 2007 ; Yessenalina et al., 2010 ; Nakagawa et al., 2010 ; Sauper et al., 2010 ; Tackström et McDonald, 2011 ; Sauper et al., 2010); Methods for blending discriminative and generative models (Lasserre et al., 2006 ; Suzuki et al., 2007 ; Agarwal et Daumé, 2009 ; Sauper et al., 2010)
Concepts clés: Supervised learning; Sentiment analysis
Données collectées (type source): For our experiments we constructed a large balanced corpus of consumer reviews from a range of domains.
A training set was created by sampling reviews from five different domains: books, dvds, electronics, music and videogames. Document sentiment labels were obtained by labeling one and two star reviews as negative (NEG), three star reviews as neutral (NEU), and four and five star reviews as positive (POS).
Définition des émotions: No definition
Ampleur expérimentation (volume de comptes): To assess the impact of fully labeled versus coarsely labeled data, we took stratified samples without replacement, of sizes 60, 120, and 240 reviews, from the fully labeled folds and of sizes 15,000 and 143,580 reviews from the coarsely labeled data. On average each review consists of ten sentences.
Technologies associées: Semi-supervised model; Fully supervised model; Partially supervised model
Mention de l'éthique: ND
Finalité communicationnelle: In this paper, we demonstrate how combining coarse-grained and fine-grained supervision benefits sentence-level sentiment analysis – an important task in the field of opinion classification and retrieval (Pang and Lee, 2008).
Résumé: We derive two variants of a semi-supervised model for fine-grained sentiment analysis. Both models leverage abundant natural supervision in the form of review ratings, as well as a small amount of manually crafted sentence labels, to learn sentence-level sentiment classifiers. The proposed model is a fusion of a fully supervised structured conditional model and its partially supervised counterpart. This allows for highly efficient estimation and inference algorithms with rich feature definitions. We describe the two variants as well as their component models and verify experimentally that both variants give significantly improved results for sentence-level sentiment analysis compared to all baselines.

Collections: Google

Auteur-es: Joshi, Dhiraj; Datta, Ritendra; Fedorovskaya, Elena; Luong, Quang-Tuan; Wang, James Z.; Li, Jia; Luo, Jiebo
Nombre Auteurs: 7
Titre: Aesthetics and Emotions in Images
Année de publication: 2011
Référence (APA): Joshi, D., Datta, R., Fedorovskaya, E., Luong, Q.-T., Wang, J. Z., Li, J., & Luo, J. (2011). Aesthetics and Emotions in Images. IEEE Signal Processing Magazine, 28(5), 94‑115. https://doi.org/10.1109/MSP.2011.941851
Mots-clés: Emotion recognition, Photography, Semantics, Data visualization, Painting, Human factors
URL: https://ieeexplore.ieee.org/document/5999579
doi: https://doi.org/10.1109/MSP.2011.941851
Accessibilité de l'article: Open access
Champ: Machine Perception
Type contenu (théorique Applicative méthodologique): Theoretical; Methodological
Méthode: Here we present plots of features of the data sets [photo resources on the web], in particular the nature of user ratings received in each case
Cas d'usage: ND
Objectifs de l'article: In this tutorial, we define and discuss key aspects of the problem of computational inference of aesthetics and emotion from images.
In this survey, we discuss research that attempts to explain the observed phenomena of aesthetics and emotions that arise from subjective judgments using known tools and knowledge about computer vision, machine learning, art, and photography.
Question(s) de recherche/Hypothèses/conclusion: Research question(s) : Aspects of the problem of computational inference of aesthetics and emotion from images. [...] key computational problems that the research community has been striving to solve and the computational framework required for solving them.; Hypothesis(es) : We strongly believe that computational models of aesthetics and emotions may be able to assist in this decision making and perhaps with time and feedback learn to adapt to expert opinion better
[...] While tutorials are typically written for relatively mature topics, we believe an early tutorial on this active topic will help summarize the existing attempts, conjure up future research directions, and ultimately lead to robust solutions.; Conclusion(s) : We also discuss future directions that researchers can pursue and make a strong case for seriously attempting to solve problems in this research domain.
Cadre théorique/Auteur.es: [aesthetic studies; aesthetics in photography; aesthetics in painting; aesthetics in other visual art forms; psychology of aesthetics; prediction of aesthetics; machine learning]; Emotion prediction (Yanulevskaya et al., 2008); Detecting and categorizing emotion in art (Yanulevskaya et al., 2008 ; USA Today, 2006); Understanding beauty, attractiveness (Valentine, 1962 ; Davis et Lazebnik, 2008 ; O'Doherty et al, 2003 ; Scheib, Gangestad, et Thornhill, 1999 ; Swaddle et Cuthill, 1995 ; Zaidel et Cohen, 2005); Feedback, personalization, and emotions in image retrieval (Wang et He, 2008 ; Fang, Geman, et Boujemaa, 2005 ; Bianchi-Berthouze, 2003); Feature extraction and image representation for semantics and image understanding (Data et al., 2008 ; Freeman, 2007 ; Axelsson, 2007 ; Datta et al., 2006 ; Peters, 2007 ; Ke, Tang, et Jing, 2006 ; Li et Chen, 2009); Aesthetics and emotions in artwork characterization (Yanulevskaya et al., 2008 ; Bianchi-Berthouze, 2003 ; Bressan, Cifarelli, et Perronnin, 2008); Recognizing emotions in images and artwork (Arapakis, Konstas, et Jose, 2009 : Davis et Lazebnik, 2008 ; Eisenthal, Dror, et Ruppin, 2006 ; Machajdik et Hanbury, 2010 ; Ramanathan et al., 2009 ; Valenti, Jaimes, et Sebe, 2010)
Concepts clés: Emotional response; Sentiment analysis
Données collectées (type source): We performed a preliminary analysis of the above data sources [Web photo resources : Photo.net, DPChallenge, Terragalleria, ALIPR] to compare and contrast the different rating patterns. A collection of images was formed, drawing at random, to create real-world data sets (to be available at http://riemann.ist.psu.edu/).
Définition des émotions: Brief description of emotions and attractiveness
Ampleur expérimentation (volume de comptes): 14,839 images from Photo.net
16,509 images from DPChallenge
14,449 images from Terragalleria
13,010 emotion-tagged images from ALIPR
Technologies associées: Computational techniques; Machine learning
Mention de l'éthique: ND
Finalité communicationnelle: Despite the challenges, various research attempts have been made and are increasingly being made to address basic understanding and solve various subproblems under the umbrella of aesthetics, mood, and emotion inference in pictures. What motivates the multidisciplinary community to make such attempts is the fact that there is much to be gained from systems that can indeed reliably infer, at least for a section of the population, what the perceptual, cognitive, aesthetic, and emotional response to a photograph or a visual artwork will be. The potential beneficiaries of this research include general consumers, media management vendors, photographers, and people who work with art. Good shots or photo opportunities may be recommended to consumers; media personnel can be assisted with good images for illustration while interior and healthcare designers can be helped with more appropriate visual design items.

We hope that this tutorial will significantly increase the visibility of this research area and foster dialogue and collaboration among artists, photographers, and researchers in signal processing, computer vision, pattern recognition, and psychology.
Résumé: In this tutorial, we define and discuss key aspects of the problem of computational inference of aesthetics and emotion from images. We begin with a background discussion on philosophy, photography, paintings, visual arts, and psychology. This is followed by introduction of a set of key computational problems that the research community has been striving to solve and the computational framework required for solving them. We also describe data sets available for performing assessment and outline several real-world applications where research in this domain can be employed. A significant number of papers that have attempted to solve problems in aesthetics and emotion inference are surveyed in this tutorial. We also discuss future directions that researchers can pursue and make a strong case for seriously attempting to solve problems in this research domain.

Collections: Google

Auteur-es: Ryokai, Kimiko; Raffle, Hayes; Kowalski, Robert
Nombre Auteurs: 3
Titre: StoryFaces: pretend-play with ebooks to support social-emotional storytelling
Année de publication: 2012
Référence (APA): Ryokai, K., Raffle, H., & Kowalski, R. (2012). StoryFaces : Pretend-play with ebooks to support social-emotional storytelling. Proceedings of the 11th International Conference on Interaction Design and Children, 125‑133. https://doi.org/10.1145/2307096.2307111
Mots-clés: storytelling, social-emotional development, video recording, children, communication tools, facial expressions
URL: https://people.ischool.berkeley.edu/~kimiko/papers/IDC.2012.Ryokai.StoryFaces.pdf
doi: https://doi.org/10.1145/2307096.2307111
Accessibilité de l'article: Open access
Champ: Human-Computer Interaction and Visualization
Type contenu (théorique Applicative méthodologique): Applicative
Méthode: StoryFaces invites children to record emotional expressions that become part of storybook illustrations.
As children watch the story being told, they see their faces bring the story to life; then they can "go backstage" to play with the story by rearranging the videos and altering the story text. More experienced children can build an interactive ebook from the ground up, creating scenes, characters and expressive faces to craft personalized narratives.
Our design rationale focuses on supporting children's exploration of emotional expression through their narrative play.
Cas d'usage: StoryFaces, digital tool for children
Objectifs de l'article: We introduce StoryFaces, a new composition and storytelling tool for children to explore the role of emotional expressions in children's narrative. Our goal is to provoke new ideas about how pretend play with digital tools can empower young children in social-emotional narrative activities.
We are interested in studying which kinds of expressions and stories children at various stages of development can successfully engage in with StoryFaces. [...] The primary goal of our study was not to quantify children's performance with StoryFaces in an empirical setting, but rather to observe the kinds of expressive activities children were able to engage in with the tool.
Question(s) de recherche/Hypothèses/conclusion: Research question(s) : Facial expression is one of the strong non-verbal communication cues we humans use in understanding one another. For young children, understanding facial expressions seems to play a role in their cognitive, social, and language development [3]. While it is assumed that children naturally develop these capabilities in interaction with other people in their environment, research suggests benefits in fostering these skills, and these skills seem to be teachable [9]. [...] There is a clear opportunity for open-ended play with digital tools today. [...] While games can teach explicit knowledge like math or spelling, they do not usually encourage children's creativity or social play and learning. But games keep children engaged; can open-ended digital play engage children as well?; Hypothesis(es) : In this paper, we argue that children's natural interest in pretend play and emotional expression can provide an appealing point of entry for children to engage in self-guided play and learning with digital tools.; Conclusion(s) : Results indicate that emotional expressions are inviting and motivating for children across this broad age range to engage in both reading and creating narrative. StoryFaces gave children's ephemeral facial expressions concrete forms with which they could manipulate, discuss, and think about the role of emotion in narratives.
Cadre théorique/Auteur.es: Developmental psychology; Facial expressions and development of social skills in children (Bergen, 2002; Declerck and Bogaert, 2008; Marsh, Kozak and Ambady, 2007; Walden and Field, 1982; Grinspan, Hemphill and Nowicki, 2003; Profyt and Whissell, 1991; Widen and Russell, 2003); Role-playing and storytelling (Goodman, 1996; Singer, 1998; Teale and Sulzby, 1986; Vygotsky, 1986); Theory of multiple intelligences (Gardner, 1982); Theory of children's co-evolution and creative tools (Raffle, 2008); Digital tools developed in Human-Computer Interaction to develop children's knowledge (Hourcade et al., 2002 ; Humphries, and McDonald, 2011 ; Kapoor, Mota and Picard, 2001 ; Raffle et al., 2010 ; Raffle et al., 2011 ; Raffle et al., 2007 ; Tseng and Ellen, 2010 ; Ryokai, Kowalski, and Raffle, 2011)
Concepts clés: Development of social skills
Données collectées (type source): In our study, we focused on this age group [preschool (age 4-5), and elementary school (age 6-10)] to observe how StoryFaces supported these emergent expressive skills in pretend play and storytelling.
All children played with StoryFaces in a pair with another playmate within the same age group. One of the premade stories, "Walking in the Woods" was used as a practice story with some assistance from a researcher. After the practice story, children were invited to play with StoryFaces by themselves. The children were told that they were free to play with it as long as they wished. Children's sessions were video recorded for later analyses.
Définition des émotions: No definition; Evokes children's understanding of facial emotions
Ampleur expérimentation (volume de comptes): 18 children participated in the study: 8 children were enrolled in a preschool (age 4-5), and 10 children were enrolled in an elementary school (age 6-10).
Each session lasted approximately 45 minutes.
Technologies associées: StoryFaces is written in ActionScript 3, which provides access to video and audio hardware as well as an animation framework
Mention de l'éthique: ND
Finalité communicationnelle: Looking ahead, we would also like to see how the findings of this research combining narrative and emotion may help children with autism spectrum disorder.
Further, as more children are using tablets in their everyday learning, we hope this research can inspire a new generation of software and content designers to use media in ways that engage children's creativity and social- emotional learning.
Résumé: We introduce StoryFaces, a new composition and storytelling tool for children to explore the role of emotional expressions in children's narrative. StoryFaces invites children to record emotional expressions that become part of storybook illustrations. As children watch the story being told, they see their faces bring the story to life; then they can "go backstage" to play with the story by rearranging the videos and altering the story text. More experienced children can build an interactive ebook from the ground up, creating scenes, characters and expressive faces to craft personalized narratives. Our design rationale focuses on supporting children's exploration of emotional expression through their narrative play. Results with eighteen children ages 4-10 indicate that emotional expressions are inviting and motivating for children across this broad age range to engage in both reading and creating narrative. StoryFaces gave children's ephemeral facial expressions concrete forms with which they could manipulate, discuss, and think about the role of emotion in narratives. Our goal is to provoke new ideas about how pretend play with digital tools can empower young children in social-emotional narrative activities.

Collections: Google

Auteur-es: Li, Fangtao; Wang, Sheng; Liu, Shenghua; Zhang, Ming
Nombre Auteurs: 4
Titre: SUIT: A Supervised User-Item Based Topic Model for Sentiment Analysis
Année de publication: 2014
Référence (APA): Li, F., Wang, S., Liu, S., & Zhang, M. (2014). SUIT : A Supervised User-Item Based Topic Model for Sentiment Analysis. Proceedings of the AAAI Conference on Artificial Intelligence, 28(1), Article 1. https://doi.org/10.1609/aaai.v28i1.8947
Mots-clés: ND
URL: https://ojs.aaai.org/index.php/AAAI/article/view/8947
doi: https://doi.org/10.1609/aaai.v28i1.8947
Accessibilité de l'article: Open access
Champ: Natural Language Processing
Type contenu (théorique Applicative méthodologique): Applicative
Méthode: We propose a novel Supervised User-Item based Topic model, SUIT model, which can simultaneously utilize the textual topic and useritem factors for sentiment analysis.
In this model, the useritem information is represented in the form of user latent factor and item latent factor.
We conduct experiment on both review dataset and microblog dataset.
Cas d'usage: ND
Objectifs de l'article: In this paper, we propose a new Supervised User-Item based Topic model, called SUIT model, for sentiment analysis. It can simultaneously utilize the textual topic and latent user-item factors.
Question(s) de recherche/Hypothèses/conclusion: Research question(s) : Probabilistic topic models have been widely used for sentiment analysis. However, most of existing topic methods only model the sentiment text, but do not consider the user, who expresses the sentiment, and the item, which the sentiment is expressed on.; Hypothesis(es) : Since different users may use different sentiment expressions for different items, we argue that it is better to incorporate the user and item information into the topic model for sentiment analysis. In this paper, we propose a new Supervised User-Item based Topic model, called SUIT model, for sentiment analysis. It can simultaneously utilize the textual topic and latent user-item factors. Our proposed method uses the tensor outer product of text topic proportion vector, user latent factor and item latent factor to model the sentiment label generalization.; Conclusion(s) : The results demonstrate the advantages of our model. It shows significant improvement compared with supervised topic models and collaborative filtering methods
Cadre théorique/Auteur.es: Sentiment analysis (Liu, 2010 ; Pang et Lee, 2009); Probabilistic topic model (Hofmann 1999 ; Blei, Ng et Jordan 2003 ; Blei et McAuliffe 2007); Sentiment modeling in an unsupervised framework (Mei et al. 2007 ; Brody et Elhadad, 2010 ; Li et al. 2010 ; Jo et Oh, 201 ; Titov et McDonald, 2008a ; Lin et He, 2009); Supervised variants of Latent Dirichlet Allocation model, LDA (Blei and McAuliffe 2007 ; Simon et al. 2008 ; Zhu et al. 2009 ; Wang et al. 2011 ; Zhao et al. 2010 ; Titov and McDonald, 2008b); Importance of user and item information (Tan et al. 2010 ; Li et al. 2011)
Concepts clés: Sentiment analysis
Données collectées (type source): We conduct our experiments on two datasets. The first is movie review data set. The second is a dataset crawled from a microblog site.
- Review dataset : the first dataset is a collection of movie reviews. Following Pang and Lee’s setting (Pang and Lee 2005), all the review stars are mapped into a 1~4 sentiment scales.
- Microblog dataset : the second dataset is crawled from a Microblog site. We first filter out the spam tweets, and then manually annotate the remaining tweets into three categories (0 ~ 2): negative (0), neutral (1), and positive (2). Negative, neutral and positive labels refer to the sentiment level that user has rated to the item. It is also necessary to filter out the spam users who only post advertisement.
Définition des émotions: Definition of emotion analysis
Ampleur expérimentation (volume de comptes): Review dataset : 15507 reviews with explicit stars. There are 458 users, 4543 products and 15507 reviews in total.

Microblog dataset : 14 movies, which are released in the last year. After filtering the spam tweets and the spam users, we finally got 387 users and 1299 tweets which include 403 negative tweets, 431 neutral tweets and 445 positive tweets.
Technologies associées: ND
Mention de l'éthique: ND
Finalité communicationnelle: These reviews are very useful for the general users, who often read product reviews before making the final decision. Companies also hope to track the customers’ opinion to improve the quality of the products or services.
Résumé: Probabilistic topic models have been widely used for sentiment analysis. However, most of existing topic methods only model the sentiment text, but do not consider the user, who expresses the sentiment, and the item, which the sentiment is expressed on. Since different users may use different sentiment expressions for different items, we argue that it is better to incorporate the user and item information into the topic model for sentiment analysis. In this paper, we propose a new Supervised User-Item based Topic model, called SUIT model, for sentiment analysis. It can simultaneously utilize the textual topic and latent user-item factors. Our proposed method uses the tensor outer product of text topic proportion vector, user latent factor and item latent factor to model the sentiment label generalization. Extensive experiments are conducted on two datasets: review dataset and microblog dataset. The results demonstrate the advantages of our model. It shows significant improvement compared with supervised topic models and collaborative filtering methods.

Collections: Google

Auteur-es: Müller, Hendrik; Sedley, Aaron
Nombre Auteurs: 2
Titre: HaTS: Large-scale In-product Measurement of User Attitudes & Experiences with Happiness Tracking Surveys
Année de publication: 2014
Référence (APA): Müller, H., & Sedley, A. (2014). HaTS : Large-scale In-product Measurement of User Attitudes & Experiences with Happiness Tracking Surveys. Proceedings of the 26th Australian Computer-Human Interaction Conference on Designing Futures: the Future of Design, 308‑315. https://doi.org/10.1145/2686612.2686656
Mots-clés: Surveys; metrics; tracking; attitudes; large scale; HaTS
URL: https://dl.acm.org/doi/10.1145/2686612.2686656
doi: https://doi.org/10.1145/2686612.2686656
Accessibilité de l'article: Open access
Champ: Human-Computer Interaction and Visualization
Type contenu (théorique Applicative méthodologique): Applicative
Méthode: Analysis of responses to user satisfaction questionnaire
We detail best Happiness Tracking Surveys (HaTS) for collecting attitudinal data at a large scale directly in the product and over time.
This method was developed at Google to track attitudes and open-ended feedback over time, and to characterize products’ user bases.
This case study of HaTS goes beyond the design of the questionnaire to also suggest best practices for appropriate sampling, invitation techniques, and its data analysis.
Cas d'usage: A dozen Google products (consumer and enterprise) to support product development and optimize UX
Objectifs de l'article: In this industry case study, we are introducing a particular survey method, referred to as Happiness Tracking Surveys (HaTS), that is designed for ongoing tracking of user attitudes and experiences within the context of real-world product usage at a large scale.
Question(s) de recherche/Hypothèses/conclusion: Research question(s) : While measuring attitudinal data at a small scale for a given design or product (i.e., in the lab or field) has been studied heavily and is widely adopted in the HCI community, there have been fewer contributions towards a model to reliably track a product’s attitudes over time and at a large scale; Hypothesis(es) : The HaTS survey method presented in this industry case study relies on these latest advances in questionnaire design and attempts to fill the gap of measuring attitudes at a large scale, in the context of real-world product usage, and over time through random sampling and the use of proactive survey invitations to aim for valid, reliable, and actionable data.; Conclusion(s) : For a variety of reasons, HaTS has proven to be a useful, high quality method for measuring, tracking and comparing users’ attitudes at a large scale, and one that can be effectively adopted by others who endeavor to better understand users’ attitudes and experiences. From the outset, HaTS has used probability sampling, the gold standard among survey researchers for achieving representative results for a given population. Users are sampled at the moment they are actually using the product, ensuring that their responses accurately reflect their true experiences, unaffected by memory bias.
Cadre théorique/Auteur.es: Product evaluation questionnaire (Brooke, 1996 ; Chin, Diehl, et Norman, 1988 ; Hassenzahl, Burmester, et Kolle, 2003 ; Kirakowski et Corbett, 1993 ; Kirakowski et Dillion, 1987); Behavioral analysis through the use of log data (Rodden, Hutchinson, et Fu, 2010); Recent work in the social sciences on the design of valid and reliable questionnaires (Coupe, 2008 ; Groves et al., 2004 ; Krosnick, 1991 ; Krosnick, 1997 ; Krosnick, Narayan, et Smith, 1996 ; Muller, Sedley, et Ferrall-Nunge, 2014 ; Saris, Krosnick, et Shaeffer, 2005 ; Smith, 1967 ; Tourangeau, Couper, et Conrad, 2004)
Concepts clés: User experience measurement
Données collectées (type source): Responses to a satisfaction questionnaire with users of Google products
Each week, a representative set of a product’s users are randomly selected to be invited to take part in HaTS.
The HaTS questionnaire follows a “funnel” approach from broad and high-level to more specific and personal questions.
In the beginning of the questionnaire, we include questions directly related to the survey topic and ask about attitudes and feedback about product as a whole (to avoid potential biases resulting from questions that ask about specific aspects of the product). These initial questions are also important to help build rapport with the respondent.
After high-level aspects have been assessed, the questionnaire then dives into common product attributes as well as productspecific tasks.
Finally, questions about respondents’ characteristics are asked about towards the end as they may be perceived as more sensitive by some.
Définition des émotions: Definition of joy
Ampleur expérimentation (volume de comptes): During any given week, a maximum of about 8% of the entire user base may be invited to the survey.
For HaTS, as a best practice, we often aim for about 400 or 1000 responses for the time period of interest.
Technologies associées: ND
Mention de l'éthique: ND
Finalité communicationnelle: HaTS has proven to be a useful, high quality method for measuring, tracking and comparing users’ attitudes at a large scale, and one that can be effectively adopted by others who endeavor to better understand users’ attitudes and experiences.

We believe other organizations can yield significant value by adopting HaTS, adjusting it to their specific needs, and continuing to refine the platform for high quality, actionable results.
Résumé: With the rise of Web-based applications, it is both important and feasible for human-computer interaction practitioners to measure a product's user experience. While quantifying user attitudes at a small scale has been heavily studied, in this industry case study, we detail best Happiness Tracking Surveys (HaTS) for collecting attitudinal data at a large scale directly in the product and over time. This method was developed at Google to track attitudes and open-ended feedback over time, and to characterize products' user bases. This case study of HaTS goes beyond the design of the questionnaire to also suggest best practices for appropriate sampling, invitation techniques, and its data analysis. HaTS has been deployed successfully across dozens of Google's products to measure progress towards product goals and to inform product decisions; its sensitivity to product changes has been demonstrated widely. We are confident that teams in other organizations will be able to embrace HaTS as well, and, if necessary, adapt it for their unique needs.

Collections: Google

Auteur-es: Yang, Rong; Singh, Sarvjeet; Cao, Pei; Chi, Ed; Fu, Bo
Nombre Auteurs: 5
Titre: Video Watch Time and Comment Sentiment: Experiences from YouTube
Année de publication: 2016
Référence (APA): Yang, R., Singh, S., Cao, P., Chi, E., & Fu, B. (2016). Video Watch Time and Comment Sentiment : Experiences from YouTube. 2016 Fourth IEEE Workshop on Hot Topics in Web Systems and Technologies (HotWeb), 26‑28. https://doi.org/10.1109/HotWeb.2016.13
Mots-clés: Sentiment, video watch time
URL: https://ieeexplore.ieee.org/document/7785813
doi: https://doi.org/10.1109/HotWeb.2016.13
Accessibilité de l'article: Open access
Champ: Machine Intelligence
Type contenu (théorique Applicative méthodologique): Applicative
Méthode: We used an analysis tool called Scarlett to perform the sentiment analysis of the comments. Scarlett summarizes text sentiment by detecting and analyzing positive and negative opinions expressed towards a given list of entities, and by finding snippets that represent the opinions in the text. For each comment, Scarlett generates two values to represent its sentiment: polarity value and magnitude value.
We looked at the daily record of video watch time as well as the sentiment of the comments.
Sentiment analysis of comment
Cas d'usage: YouTube
Objectifs de l'article: In this paper, we analyze how video watch time correlates with the comment sentiment. In particularly, we want to verify the findings in a recently published paper by Park et al, which claims that the video watch time positively correlates with the percentage of negative comments on the video
Question(s) de recherche/Hypothèses/conclusion: Research question(s) : While video watching is now an indispensable part of the general public media consumption, yet very little is known about the relationship between how users interact with each other and how that affects video consumption patterns.; Hypothesis(es) : In this paper, we explore the relationship between user commenting behavior and how that might or might not be predictive of video consumption patterns such as watch time.; Conclusion(s) : Contrary to recent findings, we found that video watch time is correlated with the positive sentiment expressed in the comments of YouTube videos. More precisely, videos with more positive sentiment on average in the comments are more likely to be watched longer; while videos with negative comment sentiment on average are more likely to have shorter watch durations. These results suggest that users prefer videos that evoke positive emotional responses.
Our analysis shows the opposite results on the correlation between video watch time and negative comment percentage. That is the video watch time is negatively correlates with the percentage of negative comments. Given that the data from the previous paper was collected using a third party web crawler, we suspect that there might be bias with the crawler itself. Moreover, the data was collected much earlier than the data we used for our analysis. It is possible that user behavior might have changed over time.
Cadre théorique/Auteur.es: View duration and comments on YouTube (Park et al., 2016)
Concepts clés: Sentiment analysis
Données collectées (type source): We looked at the daily record of video watch time as well as the sentiment of the comments. From the watch time log, we extract the following information for each viewed video :
Daily total views / Daily watch time in hours / Average watch time per view in minutes / Average watch time duration (i.e. ratio of average watch time per view to total video length)

For each of the viewed video on a given day, we extract the following sentiment related signals :
Average sentiment polarity of the comments created on the day of view / Percentage of positive comments created on the day of view / Percentage of negative comments created on the day of view
Définition des émotions: Methodological explanation of their classification (quantitative); No definition; Positive and negative labeling
Ampleur expérimentation (volume de comptes): Using one day of watch time data from over 10k videos

Overall, we included 10252 videos in the dataset. All of this videos have at least 10 comments that satisfy the sentiment magnitude condition (i.e. larger>1.0).
Technologies associées: Analysis tool called Scarlett
Mention de l'éthique: ND
Finalité communicationnelle: Understanding patterns of user engagement in YouTube, therefore, takes on paramount importance for researchers, marketers, and engineers alike.
Résumé: While video watching is now an indispensable part of the general public media consumption, yet very little is known about the relationship between how users interact with each other and how that affects video consumption patterns. In this paper, we explore the relationship between user commenting behavior and how that might or might not be predictive of video consumption patterns such as watch time. Contrary to recent findings, we found that video watch time is correlated with the positive sentiment expressed in the comments of YouTube videos. More precisely, videos with more positive sentiment on average in the comments are more likely to be watched longer, while videos with negative comment sentiment on average are more likely to have shorter watch durations. These results suggest that users prefer videos that evoke positive emotional responses. If the findings here generalizes to other social media, this suggests a motivational design finding that is useful for other system designers.

Collections: Google

Auteur-es: Marinoiu, Elisabeta; Zanfir, Mihai; Olaru, Vlad; Sminchisescu, Cristian
Nombre Auteurs: 4
Titre: 3D Human Sensing, Action and Emotion Recognition in Robot Assisted Therapy of Children With Autism
Année de publication: 2018
Référence (APA): Marinoiu, E., Zanfir, M., Olaru, V., & Sminchisescu, C. (2018). 3D Human Sensing, Action and Emotion Recognition in Robot Assisted Therapy of Children With Autism. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2018, 2158‑2167. https://doi.org/10.1109/CVPR.2018.00230
Mots-clés: ND
URL: https://openaccess.thecvf.com/content_cvpr_2018/html/Marinoiu_3D_Human_Sensing_CVPR_2018_paper.html
Accessibilité de l'article: Open access
Champ: Machine Intelligence; Machine Perception
Type contenu (théorique Applicative méthodologique): Applicative
Méthode: We analyze a large scale video dataset containing child-therapist interactions and subtle behavioral annotations.
We adapt state-of-the-art 3d human pose estimation models to this setting, making it possible to reliably track and reconstruct both the child and the therapist, from RGB data, at comparable performance levels with an industrial-grade Kinect system.
We establish several action and emotion recognition baselines, including systems based on child representations, and models that jointly capture the child and the therapist.
Cas d'usage: Robot-assisted autism treatment
Objectifs de l'article: In this paper, we introduce fine-grained action classification and emotion prediction tasks defined on non-staged videos, recorded during robot-assisted therapy sessions of children with autism
Our long term goal is to automatically interpret and react to a child’s actions in the challenging setting of a therapy session. In order to understand the child, we rely on highlevel features associated to her/his 3d pose and shape.
Question(s) de recherche/Hypothèses/conclusion: Research question(s) : We have introduced large-scale fine-grained action and emotion recognition tasks defined on non-staged videos recorded during robot-assisted therapy sessions of children with autism. The tasks are challenging due to the large numbers of sequences (over 3,700), long videos (10-15 minutes each), large number of highly variable actions (37 child action classes, 19 therapist actions), and because children are only partially visible and observed under non-standard camera viewpoints. Age variance and unpredictable behavior add to the challenges.; Hypothesis(es) : We investigated how state-of-theart RGB 3d human pose reconstruction methods combining feedforward and feedback components can be adapted to the problem, and evaluated multiple action and emotion recognition baselines based on 2d and 3d representations of the child and therapist.; Conclusion(s) : Our results indicate that properly adapted, the current 2d and 3d reconstruction methods from RGB data are competitive with industrial grade RGB-D Kinect systems. With action recognition baselines in the 40-50% performance range, the large-scale data we introduce represents a challenge in modeling behavior, with impact in both computer vision, and child-robot interaction with applications to autism.
Cadre théorique/Auteur.es: Treating autism with predictable systems (Gizzonio et al., 2014 ; Ramdoss et al., 2011 ; Moore, McGrath, et Thorpe, 2000); Interaction approaches based on humanoid robot (Esteban et al., 2017 ; Farr, Yuill, et Raffle, 2010 ; Pop et al., 2013 ; Salvador, Silver, et Mahoor, 2015 ; Wainer et al., 2014 ; Wainer et al., 2014); Facial expressions and emotion understandings (Kossaifi et al., 2017 ; Nicolaou, Gunes, et Pantic, 2010 ; Mollahosseini, Hasani, et Mahoor, 2017); Body language and emotion understandings (Lhommet et Marsella, 2015); Automatically detecting, classifying and interpreting human action from body pose features (Liu et al., 2016 ; Ke et al., 2017 ; Huang et al., 2016 ; Du, Wang, et Wang, 2015 ; Zanfir, Leordeanu, et Sminchisescu, 2013); Rely on RGBD sensors to estimate 2d and 3d human pose (Cao et al., 2017 ; Popa, Zanfir, et Sminchisescu, 2017 ; Loper et al., 2015 ; Bogo et al., 2016 ; Martinez et al., 2017 ; Pavlakos et al., 2017 ; Zhou et al., 2016
Concepts clés: Detection, classification and interpretation of human action and emotions
Données collectées (type source): The DE-ENIGMA dataset contains multi-modal recordings of therapy sessions of children with autism. [...]
A selection of recordings from multiple therapy sessions was annotated. The children selection covers a variety of gestures and interactions for typical therapy sessions. The annotation of therapy videos relies on an extensive web-based tool developed by us that can (i) select temporal extents and (ii) assign them a class label.
A video selection from [DE-ENIGMA] was also annotated with continuous emotions in a valence-arousal space by 5 specialized therapists. The valence axis specifies whether the emotion is positive or negative, whereas arousal controls its intensity.
Définition des émotions: No definition; Collaboration with 5 specialized therapists to classify emotions; Positive and negative labeling
Ampleur expérimentation (volume de comptes): A selection of recordings from multiple therapy sessions of 7 children was annotated with 37 action classes.
We have annotated a total of 3757 sequences, with an average duration of 2.1 seconds.
The experiments presented in this paper use a subset of 2031 annotated sequences spanning over 24 classes common to all children. [...] Among the annotated sequences, around a third (749 out of 2, 031) are interacting sequences.
Technologies associées: 2D et 3D ("3d skeleton data"); RGBD sensors such as Kinect; Humanoid robots; Convolutional Neural Networks; Recurrent Neural Networks (hierarchical bidirectional recurrent network baseline, HBRNN); Multitask deep neural network (DMHS)
Mention de l'éthique: ND
Finalité communicationnelle: With action recognition baselines in the 40-50% performance range, the large-scale data we introduce represents a challenge in modeling behavior, with impact in both computer vision, and child-robot interaction with applications to autism
Résumé: We introduce new, fine-grained action and emotion recognition tasks defined on non-staged videos, recorded during robot-assisted therapy sessions of children with autism. The tasks present several challenges: a large dataset with long videos, a large number of highly variable actions, children that are only partially visible, have different ages and may show unpredictable behaviour, as well as non-standard camera viewpoints. We investigate how state-of-the-art 3d human pose reconstruction methods perform on the newly introduced tasks and propose extensions to adapt them to deal with these challenges. We also analyze multiple approaches in action and emotion recognition from 3d human pose data, establish several baselines, and discuss results and their implications in the context of child-robot interaction.

Collections: Google

Auteur-es: Liu, Ariel; Sosik, Victoria Schwanda; Singh, Khadine
Nombre Auteurs: 3
Titre: Building Empathy: Scaling User Research for Organizational Impact
Année de publication: 2018
Référence (APA): Liu, A., Sosik, V. S., & Singh, K. (2018). Building Empathy : Scaling User Research for Organizational Impact. Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems, 1‑7. https://doi.org/10.1145/3170427.3174352
Mots-clés: Empathy building, usability study, industry research
URL: https://dl.acm.org/doi/10.1145/3170427.3174352
doi: https://doi.org/10.1145/3170427.3174352
Accessibilité de l'article: Open access
Champ: Human-Computer Interaction and Visualization
Type contenu (théorique Applicative méthodologique): Applicative
Méthode: Analysis of survey responses from Google employees

Analyse des impacts du programme PokerFace avec un sondage pré-test et un sondage post-test afin d'évaluer si le fait de suivre le programme Pokerface permet aux participants de développer l'empathie envers les utilisateurs et d'être plus conscients lorsqu'ils évaluent le travail et le produit de leur équipe en termes de besoins de l'utilisateur.
Cas d'usage: Pokerface, an internal program at Google to develop empathy towards users
Objectifs de l'article: The Pokerface program is a Google internal user empathy campaign[…]. Our objective is to increase user awareness with our teams and acknowledge the fact that "we aren’t the users", but by going a step further and putting product teams face to face with their users.
Question(s) de recherche/Hypothèses/conclusion: Research question(s) : While this program was successful, it was somewhat limited in scope. It was successful in reaching over 400 Googlers from more than 6 sites, but this time4 years later–our goals were much larger, targeting more than 1500 Googlers in 15 offices across the world. The increased complexity of our project brought along many additional challenges which we will discuss in this case study. In this iteration of the program we also executed a more thorough evaluation of the program’s impact, which we did not do in 2012 and which we will also report on here.
This same set of questions was also asked in the post-survey in order to evaluate whether completing the Pokerface program allows participants build user empathy and be more aware when evaluating their team’s work and product in terms of user needs.; Hypothesis(es) : The Pokerface program is intended to build user empathy among Googlers in hopes that this empathy will translate to more awareness of an adherence to Google’s design principles.; Conclusion(s) : Project Pokerface was a Google internal user empathy campaign that encouraged user-involvement as part of the product development cycle. We saw that scaling such a program across a large organization of over 1500 employees across 15 sites brought about challenges, however our evaluation showed success, as the program helped our teams worldwide break preconceived perceptions, develop new perspectives, and build more empathy for users and confidence in user-centered research.
Cadre théorique/Auteur.es: User involvement throughout the innovation process (Bosch-Sijtsema et Bosch, 2015)
Concepts clés: Empathy towards users
Données collectées (type source): Responses to a survey of Google employees (engineers, product managers, designers, analysts and program managers)
In order to measure impact [of the program], we ran 2 sets of surveys: one pre-survey sent to participants to complete upon sign-up for the program (but before starting the program), and one post-survey sent to participants to complete shortly after completing the program. The survey is intended to measure participants’ attitudes toward Google’s user centered design principles. The pre-survey was sent to all participants and asked them to rate their agreement (on a 5-point bipolar, ordinal Agreement scale) with a set of questions/constructs. This same set of questions was also asked in the post-survey in order to evaluate whether completing the Pokerface program allows participants build user empathy and be more aware when evaluating their team’s work and product in terms of user needs.
Définition des émotions: No definition
Ampleur expérimentation (volume de comptes): 1500 Googler participants at ten locations worldwide, including engineers, product managers, designers, analysts, and program managers. Of the >1500 participants, >900 participants completed both pre- and post-surveys.
Technologies associées: ND
Mention de l'éthique: Pokerface: The day of

After scripts have been prepared, participants take part in a one-hour seminar with a researcher where they learn the basic dos and don’ts of running a usability study and have a lesson in research ethics.
We ran the Pokerface program with more than 1500 Googler participants. In order to avoid bias in our survey responses, we made sure that every participant was aware that the surveys were anonymous and were not part of any team or personal evaluation. We also did not reveal the overall survey results until the teams had completed the program to ensure that the results did not influence individual responses.
Finalité communicationnelle: Research has shown that user involvement throughout the entire innovation process has been a key to success in many high-tech firms [3]. Feedback from users has helped companies uncover key user needs as well as understand how users accomplish critical journeys with products.

The Pokerface program is intended to build user empathy among Googlers in hopes that this empathy will translate to more awareness of an adherence to Google’s design principles.

The program helped our teams worldwide break preconceived perceptions, develop new perspectives, and build more empathy for users and confidence in user-centered research.
Résumé: Building user empathy in a tech organization is crucial to ensure that products are designed with an eye toward user needs and experiences. The Pokerface program is a Google internal user empathy campaign with 26 researchers that helped more than 1500 employees-including engineers, product managers, designers, analysts, and program managers across more than 15 sites-have first-hand experiences with their users. Here, we discuss the goals of the Pokerface program, some challenges that we have faced during execution, and the impact we have measured thus far.

Collections: Google

Auteur-es: Attia, Mohammed; Samih, Younes; Elkahky, Ali; Kallmeyer, Laura
Nombre Auteurs: 4
Titre: Multilingual Multi-class Sentiment Classification Using Convolutional Neural Networks
Année de publication: 2018
Référence (APA): Attia, M., Samih, Y., Elkahky, A., & Kallmeyer, L. (2018, mai). Multilingual Multi-class Sentiment Classification Using Convolutional Neural Networks. Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). LREC 2018, Miyazaki, Japan. https://aclanthology.org/L18-1101
Mots-clés: sentiment analysis, sentiment detection, multi-class
URL: https://aclanthology.org/L18-1101
Accessibilité de l'article: Open access
Champ: Natural Language Processing
Type contenu (théorique Applicative méthodologique): Applicative
Méthode: This paper describes our system for multilingual, multiclass sentiment classification using Convolutional Neural Networks (CNNs). We evaluate our system on three datasets in three different languages, and we find that state-of-art results can be achieved without language-specific features or pre-trained word embeddings.
Cas d'usage: User comments on social networks
Objectifs de l'article: Users generally express a broad variety of sentiments with a wide range of degrees, but to simplify the task, sentiment analysis approaches have traditionally classified sentiments into either positive, negative or neutral.
This paper describes our system for multilingual, multiclass sentiment classification using Convolutional Neural Networks (CNNs).
Question(s) de recherche/Hypothèses/conclusion: Research question(s) : Users generally express a broad variety of sentiments with a wide range of degrees, but to simplify the task, sentiment analysis approaches have traditionally classified sentiments into either positive, negative or neutral. This paper describes our system for multilingual, multiclass sentiment classification using Convolutional Neural Networks (CNNs).; Hypothesis(es) : The advantage of the proposed model is that it does not rely on language-specific features such as ontologies, dictionaries, or morphological or syntactic pre-processing. Equally important, our system does not use pre-trained word2vec embeddings which can be costly to obtain and train for some languages; Conclusion(s) : In this paper we have presented our systems for multi-class sentiment classification and we found that the deep neural network model can outperform traditional methods that rely on language-specific feature engineering. We show that the class imbalance in the data can lead to degradation in the system performance, and point out that oversampling can be a helpful workaround for handling this imbalance.
Cadre théorique/Auteur.es: Sentiment analysis (Pang et Lee, 2008 ; Lin et He, 2009 ; Cambria, 2016); Knowledge based sentiment analysis (Mullen et Collier, 2004 ; Boiy et Moens, 2009 ; Godbole et al., 2007 ; Gamon, 2004 ; Wilson et al., 2005 ; Esuli et Sebastiani, 2006 ; Baccianella et al, 2010 ; Cambria et al., 2016),; Statistic-based sentiment analysis (Neethu et Rajasree, 2013 ; Maas et al., 2011 ; Tripathy et al., 2016 ; Pang et al., 2002 ; Mikolov et al., 2013b ; Mikolov et al., 2013a ; Pennington et al., 2014); Deep learning for sentiment analysis (Glorot et al., 2011 ; Poria et al., 2016 ; dos Santos et Gatti, 2014); Convolutional Neural Networks (LeCun et al., 1995 ; Krizhevsky et al., 2012 ; Graves et al., 2013 ; Kim, 2014 ; Johnson et Zhang, 2014)
Concepts clés: Sentiment analysis
Données collectées (type source): Three publicly available datasets for English, German and Arabic :
- The Sanders Twitter Sentiment dataset (EN) 1 (Sanders, 2011) consists of tweets related to the products of four companies (Apple, Google, Microsoft, and Twitter). Tweets have been manually tagged as either positive, negative, neutral, or irrelevant with respect to the topic
- GermEval shared task (GE) consists of messages from various social media and web sources intended for analyzing customer reviews about “Deutsche Bahn”. The data is split roughly into 90% for training and 10% for development as shown in Table 2. They also provide two test sets. The shared task’s Subtask-B is on multi-class documentlevel polarity, which is about identifying whether the customer’s opinion of “Deutsche Bahn” or travel is positive, negative or neutral. The data was annotated by a team of six annotators, and each document was annotated by two annotators using WebAnno’s curation interface. The documents were checked for consistency by a supervisor who decided on divergences and new issues.
- The Arabic Sentiment Tweets Dataset (AR) consists of tweets which are classified as ‘positive’, ‘negative’, ‘mixed’, and ‘objective’. The tweets were collected from EgyptTrends and were not related to any particular topic, but generally included comments on diverse political issues. The tweets were annotated through the Amazon Mechanical Turk by three annotators. Tweets that were assigned the same rating by at least two annotators were accepted, otherwise rejected.
Définition des émotions: Definition of emotion analysis; Remarks on emotions, a complex object to define; Negative, neutral, positive labeling
Ampleur expérimentation (volume de comptes): The Sanders Twitter Sentiment dataset (EN) consists of 5,513 tweets related to the products of four companies: Apple, Google, Microsoft, and Twitter.

GermEval shared task (GE) consists of 21,824 messages from various social media and web sources

The Arabic Sentiment Tweets Dataset (AR) consists of 10,006 tweets which are classified as ‘positive’, ‘negative’, ‘mixed’, and ‘objective’.
Technologies associées: A simple neural network architecture of five layers (Embedding, Conv1D, GlobalMaxPooling and two Fully-Connected); Convolutional Neural Networks (CNNs)
Mention de l'éthique: ND
Finalité communicationnelle: Negative reviews circulated online may cause critical problems for the reputation, competitive power, and survival chances of any business. This in turn has led to some fundamental changes in how businesses approach their customers, gauge satisfaction, provide support and manage risks. Sentiment analysis is formally defined as the task to identify and analyze subjective information of people’s opinions in social media sources (Pham and Le, 2016; Yang et al., 2017). This field of study has recently attracted a lot of attention due to its implications for businesses and governments.
Résumé: This paper describes a language-independent model for multi-class sentiment analysis using a simple neural network architecture of five layers (Embedding, Conv1D, GlobalMaxPooling and two Fully-Connected). The advantage of the proposed model is that it does not rely on language-specific features such as ontologies, dictionaries, or morphological or syntactic pre-processing. Equally important, our system does not use pre-trained word2vec embeddings which can be costly to obtain and train for some languages. In this research, we also demonstrate that oversampling can be an effective approach for correcting class imbalance in the data. We evaluate our methods on three publicly available datasets for English, German and Arabic, and the results show that our system’s performance is comparable to, or even better than, the state of the art for these datasets. We make our source-code publicly available.

Collections: Google

Auteur-es: Karunakaran, Sowmya; Ramakrishnan, Rashmi
Nombre Auteurs: 2
Titre: Testing Grayscale Interventions to Reduce Negative Emotional Impact on Manual Reviewers
Année de publication: 2019
Référence (APA): Karunakaran, S., & Ramakrishnan, R. (2019). Testing Grayscale Interventions to Reduce Negative Emotional Impact on Manual Reviewers. The 4th Symposium on Computing and Mental Health, CHI Workshop’19. http://mentalhealth.media.mit.edu/wp-content/uploads/sites/15/2019/04/CMH2019_paper_39.pdf
Mots-clés: ND
URL: http://mentalhealth.media.mit.edu/wp-content/uploads/sites/15/2019/04/CMH2019_paper_39.pdf
Accessibilité de l'article: Open access
Champ: Human-Computer Interaction and Visualization
Type contenu (théorique Applicative méthodologique): Applicative
Méthode: We present a study measuring the emotional impact of reviewing difficult content by introducing simple image transformations such as grayscaling and blurring of content. We conduct a series of experiments on live content review queues. We maximize external validity by studying the impact on live manual review queues and test for differences in emotions, output quality, and task completion times with respect to our interventions.
Analysis of survey responses submitted to the moderators involved in the experiments
Cas d'usage: Online content moderators
Objectifs de l'article: Despite the importance of the subject, there is no prior research on the effects of technical interventions to reduce the associated emotional impact to reviewers.
To address this gap, we present a study measuring the emotional impact of reviewing difficult content by introducing simple image transformations such as grayscaling and blurring of content.
Question(s) de recherche/Hypothèses/conclusion: Research question(s) : While machines and technology play a critical role in content moderation, there continues to be a need for manual reviews where human judgement is required in interpreting borderline cases as well as generation of ground truth for ML models. It is known, however, that such manual reviews could be emotionally challenging. Despite the importance of the subject, there is no prior research on the effects of technical interventions to reduce the associated emotional impact to reviewers.; Hypothesis(es) : To address this gap, we present a study measuring the emotional impact of reviewing difficult content by introducing simple image transformations such as grayscaling and blurring of content.; Conclusion(s) : We find that simple stylistic transformations can provide an easy to implement solution to significantly reduce the emotional impact of manual content reviews
Cadre théorique/Auteur.es: Emotional response's measure "PANAS Scale" (Watson, Clark, et Tellegen, 1988)
Concepts clés: Content moderation; Emotional impact measurement
Données collectées (type source): We conduct a series of experiments on live content review queues. We maximize external validity by studying the impact on live manual review queues and test for differences in emotions, output quality, and task completion times with respect to our interventions.
Responses to a survey of moderators involved in the experiments.
Définition des émotions: No definition; Positive and negative labeling
Ampleur expérimentation (volume de comptes): ND
Technologies associées: ND
Mention de l'éthique: We refrain from collecting any personally identifiable information to keep the study fully anonymous. Reviewers had the option to opt-out of taking the PANAS survey.
Finalité communicationnelle: We find that simple stylistic transformations can provide an easy to implement solution to significantly reduce the emotional impact of manual content reviews.
Résumé: With the rise in user generated content, there has been a significant increase in content shared online every day through social networks and content platforms. This in turn has increased the need to moderate content to ensure it complies with community guidelines and policies. Content moderation relies on automated processes and manual reviews by human reviewers to determine if content displayed in the form of images, videos or text, violate the platform’s acceptable use policies. For example, on Google Drive, Photos and Blogger, in the past year, 160,000 pieces of violent extremism content were taken down [1]. While machines and technology play a critical role in content moderation, there continues to be a need for manual reviews where human judgement is required in interpreting borderline cases as well as generation of ground truth for ML models. It is known, however, that such manual reviews could be emotionally challenging.

Collections: Google

Auteur-es: Hickson, Steven; Dufour, Nick; Sud, Avneesh; Kwatra, Vivek; Essa, Irfan
Nombre Auteurs: 5
Titre: Eyemotion: Classifying Facial Expressions in VR Using Eye-Tracking Cameras
Année de publication: 2019
Référence (APA): Hickson, S., Dufour, N., Sud, A., Kwatra, V., & Essa, I. (2019). Eyemotion : Classifying Facial Expressions in VR Using Eye-Tracking Cameras. 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), 1626‑1635. https://doi.org/10.1109/WACV.2019.00178
Mots-clés: ND
URL: https://ieeexplore.ieee.org/document/9052937
doi: https://doi.org/10.1109/WACV.2019.00178
Accessibilité de l'article: Open access
Champ: Machine Intelligence; Machine Perception
Type contenu (théorique Applicative méthodologique): Applicative
Méthode: We propose a new approach aimed at classification of facial action units (AUs) [12] and ‘emotive’ expressions using only internally mounted infrared cameras within the HMD.
We are motivated by the recent availability of commercial HMDs with eye-tracking cameras [1]. This uses infrared cameras. Our model classifies user expressions using only limited periocular eye image data, which is further limited by the large amount of intra-class variation among users.
Recently convolutional neural networks (CNNs) [20, 15, 35] have performed very well on image classification tasks and are pervasive in machine learning and computer vision.
Our approach, based on deep learning, outperforms normal human accuracy and even advanced (trained users) human accuracy for categorizing select facial expressions from our dataset of only IR eye images.
Cas d'usage: HMD headsets with integrated infrared cameras
Objectifs de l'article: We propose to recognize and convey facial expressions from inside a VR HMD.
Question(s) de recherche/Hypothèses/conclusion: Research question(s) : Virtual reality equipment using head-mounted displays (HMD) makes natural expressions difficult to recognize as half the face is occluded. Thus for VR systems to provide rich social interaction, faithfully representing these expressions in some manner is absolutely critical.; Hypothesis(es) : We propose a new approach aimed at classification of facial action units (AUs) [12] and ‘emotive’ expressions using only internally mounted infrared cameras within the HMD. We are motivated by the recent availability of commercial HMDs with eye-tracking cameras [1]. This uses infrared cameras (Fig. 1B). These are used for tracking [29], but in our work we use the same input images for expression classification.; Conclusion(s) : Our primary contributions are: (1) Demonstrating that the information required to classify a variety of facial expressions is reliably present in IR eye images captured by a commercial HMD sensor, and that this information can be decoded using a CNN-based method. (2) A novel personalization technique to improve CNN accuracy on new users. Across experiments, personalization resulted in a 4% accuracy improvement on average, and was statistically significant for a set of basic ‘emotive’ expressions (p =0.018) and AUs (p =0.001) (Section 4.2). (3) The collection of a unique dataset (Section 3) of eye images paired with expression labels, collected with two separate commercial HMDs each with 23 different users. (4) We show our method can be used to generate expressive avatars in real-time, which can function as an expressive surrogate for users engaged in VR environments (Section 5.2).
Cadre théorique/Auteur.es: Expression classification from visual data (Pantic et Rothkrantz, 2000 ; Fasel et Luettin, 2003 ; Bettadapura, 2012 ; Sariyanidi, Gunes, et Cavallaro, 2015 ; Saatci et Town, 2006 ; Tian, Kanade, et Cohn, 2001); Expression classification with alternate sensors (Scheirer, Fernandez, et Picard, 1999 ; Masai et al, 2016 ; Dhall et al., 2016 ; Suzuk et al., 2016); Gaze tracking in VR (Burgos-Artizzu et al., 2015 ; Li et al., 2015 ; Olszewski et al., 2016 ; Thies et al., 2016 ; Zhao et al., 2016)
Concepts clés: Classification of facial expressions; Sentiment analysis
Données collectées (type source): We collected a subset of facial action units that influence the upper face, and could be reliably performed by multiple subjects. We also distinguish between left and right AUs, where applicable. These are Neutral (AU0), Left Brow Raise (AU1+2L), Right Brow Raise (AU1+2R), Brow Lower (AU4), Upper Lid Raise (AU5), Squint (AU44), Both Eyes Closed (AU43), Left Wink (AU46L), Right Wink (AU46R), and Cheek Raise (AU6).
We also collect ‘emotive’ expressions for basic emotions as defined by [12], which are Neutral, Anger, Surprise, and Happiness.
We collected these data by asking users to form an expression, giving them an example from an exemplar video. While this may not result in spontaneous expressions [41], it provides explicit labels for each expression. To provide a realistic exemplar, we first recorded videos of trained actors performing each expression for the participant to use as a reference. During the collection process, for each expression, we provide to the participant the name of the expression, a looped clip of an actor performing the expression, and a live video of the participant in order for them to practice the expression. [...]This continues for all expressions and AUs (these are the images in column 1 of Fig. 2). We then have them put on the HMD and repeat the process twice more, taking the headset off and putting it back on to account for slippage and variation in fit. Each of these headset repetitions constitutes a ‘session.’
Définition des émotions: No definition; Refers to the emotions defined by Ekman and Friesen.
Ampleur expérimentation (volume de comptes): We collected data with two separate HMDs [...].
23 different participants were collected with each HMD with different genders (for a total of 46), ethnicities, and hair color.
Of the 46 participants: 16 were female; 16 were aged 35 or over from an age range of 18 to 64 with a median age of 30; 11 participants had non-brown eyes and 4 had non-brown or black hair. 25 of our participants were nonwhite, with 9 Asian, 7 east Indian or south Asian, 4 two or more races, 3 Hispanic or Latino, 2 African American, and 2 preferring not to say.

Approximately 50,000 eye image pairs were collected per HMD (about 2,000 per participant).
Technologies associées: HMDs, with near IR (880nm) cameras.; Convolutional neural networks (CNN) to learn an embedding describing expressions and emotions using infrared eye images (variant of the widespread Inception architecture [37] using the TensorFlow library)
Mention de l'éthique: ND
Finalité communicationnelle: Facial expressions are essential for interpersonal communication and social interaction. They provide a means for conveying thought and emotion through visual cues that may not be easy to articulate verbally. However, virtual reality (VR) equipment using head-mounted displays (HMD) makes natural expressions difficult to recognize as half the face is occluded. Thus for VR systems to provide rich social interaction, faithfully representing these expressions in some manner is absolutely critical.

We have demonstrated using consumergrade eye-tracking cameras, which are already being included in VR headsets, a means to preserve and transmit social information among users engaged in VR
Résumé: One of the main challenges of social interaction in virtual reality settings is that head-mounted displays occlude a large portion of the face, blocking facial expressions and thereby restricting social engagement cues among users. We present an algorithm to automatically infer expressions by analyzing only a partially occluded face while the user is engaged in a virtual reality experience. Specifically, we show that images of the user's eyes captured from an IR gaze-tracking camera within a VR headset are sufficient to infer a subset of facial expressions without the use of any fixed external camera. Using these inferences, we can generate dynamic avatars in real-time which function as an expressive surrogate for the user. We propose a novel data collection pipeline as well as a novel approach for increasing CNN accuracy via personalization. Our results show a mean accuracy of 74% (F1 of 0.73) among 5 'emotive' expressions and a mean accuracy of 70% (F1 of 0.68) among 10 distinct facial action units, outperforming human raters.

Collections: Google

Auteur-es: Pham, Phuong; Wang, Jingtao
Nombre Auteurs: 2
Titre: AttentiveVideo: A Multimodal Approach to Quantify Emotional Responses to Mobile Advertisements
Année de publication: 2019
Référence (APA): Pham, P., & Wang, J. (2019). AttentiveVideo : A Multimodal Approach to Quantify Emotional Responses to Mobile Advertisements. ACM Transactions on Interactive Intelligent Systems, 9(2‑3), 8:1-8:30. https://doi.org/10.1145/3232233
Mots-clés: Affective Computing, Signal Processing, User Modeling, Computational advertising, heart rate, facial expression, mobile interfaces
URL: https://dl.acm.org/doi/10.1145/3232233
doi: https://doi.org/10.1145/3232233
Accessibilité de l'article: Open access
Champ: Human-Computer Interaction and Visualization; Machine Intelligence
Type contenu (théorique Applicative méthodologique): Applicative
Méthode: AttentiveVideo employs a combination of implicit photoplethysmography (PPG) sensing and facial expression analysis (FEA) to detect the attention, engagement,and sentiment of viewers as they watch video advertisements on unmodified smartphones.

The viewer’s photoplethysmography (PPG) signals are extracted implicitly by continuously monitoring the changes in transparency of the covering finger through the back camera.
AttentiveVideo simultaneously captures and analyzes the viewer’s facial expressions via the front camera.
From the collected PPG signals and facial expressions, the prediction module of AttentiveVideo leverages machine-learning algorithms to infer viewers’ emotional responses.
Cas d'usage: Mobile video advertising
Objectifs de l'article: Understanding a target audience’s emotional responses to a video advertisement is crucial to evaluate the advertisement’s effectiveness. However, traditional methods for collecting such information are slow, expensive, and coarse grained.
We propose AttentiveVideo, a scalable intelligent mobile interface with corresponding inference algorithms to monitor and quantify the effects of mobile video advertising in real time.
Question(s) de recherche/Hypothèses/conclusion: Research question(s) : Understanding a target audience’s emotional responses to a video advertisement is crucial to evaluate the advertisement’s effectiveness. However, traditional methods for collecting such information are slow, expensive, and coarse grained.; Hypothesis(es) : AttentiveVideo achieves implicit PPG sensing and FEA on unmodified smartphones. Our approach eliminates the additional sensor requirement for the large-scale deployment of affect sensing systems. We explore three new ideas in this work: (1) a new PPG-based feature extraction (LocalDiff), (2) reducing lens covering time, and (3) model fusion (combining different machine-learning algorithms).; Conclusion(s) : AttentiveVideo achieved good accuracy on a wide range of emotional measures (the best average accuracy = 82.6% across nine measures). While feature fusion alone did not improve prediction accuracy with a single model, it significantly improved the accuracy when working together with model fusion. We also found that the PPG sensing channel and the FEA technique have different strength in data availability, latency detection, accuracy, and usage environment.
Cadre théorique/Auteur.es: Advertising Effectiveness (Linden, Smith, et York, 2003 ; Lohtia, Donthu, et Hershberger, 2003 ; Mei, Hua, et Li, 2009 ; Broder et al, 2008); Emotions as indirect measures of the effectiveness of branding advertising (Aaker, Stayman, et Hagerty, 1986 ; Micu et Plummer, 2010 ; Stout et Leckenby, 1986) Informatique affective (Picard, 1997 ; Lang, 1990); Facial expressions (Calvo et D'Mello, 2010 ; McDuff et al., 2015 ; McDuff et al., 2014); Physiological signals (Blanchard et al., 2014 ; Calvo et D'Mello, 2010 ; Kemp et Quintana, 2013 ; Lang, 1990); Combining different modalities (Bailenson et al., 2008 ; D'Mello et Graesser, 2010 ; Hussain, Monkaresi, et Calvo, 2012); Collecting Physiological Signals without Additional Sensors (Pham et Wand 2015 ; Pham et Wang, 2016 ; Pham et Wang, 2017 ; Xiao et Wang, 2015 ; Xiao et Wang, 2016); Mobile Video Interfaces (Ganhör, 2012 ; Li, Zhang, et Yuan, 2011 ; Li et al., 2015 ; Wu et al., 2012 ; Xiao et Wang, 2015 ; Zhang et al., 2013)
Concepts clés: Affective computing; Sentiment analysis
Données collectées (type source): In the formal study, the participant watched an episode of “The Big Bang Theory” with three embedded advertising slots, each slot containing four ads. Similarly to the training session, the participant gave an emotional self-report after each advertising slot and continued with the episode. At the end of the study, the participant rated the usability of AttentiveVideo and ranked the six most liked ads

The viewer’s photoplethysmography (PPG) signals are extracted implicitly by continuously monitoring the changes in transparency of the covering finger through the back camera. Furthermore, AttentiveVideo simultaneously captures and analyzes the viewer’s facial expressions via the front camera.

In this study, we used two types of self-reporting: verbal selfreporting for discrete emotions and visual self-reporting for dimensional emotions. Participants responded to six discrete emotions related questions on the effectiveness of each advertisement, i.e., Attention, Share, Touching, Rewatch, Recall, and Amusing. We used Touching as a warmth emotion in advertising, i.e., the viewer feels moved by the ad. We also used the Self-Assessment Manikin (SAM) [34] to collect responses for two dimensional emotions, Valence and Arousal. These ratings are in a 7-point Likert scale format (1: highly disagree;7:highly agree). In addition, participants rated the Like measure by ranking the 6 most liked ads at the end of the study. In total, we collected measures of nine emotional states that can be grouped into three categories: attention, engagement, and sentiment (Table 3).
Définition des émotions: No definition; Description of facial expressions and emotions observed
Ampleur expérimentation (volume de comptes): 24 participants (13 females) from a local university. The average age was 25.58 (σ = 3.01); the ages ranged from 21 to 33.
Technologies associées: AttentiveVideo collects a viewer’s PPG signals through the back camera (AttentiveVideo extended the LivePulse algorithm [15] to identify the interbeat intervals (NN) from PPG signals).; AttentiveVideo utilizes the front camera to capture the user’s facial expressions as she watches advertisements. AttentiveVideo used Affdex SDK [27] to analyze facial expressions from the recorded clips.; Use of machine-learning algorithms to infer the user’s emotional responses to the advertisement
Mention de l'éthique: ND
Finalité communicationnelle: AttentiveVideo can help advertisers gain a richer and more fine-grained understanding of users’ emotional responses toward video advertisements. AttentiveVideo can also help viewers to enjoy more high-quality video materials for free via subsidized video ads.
AttentiveVideo can additionally be applied to personalized advertising.
Résumé: Understanding a target audience's emotional responses to a video advertisement is crucial to evaluate the advertisement's effectiveness. However, traditional methods for collecting such information are slow, expensive, and coarse grained. We propose AttentiveVideo, a scalable intelligent mobile interface with corresponding inference algorithms to monitor and quantify the effects of mobile video advertising in real time. Without requiring additional sensors, AttentiveVideo employs a combination of implicit photoplethysmography (PPG) sensing and facial expression analysis (FEA) to detect the attention, engagement, and sentiment of viewers as they watch video advertisements on unmodified smartphones. In a 24-participant study, AttentiveVideo achieved good accuracy on a wide range of emotional measures (the best average accuracy = 82.6% across nine measures). While feature fusion alone did not improve prediction accuracy with a single model, it significantly improved the accuracy when working together with model fusion. We also found that the PPG sensing channel and the FEA technique have different strength in data availability, latency detection, accuracy, and usage environment. These findings show the potential for both low-cost collection and deep understanding of emotional responses to mobile video advertisements.

Collections: Google

Auteur-es: Sun, Jennifer J.; Liu, Ting; Prasad, Gautam
Nombre Auteurs: 3
Titre: GLA in MediaEval 2018 Emotional Impact of Movies Task
Année de publication: 2019
Référence (APA): Sun, J. J., Liu, T., & Prasad, G. (2019, novembre). GLA in MediaEval 2018 Emotional Impact of Movies Task. MediaEval 2018 Multimedia Benchmark Workshop. MediaEval’18, 29-31 October 2018, Sophia Antipolis, France. https://doi.org/10.48550/arXiv.1911.12361
Mots-clés: ND
URL: http://arxiv.org/abs/1911.12361
doi: https://doi.org/10.48550/arXiv.1911.12361
Accessibilité de l'article: Open access
Champ: Machine Intelligence; Machine Perception
Type contenu (théorique Applicative méthodologique): Applicative
Méthode: This task, using the LIRISACCEDE dataset, enables researchers to compare different approaches for predicting viewer impact from movies. Our approach leverages image, audio, and face based features computed using pretrained neural networks. These features were computed over time and modeled using a gated recurrent unit (GRU) based network followed by a mixture of experts model to compute multiclass predictions. We smoothed these predictions using a Butterworth filter for our final result.

We first used pre-trained networks to extract features. To model the temporal aspects of the data, the methods we evaluated included long short-term memory (LSTM), gated recurrent unit (GRU) and temporal convolutional network (TCN).
Cas d'usage: ND
Objectifs de l'article: Towards a better understanding of viewer impact, we present our methods for the MediaEval 2018 Emotional Impact of Movies Task to predict the expected valence and arousal continuously in movies. This task, using the LIRISACCEDE dataset, enables researchers to compare different approaches for predicting viewer impact from movies.
Question(s) de recherche/Hypothèses/conclusion: Research question(s) : Movies can cause viewers to experience a range of emotions, from sadness to relief to happiness. Viewers can feel the impact of movies, but it is difficult to predict this impact automatically. [...] The Emotional Impact of Movies Task provides participants with a common dataset for predicting the expected emotional impact from videos. We focused on the first subtask in the challenge: predicting the expected valence and arousal continuously (every second) in movies.; Hypothesis(es) : Our method’s novelty lies in the unique set of features we extracted including image, audio, and face features (capitalizing on transfer learning) along with our model setup, which comprises of a GRU combined with a mixture of experts.
We approached the valence and arousal prediction as a multivariate regression problem. Our objective is to minimize the multilabel sigmoid cross-entropy loss and this could allow the model to use potential relationships between the two dimensions for regression.; Conclusion(s) : We found that precomputed features modeling image, audio, and face in concert with GRUs provided the optimal performance in predicting the expected valence and arousal in movies for this task. [...] We found some evidence that recurrent models performed better than TCN. [...] The pre-computed features we used to model image, audio, and face information showed better performance when compared with the VGG16+openSMILE baseline.
Cadre théorique/Auteur.es: References to articles by previous MediaEval participants (Dellandréa et al., 2018 ; Jin et al., 2017 ; Liu, Gu, et Ko, 2017)
Concepts clés: Emotion prediction
Données collectées (type source): The dataset provided by the task is the LIRIS-ACCEDE dataset [3, 4], which is annotated with self-reported valence and arousal every second from multiple annotators.
Définition des émotions: No definition
Ampleur expérimentation (volume de comptes): We optimized the hyperparameters of our models to have the best performance on the validation set, which consists of 13 movies from the development set. We then trained our models on the entire development set to run inference on the test set. Our setup used a batch size of 512.
Technologies associées: Transfer learning; Inception network pre-trained on ImageNet; AudioSet, a VGG-inspired model pre-trained on YouTube-8M; An Inception based architecture trained on faces; Artificial neural network (LSTM, GRU); Temporal Model (TCN)
Mention de l'éthique: ND
Finalité communicationnelle: Movies can cause viewers to experience a range of emotions, from sadness to relief to happiness. Viewers can feel the impact of movies, but it is difficult to predict this impact automatically. The ability to automatically predict movie evoked emotions is helpful for a variety of use cases [7] in video understanding.
Résumé: The visual and audio information from movies can evoke a variety of emotions in viewers. Towards a better understanding of viewer impact, we present our methods for the MediaEval 2018 Emotional Impact of Movies Task to predict the expected valence and arousal continuously in movies. This task, using the LIRIS-ACCEDE dataset, enables researchers to compare different approaches for predicting viewer impact from movies. Our approach leverages image, audio, and face based features computed using pre-trained neural networks. These features were computed over time and modeled using a gated recurrent unit (GRU) based network followed by a mixture of experts model to compute multiclass predictions. We smoothed these predictions using a Butterworth filter for our final result. Our method enabled us to achieve top performance in three evaluation metrics in the MediaEval 2018 task.

Collections: Google

Auteur-es: Lu, Zhiyun; Cao, Liangliang; Zhang, Yu; Chiu, Chung-Cheng; Fan, James
Nombre Auteurs: 5
Titre: Speech Sentiment Analysis via Pre-Trained Features from End-to-End ASR Models
Année de publication: 2020
Référence (APA): Lu, Z., Cao, L., Zhang, Y., Chiu, C.-C., & Fan, J. (2020). Speech Sentiment Analysis via Pre-Trained Features from End-to-End ASR Models. ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 7149‑7153. https://doi.org/10.1109/ICASSP40776.2020.9052937
Mots-clés: Speech sentiment analysis, ASR pretraining, End-to-end ASR model
URL: https://ieeexplore.ieee.org/document/9052937
doi: https://doi.org/10.1109/ICASSP40776.2020.9052937
Accessibilité de l'article: Open access
Champ: Speech Processing
Type contenu (théorique Applicative méthodologique): Applicative
Méthode: We propose to use pre-trained features from e2e ASR model to solve sentiment analysis (the best performed sentiment decoder is RNN with self-attention)
Cas d'usage: ND
Objectifs de l'article: In this paper, we propose to use pre-trained features from end-to-end ASR models to solve speech sentiment analysis as a down-stream task.
Question(s) de recherche/Hypothèses/conclusion: Research question(s) : The key challenge in speech sentiment analysis is how to learn a good representation that captures the emotional signals and remains invariant under different speakers, acoustic conditions, and other natural speech variations. In this work, we introduce a new direction to tackle the challenge.; Hypothesis(es) : We propose to use end-to-end (e2e) automatic speech recognition (ASR) as pre-training, and solve the speech sentiment as a down-stream task. This approach is partially motivated by the success of pre-training in solving tasks with limited labeled data in both computer vision and language. Moreover, the e2e model combines both acoustic and language models of traditional ASR, thus can seamlessly integrate the acoustic and text features into one representation.
We hypothesize that the ASR pre-trained representation works well on sentiment analysis.; Conclusion(s) : We evaluate the performance of pre-trained ASR features on both IEMOCAP and SWBD-sentiment. On IEMOCAP, we improve the state-ofthe-art sentiment analysis accuracy from 66.6% to 71.7%. On SWBD-sentiment, we achieve 70.10% accuracy on the test set, outperforming strong baselines.
Cadre théorique/Auteur.es: Speech sentiment analysis (Li et al., 2019 ; Li et al., 2018 ; Wu et al., 2019 ; Xie et al., 2019 ; Tzirakis, Zhang, et Schuller, 2018); Speech and text sentiment analysis (Kim et Shin, 2019 ; Prabhavalkar et al, 2017 ; Cho et al., 2018 ; Gu et al., 2018); End-to-end automatic speech recognition (Prabhavalkar et al., 2017 ; Rao, Sak, et Prabhavalkar, 2017 ; Chiu et al., 2018 ; He et al., 2019)
Concepts clés: Speech sentiment analysis
Données collectées (type source): We use two datasets IEMOCAP and SWBD-sentiment in the experiments. IEMOCAP [18] is a well-benchmarked speech emotion recognition dataset. Following the protocol in [1, 7, 9, 10], we experiment on a subset of the data, which contains 4 emotion classes {happy+excited, neutral, sad, angry}, with {1708, 1084, 1636, 1103} utterances respectively.

To further investigate speech sentiment task, we annotate a subset of switchboard telephone conversations [17] with three sentiment labels, i.e. negative, neutral and positive, and create the SWBD-sentiment dataset.
Définition des émotions: No definition; Use of sentiment categories/groups; Negative, neutral, positive labeling
Ampleur expérimentation (volume de comptes): IEMOCAP : contains approximately 12 hours audiovisual recording of both scripted and improvised interactions performed by actors.

SWBD-sentiment : over 140 hours of speech which contains approximately 49.5k utterances.
Technologies associées: Automatic Speech Recognition (ASR); End-to-end Automatic Speech Recognition; SpecAugment
Mention de l'éthique: ND
Finalité communicationnelle: Speech sentiment analysis is an important problem for interactive intelligence systems with broad applications in many industries, e.g., customer service, health-care, and education.
Moreover, we create a large-scale speech sentiment dataset SWBD-sentiment to facilitate future research in this field. Our future work includes experimenting with unsupervised learnt speech features, as well as applying end-to-end ASR features to other down-stream tasks like diarization, speaker identification, and etc.
Résumé: In this paper, we propose to use pre-trained features from end-to-end ASR models to solve speech sentiment analysis as a down-stream task. We show that end-to-end ASR features, which integrate both acoustic and text information from speech, achieve promising results. We use RNN with self-attention as the sentiment classifier, which also provides an easy visualization through attention weights to help interpret model predictions. We use well benchmarked IEMOCAP dataset and a new large-scale speech sentiment dataset SWBD-sentiment for evaluation. Our approach improves the-state-of-the-art accuracy on IEMOCAP from 66.6% to 71.7%, and achieves an accuracy of 70.10% on SWBD-sentiment with more than 49,500 utterances.

Collections: Google

Auteur-es: Demszky, Dorottya; Movshovitz-Attias, Dana; Ko, Jeongwoo; Cowen, Alan; Nemade, Gaurav; Ravi, Sujith
Nombre Auteurs: 6
Titre: GoEmotions: A Dataset of Fine-Grained Emotions
Année de publication: 2020
Référence (APA): Demszky, D., Movshovitz-Attias, D., Ko, J., Cowen, A., Nemade, G., & Ravi, S. (2020). GoEmotions : A Dataset of Fine-Grained Emotions. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 4040‑4054. https://doi.org/10.18653/v1/2020.acl-main.372
Mots-clés: ND
URL: https://aclanthology.org/2020.acl-main.372
doi: https://doi.org/10.18653/v1/2020.acl-main.372
Accessibilité de l'article: Open access
Champ: Natural Language Processing
Type contenu (théorique Applicative méthodologique): Applicative
Méthode: We compiled GoEmotions, the largest human annotated dataset of Reddit comments.
We design our emotion taxonomy considering related work in psychology and coverage in our data.
We include a thorough analysis of the annotated data and the quality of the annotations. Via Principal Preserved Component Analysis (Cowen et al., 2019b)
Build an emotion classification model.
We provide a strong baseline for modeling finegrained emotion classification.
We conduct transfer learning experiments with existing emotion benchmarks.
Cas d'usage: ND
Objectifs de l'article: In the past decade, NLP researchers made available several datasets for language-based emotion classification [...]. However, existing available datasets are (1) mostly small, containing up to several thousand instances, and (2) cover a limited emotion taxonomy, with coarse classification into Ekman (Ekman, 1992b) or Plutchik (Plutchik, 1980).
Create a large-scale, consistently labeled emotion dataset over a fine-grained taxonomy, with demonstrated high-quality annotations.
Question(s) de recherche/Hypothèses/conclusion: Research question(s) : Need for a large-scale, consistently labeled emotion dataset over a fine-grained taxonomy, with demonstrated high-quality annotations.; Hypothesis(es) :; Conclusion(s) : We present GoEmotions, a large, manually annotated, carefully curated dataset for fine-grained emotion prediction. We provide a detailed data analysis, demonstrating the reliability of the annotations for the full taxonomy. We show the generalizability of the data across domains and taxonomies via transfer learning experiments. We build a strong baseline by fine-tuning a BERT model, however, the results suggest much room for future improvement.
Cadre théorique/Auteur.es: Emotion Datasets (Bostan et Klinger, 2018 ; CrowdFlower, 2016); Emotion Taxonomy (Ekman, 1992a ; Russell, 2003 ; Cowen et al., 2019a ; Cowen et Keltner, 2017 ; Cowen et al., sous presse ; Cowen et Keltner, 2019 ; Cowen et al., 2019b ; Cowen et al., 2018); Emotion Classification Models (Mohammad, 2018 ; Devlin et al., 2019)
Concepts clés: Sentiment analysis; “Semantic space” of emotion
Données collectées (type source): We use a Reddit data dump originating in the redditdata-tools project, which contains comments from 2005 (the start of Reddit) to January 2019. We select subreddits with at least 10k comments and remove deleted and non-English comments.
Définition des émotions: Explanation of their taxonomy; Evokes Ekman's taxonomy; List and define the 27 emotions
Ampleur expérimentation (volume de comptes): Our dataset is composed of 58K Reddit comments, labeled for one or more of 27 emotion(s) or Neutral.
Technologies associées: Principal Preserved Component Analysis (Cowen et al., 2019b)
Mention de l'éthique: ND
Finalité communicationnelle: Understanding emotion expressed in language has a wide range of applications, from building empathetic chatbots to detecting harmful online behavior. Advancement in this area can be improved using large-scale datasets with a fine-grained typology, adaptable to multiple downstream tasks.
Our taxonomy includes a large number of positive, negative, and ambiguous emotion categories, making it suitable for downstream conversation understanding tasks that require a subtle understanding of emotion expression, such as the analysis of customer feedback or the enhancement of chatbots.
Résumé: Understanding emotion expressed in language has a wide range of applications, from building empathetic chatbots to detecting harmful online behavior. Advancement in this area can be improved using large-scale datasets with a fine-grained typology, adaptable to multiple downstream tasks. We introduce GoEmotions, the largest manually annotated dataset of 58k English Reddit comments, labeled for 27 emotion categories or Neutral. We demonstrate the high quality of the annotations via Principal Preserved Component Analysis. We conduct transfer learning experiments with existing emotion benchmarks to show that our dataset generalizes well to other domains and different emotion taxonomies. Our BERT-based model achieves an average F1-score of .46 across our proposed taxonomy, leaving much room for improvement.

Collections: Google

Auteur-es: Egelman, Carolyn D.; Murphy-Hill, Emerson; Kammer, Elizabeth; Hodges, Margaret Morrow; Green, Collin; Jaspan, Ciera; Lin, James
Nombre Auteurs: 7
Titre: Predicting developers' negative feelings about code review
Année de publication: 2020
Référence (APA): Egelman, C. D., Murphy-Hill, E., Kammer, E., Hodges, M. M., Green, C., Jaspan, C., & Lin, J. (2020). Predicting developers’ negative feelings about code review. Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering, 174‑185. https://doi.org/10.1145/3377811.3380414
Mots-clés: code review, interpersonal con ict
URL: https://dl.acm.org/doi/10.1145/3377811.3380414
doi: https://doi.org/10.1145/3377811.3380414
Accessibilité de l'article: Open access
Champ: Software Engineering
Type contenu (théorique Applicative méthodologique): Applicative
Méthode: To study negative behaviors in code reviews, we combine qualitative and quantitative methods using surveys and log data from Google. We developed three log-based metrics, informed by interviews with a diverse group of 14 developers, to detect feelings of pushback in code reviews and validated those metrics through a survey.
Analysis of developer interviews
Analysis of developer survey responses
Cas d'usage: Google developers
Objectifs de l'article: This paper seeks to understand and measure negative experiences in code review. Measurement enables understanding of the prevalence of the bad experiences, whether negative experiences are occurring at different rates in subpopulations of developers, and whether initiatives aimed at reducing negative experiences, like codes of conduct, are working.
Question(s) de recherche/Hypothèses/conclusion: Research question(s) : We asked the following research questions: 1) How frequent are negative experiences with code review? 2) What factors are associated with pushback occuring? 3) What metrics detect author-perceived pushback?; Hypothesis(es) : In the process, developers may have negative interpersonal interactions with their peers, which can lead to frustration and stress; these negative interactions may ultimately result in developers abandoning projects.; Conclusion(s) : Our results suggest that such negative experiences, which we call “pushback”, are relatively rare in practice, but have negative repercussions when they occur. Our metrics can predict feelings of pushback with high recall but low precision, making them potentially appropriate for highlighting interactions that may bene t from a self-intervention
Cadre théorique/Auteur.es: Interpersonal conflict (Schieman et Reid, 2008); Developer's professional experiences (Singh Arneja, 2015 ; Dietrich, 2018 ; Ranzhin, 2019 ; Graziotin et al., 2019)
Concepts clés: Interpersonal conflict
Données collectées (type source): Developer log-based metrics data from Google
Responses to interviews with Google developers
Responses to a survey of Google developers
Définition des émotions: No definition; Use of sentiment categories/groups
Ampleur expérimentation (volume de comptes): 14 one-hour semi-structured interviews
2,500 contacted, 1,317 responded to the first section; 606 authors and 573 reviewers responded to the second section on their own code reviews; and 1,182 responded to the third section on third-party code reviews. 78% of survey respondents worked in an office in the USA; 16% in Europe, the Middle East or Africa; 4% in the Asia-Pacific region; and 2% in the Americas outside the USA.
Technologies associées: Google’s code review process
Mention de l'éthique: About the objective of research :
Unfortunately, we have little systematic understanding of what makes a code review go bad. This is important for three reasons. First, from an ethical perspective, we should seek to make the software engineering process fair and inclusive.

About the interviews :
Given the sensitivity of the interview topics, we conducted interviews with informed consent and according to a strict ethical protocol.1
1. We note specifically that we provided participants the option to participate with or without audio recording, and we paused recording when starting to discuss topics the interviewer or participant identified as potentially sensitive. We also provided materials on organizational resources for dealing with workplace concerns, and described the responsibilities the researchers and notetakers had related to reporting policy violations to reduce any ambiguity about what to expect during or after the session.
Finalité communicationnelle: While our predictions are far from perfect, we believe such predictions are needed to support future interventions designed to help reduce pushback so that we can monitor the effectiveness of those interventions. Detecting pushback may also help identify any subpopulations where pushback may be more prevalent.
Résumé: During code review, developers critically examine each others' code to improve its quality, share knowledge, and ensure conformance to coding standards. In the process, developers may have negative interpersonal interactions with their peers, which can lead to frustration and stress; these negative interactions may ultimately result in developers abandoning projects. In this mixed-methods study at one company, we surveyed 1,317 developers to characterize the negative experiences and cross-referenced the results with objective data from code review logs to predict these experiences. Our results suggest that such negative experiences, which we call "pushback", are relatively rare in practice, but have negative repercussions when they occur. Our metrics can predict feelings of pushback with high recall but low precision, making them potentially appropriate for highlighting interactions that may benefit from a self-intervention.

Collections: Google

Auteur-es: Thomas, Kurt; Akhawe, Devdatta; Bailey, Michael; Boneh, Dan; Bursztein, Elie; Consolvo, Sunny; Dell, Nicola; Durumeric, Zakir; Kelley, Patrick Gage; Kumar, Deepak; McCoy, Damon; Meiklejohn, Sarah; Ristenpart, Thomas; Stringhini, Gianluca
Nombre Auteurs: 14
Titre: SoK: Hate, Harassment, and the Changing Landscape of Online Abuse
Année de publication: 2021
Référence (APA): Thomas, K., Akhawe, D., Bailey, M., Boneh, D., Bursztein, E., Consolvo, S., Dell, N., Durumeric, Z., Kelley, P. G., Kumar, D., McCoy, D., Meiklejohn, S., Ristenpart, T., & Stringhini, G. (2021). SoK : Hate, Harassment, and the Changing Landscape of Online Abuse. 2021 IEEE Symposium on Security and Privacy (SP), 247‑267. https://doi.org/10.1109/SP40001.2021.00028
Mots-clés: ND
URL: https://ieeexplore.ieee.org/abstract/document/9519435
doi: https://doi.org/10.1109/SP40001.2021.00028
Accessibilité de l'article: Open access
Champ: Security, Privacy and Abuse Prevention
Type contenu (théorique Applicative méthodologique): Applicative
Méthode: We collate over 150 research papers and prominent news stories related to hate and harassment and use them to create a taxonomy of seven distinct attack categories.
Analysis of responses to public survey
Wherever possible, we compare our results to similar surveys.
Cas d'usage: Online hate and harassment
Objectifs de l'article: We propose a taxonomy for reasoning about online hate and harassment.
Question(s) de recherche/Hypothèses/conclusion: Research question(s) : In this work, we explore how online hate and harassment has transformed alongside technology and make a case for why the security community needs to help address this threat.; Hypothesis(es) : In this work, we argued that security, privacy, and anti-abuse protections are failing to address the growing threat of online hate and harassment.; Conclusion(s) : We proposed a taxonomy, built from over 150 research articles, to reason about these new threats. We also provided longitudinal evidence that hate and harassment has grown 4% over the last three years and now affects 48% of people globally. Young adults, LGBTQ+ individuals, and frequent social media users remain the communities most at risk of attack. We believe the computer security community must play a role in addressing this threat. To this end, we outlined five potential directions for improving protections that span technical, design, and policy changes to ultimately assist in identifying, preventing, mitigating, and recovering from hate and harassment attacks.
Cadre théorique/Auteur.es: Attack characteristics (Matthews et al., 2017 ; Sambasivan et al., 2019 ; Chatterjee et al., 2018); Motivation of attackers (Citron, 2014); Online hate and harassment(Slonje et Smith, 2008 ; Kowalski et al..., 2016); Cybercrime (Anderson, 2012); Violent extremism and emerging technologies (Brachman, 2006 ; Edwards et Gribbon, 2013 ; Lima et al., 2018); Disinformation and misinformation (Starbird, Arif et Wilson, 2019 ; )
Concepts clés: Online hate and harassment
Données collectées (type source): Taxonomy :
Examination of the last five years of research from IEEE S&P, USENIX Security, CCS, CHI, CSCW, ICWSM, WWW, SOUPS, and IMC, on topics related to hate speech, harassment, trolling, doxing, stalking, non-consensual image exposure, disruptive behavior, content moderation, and intimate partner violence. We then manually searched through the related works of these papers for relevant research, including findings from the social sciences and psychology communities (though restricted solely to online hate and harassment, rather than hate speech or bullying in general). Additionally, we relied on the domain expertise of the authors to identify related works and major recent news events.

Survey :
Our survey asked participants “Have you ever personally experienced any of the following online?” and then listed a fixed set of experiences that participants could select from. [The survey also] asking if this behavior was experienced (prevalence only) and not measuring frequency or severity. We did expand the set to include eight other experiences related to lockout and control, surveillance, content leakage, impersonation, and a deeper treatment of toxic content beyond just name calling (as used by earlier works).
Définition des émotions: Definition of hate and harassment
Ampleur expérimentation (volume de comptes): Taxonomy: Review of over 150 press articles and scientific papers on the subject of online hate and harassment

Survey: 50,000 respondents in 22 countries (Brazil, China, Colombia, France, Germany, India, Indonesia, Ireland, Japan, Kenya, Mexico, Nigeria, Poland, Russia, Saudi Arabia, South Korea, Spain, Sweden, Turkey, United Kingdom, United States, Venezuela)
Technologies associées: Nudges, indicators, and warnings; Human moderation, review, and delisting; Automated detection and curation
Mention de l'éthique: B. Challenges for researchers
Researcher safety and ethics.
Currently, there are no best practices for how researchers can safely and ethically study online hate and harassment. Risks facing researchers include becoming a target of coordinated, hostile groups, as well as emotional harm stemming from reviewing toxic content (similar to risks for manual reviewers) [2]. Likewise, researchers must ensure they respect at-risk subjects and do not further endanger targets as they study hate and harassment.
Finalité communicationnelle: We outlined five potential directions for improving protections that span technical, design, and policy changes to ultimately assist in identifying, preventing, mitigating, and recovering from hate and harassment attacks.
Résumé: We argue that existing security, privacy, and antiabuse protections fail to address the growing threat of online hate and harassment. In order for our community to understand and address this gap, we propose a taxonomy for reasoning about online hate and harassment. Our taxonomy draws on over 150 interdisciplinary research papers that cover disparate threats ranging from intimate partner violence to coordinated mobs. In the process, we identify seven classes of attacks—such as toxic content and surveillance—that each stem from different attacker capabilities and intents. We also provide longitudinal evidence from a three-year survey that hate and harassment is a pervasive, growing experience for online users, particularly for at-risk communities like young adults and people who identify as LGBTQ+. Responding to each class of hate and harassment requires a unique strategy and we highlight five such potential research directions that ultimately empower individuals, communities, and platforms to do so.

Collections: Google

Auteur-es: Kelley, Patrick Gage; Yang, Yongwei; Heldreth, Courtney; Moessner, Christopher; Sedley, Aaron M.; Woodruff, Allison
Nombre Auteurs: 6
Titre: “Mixture of amazement at the potential of this technology and concern about possible pitfalls”: Public sentiment towards AI in 15 countries
Année de publication: 2021
Référence (APA): Kelley, P. G., Yang, Y., Heldreth, C., Moessner, C., Sedley, A. M., & Woodruff, A. (2021). “Mixture of amazement at the potential of this technology and concern about possible pitfalls” : Public sentiment towards AI in 15 countries. Bulletin of the IEEE Computer Society Technical Committee on Data Engineering, 44(4), 28‑46.
Mots-clés: ND
URL: http://sites.computer.org/debull/A21dec/A21DEC-CD.pdf#page=30
Accessibilité de l'article: Open access
Champ: Human-Computer Interaction and Visualization
Type contenu (théorique Applicative méthodologique): Applicative
Méthode: Analysis of responses to public survey
Cas d'usage: Articifial Intelligence
Objectifs de l'article: Explorations of public perception of AI
Question(s) de recherche/Hypothèses/conclusion: Research question(s) : In this paper we focus on the following research objective: What sentiment do respondents have towards AI? Specifically, we present emergent themes, descriptive statistics, and illustrative quotes for the following open-ended question about sentiment: ‘What feelings or emotions come to mind when you hear the phrase Artificial Intelligence (AI)?’; Hypothesis(es) : Rather than presupposing particular sentiment, we began with open-ended responses and looked for emergent themes.; Conclusion(s) : Our findings revealed sentiment groups as a distinguishing feature, with respondents in different countries finding AI to be Exciting, Useful, Worrying, and Futuristic to varying degrees. These groups provide one nuanced alternative to understanding people’s feelings towards AI, rather than considering their orientation to AI as simply positive or negative.
Cadre théorique/Auteur.es: Survey-bsased research on public perception of AI (Cave, Coughlan et Dihal, 2019 ; Zhang et Dafoe, 2022); Sentiment analysis based research on public perception of AI (Fast et Horvitz, 2017 ; Chuan, Tsai et Cho, 2019); Narrative analysis based research on public perception of AI (Cave et al. 2018 ; Cave, Dihal, et Dillon, 2020); Our work sits within a growing body of research on people’s perceptions of AI, across disciplines including HCI, critical studies, law, marketing, policy, psychology, and more. This topic is highly complex, multi-dimensional, and far from fully understood.
Concepts clés: Public perception of AI
Données collectées (type source): Responses to a general public survey
Définition des émotions: No definition; Use of sentiment categories/groups
Ampleur expérimentation (volume de comptes): 17,000 respondents in 15 countries (Germany, Australia, Finland, Singapore, Belgium, Canada, USA, South Korea, Spain, France, Poland, Brazil, China, India and Nigeria)
Technologies associées: Artificial Intelligence
Mention de l'éthique: Beyond that, however, our findings align with calls to develop technology that supports public values. For example, many respondents were concerned about negative impacts of AI on privacy, reinforcing the value of continued emphasis on designing and developing AI with privacy in mind, concordant with discussion of privacy by design in the EU General Data Protection Regulation (GDPR).9 The privacy discussion continues to evolve quickly, and best practices for AI technologies continue to be actively explored in the academic, legal, and policy communities, offering many opportunities for advances in this area.
Finalité communicationnelle: Further, our findings also suggest ways in which the design and development of particular technologies may have a favorable impact on public opinion. For example, our findings point to the value of emphasizing AI’s application to healthcare in product and research investments as well as communications.
Résumé: Public opinion plays an important role in the development of technology, influencing product adoption, commercial development, research funding, career choices, and regulation. In this paper we present results of an in-depth survey of public opinion of artificial intelligence (AI) conducted with over 17,000 respondents spanning fifteen countries and six continents. Our analysis of open-ended responses regarding sentiment towards AI revealed four key themes (exciting, useful, worrying, and futuristic) which appear to varying degrees in different countries. These sentiments, and their relative prevalence, may inform how the public influences the development of AI.

Collections: Google

Auteur-es: Thomas, Kurt; Kelley, Patrick Gage; Consolvo, Sunny; Samermit, Patrawat; Bursztein, Elie
Nombre Auteurs: 5
Titre: “It’s common and a part of being a content creator”: Understanding How Creators Experience and Cope with Hate and Harassment Online
Année de publication: 2022
Référence (APA): Thomas, K., Kelley, P. G., Consolvo, S., Samermit, P., & Bursztein, E. (2022). “It’s common and a part of being a content creator” : Understanding How Creators Experience and Cope with Hate and Harassment Online. Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, 1‑15. https://doi.org/10.1145/3491102.3501879
Mots-clés: Security and privacy, hate, harassment, creators, content moderation
URL: https://dl.acm.org/doi/10.1145/3491102.3501879
doi: https://doi.org/10.1145/3491102.3501879
Accessibilité de l'article: Open access
Champ: Security, Privacy and Abuse Prevention
Type contenu (théorique Applicative méthodologique): Applicative
Méthode: Analysis of survey responses from content creators
Cas d'usage: American content creators on Facebook, Instagram, TikTok, Twitter or YouTube
Objectifs de l'article: We surveyed creators to understand their personal experiences with attacks (including toxic comments, impersonation, stalking, and more), the coping practices they employ, and gaps they experience with existing solutions (such as moderation or reporting)
Question(s) de recherche/Hypothèses/conclusion: Research question(s) : In order to prevent or mitigate attacks, creators and platforms rely on a variety of levers including community guidelines [53], content moderation tools [32, 33, 52, 71], and abuse reporting [14, 34], with an eye towards increasingly automated detection [21, 29]. However, multiple critiques have been leveled at these systems, including fragility to evasion or bias [5, 18, 35] and a lack of transparency for what policies are enforced [8]. Equally challenging, protections for hate and harassment are often disjointed, requiring unique interventions for a creator experiencing toxic comments, versus a creator whose personal information was leaked, or a creator being overloaded with negative reviews and ratings [64].; Hypothesis(es) : As such, we argue there is a gap in the community’s current understanding for which solutions best protect creators, and how to prioritize improvements based of the frequency of attacks and their resulting harms.; Conclusion(s) : Nearly every creator in our study experienced some form of hate and harassment, and for one in three, such experiences were a regular occurrence. As such, creators represent a population at-risk of hate and harassment compared to general internet users. Creators frequently relied on content moderation, and to a lesser extent reporting, for responding to attacks. However, when these platform-provided tools fell short, some creators engaged in protective practices such as self-censoring their personal attributes and beliefs, or leaving platforms and communities entirely, in order to avoid further harm.
Cadre théorique/Auteur.es: Tools used by platforms to combat hate and harassment (Pater et al., 2016); Shortcomings of these control measures (Pater et al., 2016 ; Crawford et Gillespie, 2016); Estimates of hate and harassment online (PEW Research Center, 2017); Research about people who are at a risk (Warford et al., 2021)
Concepts clés: Content moderation; Online hate and harassment
Données collectées (type source): Responses to a survey submitted to US content creators on Facebook, Instagram, TikTok, Twitter or YouTube
Définition des émotions: No definition; List of harassment experiences
Ampleur expérimentation (volume de comptes): 198 contacted, 145 replies, 135 retained
Technologies associées: Squadbox [42], which allows a person to appoint family members, friends, or community members to assist with review.; keyword lists like Hatebase [26] and community-generated blocklists of abusive accounts like Blocktogether [9].; HeartMob, which connects targets of harassment with supporters [8].
Mention de l'éthique: 3.4 Research ethics & anonymization
To ensure our work did not put participants at undue risk when recalling past sensitive experiences, our study plan was reviewed by a set of experts at Google in domains including ethics, human subjects research, policy, legal, security, privacy, and anti-abuse. We note that Google research does not require IRB approval, though we adhere to similarly strict standards. We alerted participants that our survey collected sensitive demographic data in our consent form. Additionally, all demographic questions included an option to “Prefer not to say”, and did not require any answer within our survey tool.
We took multiple steps to ensure the anonymity of participants. We note that our survey instrument never collected names, email addresses, social media handles, or other public identifers of creators. Distribution and compensation was handled solely by the organizers of the residency program who had access to identifying contact information, while the researchers involved in the study were the only parties with access to raw response data. Throughout the paper, the quotes we provide are the unedited responses of participants. We have only removed identifying information, including specifc platform and community names or features to protect the participants from de-anonymization.
Finalité communicationnelle: As creators represent an at-risk population, it is our belief that their lived experiences act as a portent for how hate and harassment will evolve online. Lessons and protective practices that emerge for creators can thus inform the broader solution space for general internet users.
Résumé: Content creators—social media personalities with large audiences on platforms like Instagram, TikTok, and YouTube—face a heightened risk of online hate and harassment. We surveyed 135 creators to understand their personal experiences with attacks (including toxic comments, impersonation, stalking, and more), the coping practices they employ, and gaps they experience with existing solutions (such as moderation or reporting). We find that while a majority of creators view audience interactions favorably, nearly every creator could recall at least one incident of hate and harassment, and attacks are a regular occurrence for one in three creators. As a result of hate and harassment, creators report self-censoring their content and leaving platforms. Through their personal stories, their attitudes towards platform-provided tools, and their strategies for coping with attacks and harms, we inform the broader design space for how to better protect people online from hate and harassment.

Collections: Google

Meta

The spread of emotion via Facebook

Textless Speech Emotion Conversion using Discrete & Decomposed Representations

Modeling Self-Disclosure in Social Networking Sites

Textless Speech Emotion Conversion using Discrete & Decomposed Representations

“That’s so cute!”: The CARE Dataset for Affective Response Detection

Once More with Feeling: Supportive Responses to Social Sharing on Facebook

Ensemble of Generative and Discriminative Techniques for Sentiment Analysis of Movie Reviews

Modeling Self-Disclosure in Social Networking Sites

Joint Modelling of Emotion and Abusive Language Detection

Towards Pleasant Touch: Vibrotactile Grids for Social Touch Interactions

What’s in a Like? Attitudes and Behaviors Around Receiving Likes on Facebook

Once More with Feeling: Supportive Responses to Social Sharing on Facebook

Textless Speech Emotion Conversion using Discrete & Decomposed Representations

Does Visual Self-Supervision Improve Learning of Speech Representations for Emotion Recognition?

Exploring Deep Multimodal Fusion of Text and Photo for Hate Speech Classification

Bringing Portraits to Life

Textless Speech Emotion Conversion using Discrete & Decomposed Representations

Does Visual Self-Supervision Improve Learning of Speech Representations for Emotion Recognition?

Joint Modelling of Emotion and Abusive Language Detection

Exploring Deep Multimodal Fusion of Text and Photo for Hate Speech Classification

Towards Pleasant Touch: Vibrotactile Grids for Social Touch Interactions

“That’s so cute!”: The CARE Dataset for Affective Response Detection

The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes

Social Comparison and Facebook: Feedback, Positivity, and Opportunities for Comparison

The Dialogue Dodecathlon: Open-Domain Knowledge and Image Grounded Conversational Agents

Joint Modelling of Emotion and Abusive Language Detection

Communicating Socio-Emotional Sentiment Through Haptic Messages

Lemotif: An Affective Visual Journal Using Deep Neural Networks

“That’s so cute!”: The CARE Dataset for Affective Response Detection

Textless Speech Emotion Conversion using Discrete & Decomposed Representations

Does Visual Self-Supervision Improve Learning of Speech Representations for Emotion Recognition?

TOKEN: Task Decomposition and Knowledge Infusion for Few-Shot Hate Speech Detection

An Online Survey on the Perception of Mediated Social Touch Interaction and Device Design

Affective Signals in a Social Media Recommender System

Linking Haptic Parameters to the Emotional Space for Mediated Social Touch

“That’s so cute!”: The CARE Dataset for Affective Response Detection

Textless Speech Emotion Conversion using Discrete & Decomposed Representations

SS-VAERR: Self-Supervised Apparent Emotional Reaction Recognition from Video

Google

StoryFaces: pretend-play with ebooks to support social-emotional storytelling

A comparison of classifiers for detecting emotion from speech

Comparative experiments on sentiment classification for online product reviews

Structured Models for Fine-to-Coarse Sentiment Analysis

Building a Sentiment Summarizer for Local Service Reviews

A Joint Model of Text and Aspect Ratings for Sentiment Summarization

Emotional Memory and Adaptive Personalities

Sentiment summarization: evaluating and learning user preferences

What's great and what's not: learning to classify the scope of negation for improved sentiment analysis

Discovering Fine-Grained Sentiment with Latent Variable Structured Prediction Models

Semi-supervised latent variable models for sentence-level sentiment analysis

Aesthetics and Emotions in Images

StoryFaces: pretend-play with ebooks to support social-emotional storytelling

SUIT: A Supervised User-Item Based Topic Model for Sentiment Analysis

HaTS: Large-scale In-product Measurement of User Attitudes & Experiences with Happiness Tracking Surveys

Video Watch Time and Comment Sentiment: Experiences from YouTube

3D Human Sensing, Action and Emotion Recognition in Robot Assisted Therapy of Children With Autism

Building Empathy: Scaling User Research for Organizational Impact

Multilingual Multi-class Sentiment Classification Using Convolutional Neural Networks

Testing Grayscale Interventions to Reduce Negative Emotional Impact on Manual Reviewers

Eyemotion: Classifying Facial Expressions in VR Using Eye-Tracking Cameras

AttentiveVideo: A Multimodal Approach to Quantify Emotional Responses to Mobile Advertisements

GLA in MediaEval 2018 Emotional Impact of Movies Task

Speech Sentiment Analysis via Pre-Trained Features from End-to-End ASR Models

GoEmotions: A Dataset of Fine-Grained Emotions

Predicting developers' negative feelings about code review

SoK: Hate, Harassment, and the Changing Landscape of Online Abuse

“Mixture of amazement at the potential of this technology and concern about possible pitfalls”: Public sentiment towards AI in 15 countries

“It’s common and a part of being a content creator”: Understanding How Creators Experience and Cope with Hate and Harassment Online