Towards Empathetic Open-domain Conversation Models: a New Benchmark and Dataset

Auteur-es

Hannah Rashkin, Eric Michael Smith, Margaret Li, Y-Lan Boureau

Nombre Auteurs

4

Titre

Towards Empathetic Open-domain Conversation Models: a New Benchmark and Dataset

Année de publication

2019

Référence (APA)

Rashkin, H., Smith, E. M., Li, M., & Boureau, Y.-L. (2019). Towards Empathetic Open-domain Conversation Models : A New Benchmark and Dataset. https://doi.org/10.18653/v1/P19-1534

résumé

One challenge for dialogue agents is recognizing feelings in the conversation partner and replying accordingly, a key communicative skill. While it is straightforward for humans to recognize and acknowledge others’ feelings in a conversation, this is a significant challenge for AI systems due to the paucity of suitable publicly-available datasets for training and evaluation. This work proposes a new benchmark for empathetic dialogue generation and EMPATHETICDIALOGUES, a novel dataset of 25k conversations grounded in emotional situations. Our experiments indicate that dialogue models that use our dataset are perceived to be more empathetic by human evaluators, compared to models merely trained on large-scale Internet conversation data. We also present empirical comparisons of dialogue model adaptations for empathetic responding, leveraging existing models or datasets without requiring lengthy retraining of the full model.

URL

https://research.facebook.com/file/537241710847159/Towards-Empathetic-Open-domain-Conversation-Models-a-New-Benchmark-and-Dataset.pdf

doi

https://doi.org/10.18653/v1/P19-1534

Accessibilité de l'article

Libre

Champ

Artificial Intelligence, Natural Language Processing & Speech

Type contenu (théorique Applicative méthodologique)

Applicatif

Méthode

The method involves collecting a dataset of 25k dialogues grounded in situations prompted by specific emotion labels. The dataset is used to provide retrieval candidates or fine-tune conversation models to generate more empathetic responses.

Cas d'usage

N/A

Objectifs de l'article

The objectives of the article are to introduce a new dataset for training and evaluating dialogue models that can recognize and respond to emotions in a conversation partner, and to demonstrate the effectiveness of this dataset in improving the empathetic quality of dialogue models.

"This work aims to facilitate evaluating models’ ability to produce empathetic responses. We introduce a new task for dialogue systems to respond to people discussing situations that cover a wide range of emotions, and EMPATHETICDIALOGUES (ED), a novel dataset with about 25k personal dialogues."

Question(s) de recherche/Hypothèses/conclusion

The research question is how to integrate empathetic responding into more general dialogue when the needs for empathy have to be balanced with staying on topic or providing information.

The hypothesis is that using the EMPATHETICDIALOGUES dataset to train dialogue models will lead to responses that are evaluated as more empathetic.

The conclusions are that using the EMPATHETICDIALOGUES dataset to provide retrieval candidates or fine-tune conversation models leads to responses that are evaluated as more empathetic, and that this dataset and the results of the experiments will stimulate more research in the direction of making dialog systems more empathetic.

Cadre théorique/Auteur.es

The theoretical framework of the article includes concepts from psychology and natural language processing. The main authors cited include Klaus R. Scherer, Harald G. Wallbott, Amy Skerry, Rebecca Saxe, Carlo Strapparava, and Rada Mihalcea.

Concepts clés

Emotion recognition, Empathy, Open-domain conversation

Données collectées (type source)

The data collected consists of personal conversations grounded in situations related to a given feeling. The situations are associated with a given emotion label, chosen from a list of 32 labels that cover a broad range of positive and negative emotions.

Human evaluations were collected on MTurk.

Définition des émotions

Non

Ampleur expérimentation (volume de comptes)

25k conversations

100 ratings per model

221 US workers rated

Technologies associées

Natural language processing, Machine learning, DL, FastText model, Transformer networks/architecture, BERT retrieval models

Mention de l'éthique

Non. Anonymous reviewers.

Finalité communicationnelle

Making dialog systems more empathetic.

"Future work will investigate how to integrate empathetic responding into more general dialogue when, for example, the needs for empathy have to be balanced with staying on topic or providing information."

Commentaires

Financed by National Science Foundation Graduate Research Fellowship Program under Grant No. DGE-1256082. - exigences en terme d'éthique ?