HaTS: Large-scale In-product Measurement of User Attitudes & Experiences with Happiness Tracking Surveys · EMOTICONES

Auteur-es

Müller, Hendrik; Sedley, Aaron

Nombre Auteurs

2

Titre

HaTS: Large-scale In-product Measurement of User Attitudes & Experiences with Happiness Tracking Surveys

Année de publication

2014

Référence (APA)

Müller, H., & Sedley, A. (2014). HaTS : Large-scale In-product Measurement of User Attitudes & Experiences with Happiness Tracking Surveys. Proceedings of the 26th Australian Computer-Human Interaction Conference on Designing Futures: the Future of Design, 308‑315. https://doi.org/10.1145/2686612.2686656

Mots-clés

Surveys; metrics; tracking; attitudes; large scale; HaTS

URL

https://dl.acm.org/doi/10.1145/2686612.2686656

doi

https://doi.org/10.1145/2686612.2686656

Accessibilité de l'article

Open access

Champ

Human-Computer Interaction and Visualization

Type contenu (théorique Applicative méthodologique)

Applicative

Méthode

Analysis of responses to user satisfaction questionnaire
We detail best Happiness Tracking Surveys (HaTS) for collecting attitudinal data at a large scale directly in the product and over time.
This method was developed at Google to track attitudes and open-ended feedback over time, and to characterize products’ user bases.
This case study of HaTS goes beyond the design of the questionnaire to also suggest best practices for appropriate sampling, invitation techniques, and its data analysis.

Cas d'usage

A dozen Google products (consumer and enterprise) to support product development and optimize UX

Objectifs de l'article

In this industry case study, we are introducing a particular survey method, referred to as Happiness Tracking Surveys (HaTS), that is designed for ongoing tracking of user attitudes and experiences within the context of real-world product usage at a large scale.

Question(s) de recherche/Hypothèses/conclusion

Research question(s) : While measuring attitudinal data at a small scale for a given design or product (i.e., in the lab or field) has been studied heavily and is widely adopted in the HCI community, there have been fewer contributions towards a model to reliably track a product’s attitudes over time and at a large scale

Hypothesis(es) : The HaTS survey method presented in this industry case study relies on these latest advances in questionnaire design and attempts to fill the gap of measuring attitudes at a large scale, in the context of real-world product usage, and over time through random sampling and the use of proactive survey invitations to aim for valid, reliable, and actionable data.

Conclusion(s) : For a variety of reasons, HaTS has proven to be a useful, high quality method for measuring, tracking and comparing users’ attitudes at a large scale, and one that can be effectively adopted by others who endeavor to better understand users’ attitudes and experiences. From the outset, HaTS has used probability sampling, the gold standard among survey researchers for achieving representative results for a given population. Users are sampled at the moment they are actually using the product, ensuring that their responses accurately reflect their true experiences, unaffected by memory bias.

Cadre théorique/Auteur.es

Product evaluation questionnaire (Brooke, 1996 ; Chin, Diehl, et Norman, 1988 ; Hassenzahl, Burmester, et Kolle, 2003 ; Kirakowski et Corbett, 1993 ; Kirakowski et Dillion, 1987)

Behavioral analysis through the use of log data (Rodden, Hutchinson, et Fu, 2010)

Recent work in the social sciences on the design of valid and reliable questionnaires (Coupe, 2008 ; Groves et al., 2004 ; Krosnick, 1991 ; Krosnick, 1997 ; Krosnick, Narayan, et Smith, 1996 ; Muller, Sedley, et Ferrall-Nunge, 2014 ; Saris, Krosnick, et Shaeffer, 2005 ; Smith, 1967 ; Tourangeau, Couper, et Conrad, 2004)

Concepts clés

User experience measurement

Données collectées (type source)

Responses to a satisfaction questionnaire with users of Google products
Each week, a representative set of a product’s users are randomly selected to be invited to take part in HaTS.
The HaTS questionnaire follows a “funnel” approach from broad and high-level to more specific and personal questions.
In the beginning of the questionnaire, we include questions directly related to the survey topic and ask about attitudes and feedback about product as a whole (to avoid potential biases resulting from questions that ask about specific aspects of the product). These initial questions are also important to help build rapport with the respondent.
After high-level aspects have been assessed, the questionnaire then dives into common product attributes as well as productspecific tasks.
Finally, questions about respondents’ characteristics are asked about towards the end as they may be perceived as more sensitive by some.

Définition des émotions

Definition of joy

Ampleur expérimentation (volume de comptes)

During any given week, a maximum of about 8% of the entire user base may be invited to the survey.
For HaTS, as a best practice, we often aim for about 400 or 1000 responses for the time period of interest.

Technologies associées

ND

Mention de l'éthique

ND

Finalité communicationnelle

HaTS has proven to be a useful, high quality method for measuring, tracking and comparing users’ attitudes at a large scale, and one that can be effectively adopted by others who endeavor to better understand users’ attitudes and experiences.

We believe other organizations can yield significant value by adopting HaTS, adjusting it to their specific needs, and continuing to refine the platform for high quality, actionable results.

Résumé

With the rise of Web-based applications, it is both important and feasible for human-computer interaction practitioners to measure a product's user experience. While quantifying user attitudes at a small scale has been heavily studied, in this industry case study, we detail best Happiness Tracking Surveys (HaTS) for collecting attitudinal data at a large scale directly in the product and over time. This method was developed at Google to track attitudes and open-ended feedback over time, and to characterize products' user bases. This case study of HaTS goes beyond the design of the questionnaire to also suggest best practices for appropriate sampling, invitation techniques, and its data analysis. HaTS has been deployed successfully across dozens of Google's products to measure progress towards product goals and to inform product decisions; its sensitivity to product changes has been demonstrated widely. We are confident that teams in other organizations will be able to embrace HaTS as well, and, if necessary, adapt it for their unique needs.