👋 I'm happy to be making my first contribution to the Quant UX Blog! I'm Chris's co-author on the Quant UX Research book, and I'm excited to join him in sharing perspectives that complement the book.
I led the early Quant UX Research team at Google, where we came up with the HEART framework to help teams define metrics of user experience. More than a decade ago, I wrote this
blog post about UX metrics
that introduced HEART to a wider audience outside of Google.
Since then, it has been applied by teams across the tech industry, and has found its way into many introductory resources about user experience and product management, including the popular book Escaping the Build Trap. Those resources generally cover similar ground to my original blog post, and there is little published material about what happens when teams actually apply HEART to real projects.
In this post, I'll go beyond those basic concepts and share some new perspectives on how to apply HEART in practice, based on material from Chapter 7 of the Quantitative UX Research book.
It's divided into four sections:
- What is the HEART framework?
- Don't skip the Goals-Signals-Metrics process
- Avoid individual pitfalls
- Look out for organizational challenges
With this knowledge, you'll have a better chance of helping your team reach a successful implementation of user experience metrics.
What is the HEART Framework?
HEART helps teams break down the broad concept of "user experience" into more specific, measurable outcomes. It also encourages teams to consider multiple aspects of the user experience when defining metrics, although it does not cover every possible aspect.
The acronym stands for:
Happiness: Measures of user attitudes, often collected via surveys. This might include satisfaction or perceived ease of use.
Engagement: The level of user involvement with a product, typically measured by frequency, intensity, or depth of interaction. For example, the number of visits during a certain time period, or the usage of key features.
Adoption: How many new users start using a product or feature. Making an explicit distinction between new and existing users helps a team to understand growth.
Retention: The rate at which existing users return to the product. This can be thought of as a long-term version of engagement. Some teams focus more specifically on failure to retain, which is known as "churn".
Task success: The efficiency, effectiveness, and error rate of user actions. This category often yields the most useful metrics for UX changes, provided that task-specific data is available.
Don't Skip the Goals-Signals-Metrics Process
HEART 💖 has a fun acronym that makes it easy to remember. Teams often get enthusiastic about it and jump straight to brainstorming metric ideas, because they want to get started on building a dashboard.
However, this is very unlikely to lead to a successful outcome. Metrics are not useful unless they are aligned closely with the team's high-level goals... and many teams are surprisingly unclear about what those are, especially in terms of user experience.
The Goals-Signals-Metrics process is designed to help with this problem by encouraging teams to start by thinking at a higher level.
Goals: Using the HEART framework as inspiration, define the overarching objectives for your product or feature. This step involves team discussions to align on priorities and user experience goals, including explicitly addressing the inevitable disagreements. Omit HEART categories that are less relevant to your project.
Signals: For each goal, identify possible signals — ways that success or failure might manifest in user behavior or attitudes. Map the goals to the data that you are (or could be) collecting about user experience. Consider both the ease of tracking these signals and their likely sensitivity to design changes.
Metrics: Develop specific, quantifiable measurements based on the signals. This involves steps like figuring out how to analyze the low-level signals (e.g., using averages or percentages) and deciding on appropriate time periods for aggregation.
By following this process, teams can create meaningful metrics that align closely with their product goals and user experience priorities. To give a simple example, a goal of "make the upload process easier" might map to signals relating to completion of each stage of the process, or, alternatively, to responses to an inline survey question about ease of use. The signal about completion could translate to a specific metric like "the percentage of times a user finishes the upload flow successfully, having started it in the past 7 days".
Two important aspects of the process that I want to emphasize:
You Must Prioritize: Focus on implementing metrics related to your top goals. It's better to have a few well-chosen metrics than an overwhelming dashboard.
You Must Iterate: When you've gone through the process the first time, you are not done. As you collect data and gain insights, be prepared to refine your choices over time.
Avoid These Individual Pitfalls
Certain issues come up over and over again for individual quantitative UX researchers when they try to apply HEART and Goals-Signals-Metrics. I still make some of these mistakes myself!
1. Neglecting Team Involvement
While it might seem efficient to develop metrics independently, this approach can backfire. Engaging your team throughout the process:
- Increases ownership and buy-in
- Improves the quality of metric ideas
- Enhances the impact of the eventual assessments
I suggest scheduling a collaborative session to work through the first part of the Goals-Signals-Metrics process with key members of your team: agree on goals, and brainstorm possible signals.
2. Starting Too Ambitiously
The HEART framework is most useful when applied to specific projects with engaged teams. Avoid the temptation to create organization-wide dashboards immediately. Instead:
- Begin with a single, receptive team
- Focus on details and learn from the experience
- Use this initial project as a case study to inspire other teams
3. Underestimating the Rest of the Process
HEART and Goals-Signals-Metrics are only the beginning of a long process. Just because you used them to come up with a metric idea, that doesn't mean that it's a good or a useful metric, or that it has any correlation with quality of user experience.
Be prepared for:
- Data analysis to refine your signals and metrics
- Iterating on your chosen metrics as you learn more
- Implementation challenges, including instrumentation and dashboard creation
Allocate sufficient time and resources for these crucial next steps.
4. Metric Overload
While HEART can help you generate numerous ideas, implementing too many metrics can be counterproductive. To avoid overwhelming your team and other stakeholders:
- Clearly prioritize your most important metrics
- Consider implementing secondary metrics on a separate dashboard
- Remember that you don't need to use all HEART categories – focus on what's most relevant to your project
Look Out For These Organizational Challenges
Every project takes place in context, and different organizations and teams have different challenges that you can attempt to proactively address, or keep in mind when selecting the right projects to apply HEART.
1. Fear of Evaluation
Introducing targeted metrics can expose project shortcomings, which may create anxiety within teams. To address this:
- If you are in a leadership role, foster a culture of learning from failures (e.g., blameless post-mortems); otherwise, prioritize working with teams who already have this kind of culture
- Involve stakeholders in the metric definition process to build trust and ownership
- Emphasize the value of being informed by data, even when results are unexpected
2. Single Metric Tunnel Vision
While having too many metrics is problematic, focusing on a single metric is also detrimental, but leaders are often drawn to that in an attempt to create clarity and focus. Chris has a separate post about the problems with "North Star" metrics. Remember:
- No metric can be a perfect representation of user experience
- Additional key metrics provide essential checks and balances
- Optimizing for one metric can render it useless for evaluation (Goodhart's Law)
3. Not Considering Ethical Implications
UX metrics are only proxies for user experiences. No metric of user engagement can actually identify how truly engaged a user is, or whether that engagement represents a positive experience for them. For example, time spent in a product is often used as a default metric of engagement, but this may not be appropriate for a given product, especially if unhealthy overuse is a possibility.
When implementing metrics:
- Consider long-term outcomes and potential negative consequences
- Use qualitative research to gain deeper insights into actual user experiences
- Be willing to iterate on metrics as you learn more about what constitutes a positive user experience
Key Takeaways
HEART is a useful tool to help teams focus on the user experience when defining metrics. Here's a summary of the main things to keep in mind when applying it in practice:
- Use HEART as a starting point to consider various aspects of user experience, but don't use all of the categories or force goals to fit into them
- Follow the Goals-Signals-Metrics process to create meaningful, prioritized metrics
- Collaborate with your team to define goals, signals, and metrics
- Start small and grow your approach over time
- Be prepared for the work that comes after initial metric definition
- Prioritize your most important metrics to avoid overload, but don't narrow down to a single metric
- Look out for the organizational challenges that come with negative results
- Consider ethical implications and long-term outcomes
For more detail on all of this, and a case study of a Gmail project, see Chapter 7 of the Quantitative UX Research book. I'm also available for consulting or speaking on these topics 😎
Have you applied HEART with your team? If you have, I encourage you to write about your experiences, so that the rest of the community can build on what you've learned.
