First, a few caveats. No one here knows I'm recommending them, and I'm not incentivized to do so. Still, if you contact them, it's fine to mention my recommendation (or not). Please don't infer anything from the absence of anyone. It might be that I don't recommend them, but more likely I don't know of them, forgot about them, or the list was too long to include them. Finally, this list reflects my own needs and history. It may not align perfectly with anyone else.

In a nutshell, these are folks I call first. If you know what a Rolodex is, this is mine. Working with these folks will help your projects, too!

General Software Tools

Before recommending suppliers, these are the baseline tools I use everyday.

R and RStudio. This is pretty obvious, especially given my R book. I mostly use base R instead of tidyverse for simplicity and long term stability. I have nothing against Python, though. My Quant UX book was actually written inside RStudio, from draft through near-final PDF! BTW, I reported empirical observations of my most-used R packages in this post.

Google Docs. Again, pretty obvious. I will call out Docs' speed, reliability, and seamless abilities to collaborate and co-edit. Yes, there are downsides and I wish Proton was a more full-featured alternative but it isn't. (In case you're wondering, no, I do not use Excel or Word.)

TexShop / LaTeX. The R book and the Python book were both written start-to-finish using TeXShop, all the way to a near camera ready PDF handed off to the publisher. The Quant UX book was also LaTeX but using RStudio instead of TeXShop. The start-up effort of LaTeX is high but the power is unmatched. I love to see things "camera ready" as I write.

GitHub / BitBucket. My coauthors and I collaborate using GitHub (or BitBucket, depending) so we can work on drafts simultaneously, and have backups and versioning. If you code, or write using any local file system tool (e.g., Scrivener), you should learn git.

BTW, my laptops have all been Macs for the past 15 years. In addition to other features, Macs are especially well-suited for R and LaTeX users. I'm writing this with a MacBook Pro M5.

Survey Research Suppliers

Platforms and people I rely on for great survey research authoring and fielding.

Sawtooth Discover. This is my all-purpose survey authoring & hosting platform. Sawtooth has long been the leader for MaxDiff and conjoint analysis surveys, but even when I'm just doing a general survey I use their platform. It's free to use for up to N=50 respondents.

Cloud Research. Their Connect product is my most-used provider to obtain general "self service" panel responses to surveys. It is exceptionally easy to use and transparent on pricing, and they have done a lot of work on respondent quality.

Verasight. They are a new player in the space of quality sampling and research with probability-based survey panels. I have not used them yet, but they do great research on the current sample landscape, and they are now on my short list for DIY sample needs.

Answers Research. For 15+ years, when I've needed someone to manage survey research I can't handle fully — complex fielding, multiple languages, authoring with stakeholders, fielding, qualitative add-ons, and/or analytics — Answers is the first partner I've asked.

Unabashed Research. This is a different kind of recommendation. When I want someone to directly partner with me to scope, lead, run, and report research, my first call is to James Alford at Unabashed. I've worked with him on dozens and dozens of projects for 20 years.

General Research Suppliers

The folks I call when I need someone to lead specific projects, especially in-person research. I have used these groups for many projects, each for 15+ years. Besides the US, I have done a lot of research in Japan; it is an unusually rewarding and insightful locale.

Caelus (US). I've known the folks at Caelus for 25 years and they handle everything from in-person research, to validated recruiting, to online survey and related projects. They are unparalleled at assessing research needs and helping to form honest, transparent plans.

Blink UX (US). Blink UX is a large agency able to handle all aspects of UX product work from research, to concept testing, to design iteration. Several of their leaders are former close colleagues of mine, and I highly recommend them.

R2 Insights (Japan). A boutique, highly personalized research service provider for Japan. For English-speaking product teams, R2 is the ideal partner to help you understand the Japan market. I've done 20+ projects with them over the years.

Sugata Research (Japan). My other favorite Japan research agency is Sugata, with whom I've also done many projects over the years. They are larger than R2 Insights and may be able to support you with a larger team when that is needed (e.g., parallel, multi-city research).

Qualitative Research Facilities

When I run field research myself (as opposed to have one of the above suppliers lead it) — whether it involves in-depth interviews, home visits, or focus groups — I call these folks for recruiting and local facilities. (For Japan, see R2 Insights and Sugata Research above.)

Fieldwork (US). These are top-notch research facilities all around the US. They have layouts suitable for in-depth interviews, focus groups, usability and concept tests, and specialty projects such as cooking tests. Very easy to coordinate efficient, multi-city projects.

Sago (Europe). I've used their network and partners many times in Munich, Paris, Berlin, London, Hamburg, and Rome. In the whole world, my personal all-time favorite research facility is their location in old city Munich!

Acorn Asia (Asia Pacific generally). I've used Acorn Asia for multiple projects; if I recall correctly, in China, South Korea, Indonesia, and the Philippines. Similar to Fieldwork and Sago, it is easy to coordinate multiple locations as part of a single efficient project plan.

P.S. A Plug for International Research

It is shockingly common for US-based companies to skip international customer research, despite the fact that customer needs, perceptions, considerations, and behaviors in other countries may vary substantially from the US.

The usual reasons not to do international research are:

• the US market is the largest

• we'll launch first in the US

• any other single market is too small

• it is too expensive to do international research

• we don't have the time for international research

My responses to those claims are:

• the US market is the largest ==> OK, and the rest of the world is larger when combined

• we'll launch first in the US ==> it's better to find international problems before the product is locked-in for any market

• any other single market is too small ==> the point is not to test or develop for any single market, but to use one or two locations to "pressure test" the product and needs

• it is too expensive to do international research ==> don't do international testing all the time, but allocate some money; the risk of not doing it is higher than cost of doing some

• we don't have the time for international research ==> with efficient research partners as above, it will take very little additional time

As a general high-level matter, as a US-based researcher for US-based companies, I tended to allocate around 20% of projects to international locations. That ensured that we understood them at a baseline level, with enough "sample" to be confident we would catch big issues. Yet it was a small enough proportion to ensure that we placated stakeholders about the overall level, and it did not claim excessive time.

A shorter way to explain it: if you're building for the whole world (and you probably are), then you owe it to the world to consider their needs and issues. Ultimately, those customers are paying you — and you should devote some research to them!

I propose a stack with 4 levels of how I view “rigor” … and first, I’ll start with what is not rigor.

What Is Not Rigor?

Rigor is not a synonym for advanced methods. In fact, I believe that using so-called advanced methods (such as, say, causal modeling, deep learning, multilevel regression) is too often a crutch that people grab when they are confronted with a serious problem apart from analytics. A few of those problems are:

• Insecurity. They want to use advanced methods to say “look how smart I am”

• Poor data. They hope to rescue something when the data set is a mess

• Wanting to learn something. They’d like to stretch the boundaries of their knowledge

I am not saying those problems are bad or wrong to address. Also, I am not saying that every use of an advanced method is wrong. Far from it! Quants should happily use any method that is appropriate. Rather, I am simply observing that complex methods are often chosen for reasons other than statistical suitability. And even when they are suitable, a complex analysis is not inherently more rigorous than a simple one.

Rigor is not statistical significance or p-values. First of all, as a Bayesian, I don’t find the concepts of statistical significance and p-values to be very useful, for reasons similar to those described by Frank Harrell (who TBH is a much more knowledgable statistician than I am). Setting Bayesian stats aside, even in the frequentist world statistical significance is closely tied to null hypothesis significance testing (NHST) … and in my opinion that’s not what quants should do most of the time (see Cohen’s famous takedown of p-values and NHST).

Instead, I believe quant UXRs should use data to learn and inform decisions. Most of that process cannot be forced into an NHST paradigm, and assigning a p-value to some portion is misleading at best.

In short, statistical significance is a misleading concept. Unfortunately, those who ask about it most often may understand it the least. Again, that’s not 100% of the time and NHST has occasional uses (some forms of A/B and multivariate testing sometimes are exceptions).

Rigor is not massive data. I often see studies that attempt to collect massive data, either at a sample level (such as 10000s of survey responses, or 100Ms of online users) or at a response level (such as collecting 100s or 1000s of variables per observation). The problem is that massive data is not necessarily good data, and analysis of it may increase the odds of finding spurious associations and misleading results.

In other words, a bigger load of garbage is still a load of garbage, and “advanced methods” (see above) won’t save it. Getting more data is not more “rigorous,” it is just more data, good or bad.

Rigor is not an unwavering protocol. This is a problem I sometimes see when a researcher transitions from academia to industry. One common form is to put every UX research participant through an identical set of instructions, tasks, and data collection methods. This may occur in interviews, usability tests, or focus groups, as well as surveys. The usual motivation is to collect identical, consistent, and “unbiased” data.

The problem with a rigid protocol is this: it maximizes the defensibility of a data against criticism from other researchers (such as journal reviewers) but it does not maximize what we learn from our participants. I vastly prefer to engage with participants and adapt an interview or other protocol in response to what I hear. If that gives “incomparable data,” then that is my problem to determine how to have impact with stakeholders … but however I resolve that, I will have learned more from participants than I would by subjecting them to a tedious, unwavering test. And that maximizes business value of the research.

In summary, rigorous research does not imply or necessitate:

• Advanced methods

• A focus on hypothesis testing, “statistical significance,” or p-values

• Massive data sets

• Unwavering protocols for interviews or other data collection

What is Rigor?

You may have noticed above that I repeatedly suggested that the point of research is to learn from our users, customers, and research participants and to inform product or other business decisions. To do that rigorously is to learn deeply and to focus relentlessly on making decisions well.

To those ends, I propose the following four area of research rigor. I believe there is some degree of logical order to these areas such that each one presupposes and is built on the previous one, yet we could also view them as complementary rather than strictly nested.

Rigor 1. Focus on Learning and Decisions

The first area of rigor is to focus on decisions that we need to make, and how to learn appropriately from customers, users, and others to make those decisions.

This may seem obvious and yet it often goes wrong in industry UX practice. For example, a classic usability test to find usability issues may go wrong if there is no process in place to act on and fix the issues that are found, or if management will simply dismiss them as “by design” or “won’t fix.” Likewise, a customer satisfaction (CSat) tracker is of no use unless there is a process in place to do something with the results.

To focus on decisions doesn’t mean to reduce everything to an A/B test or a specific product decision. A “decision” may mean a product action such as changing a feature but it may also mean to take some other action. For instance, if CSat drops on a tracking survey, an appropriate decision may be to launch a follow-up survey, to do usability testing, or to conduct depth interviews.

The important point is to have a decision process in place before research commences. That helps both to frame the research and to know what to do with it rather than simply hoping for “impact.”

Rigor 2. Quality Data and Methods

Once we know what decision(s) to influence, we need high quality data and appropriate research methods. This area is most similar to traditional conceptions of “rigor” although I would emphasize a few differences. For one thing, as I noted above, it doesn’t imply rigid data collection or advanced methods.

Also, to address a decision point rigorously, we typically need multiple kinds of information. The proverbial “quant-qual sandwich" (alternating quantitative and qualitative projects) is one approach. Perhaps the most common question in this realm is “why?” Suppose customers prefer some potential feature on a survey. That answer is insufficient; we also need to know “why?” We may also wish to know, “what else?” and insight into “instead of what?” … for example, whether their preference comes at the expense of brand perception, another product of ours, an improved competitive position, and so forth.

From a methods point of view, this implies two things. First, the methods required may be straightforward. If multiple sets of data agree or disagree, there is no need for fancy analysis. Second, it implies that simply “getting an answer” is not enough. The point of having a specific decision frame (“Rigor 1” above") is not to exclude broader learning; rather, it is to set a target that we can hit directly and, ideally, surround with complementary insight.

Rigor 3. Effective Stakeholder Engagement

The third aspect of rigor is to ensure that you have both the relevant processes and skills to engage with stakeholders. Too often, I meet Quant UX researchers who hope that “the data speak for themselves.” But data sets don’t speak — that is our job.

In an ideal situation, a research team may have multiple members who divide the tasks of stakeholder engagement, research operations, analytics, and so forth. It is unreasonable to expect a single researcher to be highly competent in all of those areas.

Now, you might wonder, “Wait? Why is stakeholder engagement at level 3 instead of being a baseline level 1?” The reason is simple: I’ve met too many UXRs, marketing researchers, and even academic researchers who are great at stakeholder engagement but who do not have the technical skills to deliver quality research. Without quality research, great stakeholder engagement and storytelling too often end up as random walks or exercises in self-congratulation.

What is effective engagement? I would identify a few aspects. It is decision-oriented and the results are both clear and directly informative of the decisions to be made. Second, it pushes back on stakeholders to clarify the questions and say “no” when a proejct or question is infeasible. Third, it is unafraid to deliver negative results even when they are unpopular, such as, “No, users just don’t want that.” Fourth, it does this in a way that stakeholders will hear and value the results, and will want more in the future.

Is all of that a tall order and difficult to accomplish? Yes, and that’s why it is level 3 in my stack of “rigor”.

One other note: too many organizations gaslight researchers into believing that it is a researcher’s fault for not having enough “impact” or poor stakeholder engagement. These relationships require two parties (at least) and failure, poor fit, or less-than-useful results do not arise because of researchers alone. Sometimes the best way to have better stakeholder relations is to change stakeholders (i.e., teams, organizations, jobs, or industries).

As a side note, I’ll call out an implication of areas 1 + 2 + 3 when they are taken together: Rigor is not about the quantity of research, but about how well it addresses crucial problems.

It is better to determine the most important stakeholder questions and address them head-on than to produce an endless streaming of learning about users or answering small-stakes questions. And that leads us to the final area …

Rigor 4. Attention to Higher-Order Strategic Decisions

The final area of “rigor” builds on effective research and stakeholder engagement to deliver broader insight for a business. Unlike areas 1, 2, and 3 above, this is a relatively rare area but one that is rewarding to both organizations and researchers when and if (and only if) they are prepared for it.

This involves attention to issues including:

• Opportunity costs of both product decisions and research engagement. If we do X, what does that mean for all of the things, T, U, V, and W that we did not do? If we spend time researching hot question of the day, what are we are not learning that could be more valuable?

• The asymmetric payoff matrix for all decisions and findings. For any given research result, there is some chance we will be wrong; and for any decision there is an expected benefit as well as potential cost. When we put those together, we can better inform product and business decisions. A full consideration of this area is far beyond the scope here, but I will note two extremely common failure points in this area: (a) a presumption that a product, feature, service, or business will happen, paired with a research demand to “validate” it, failing to consider the alternative that the business may be unneeded or impossible; and (b) a presumption that research should occur just because a question is “important.”

• Models for research allocation and staffing. With good stakeholder engagement, credibility, and skills in broad research for decision making, we can turn those skills to address questions such as, “What should we be doing with research?”, “What are the things we don’t know?”, and “What is the right mix of research staffing?” The skills needed to answer those questions are exactly the same as to answer any other research question. If you’re thinking, “wait! that would be a massive research project” then I’ll point out again that research does not require “statistical significance” or large data sets. Instead, it requires an important decision plus learning to answer it … and even when an answer is imperfect, the expected value of not answering it is much worse than getting an approximate and useful answer.

There is one particular failure mode of “strategic UX research” that I would like to call out: research into product strategy that is untethered to any particular decision or decision process, and that instead is merely hoping to find influence by calling itself strategic. Too many projects in “foundational research” fall into this category. They learn a lot about customers but do not have the decision orientation, stakeholder buy-in, quality data, and attention to asymmetric results that are required actually to inform strategy. Sadly, such projects all-too-often end up with the UX researchers showing up like junior product managers advocating for some particular direction in hopes they will win a battle for influence.

A common question about Level 4 strategic research is, “how do I get there? how can I sell it?” My take is that it can’t and shouldn’t be pressed as a direction by researchers. Instead, it requires (a) building the skills needed to do it, such as depth in game theoretic thinking, (b) great stakeholder engagement based on prior, more focused success, and (c) waiting for the right time and set of questions.

Conclusion

I’ll boil this post down to two sentences:

UX Research rigor not a property of methods or data. Instead, it is about informing effective decisions, in conditions of uncertainty, through appropriate learning from our users and customers.

If I had to emphasize one skill for effective and rigorous research — and really, it is a frame of thought more than a skill as such — it would be to ask “what if?” incessantly. What if … I used 4 small sample methods instead of 1 large project? What if … the answer from customers is “no”? What if … we didn’t do this project? What if ... we shipped this feature or this product and we are wrong?

When that orientation is paired with individual skill in data collection and analysis, you will be well on the way to having truly “rigorous” research!

[Note from Chris: This week I've invited a post from Kelly Moran, a longtime colleague and exceptionally thoughtful and experienced qualitative researcher and anthropologist. Many of us are familiar with the qual/quant "sandwich" that alternates research approaches. Kelly shares deeper thoughts about how to get the sequence right, and I recommend these considerations to all UX researchers.]

In the data-obsessed world we live in, companies are constantly reaching for numbers as the foundation for improving the performance of their products and services. A commitment to gathering and analyzing user data is a critical component for building an understanding of how, and whether, we’re meeting customer needs. Harnessing this information allows teams to move beyond assumptions, identify pain points, and strategically design solutions that enhance usability, satisfaction, and ultimately, business outcomes. We know in UX research that numbers are often only part of the story.

My name is Kelly Moran, and I’ve been working on applying research to business and consumer problems since the early 00’s. I have an MS degree in Applied Anthropology and I’ve worked in both consulting and in-house, most recently on Google Search. I’m currently leading Experience Research at the global experience consulting company, Geniant.

Mixing Methods

Combining quantitative (quant) and qualitative (qual) research methods is a hot topic right now, and for good reason. A common question is which one should come first. I often hear the perspective that starting with quant will bring up questions that the team can then dig into with qual research for a deeper understanding of the “why.” This is a great approach. By identifying what is happening through quantitative data (the numbers and counts), teams can then use qualitative methods (the rich, descriptive, word-based data) to explore the motivations, context, and underlying reasons behind those metrics. It allows for a targeted investigation into the phenomena the numbers highlight. But sometimes flipping that script can ensure you're measuring the right things in the first place, or perhaps more accurately, that the quantitative measurements you are taking truly reflect the real-world behaviors and experiences you intend to capture.

This alternative approach (starting with qualitative research) can be incredibly powerful for checking your underlying assumptions and refining your data collection instruments before a large-scale quantitative study is even launched. By first engaging with a smaller group through observation, interviews, or other qualitative methods, you gain a deep, nuanced understanding of the landscape. This initial insight can reveal blind spots in your current metrics or suggest entirely new categories of data that should be tracked. In essence, starting with qual lets you refine the questions you ask in your quant research, ensuring the resulting data is both accurate and meaningful.

I have an illustrative example from some consulting work that demonstrates the value of a qualitative-first approach. I start by laying out the problem as business saw it, the response research took, what we learned, and how we recommended a larger change to the way the business approached understanding and building for their own team.

Responding to Poor Metrics

A company that services loans wanted to improve the metrics coming in from their call center. They had a team of agents for borrowers to reach out to by phone with questions or for general service on their loans, and at the end of each call, agents were required to enter a reason for the call into their call management system. They had a list of about half a dozen options to choose from. The reason “make a payment” was selected over 80% of the time. And this made sense. Paying a loan is important, and hopefully every borrower is making frequent and timely payments. But the company had put a lot of work into making it easy for people to pay their loans online. Loan payment should be a straightforward process, yet according to the metrics, it was taking a lot of time for call center agents to get through all these calls processing payments for customers over the phone. Relieving this load from the agents would decrease the wait time for borrowers calling in about other matters.

So they kept working on improving the online payment process.

And they kept seeing “make a payment” as the dominant reason borrowers called in.

They began to wonder if they needed to work this from another angle. Shifting your mindset is a great way to approach a sticky problem. They decided making the software easier on the agent side might at least help agents get through those payments faster.

They engaged with my team to observe agents in the call center so we could make context-informed design recommendations for the agent-facing software.

What we learned was surprising

We ended up listening in on 98 calls over a few days of observation with several agents. We did see opportunities to improve the design of the agent software, but we also learned something else.

In our observations, the frequency of calling in to make a payment was much lower than 80%. In fact it was under 50%. Instead, many borrowers were calling to check on a payment. These customers had used the website to make a payment but the system had a delay in showing funds applied (a banking issue we recommended they address), so borrowers were calling to be sure their payment had gone through or at least to ask that it not be counted as late, as the delay was out of their control.

Agents would assure the customer that all was well, and at the end of the call, they’d check the “Make a Payment” box because “Reassure the Borrower” was not an option. Nothing else on the reasons list was close enough.

The data was wrong because the selection options did not accurately reflect the real world.

The fix was multi-faceted. On the one hand, they needed to either address the delay in applying payments or at least provide clearer messaging that a payment was being applied and that no late fees would result from the processing time. A sophisticated solution would include a call routing system that identified a caller by phone number, registered that their account had recently had a payment submitted online, and provided a recorded message that the payment was in process. This alone could have brought their “make a payment” numbers down, as perhaps some callers would hang up before reaching an agent.

The agent-side fix would not only improve their “make a payment” metric but also provide a more accurate picture of why customers call in by improving the data being collected. They needed to reassess the list of reasons for calls, which our qualitative data provided a great starting point for. On the other hand if they had begun their original effort to collect data on "reasons for call” with a qualitative review of those calls they could have had a more accurate picture from day 1 of metrics collection.

There is real power in rotating between qual and quant

The benefits of pairing quantitative and qualitative research methodologies is clear, but the sequencing can feel challenging. While the quant-first approach, using numbers to raise questions for qual to answer, is valuable for targeted follow-up, the example from the loan servicing company demonstrates a practicality in leading with qualitative inquiry. Had the company implemented a qualitative discovery project before they implemented their call categorization metrics they could have saved a lot of time trying to fix an online payment system that was already working (albeit missing critical follow-up messaging). And without my team happening to be in the right place observing for a different design purpose, the company would have continued to invest in the wrong solutions.

Qualitative research can serve as a vital diagnostic tool, ensuring your quantitative instruments are calibrated to the real world. By doing qual first, you don't just dig deeper into existing data; you ensure you are measuring the right things in the first place, leading to truly effective, context-informed solutions.

So which should you do first?

As always with research, it depends. Is your team confident they have a clear enough understanding to create meaningful categories for your quantitative data? Is there any reason why they could not pause for a qualitative review of the landscape? Sometimes secondary datasets can provide useful context for this work. And criticality, what risks might there be in collecting inaccurate or incomplete data? Talk through these questions with your team and decide how to create the best combined quant/qual plan for your workstream. If confidence is high, timing is tight (and second guess that decision, always), and the risk is low, suggest circling back once metrics start rolling in to see how a qualitative project could add clarity. Otherwise, get out there and start defining your customers’ reality with descriptive data.

The cover image is adapted from a photo by Jem Sahagun found on Unsplash.

In this post, I want to share what I’m hearing. This is reporting and I’m not arguing for or against any particular claim. Each claim could be a contentious discussion, and I’m not inclined nor do I feel sufficiently broadly informed to engage in the arguments in detail.

Put differently, my goal is simply awareness — and closely related, I hope that UXRs who feel isolated, gaslit, anxious, or confused will understand that they are not alone.

I will preface this with a caveat: these themes are what I am hearing, but I am not a random or representative sample. My connections and I self-select to chat, exactly because we often agree. Nevertheless, the views I’m hearing from senior UXRs are important in the UX industry.

(1) The Job Market is Chaotic & Unpredictable for UXRs

I expect that no one will be surprised by this section.

For various reasons — ranging from layoffs to dissatisfaction — many of the UXRs I talk with are looking for new positions, and are quite frustrated by the experience. I’m hearing four primary trends:

• A feeling that “there are no jobs”

• Over leveling, with everyone seeking higher positions

• More emphasis on specific skills, especially “quant” and “mixed methods” (maybe without understanding them)

• Extreme asymmetry in openings between tech hubs — especially Silicon Valley, and to some extent Seattle and New York — versus most other locations and remote

As for over leveling, there is a trend towards acceleration of career levels, with folks expecting promotion every few years at least. This leads to two problems: (1) folks who are in over their heads when their breadth of experience (in methods, approaches, products, and politics) doesn’t match expectations of senior stakeholders, and (2) fewer job openings as they move up, paired with higher demands, leading to lower satisfaction. We say much more about that in Chapter 14 of the Quant UX book.

I am also hearing a lot of demand for quant skills, with positions often listing something “quant” added onto general UX research skills. However, it’s difficult to sort out the extent to which hiring managers actually understand what they are seeking in these cases. Some managers may expect little more than general attention to UX metrics (see Kerry Rodden’s excellent post) while others may imagine — quite unrealistically — that they can find a UXR who spans everything from qualitative fieldwork to data science logs analysis. Mostly, my advice here is for hiring managers to be very specific (see the Quant UX book) and for candidates to ask lots of questions and not oversell their quant skills. To be fair, what folks tell me about this may be biased due to my own experience.

There are some signs of recovery in the market but it is appearing tepid. My main recommendation for folks outside the Bay Area is to consider the idea of relocating there or at least to Seattle or another major tech hub. Yes, I know relocation is painful, expensive, and perhaps even unfair. I’m only saying that because it is the reality I am seeing.

TBH, I took the relocation advice once myself. In 2011, I was looking for new positions and the market in Seattle was bad at that time. Google offered me positions in Mountain View and New York, and I ended up moving — for 3 years, until transferring back to Seattle. In retrospect, it was a great career choice.

(2) Yet Hiring Managers Don’t See the Candidates They Expect

In social media (e.g., LinkedIn) there is a pervasive sense that UXR jobs are few, difficult to land, and swamped with applications. Given that, I was surprised by several hiring managers (HMs) who perceive a lack of UXR candidates. Not a single HM said they have too many candidates.

The three main things I heard about hiring are:

1. There are some jobs although headcount is tight

2. HMs don’t see enough candidates

3. Candidates don’t have the right skills

The first point is pretty obvious. Hiring managers often report that they are able to hire again, but they have substantially reduced headcount, compared to highpoint years such as 2022.

The second claim, from several folks, greatly surprised me: that they do not have enough candidates. For instance, I heard of one senior position at a MAMANG company that received fewer than 10 candidates, and among those applicants, the HM judged that none was qualified. To be clear, this is not due to an overall lack of applicants; I’ve heard this from managers at two of the top five largest tech companies, where the application pools are overflowing with candidates. And yet few filter through to HMs.

I don’t have a good explanation for this, although one important factor is how recruiting has changed since COVID and after the big rounds of layoffs in 2023 & 2024. Many long-tenured and knowledgeable recruiters lost their jobs or changed companies.

By the time hiring started to resume, companies had lost the recruiting skills and knowledge needed to understand, screen, assess, and attract specialized candidates such as Quant UXRs. It is not a position that can be reduced to a list of qualifications such as knowledge of methods. Instead, it is crucial for recruiters to understand how quant skills relate to stakeholder needs. When they understand that, they can improve screening by lightly probing candidates in initial interviews to see whether they are able to adapt and explain concepts and impact clearly. It takes time, experience, and support for recruiters to develop those skills. Additionally, as recruiter and hiring managers turn over, they lose the connections and understanding that previously led to alignment on potential candidates.

Meanwhile, the deluge of job-seekers — multiplied by AI and resume-padding — has overwhelmed recruiters. The difficulty for recruiters is compounded by rapidly changing requirements from upper management, fluctuating numbers of openings, and consequent pressure from hiring managers to fill positions quickly. An open position one week may be repurposed the next week, abandoning candidates who are in process … and discouraging recruiters from investing time in the next round of candidates.

The net result: the candidates who make it to hiring managers are far more random, and less well-screened than they were in pre-COVID times.

Recommendation for current, senior Quant UXRs: have you done 10+ interviews with candidates? Reach out to your recruiter(s) and ask whether they would sync with you about the experience. Perhaps you could sit down for an hour to review candidates in their pipeline. You might discuss the things you would look for on CVs, or simple phone screen questions with good and not-so-good example answers. This may boost the quality of candidates, and make a recruiting friend!

(3) Many Senior UXRs Are Disillusioned by Colleagues

I love the UXR community — so this section is painful for me. But I am sharing it as I heard it.

This is a short section and makes me sad. I have increasingly heard from senior UXRs (L6 aka “Staff Researchers” and up) that they view colleagues and junior researchers as lacking skills. They claim things like:

• Research is rudimentary and responds only to direct stakeholder questions, missing the larger picture

• Research obtains obvious results and then is oversold as being very important

• Research design has significant flaws such as skewed samples, leading questions, bad survey panel data, dubious methods, and the like

• Specifically on the quant side, research may be primarily “procedural” in applying advanced methods or a great deal of code and analysis, but is applied to low quality or largely irrelevant data

• Colleagues use questionable tools — LLMs are often called out — to write reports, deliver unreliable findings, and the like

This is often summed up as, “they have no idea what they’re doing” or “they just crank out meaningless reports.”

Now, I’m not making those claims, just reporting them. My personal impression is that we’re seeing a combination of several things:

• Everyone is stressed and stretched. Junior UXRs do not have the mentoring or time to develop skills on the long time horizons that used to be common. Senior UXRs take out their frustrations by complaining about research quality.

• Demands are increasing, as described above about management expectations

• Research is becoming more difficult to do, with less support. This comes from a combination of difficulty recruiting participants, low quality survey panels, and shrinking budgets for research.

• The breadth of UX methods has grown such that no one can be expert in everything; and yet stakeholders may well demand that any UXR tackle any kind of research.

• As mentioned above, over-leveling causes both real problems and also interpersonal resentment.

My main recommendation here is for everyone to try to look for the best in others.

Managers: help your team do less in terms of breadth, and instead with more partnership in depth. Instead of having 4 UXRs run 8 different projects … try having 2 teams of 2 UXRs focus on 4 more important projects. You will have less “coverage” but that will be strongly offset by having much greater depth on the most important projects. 2 UXRs working together have a multiplicative effect. And it will build cohesion, develop skills, and insulate your team from shocks of turnover and the like.

(4) ... and UXRs are too often disillusioned, especially LLM hype from upper management

LLMs were an important topic in every single conversation with senior UXRs. Typically AI / LLMs emerged as the #1 issue of discussion as a conversation progressed.

The most common concern is this: UXRs feel that LLMs are being pushed into products without any clear understanding of user needs for them.

Specifically, UXRs often see, feel, or believe one or more of the following:

• Teams are told to “find a need” under a predetermined assumption that LLMs must be valuable

• If UXRs find that customers do not want an LLM feature, management won’t accept the finding

• If UXRs push back, they believe their own jobs are at risk in an awaited next round of layoffs

• They worry that management doesn’t understand LLMs to begin with, so it is all an exercise in futility

• They fear (or observe) this LLM push will erode product value, user satisfaction, and brand trust

These UXRs view the continual search for “ways to use LLMs” as a fool’s errand when there are urgent and obvious unmet needs and pain points in products. They may be seeing decreasing user satisfaction, pushback from users about AI, security threats, worsening UI experience, declining overall code quality, and immediately addressable needs that users are requesting — all of which are ignored in the promise of some AI utopia to come. Doing research on yet-another-AI-concept can be demoralizing in the face of actual user needs.

To be clear, I’m not saying that there are no uses for LLMs. I’m saying that AI does not change everything overnight, and that management expectations for AI are often out of alignment with users and users’ needs.

This mismatch between management expectation and the reality of interacting with users can lead to UXRs responding with helplessness, apathy, cynicism, dishonesty such as telling management what it wants to hear, and similar and other ultimately self-destructive responses.

Recommendation: if your research projects appear unrealistic, short-sighted, unlikely to ship or have impact, or even border on being delusional, find a way to add something that delivers value in another way. That might be foundational research you can do “for free” (and perhaps not even report), it might be learning about method to build your skills, or it might be following up on prior research. Research plans almost always offer the flexibility to learn more than simply meeting immediate goals. Best case, the team learns something. But the worst case is also good: you learn something!

But UXRs Love Being UXRs: So, Cautious Optimism

The final thing I heard across my conversations was this: UXRs love being UXRs. Much of the anxiety concerned whether they would be able to continue as UXRs (or perhaps have to look for jobs in data science or PM); and whether the role of UXR could continue to look the way it has in the past, with the excitement and enjoyment of learning about users from direct interaction as well as data.

I’ve often said that UX Researcher (and its variants such as Quant UXR) are the best job in the world … for the right person. For people who love learning about many different products, tacking ambiguous research goals, doing research well, meeting customers and helping to solve their needs, and working in generally enjoyable and good positions, it is difficult to imagine a better career. Ultimately, learning from users and delivering better products must deliver high value on average.

So I expect the UXR situation to improve both in terms of jobs and research demands. When? That’s harder to say … but each of us can help that day come sooner. I hope these reflections and recommendations might help.

Final personal note: to increase well being and generally build community, I’ve organized a Zen meditation group for the Tech — especially UXR — community. This is not a sales pitch (and we don’t sell anything!) but just my personal note. There are many options to well being apart from Zen, but if you’re interested, details are at https://tczen.org Join a session or sign up for the newsletter.

More generally, treat your colleagues, the world, and yourself kindly. We’re all in this together!

For much more discussion, to meet other UXRs, and to hear different perspectives, I also encourage you to join us for Quant UX Con 2025 in November. Find details here!

I sort these articles into categories of:

• Tech Community Reflections

• Quant Methods

• Method Critiques

• Career Advice

• Distinguished Guests

I’ve added a brief note about each topic as a preview or comment. I hope you find something new and provocative in these articles!

Tech Community Reflections

The End of Tech as a Big Family: https://quantuxblog.com/the-end-of-tech-as-a-big-family. This post reflects on the personal anxieties and changes in Big Tech that were exposed during recent layoffs.

We’re Far from AGI: https://quantuxblog.com/were-far-from-agi. As a psychologist and sometimes philosopher, I explain why proponents of AGI and “superintelligence” don’t understand what they’re claiming (or more precisely, they don’t understand what they are not discussing).

Quant Methods

Multidimensional Sentiment Analysis: Part 1, Part 2. If I were to sort methods into quadrants on a 2×2 grid, arranging them by powerfulness vs. frequency of usage, then multidimensional sentiment analysis would score as a “high opportunity” — very powerful but surprisingly rarely used. I’d like to change that!

Easy MaxDiff in R: https://quantuxblog.com/easy-maxdiff-in-r . If you’d like to understand MaxDiff from the ground up, this short code explainer may help (note: the code is for educational purposes, not production). It looks at the observations that MaxDiff collects, and basic statistics for that data.

Individual Scores in Choice Models, Part 3: Respondent Segments: https://quantuxblog.com/individual-scores-in-choice-models-part-3-respondent-segments. I examine one way to segment MaxDiff data. But even more, this article is about how segmentation is usually misunderstood and what it actually does.

Individual Scores in Choice Models, Part 4: Inspecting Model Fit with RLH: https://quantuxblog.com/individual-scores-in-choice-models-part-4-inspecting-model-fit-with-rlh. I look at the RLH fit statistic in conjoint analysis and MaxDiff. Using basic math, theory, and code, I show why common heuristics to interpret RLH are misleading. RLH is diagnostic but cannot “filter” respondents.

Method Critiques

Surveys and the “True Score” Mistake: https://quantuxblog.com/surveys-and-the-true-score-mistake. This article explains why surveys are not about finding “the truth”, aka a latent score, in people's heads. Instead surveys are about listening to people. (This article is also one of the ground for my criticism of synthetic data in the next article.)

Synthetic Survey Data? It’s Not Data: https://quantuxblog.com/synthetic-survey-data-its-not-data This article explains why I believe the concept of “synthetic data” is impossibly flawed in multiple ways, including basic logic, statistics, and in regards to the scientific method.

Critical Assessment of the Kano Model: Part 1, Part 2. If you are tempted to use the Kano Model to identify “delighter” products … I don’t recommend it. This pair of articles explains why not, and gives alternatives.

Career Advice

Skills Combination for Quant UX Applications: https://quantuxblog.com/skills-combination-for-quant-ux-applications. I’ve interviewed hundreds of candidates and, for several years, coordinated the standard Quant UX hiring criteria at Google. This article summarizes the skills needed to be a Quant UXR, and is also a good preview of the Quant UX book!

Recommendations for Quant UX Interview "Portfolio" Presentations: https://quantuxblog.com/quant-ux-interview-portfolio-presentations-recommendation. If you are asked to give a Quant UX research presentation — sometimes confusingly called a “portfolio” presentation — this is how to approach it. The short version: give a research presentation that interests you; don’t try to guess some “right” content.

Distinguished Guests!

How to make HEART metrics work in practice: https://quantuxblog.com/how-to-make-heart-metrics-work-in-practice One of the founders of the Quant UX discipline, Kerry Rodden examines the evergreen and useful HEART framework, detailing practical advice to apply it. HEART is my all-time single favorite Quant UX framework, and a memorable heuristic to draw on in many, many research contexts.

Be a T-shaped Quant UXR: How Doing Qualitative Research Made Me a Better Quantitative UX Researcher: https://quantuxblog.com/be-a-t-shaped-quant-uxr-how-doing-qualitative-research-made-me-a-better-quantitative-ux-researcher Kitty Xu, cofounder of the Quant UX Conference, explains why Quant UXRs should also engage personally with qualitative research.

More Reading?

Each article above has pointers to additional articles or references. Or check out:

• Hundreds of talks in archives of the Quant UX Conference … or join us in November 2025!

• With my coauthors: the Quant UX book, the R book, and/or the Python book

• And I hope you’ll subscribe to this blog, if you haven’t already

Happy reading!

I was fortunate to study writing with the late SF Grand Master, James Gunn. He often made two points: that science fiction might save the world (by letting us explore hypotheticals to avoid and plan potential responses), and that all fiction should be entertaining (otherwise what is the point?)

To Gunn's points, each of these books is entertaining — and maybe each is helpful to save the world. All of these authors offer more to read if you like them. Several of the titles anchor series with 3+ books.

I’ll also acknowledge the counter-narrative to Gunn’s positive claims for SF: it also contributes to expectations and fantasies — such as those about AI — that are destructive when we absorb them uncritically. (IMO this is usually due to lack of understanding, when readers don’t understand that SF is not about predicting the future; it’s about exploring themes that are happening now.)

Jemisin's and Scalzi's books are reportedly filming as TV series (Corey's and Chiang's already have been filmed). Read the books first!

Ted Chiang. Stories of Your Life and Others. One of the best short story collections I've read. The title story is an astounding look at the limits of human experience. Read it first and then watch the movie adaptation, Arrival. Amazon, Powells.

James S.A. Corey. Leviathan Wakes. First in a series filmed as The Expanse. A great look at a future that feels much more probable than than those often portrayed much of SF, with battling factions, politics, economies, billionaires, and solar system conflict. Read and then watch The Expanse. Amazon, Powells.

N.K. Jemisin. The Fifth Season (The Broken Earth Trilogy). It takes a while — perhaps 100 pages — to "learn how to read" this book. Stick with it! Is it fantasy or science fiction? Either way the entire series deserves its 3-in-a-row Hugo awards. Amazon, Powells.

Stephen Markley. The Deluge. This title is not genre SciFi; it is literary fiction set in the near future, but it is close enough for me to include it. A sprawling book about climate change, societal breakdown, power, and political action in the near term US, with multiple story lines that converge dramatically. Amazon, Powells.

John Scalzi. Old Man's War. This has one of the best and most memorable opening pages of any book. It is military style SF, but even if that's not your thing, you'll likely enjoy it. It is a fun read that maxes out on the factors of fast plot and entertainment in SF. Amazon, Powells.

Adrian Tchaikovsky. Children of Time. Besides being great space opera, this series has outstanding portrayals of non-human cognition. For those interested in AI and AGI, I recommend reading the series and considering what those themes tell us about AI. Amazon, Powells.

Robert Charles Wilson. Spin. I love Wilson's books for their portrayal of near-future society. Even more, his writing is especially humane in its focus on people. Spin is a great tale that starts one night when the stars all go dark. Amazon, Powells.

Read them already? How about ... ?

If you like the books above, try some of these authors, according to similarities I see:

Octavia Butler ... may appeal if you like Wilson or Jemisin. At least read Parable of the Sower.
David Louis Edelman ...may appeal if you like Tchaikovsky
Greg Egan... may appeal if you like Chiang (nb, Egan is sometimes very technical)
Ann Leckie ...may appeal if you like Corey
Alaistair Reynolds... may appeal if you like Corey or Tchaikovsky
Kim Stanley Robinson... may appeal if you like Wilson. Great mix of hard science and human themes.
Neal Stephenson ... no near neighbor. Uneven IMO, suggest starting with Snow Crash or Cryptonomicon
Peter Watts ... may appeal if you like Tchaikovsky (although Watts’s worlds are substantially darker)
Martha Wells ... may appeal if you like Scalzi. The first Murderbot book has filmed & aired on Apple TV.
Connie Willis ... may appeal if you like Wilson

My Science Fiction Short List, to Read Soon

R.F. Kuang: Babel. Have heard superlatives; saving it for the right time to read.
Arkady Martine: A Memory Called Empire. Said to be a space opera plus and diplomacy SciFi.

Happy reading!

P.S. I've also recommended literary fiction. Sometime I'll recommend mysteries, too!

Finally, as always, this post was …

First, if you’re wondering whether you can do it … you can! Quant UX Con is intended to be useful, and does not require that presentations are academic “contributions”, or theoretical, or advanced methods, or anything like that. As long as a presentation is something of interest to UX Researchers who use quant methods, it fits.

Second, the Quant UX Con audience is friendly and welcoming, and new presenters are especially welcomed! And uou can present from anywhere worldwide, around the clock in any timezone. Or join live at a watch party location.

Third, if you’re thinking, “my company doesn’t care about publishing, it won’t help me” … I can say this: presenting will help you! Maybe it won’t help in your current job, but it will help your confidence, learning, meeting people, and will quite possibly your next job. (My own publishing was quite important when I made two job changes in my career, to Google and to Amazon!)

“I don’t have anything to present!”

Ask yourself this question: what is something I learned in the past two years, that I wish I had known before? If you can answer that, then you probably have a presentation that will interest others!

Your talk might walk through a method, or give a literature review, or a case study, or a career reflection, or put together a panel discussion, or a code walkthrough … all of those are welcomed! You can find past talks here: https://www.quantuxcon.org/quant-ux-con/past-conferences

Want more specific ideas? Here are some ideas I would personally like to see, off the top of my head:

• A walkthrough of a favorite but little-known R (or Python) package

• How to set up a local SQL server and IDE, so you can learn SQL on your own

• Scanning old PDFs to get data from them

• How we tackled [difficult problem X] and the lessons we learned

• Getting started with GitHub: how and why to set up your own code repo

• Integrated reporting with Quarto: creating reports with reproducible code

• Why Bayes?

• Connecting Excel to R — using the power of R directly from Excel

• Kill the chart clutter — how to make cleaner, better, more useful charts

• Round table on hiring: 4 hiring managers [or 4 candidates] discuss the UXR job market today

• Keeping up with the bots, addressing survey panel issues today

• An alternative to Powerpoint: making slides in Beamer / R markdown / Quarto …

• Put it in a notebook! Boost your collaboration with Jupyter (or Colab, or Quarto, etc.)

... and of course, traditional research methods discussions, innovations, think pieces, or specific research findings.

Do It Now … “But wait, I don’t have time!”

Yes, the CFP is due on Tuesday July 1 … but that is doable! Proposals require only an abstract with these pieces:

• Summary: 2-5 paragraph abstract that describes the presentation

• Audience: who will be interested?

• Goal / takeaways: what will the audience get from the talk?

Once you have a concept, I’d estimate the CFP itself might take 30-45 minutes. Perhaps 60 or 120 if you really want to polish it. (FWIW, I timed myself and wrote up my own very complete proposal, directly in the CFP form, in 20 minutes.)

And you’ll hear it here first: we might extend the CFP by a few days. That has not been decided, so don’t count on it — and go ahead and start. But if it takes you slightly longer, you may be OK! Or go ahead and submit now, and then update it after you polish it.

Or worst case, write it up now and save it for next year :) But better, send it in!

How?

Find the CFP form at: https://app.oxfordabstracts.com/stages/78930/submitter

Thanks for considering it — we really do hope you’ll share ideas, and of course, please join us in November!

There is widespread discussion of LLM-generated synthetic survey data and its utility. Some survey panel providers promote synthetic data as being fast and cheap and, they claim, as a way to “boost the prevalence” of infrequent respondent groups.

In this post I share my point of view on synthetic survey data — not because I want to wade into any AI wars, but because the topic arises repeatedly and colleagues have asked me to write a post.

Among other things, I argue in this article that:

• the concept of synthetic data is logically flawed

• synthetic data fails empirically to emulate human data

• synthetic data cannot overcome sampling limitations

I include some references but this is not a comprehensive review. It only compiles my thoughts, in varying degrees of completeness, plus a few pointers to other folks’ work. You may agree with some arguments more that others — and that’s fine, I only hope you agree with something :)

Side note: this post discusses the utility of synthetic data, but that is only one consideration. I’ve described elsewhere why it is important to consider ethics, externalities, aesthetics, and social structures alongside the utility of AI. However, proponents of synthetic data discuss it in terms of utility, so that’s where I focus discussion in this article.

The Concept of Synthetic Data is Logically Flawed

I see three fundamental logical problems with LLM synthetic survey data:

• It rests on the common, yet I believe incorrect, assumption that surveys are about sampling a “true state of affairs” in the world. That view is imported from classical psychometrics where it is a simplifying assumption. But it is not what surveys do in the real world. As I have explained elsewhere, surveys are a form of motivated communication and they must be interpreted as such. They cannot be viewed as measurements of any particular “reality” that is accessible or meaningful apart from considering motivation.

Thus, unless we are interested in the motivations of LLM systems, their data logically cannot replace human data. It doesn’t even matter whether their answers might in some way be “the same” … because the point of surveys is not to measure “the answer” in the first place. The point is to listen to people, and that requires … well, listening to people.

• The second logical problem is temporal: LLMs are trained on past data, whereas the goal of a survey should be to listen to people now. Even if an LLM has historical data aligning with our question, it is outdated as soon as it has been trained. Thus, synthetic data has no determinable relationship to what we want to know now. It might or might not be relevant; we don’t know.

You might object, “what if I want data about something that other researchers have asked in the past?” That’s fine … but if those data already exist, you don’t need to go through the convoluted process of writing a survey that you hope will align with those data, and then subject that survey to an LLM that you hope will use the data to give responses to your survey, which you then hope will recreate the data. Instead, simply Google your question and access existing data directly.

• The third problem concerns the domain of inquiry: the space of potential business questions is infinite, but existing data is finite. Thus we may expect that most questions (the infinite space) have not been answered … especially when we are working in a new product area. Although LLMs create novel data on demand, there is no logical reason to expect that their statistical models will infer any particular novel truth — and, indeed, LLMs are not even designed to represent truth within their training sets. [Mathematical note: there are infinities of differing sizes. Even if an LLM can infer within one infinity of data, it doesn’t necessarily infer within all infinities of data. I’ll set that aside.]

To summarize: there is no need to do research when the answer to your question already exists. But you can’t find out whether an answer to your question exists by asking an LLM to create an answer. You need to find an actual data set or else collect new data.

Synthetic Data Fails Empirically (yet that’s not the right question)

There have been various empirical studies assessing whether LLM synthetic data aligns with real data. I believe that is somewhat of a wrong question, which I’ll explain later. Meanwhile, empirical results contradict the claim that synthetic data is similar to human data. Here are a few examples.

• Bisbee et al (2024) demonstrated that ChatGPT survey results are unstable and are not representative of human survey answers. They found that, “sampling by ChatGPT is not reliable for statistical inference … [Also] the distribution of synthetic responses varies with minor changes in prompt wording, and … the same prompt yields significantly different results over a 3-month period.”

• Paxton & Yang (2024) found that LLMs and humans report strongly differing “attitudes” about technology products. They found that “language model responses diverge from human responses—often dramatically … [These] divergent results are robust to multiple prompt variations, model families (Gemini, GPT, etc.), and major updates to the models … [Therefore] language model responses should not be used to replace or augment human survey responses at this point in time.”

The following table is one snapshot from Paxton & Yang, assessing the correlations in attitudinal ratings obtained from ChatGPT, Google Gemini, and human raters. Among 8 attitudes, it was only for ratings of “helpfulness” that the correlation exceeds r\=0.20; and even then, it was only r\=0.28-0.30, a weak correlation. The median correlation between LLMs and human attitudes was r\=0.10. This means that LLMs reproduced a median of only 1% (r²) of the pattern of attitudinal ratings of human respondents.

• Samoylov (2024) noted multiple problems with LLM-created data, especially that LLM results vary dramatically, across many dimensions, in response to relatively simple rephrasing of prompts. Similar to my point above, he further noted that it is impossible to know when one’s domain of interest is covered by LLM training data. He wrote, “how [a prompt] was worded massively affected the results. This is a demonstration of the test-retest unreliability of using LLM-generated responses … because you do not know what most of the LLMs were trained on, you do not know what kind of knowledge they encode … this one observation is enough to make anyone interested in getting real data look the other way”.

An especially nice feature of Samoylov’s article is the inclusion of R code to demonstrate the unreliability of LLM synthetic data. You can update his prompts and examine for yourself whether, in your domain, the answers from an LLM are reliable. The code is at the end of his article.

In short, these empirical studies demonstrate that:

• Results vary from LLM model to model, time to time, and prompt to prompt.
  Implication: synthetic data are not reliable in the way we expect data to be reliable.

• Results do not agree with human responses or the patterns of human responses.
  Implication: synthetic data do not have construct validity to mimic human responses in the way they claim. (Read more about construct validity here.)

In the post so far, we’ve seen: (1) that the concept of synthetic data misunderstands how surveys work as motivated communication so the data don’t make sense logically; and, (2) even if we set that problem aside, the resulting data empirically do not align with human survey data. Next, I’ll examine why empirical claims about synthetic data are unscientific in any case.

… Yet empirical evaluation is not the right question

[tl/dr; This is a long sub-section, but I hope it will be worth your time to consider the logic here!]

Although the empirical work above is admirable, it reflects a whack-a-mole strategy. Empirical investigations have no particular theoretical justification; instead, they respond to claims by AI proponents. Such proponents claim synthetic data “works” and demand proof otherwise.

Let’s assume the opposite of the previous empirical results for a moment. Let’s suppose — for the sake of argument — that synthetic responses to survey items are expected to be universally reliable and valid. What would that mean? It would imply that an LLM could, in principle, be expected to answer any question, on any survey, similarly to how a human would answer.

To make that more explicit, let’s look again at the general claims for synthetic data. Suppose we want to know about the overall population likelihood that people will purchase our new product. A traditional survey of purchasing intent will:

• Target a group of people and ask about each person’s intention to purchase our product

• Expect that the intention will be somewhat indicative of future behavior

• Sample a population so we can estimate the aggregate behavior of the targeted group

The claim for synthetic data is the same: that we can use synthetic responses to “sample” a group and estimate its aggregate intentions or behavior. For example, we might estimate whether synthetic “purchasers” are likely to purchase our product. And then we project that estimate to humans.

When we think about that for a moment, we might notice two things. First, it seems highly implausible. If a data provider has an LLM with such capability — to credibly answer any survey question similarly to humans would answer, about future behavior — why would they use it to sell survey panel responses?

Imagine, as one example, that we can accurately assess the intentions of executives and stock traders. For instance, we could envision survey items about their intended behaviors, such as these:

• Tomorrow, will you sell XYZ stock?

• Today, will you increase your hedge of ABC currency?

• In the next hour, will you short UVW stock?

Just to be clear, these questions do not ask for predictions about future stock prices or the like. They ask only about a person’s intended behavior. In exactly the same way we might survey consumers about intentions to purchase a product, we could ask traders about intentions to purchase stocks, commodities, or currency. We can use the aggregate of those responses to infer likely population behavior.

If we could accurately assess such behavioral intentions, we could use that aggregate information quite profitably to predict stock, currency, or commodity futures — and that would be a much better business than selling survey responses. It is easy to imagine similar examples of valuable data in the domains of politics, medicine, pharmaceuticals, military affairs, logistics, shipping, and the like. If we could use synthetic data to estimate intended behaviors of groups of people in those domains, it would have value that far exceeds that of selling survey panel data.

“But wait!” : Side note about such a hypothetical survey

Let’s set aside LLMs for a moment. Do you object that executives and traders would not answer such items on a survey? That they would not report their intended trades? Or that they would not answer honestly? Do you worry that any answers they give wouldn’t predict real behavior?

Good! If you think that, your views may align with interpreting surveys as being about motivated communication, and not as about sampling any particular “reality” apart from that. And that implies that the core premise of synthetic data is illogical, as I noted at the outset of this post (you might return to the top of this post and re-read my discussion; or see this separate post for more.)

For this first point, my inference is this: synthetic data providers do not believe that their systems can do anything like this or that their data have such value. Instead, they sell such data because it has low value. Besides their own behavior, they reveal their belief in the value of such through their pricing, and in their statements about how cheap it is.

Now, maybe you’re thinking, “I understand the argument, but perhaps providers do use the LLMs profitably in the way you describe, but they do so secretly while they also sell survey responses as a second business.” That is logically possible, yet: (1) why would they give away a competitive advantage of that kind and let others match what they are doing? (2) where are the case studies to prove such capability ? (3) why is their pricing so low? Overall, this possibility seems extremely unlikely to be the case, for all the reasons outlined here.

Second, the claim that any survey question might credibly be answered by an LLM is not a claim that can be tested empirically. How would one go about testing that? One would need to define the space of all possible survey questions, determine a sampling strategy for that space, and determine a way to find the “ground truth” of human responses across that entire space to compare to LLM responses. And would need to show that the infinity of possible spaces maps to a majority of its constituent subspaces.

I have no idea how one could do that in a general way. Therefore, the question of empirically assessing LLMs is necessarily limited to specific domains (as in each of the examples above) and cannot be evaluated generally.

Consider, third, that when we evaluating the reliability of synthetic data in any particular domain, an LLM proponent could always claim “that is just one example.” This shows that an empirical assessment strategy cannot answer the general claim about answering survey items, because that general claim is not an empirical claim. Instead, it is a belief disguised as a claim.

In summary, the “hypothesis” that is supposedly being tested in empirical evaluation of synthetic data — namely, that it can replace human data — is a marketing claim and not a scientific hypothesis.

Next we’ll look at another marketing claim: that synthetic data can replace hard-to-reach subgroups of respondents.

Synthetic Data Cannot Overcome Sampling Limitations

LLM-generated data providers postulate that synthetic data can emulate “hard to reach audiences” (example). The term “hard to reach” is rarely defined by providers. Instead, they use vague references to imprecise notions such as “increasing the diversity” of respondents.

From discussions I’ve had, this is often interpreted in two ways. First, it may mean niche audiences who have low prevalence in panels (corporate executives, physicians, developers, users of one specific product, etc). Second, it may mean historically marginalized respondents such as ethnic minority groups, people with disabilities, different language groups, and others.

In either case, the premise is that these are people who do not answer our surveys as often as we researchers would like them to. (Remember my argument that surveys reflect motivated communication? This premise also aligns with that view.) Platforms claim that they can “boost” the responses for such people using synthetic data that either supplements or completely replaces real data.

Unfortunately, there is no reason to believe that LLMs can accomplish the goal of representing difficult-to-reach audiences, and there are several reasons to expect otherwise:

• We saw above that LLM data does not agree with general population responses. Why would an LLM give better data for subgroups than it does for larger populations? There is no logical or statistical reason to expect that.

• We know that LLM training data over-represents some groups (English speaking, white, educated, western, technology-interested, affluent) and under-represents other groups (non-English speaking, other than white, non-western, etc.). There are algorithmic approaches to reduce bias, but — even if a provider has implemented some of them — how would we know that bias has been eliminated for some potentially unique group of interest that we want to sample? There is no a priori bias reduction that works in advance, for targeted samples that will only be specified later. (This is similar to the empirical questions above; it cannot be answered in a general way but only on a per-domain basis.)

• Even when subgroup bias has been addressed in some way, the approaches are fragile and non-generalizable. Ferrara (2024) has written of this as the “butterfly effect,” that small changes to algorithms, training data, or prompts can lead to substantial changes in the output of LLM models that magnify the effect of biases. What does that imply? Even if an LLM system generates reliable synthetic data that overcomes biases at one point in time and for one group, we have no reason to expect that it will do so at a later time, or with a different group, or after an algorithm update.

In short, the claim that LLM synthetic data can amplify otherwise underrepresented groups appears to be highly unlikely as a statistical matter, and is impossible to prove as a general expectation.

There is another sampling problem that I’ve written about separately: the basic concepts and statistics of sampling do not apply to LLMs. This is additional to the concerns in this post, and it means that, besides the concerns here, any “sample” from an LLM has no particular statistical meaning. It is, in fact, not a sample at all but output of a mostly uncharacterized stochastic process. (This implies, for example, that we cannot place meaningful confidence intervals on statistics from synthetic data; the estimates certainly are not exact values, yet we have no way to assess our degree of sampling certainty about them.)

Side Note: Questions for Research Ethics

Although I don’t have space here to consider all aspects of research ethics, there a core question that I encourage every researcher to ask: does synthetic data meet ethical obligations to society and ourselves?

Various governmental and professional organizations outline ethical requirements for research. These include legal definitions such the US Code of Federal Regulations (42 CFR 93.234) and professional standards such as the Ethics Code of the American Psychological Association (2017). The legal definition in US 42 CFR is relatively typical:

Research misconduct means fabrication, falsification, or plagiarism in proposing, performing, or reviewing research, or in reporting research results.

So the question is: are synthetic data fabricated, falsified, or plagiarized?

I don’t propose an answer here because the set of considerations — from models of ethics such as deontology vs. consequentialism, to the definition of individual words and the functioning of specific LLM systems with respect to plagiarism (e.g., Reisner, 2025) — is too large to tackle in this post.

Instead, I believe that the question is important (and sometimes mandated) for each of us to consider and answer for ourselves. We should be confident that our research aligns with ethical requirements!

Other Uses for Synthetic Data: Unfortunately Unconvincing

Here are four common claims for synthetic data other than reporting it, with my brief rebuttals.

• Synthetic data accelerates research. Rebuttal: synthetic data is not data; using it is not research.

• Synthetic data can pre-test a survey and analyses. Rebuttal: random responses, as many platforms provide, are preferable. Random responses are free of assumptions and correlated patterns, giving more comprehensive and unbiased tests. Random data can also be used to help assess data validity.

• Synthetic data can preview expected results with stakeholders. Rebuttal: such data doesn’t preview anything. Better — and much less risky — is for a researcher to use a combination of domain knowledge and stakeholder engagement to create a few scenarios reflecting potential outcomes. We can discuss those with high specificity and relevance, without fabricating data.

• Other colleagues will use such data, but I can use it more carefully. Rebuttal: they shouldn’t use it, either. We can’t let research ethics and practice be defined by what non-researchers might do.

Overall, I don’t view any of these claims as a good reason to use synthetic data. Let’s consider the benefits and risks. On the benefits side: synthetic data shows low value and there are preferable alternatives. On the risks side: creating synthetic data could lead to it being accidentally used, reported, or demanded by executives. In my opinion, the low benefits are strongly outweighed by the substantial risk.

Although it is great to pre-test a survey and its analyses, and it’s also great to preview potential results with stakeholders, those goals do not require synthetic data and are accomplished better without it.

Conclusion

My personal conclusion — as a matter of logic, empirical findings, statistical reasoning, and scientific principles — is that synthetic data has no place in survey research. I also believe the purported use cases for synthetic data are unconvincing; alternative approaches are superior in their results and are less risky in practice. And there are important questions about research ethics that each of us should consider.

Disagree? Publish your reasoning and results! (For a venue, check out quantuxcon.org)

All of this poses a question: why is there so much interest in synthetic data? Samoylov (2024) argues that it reflects a “snake oil” industry that is intent to sell products to naive customers.

For my part, I see the hope placed in synthetic data as a form of magical thinking. It is certainly appealing to believe that a data genie can magically create the data I need! And it is even more appealing to believe that it can free me from the difficulties of collecting real data, while getting results faster.

But more likely, we can’t escape the real work of survey research: collecting good data that informs unique decisions. On a happy note, actual research — which is to say, learning from people — is not only informative about the real world, it is also an enjoyable and rewarding enterprise.

And that process of learning from people will always deliver enough value to exist!

References

American Psychological Association (2017). Ethical Principles of Psychologists and Code of Conduct. At https://www.apa.org/ethics/code

Bisbee J, Clinton JD, Dorff C, Kenkel B, Larson JM. Synthetic Replacements for Human Survey Data? The Perils of Large Language Models. Political Analysis. 2024; 32(4):401-416. doi:10.1017/pan.2024.5

Ferrara E (2024). The Butterfly Effect in artificial intelligence systems: Implications for AI bias and fairness. Machine Learning with Applications, Volume 15. https://doi.org/10.1016/j.mlwa.2024.100525. At https://www.sciencedirect.com/science/article/pii/S266682702400001X.

Paxton J, Yang Y. (2024). “Do LLMs simulate human attitudes about technology products?” In Proceedings of the 2024 Quantitative User Experience Conference. At https://drive.google.com/file/d/16F_JZv4eHNiDMJT6BT7F6m97C2rBX8-7/view?usp=sharing

Reisner A (2025). “Search LibGen, the Pirated-Books Database That Meta Used to Train AI”. The Atlantic, at https://www.theatlantic.com/technology/archive/2025/03/search-libgen-data-set/682094/

Samoylov N (2024). Synthetic respondents are the homoeopathy of market research. At https://conjointly.com/blog/synthetic-respondents-are-the-homeopathy-of-market-research/

US Code of Federal Regulations (2024). Research misconduct. 42 CFR 93.234. https://www.ecfr.gov/current/title-42/section-93.234

Finally, as always, this post was …

At a high level, TURF answers this question: after we do the #1 best thing — whether that is a product, message, placement, etc. — that customers desire most, what should we do next to reach additional customers, beyond those who want the #1 thing? It is not necessarily the #2 thing from our list.

TURF stands for “total unduplicated reach and frequency” (Miaoulis et al, 1990). That means that it find combinations of items that will reach the largest total number of people, while having as few duplications [multiple exposures per person] as possible . And, secondarily, it will maximize the frequency — average number of reaches per person — after reaching the most unique people.

A classic TURF example is to reach as many unique viewers as possible with advertisements. If you ignore the question of pricing, you would want first of all to place an ad into the most popular channel (TV show, magazine, etc.) That would have the largest audience (“reach”) for a single placement. But after that placement, what is the best second choice? Is it the 2nd most popular channel? Not necessarily, because the second most popular channel may have high overlap with the first channel. For instance, visitors to the most popular websites also tend to visit many of the most popular websites; they are heavy internet users across many sites.

If you just go by ranked popularity, you would often be advertising to — reaching — the same people again. It may be better to go farther down the list of popularity and find a channel that is smaller but that reaches a unique audience, relative to the #1 channel.

My example is this: when scheduling practice times for an online Zen group, what is the best set of times that will reach the most people, so at least one time in the set will work for as many people as possible? Respondents took a survey and said when they are available. Given that, I want to find a small number of times that will make practice available to as many unique people as possible. TURF is a way to do that.

In this post, I share a data set — after protecting privacy as noted below — and R code for TURF analysis. In this data set, it would be possible to do TURF by hand. However, it is a great example for R analysis, and would easily scale to larger data sets. As always, I share R code along the way and compile it at the end.

The Data

I fielded a survey that asks respondents what times they are available on weekdays. Each respondent answered for their local time zone, which was later converted to standard time. (BTW, I also asked about weekends, and preferred days of the week. But for this post, we’ll only look at weekday times.) The results are a grid of availability by respondent, adjusted for time zone, mapped to 24 hours of the day.

Unlike many of my surveys, these data are private. However, I have created a “permuted,” simulated data set that I can share. It preserves enough high-level characteristics to give an identical TURF answer.

More detail. In these data, using random permutation of rows and columns, no observation is identical to the original data, and all sets of individual answers — and the relationships among times for every person — are altered randomly. This way, the data set preserves privacy while also reproducing the overall counts of availability by row and column. It gives identical TURF answers as my real data. The permuted data set was generated by the R vegan::permatswap() function (Oksanen et al, 2025).

Why don’t I just use fake data? Because this way, I can share results with the Zen community as well as the Quant community, and show exactly how the schedule was decided.

The permuted data set is small enough to share it in code, with no need to download it. I start with output obtained from the R dput() function, giving the data in text format. To create our data set hourgrid:

# the following was obtained from R dput() after survey import, data permutation, etc.
hourgrid <- structure(c(0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 
  0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 
  0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 
  0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 
  0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 
  0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 
  0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 1L, 
  0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 
  0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 1L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 1L, 
  0L, 0L, 0L, 1L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 
  0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 1L, 1L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 
  0L, 1L, 0L, 0L, 1L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 1L, 1L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 1L, 0L, 
  0L, 1L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 1L, 1L, 1L, 1L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 1L, 0L, 
  0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 
  0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 1L, 0L, 0L, 1L, 0L, 1L, 0L, 0L, 0L, 1L, 0L, 1L, 0L, 0L, 1L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 
  0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 
  0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 
  0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 1L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 
  0L, 1L, 0L, 0L, 1L, 0L, 1L, 1L, 1L, 0L, 0L, 1L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 1L, 0L, 1L, 0L, 0L, 0L, 0L, 
  0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 1L, 0L, 1L, 1L, 
  0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 
  0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 1L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 
  0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 1L, 
  0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 1L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 
  0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 
  0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 
  0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 
  1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 
  0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 
  0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L), 
  dim = c(39L, 24L), 
  dimnames = list(
    c("1", "2", "3", "4", "5", "6", "7", "8", 
      "9", "10", "11", "12", "13", "14", "15", "16", "17", "18", "19", 
      "20", "21", "22", "23", "24", "25", "26", "27", "28", "29", "30", 
      "31", "32", "33", "34", "35", "36", "37", "38", "39"), 
    c("1:00", "2:00", "3:00", "4:00", "5:00", "6:00", "7:00", "8:00", "9:00", 
      "10:00", "11:00", "12:00", "13:00", "14:00", "15:00", "16:00", 
      "17:00", "18:00", "19:00", "20:00", "21:00", "22:00", "23:00", "0:00")
    )
  )

Next I structure it as a data.frame and clean it up a bit:

# set that up as a nice dataframe and keep colnames as "hours"
tmp.names       <- colnames(hourgrid)   # save the names instead of what data.frame() does
hourgrid        <- data.frame(hourgrid)
names(hourgrid) <- tmp.names     
hourgrid$ID     <- 1:nrow(hourgrid) # add ID variable needed later by TURF
# check its structure
head(hourgrid)

With the head() command, we see that each row shows a person’s availability for each hour of the day. It is coded as 1 if they said they are available at that time. (as noted, it asked for responses in the local time for each respondent. Those were standardized into Pacific time before this analysis started.)

We can find the most popular times by summing the 24 columns representing each hour:

# count of preferred times (omitting ID column)
colSums(hourgrid[ , 1:24])

We see that 09:00 and 15:00 (not shown here) are the most popular times, followed by 10:00 and 12:00:

Even better is a data visualization. For this data set, we can do a heatmap of times by respondent. Here’s R code for that:

library(ggplot2)
library(reshape2)
# melt the data for nice ggplot structure
hours.m <- melt(subset(hourgrid, rowSums(hourgrid[ , 1:24]) > 0), 
                id.vars = "ID")
names(hours.m) <- c("Respondent", "Time", "Available")

# plot it
p <- ggplot(data=hours.m, 
            aes(x=Respondent, y=Time, fill=Available)) +
  geom_tile(color = "grey90",
            lwd = 0.5,
            linetype = 1) +
  scale_fill_gradient(low = "white", high = "darkblue") +
  scale_y_discrete(limits = rev(levels(hours.m$Time))) +
  coord_fixed() +
  theme_minimal() +
  theme(panel.grid.major = element_blank(), panel.grid.minor = element_blank()) +
  theme(legend.position = "none") +
  xlab("Respondent (permuted data)") +
  ylab("Times Selected by Respondent (permuted data)")

p

Two small notes on that code. First, when melt()‘ing the data, I keep only respondents with at least 1 time selected (nb, in ggplot() this can lead to blank columns; solve by making ID an ordered factor, or create a new ID without gaps). Second, I reverse the Y axis using scale_y_discrete(limits = rev(levels(…))) so the times are shown in a more natural order. Here’s the plot:

We can see a few characteristics of the data. First of all — remembering that these are permuted data with the same high level structure but not people’s actual individual responses — most respondents only selected a few times they are available, commonly 1, 2, or 3 times on a given day. Second, we see that the times are distributed across almost all hours of the day (which reflects the worldwide community of respondents). Thus, the overall grid is relatively sparse.

FWIW, we can calculate the sparseness. Add up all the selected times and divide by the size of the grid:

# how sparse is it?
sum(hourgrid[ , 1:24]) / nrow(hourgrid) / 24

The answer is 0.108, i.e., the “average” respondent selected 10.8% of the available times.

All of this tells us that the single most popular time would be either 09:00 or 15:00 Pacific. But what if we want to schedule 2 or 3 times that maximize availability for the largest number of people, where each of them has at least one time that would work? That’s where TURF comes in.

What about days of the week? For purposes here, I’m ignoring that. This grid has responses for weekdays (weekends were asked separately). Among all 7 days of the week, the most popular days were Monday and Tuesday. Of those, only Tuesday works for me to schedule the meetings. Thus, in this post, I assume that we’re talking about times on Tuesdays.

BTW, you might wonder: why not ask about all times, across all days using a 7 day x 24 hour grid? Answer: in that format, respondents tend to select only a small number and data are extremely sparse. However, we could still use TURF with data of that kind.

TURF Code and Results

It is relatively straightforward to write one’s own TURF function, but we don’t need to; there is already an easy-to-use turfR library for R (Horne, 2014). First, we slightly reformat the data to match its expectation:

# TURF Analysis
library(turfR)

# set up the data for TURF package
# add ID variable so respondents are identified to TURF
turf.dat <- data.frame(
  ID = hourgrid[ , "ID"],           # respondent ids
  Weight = rep(1, nrow(hourgrid)),  # the weight / importance for each respondent
  hourgrid[, 1:(ncol(hourgrid)-1)]  # columns of data to use; all except "ID"
)
head(turf.dat)

This format is almost identical to our original format, except that it adds a Weight for each respondent. The weight is the “importance” to give each observation. For example, if you are working with a customer database, you might want to weight customers according to their observed revenue, longevity, retention, segment membership or the like; or use similar estimated from a regression model; or from responses to a survey item such as their level stated interest. For our data, I assign equal weight of 1 to all observations.

Here’s what the turfR data object looks like:

Although our data has simple values of 1 and 0 for people’s availability, binary data is not required for TURF. TURF can work with continuous values (see note at the end), such as likelihood scores from a database, regression model, or simple Likert responses; estimated preference values (e.g., from a choice model survey); and so forth. In our case, the data come from checkboxes of availability on a survey, so 1/0 is the natural format.

With that, we can now fit a TURF model. I’ll start by looking for the best 2 times (k=2) and ask for the top 10 results (keep=10). We tell it that we have 24 items (times) to consider (n=24). Here’s the code and result:

# run the TURF analysis for best 2 times
(turf.2 <- turf(turf.dat, n=24, k=2, keep=10))$turf

I’ll interpret results in the next section, but will note here an unusual but happy coincidence in these data: the item & column numbers (1—24) correspond exactly to the hours of the day (01:00—24:00/0:00). That makes the results easy to read, but it is a unique situation for this data set and question. In most cases, you would have to match those numbers to interpretable names of items, features, messages, etc.

For my purposes, I’ll use TURF results for the tentative 3 best times to schedule. Here’s how to get those, asking for the 3 best times (k=3) and keeping the top 15 results (keep=15):

# what if we scheduled 3 times instead of 2? 
(turf.3 <- turf(turf.dat, n=24, k=3, keep=15))$turf

BTW, in this simple case, the algorithm does a full expansion to consider all possible combinations. If we had larger sets to consider, that would become computationally intractable (e.g., choosing the best sets of 20 ads to rotate out of a set of 100 options). In such cases, turfR offers Monte Carlo sampling options.

Selecting Among the Results

For my scheduling problem, I will start by scheduling meetings 1 time per week and then soon expand to 2 and eventually perhaps 3 times per week. Given that plan (i.e., “strategic goal”), I’ll use the k=3 results for the optimal 3 meetings per week, and develop the schedule in light of those results.

Following is the k=3 result again (first 6 solutions). Before jumping into the schedule choice, I’ll explain the first 3 columns: combo, rchX, and frqX. The combo is simply the internal solution number that turfR considered. We can ignore that unless we want to dig deeply into its solution design matrix.

The next two columns, rchX and frqX, are the heart of the TURF solution: the reach (rchX) and frequency (frqX) that the solutions reach, sorted in decreasing order of reach. Specifically, rchX is the proportion of respondents who are reached at least once by a given solution, which frqX is how often the average respondent is reached.

Consider the first solution, which is to offer meeting times at 09:00, 12:00, and 18:00 Pacific Time. That set of times is estimated to reach 69% of the respondents — meaning that at least 1 of those times would work for 69% of people who replied to the survey. However, some people are available for multiple times within that set. The frequency estimate tells us that this set of 3 times would offer 0.74 available meeting times per week to the average respondent. I.e., some of the 69% would have multiple options.

You might wonder, “what about confidence intervals (CIs)? Are there meaningful differences between 69% for the first solution and 67% for the second one?” Like anything involving CIs, it is a complex question. In this case, it is unclear to what extent we should consider these data to be a sample of a population — as a CI would assume — as opposed to being a descriptive census of a community. However, if we assume that it is a random sample, we could find CIs either with a bootstrap operation (best) or using a quick approximation for the binomial proportions. Given N=39 respondents and looking for an 80% confidence interval (±1.28 standard errors), we get a simple estimate of:

# est'd CI for the TURF #1 solution
# get the proportion reached for solution #1
p.turf <- turf.2$turf[[1]]$rchX[1]
# estimate of 80% confidence interval (swag; bootstrap etc would be better)
# set the Z value (standard errors) as appropriate; in this case "80% CI" Z == 1.28
ci     <- 1.28 * sqrt(p.turf * (1-p.turf) / nrow(hourgrid))
c(p.turf - ci, p.turf + ci)

The answer is that our CI is a range of 0.598—0.787, meaning that solution #1 is estimated to reach approximately 60-79% of respondents. The other 14 solutions all fall within that range for reach. So, as a first approximation, none of those 14 are dramatically worse than solution #1. Here’s the CI computation using simple binomial approximation at the 80% level:

With all of this in place, let’s look at the first 6 solutions. Then I’ll discuss how I chose among them with respect to overall goals. The solutions are:

Examining the patterns, a few things stand out. First, the 09:00 time appears in all 6 of the top results. Second, we see that solutions 2—5 are all tied for reach with one another, and are only slightly behind solution #1. Third, solutions 2 & 3 — and to a lesser degree solutions 4 & 5 — score somewhat higher on frequency than solution #1. Finally, some of the solutions including #1 and #5 have times spread out across the day (e.g., 09:00, 12:00, 18:00) whereas others are more compact, such as #4 that has times at 09:00, 10:00, and 12:00.

Here’s how I assessed the possibilities and made a choice for the schedule.

1. For the first, initially scheduled time, 09:00 is a clear winner. Although it is slightly less popular on its own than 15:00 (N=12 vs. N=13), the 09:00 time appears in all of the top solutions. There it is a good choice of a foundational time, while allowing the schedule to evolve later.

2. Beyond that, I tentatively select 10:00 as the second, expected time slot. 10:00 appears in solutions #2 and #4 and thus is an excellent second choice. Additionally it has the advantage of being compact in time (assuming both sessions occur on the same day) which is logistically simple. It also appears in solutions with relatively higher frequency, meaning that it will give some flexibility to people.

3. If and when the community needs to have a third time scheduled, I’d want to see how 09:00 and 10:00 are going and then decide. Tentatively, 12:00 or 15:00 look promising; another option might be 18:00.

This process of selecting 09:00 — while tentatively planning 10:00, and considering a more complete schedule over time — is a good example of a cautious process.

At Amazon (and elsewhere) this is often described as the difference between a “two-way door” and a “one-way door”. A one-way door is a decision that is difficult to reverse. For example, announcing a scheduled time of 09:00 and scheduling practice then is difficult to reverse because people come to expect it. Similarly shipping a product, buying a house, and getting married are all one-way doors.

By contrast, a two-way door is relatively easier to reverse. Deciding tentatively on 10:00 as a planned next step is easy to change later. It is similarly “two way” and reversible to test a product prototype, rent an apartment, or to date someone as opposed to getting married.

A Few Applications for TURF

In the worlds of Quant UX and marketing research, TURF analysis has many applications. Here are a few that occur to me somewhat randomly:

• Constructing a restaurant or other “menu”: given preference data (such as a MaxDiff assessment), which smallest set of items will often the most choices that appeal to as many customers as possible? (Want to try it? Get the Quant UX Association class preference data and run TURF on the estimates!)

• Planning conference or workshop locations over time: which set of locations will maximize the appeal — on at least one occasion — for as many people as possible?

• Selecting marketing messages: if we can feature only a few messages, such as 3 claims on a product package or website, which set of messages will catch attention from as many people as possible?

• Offering employee perks or gifts: suppose we want a small number of things to offer to employees. Which set will be most likely to have “something for everyone”?

• Prioritizing customer outreach efforts: suppose our sales team wants to call customers who are highly interested in at least one of our new features. We want to select a limited number to call, prioritized jointly by feature interest [reach] and account size [weight]. Who are they?

• Pharmaceutical formulary: which combination of prescription drugs will offer the greatest coverage, to as many of our patients and/or physicians as possible?

• Excursions for a cruise ship: what is the smallest combination of outings and on-ship activities that will interest the largest number of passengers?

• Similarly: selecting political party platforms or legislative priorities; books for a mobile library; design templates for a app; destinations to fly to be a “complete” airline; and of course, the original goal of optimizing ad placement for reach; among many others.

Remember that TURF can use continuous estimates for “reach" (see note at the end) and thus can take advantage of any sort of estimates for interest or preference; and it can also weight respondents. The reach estimates may come from respondent choices, as in the data here, but also may come from observed data as in product instrumentation or a CRM system, or from a regression model. With so many options, there are many applications for TURF!

Conclusion

As we saw, TURF analysis provides potential solutions to a very practical question: how to schedule a small number of events (weekly meeting times) for a group of people. Given a few candidate solutions, I was able to consider other factors such as practicality and create an initial schedule.

We discussed how TURF can be extended to larger and more complex cases, including ones where we have continuous (real number) estimates of interest instead of binary indications, and could weight respondents by some kind of importance estimate.

Could I have done this analysis by hand? With this small data set and number of options, yes. OTOH, by doing it in code, I have an answer that is more scalable and will be easy to apply again as new data arrive. Plus I like coding (and also like blogging and explaining code).

Learning More. Although TURF is widely used, I’m not aware of a great single source to learn more about it. The original paper was by Miaoulis et al (1990), listed in “Citations” below. To find more applications, I’d suggest to check that paper’s citations in Google Scholar and branch out from there. The turfR library also has helpful documentation and references.

Try it! If you’d like to try TURF with another, more complex data set, here’s a homework exercise. Get the public MaxDiff data set with individual preferences for classes to be offered by the Quant UX Association. What is the best set of 3, 4, or 5 classes that would reach the most people? (Note: some choice modeling software offers TURF analysis built-in, but by adapting the code here, you can do it on your own in R!)

Best wishes!

Citations

Horne J (2014). turfR: TURF Analysis for R. R package version 0.8-7, https://CRAN.R-project.org/package=turfR.

Miaoulis, G., Parsons, H., & Free, V. (1990). TURF: A New Planning Approach for Product Line Extensions. Marketing Research, 2(1).

Oksanen J, Simpson G, Blanchet F, Kindt R, Legendre P, Minchin P, O'Hara R, Solymos P, Stevens M, Szoecs E, Wagner H, Barbour M, Bedward M, Bolker B, Borcard D, Carvalho G, Chirico M, De Caceres M, Durand S, Evangelista H, FitzJohn R, Friendly M, Furneaux B, Hannigan G, Hill M, Lahti L, McGlinn D, Ouellette M, Ribeiro Cunha E, Smith T, Stier A, Ter Braak C, Weedon J, Borman T (2025). vegan: Community Ecology Package. R package version 2.6-10, https://CRAN.R-project.org/package=vegan.

Wickham H (2007). Reshaping Data with the reshape Package. Journal of Statistical Software, 21(12), 1-20. URL http://www.jstatsoft.org/v21/i12/.

Wickham H (2016). ggplot2: Elegant Graphics for Data Analysis. Springer-Verlag New York.

Appendix: Note on Continuous Estimates with TURF

TURF in general can work with continuous estimates if you define an appropriate conversion and/or estimation value for it to optimize. However, the most recent version of the turfR package doesn’t support that. It only uses 0 and 1 indicators. If you want to work with continuous data, there are at least three options:

• Convert your data to 0/1 based on some sort of cutoff value. This is by far the most common option in practice, in my experience. Then use turfR exactly as shown above.

  • For example, following is an R command that will take a data frame with continuous values and give a transformation such that the top 25% of values for each respondent are coded as “1” while all others are coded as “0”. (There are many, many variations and alternatives for such transformations. I’m sure there is a nice tidy function to do this, too, but I write base R by habit!)

  • This code uses quantile() to find a cutting point, and uses apply() to apply a binary cutting point function to each respondent (margin=1). Finally it uses t() to reshape that back to the expected row-x-column layout.

•   data.bin <- t(apply(data, margin=1, 
                        function(x) { as.numeric(x >= quantile(x, probs=0.75)) } ))
  

• Write your own TURF function or find one online. I’ve written TURF code at work — although unfortunately that lives with a former employer and I can’t share it — and will say it is not terribly complex. You could use the code in turfR as a starting point, or a source like this one. Just find the point where the comparison is made to 0/1 data or combinations and insert the continuous function you want instead (such as identifying the combo with the highest sum of values, etc.).

• Use a platform that implements TURF for the data format you have. For example, if you want preference estimates from MaxDiff or Conjoint Analysis, Sawtooth Software’s Lighthouse Studio and Discover products will do TURF with continuous estimates.

All the Code

Following is all of the R code from this post, including inline code that creates the small data set.

####
# load our data
# note, data have been permuted from original responses,
# preserving overall characteristics while changed for each respondent (cf. vegan::permatswap)
hourgrid <- structure(c(0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 
  0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 
  0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 
  0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 
  0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 
  0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 
  0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 1L, 
  0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 
  0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 1L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 1L, 
  0L, 0L, 0L, 1L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 
  0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 1L, 1L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 
  0L, 1L, 0L, 0L, 1L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 1L, 1L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 1L, 0L, 
  0L, 1L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 1L, 1L, 1L, 1L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 1L, 0L, 
  0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 
  0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 1L, 0L, 0L, 1L, 0L, 1L, 0L, 0L, 0L, 1L, 0L, 1L, 0L, 0L, 1L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 
  0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 
  0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 
  0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 1L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 
  0L, 1L, 0L, 0L, 1L, 0L, 1L, 1L, 1L, 0L, 0L, 1L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 1L, 0L, 1L, 0L, 0L, 0L, 0L, 
  0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 1L, 0L, 1L, 1L, 
  0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 
  0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 1L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 
  0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 1L, 
  0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 1L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 
  0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 
  0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 
  0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 
  1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 
  0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 
  0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L), 
  dim = c(39L, 24L), 
  dimnames = list(
    c("1", "2", "3", "4", "5", "6", "7", "8", 
      "9", "10", "11", "12", "13", "14", "15", "16", "17", "18", "19", 
      "20", "21", "22", "23", "24", "25", "26", "27", "28", "29", "30", 
      "31", "32", "33", "34", "35", "36", "37", "38", "39"), 
    c("1:00", "2:00", "3:00", "4:00", "5:00", "6:00", "7:00", "8:00", "9:00", 
      "10:00", "11:00", "12:00", "13:00", "14:00", "15:00", "16:00", 
      "17:00", "18:00", "19:00", "20:00", "21:00", "22:00", "23:00", "0:00")
    )
  )
# set that up as a nice dataframe and keep colnames as "hours"
tmp.names <- colnames(hourgrid)
hourgrid <- data.frame(hourgrid)
names(hourgrid) <- tmp.names
hourgrid$ID <- 1:nrow(hourgrid) # add ID variable needed later by TURF
# check its structure
head(hourgrid)

# basic stats
# count of preferred times
colSums(hourgrid[ , 1:24])
# number of times chosen, per respondent
rowSums(hourgrid[ , 1:24])
# how sparse is it?
sum(hourgrid[ , 1:24]) / nrow(hourgrid) / 24

# heat map of times
# make heatmap
library(ggplot2)
library(reshape2)
library(car)
# melt the data for nice ggplot structure
hours.m <- melt(subset(hourgrid, rowSums(hourgrid[ , 1:24]) > 0), 
                id.vars = "ID")
names(hours.m) <- c("Respondent", "Time", "Available")
# some(hours.m, 15)

# plot it
p <- ggplot(data=hours.m, 
            aes(x=Respondent, y=Time, fill=Available)) +
  geom_tile(color = "grey90",
            lwd = 0.5,
            linetype = 1) +
  scale_fill_gradient(low = "white", high = "darkblue") +
  scale_y_discrete(limits = rev(levels(hours.m$Time))) +
  coord_fixed() +
  theme_minimal() +
  theme(panel.grid.major = element_blank(), panel.grid.minor = element_blank()) +
  theme(legend.position = "none") +
  xlab("Respondent (permuted data)") +
  ylab("Times Selected by Respondent (permuted data)")

p

# TURF Analysis
library(turfR)

# set up the data for TURF package
# add ID variable so respondents are identified to TURF
turf.dat <- data.frame(
  ID = hourgrid[ , "ID"],           # respondent ids
  Weight = rep(1, nrow(hourgrid)),  # the weight / importance for each respondent
  hourgrid[, 1:(ncol(hourgrid)-1)]  # columns of data to use; all except "ID"
)
head(turf.dat)

# run the TURF analysis for best 2 times
(turf.2 <- turf(turf.dat, n=24, k=2, keep=10))$turf

# est'd CI for the TURF #1 solution
# get the proportion reached for solution #1
p.turf <- turf.3$turf[[1]]$rchX[1]
# estimate of 80% confidence interval (swag; bootstrap etc would be better)
# set the Z value (standard errors) as appropriate; in this case "80% CI" Z == 1.28
ci     <- 1.28 * sqrt(p.turf * (1-p.turf) / nrow(hourgrid))
c(p.turf - ci, p.turf + ci)

# what if we scheduled 3 times instead of 2? 
(turf.3 <- turf(turf.dat, n=24, k=3, keep=15))$turf

If you’re only here for Quant UX discussion, you can skip the rest of this newsletter. However, I’ve heard from some readers interested in Zen. If you are one, keep reading. Or visit Tech Community Zen!

Brief Background

Apart from my work and writing, I’m an ordained Zen teacher (aka “priest”). I started Zen practice in 2001, was ordained as a Dharma Teacher in 2015, and received additional teaching authorization (“Inka”) in 2023.

I’m starting Tech Community Zen for a simple reason: the tech community can benefit from Zen. I see the tech community experiencing chaos, with many people hoping for something stable and fulfilling. Zen practice can be part of an answer to that chaos. Its benefits extend to our colleagues, families, communities, and the world around us.

Zen teaches deep attention, helping others, compassion, and counteracting destructive desire and anger. In these times, the tech community — and the whole world — needs those.

What is Tech Community Zen?

Tech Community Zen practices authentic Zen online as a community … and will develop from there. This includes group practice (Zazen; meditation) and, for those interested, individual koan practice (if that doesn’t mean anything to you, don’t worry; I’m mentioning it for anyone who wonders).

Depending on interest over time, we might add other online or in-person events such as intensive retreats. We’ll see!

Why does this focus on the tech community? Attention to the tech community is beneficial for two reasons: the community needs it, and the focus helps with teaching, programming, expectations, and community interaction. At the same time, everyone is welcomed and no one is excluded.

Interested? Help with the Planning!

You can find out more — and help plan the schedule and activities — in two ways:

1. Share your preferences, especially for days & times for online practice, in this short survey:
   Tech Community Zen Interest Survey

2. Sign up for the Tech Community Zen newsletter:
   https://tech-community-zen-newsletter.beehiiv.com/subscribe

More Questions?

Visit the FAQ and other information pages at tczen.org (== techcommunityzen.org). Or email and ask: techcommunityzen@gmail.com

Cheers, and I’ll return to Quant UX specific topics in the next post :)

I structure the conversations I’ve recently had with UXRs about LLMs into four thematic areas, giving a few pro & con arguments in each. I’m writing this not to advance specific claims but instead to share a schema that has been useful to me.

Thanks to corporate interests, there is a lot of “pro AI” discussion, and the claims on that side are well known. On the other hand, social pressures act to obscure “con” arguments, and I spend slightly more time on them. Although I outline both pro & con, I’m not advocating for either “both sides” or “no one knows.” Instead, my hope here is that any reader (and especially me) will gain clarity from the considerations and can thereby have a stronger, more specific, and more coherent framework wherever they land.

I outline the following four general areas:

• Utility

• Aesthetics

• Social Context & Reaction

• Ethics & Externalities

I present this in terms of “what UXRs are thinking”. That is only because my conversations are most often with UXRs. By understanding these areas, I hope we can disentangle arguments about LLMs, such as when claims of utility are pitted against aesthetic or ethical perceptions. After laying out the framework, I recount a small illustration of how it may help us think about claims that people make.

Theme 1: Utility

The first pattern is the most obvious for UX researchers: consideration of the utility of AI in delivering use cases for a product, or similarly, in helping with UXR activities.

On the pro side there are utility claims that LLMs deliver new experiences, assists in running surveys, can analyze user responses, summarize documents, meeting, or transcripts, draft reports, write code, and so forth.

On the negative side, I see two typical patterns:

• Claims that LLMs don’t make things better, or not much better, or are not worth the effort (cf. my article on R coding)

• Claims that LLMs trade off short term gains for long-term harms in utility (cf. some studies in this review of AI for developer productivity, or in this study for general cognition).

A common view is that any AI application has a “game theory” payoff matrix:

• Potential benefits when AI is correct or helpful

• vs. Potential risks when AI is incorrect or misleading

Some activities have low risk (e.g., recommender systems, role playing chat, games) whereas others have high risk (e.g., finance, health case, security). A complete game theoretic model could additionally include the risks and benefits from ethical considerations and externalities as noted below.

A criticism of that sort of game theoretic view is that it is based on (1) an assumption of a “net profit” model where an LLM delivers some positive value, and the question is how problems trade off against that value; and (2) that assumes that such trade-offs can be made at all. As I’m arguing here, some of the trade-offs are between incommensurate areas, such that a game theory approach — although not “wrong” as such — is limited at best and strongly misleading at worst.

Overall, utility is clearly the #1 area of discussion for UXRs and product teams generally.

Side note: unlike classic utilitarian philosophy, “utility” in the AI context is often used reductively to mean “[individual] productivity.” To the extent that such claims align with classic, longer-run utilitarian considerations, they often fall into the problems of so-called “effective altruism” and related movements. That caveat is worth noting although depth discussion is out of scope here.

Theme 2: Aesthetics

The second area I see is aesthetic evaluation of AI (again, for purposes here, meaning LLMs). By this, I mean the evaluation of the beauty, tastefulness, appeal, and similar properties of AI output. This is not restricted to AI-generated art but applies to any AI output, such as the “beauty” of AI-written code, reports, texts, etc.

Although many people are using AI-generated texts and graphics, I’m not aware of positive claims that the aesthetics of those outputs are actually good (related study). Instead, AI output appears to be regarded by its users as aesthetically tolerable. They often comment that AI results are easy or “democratized”.

On a closely related positive note, there are many UXRs and others who are fascinated by AI interactions and who enjoy engaging with AI systems at length. I would not categorize that enjoyment as aesthetic but rather as more generally emotionally engaging.

On the negative side, I observe three primary areas of aesthetic complaint:

• Claims that AI products are aesthetically bad, such as being ugly, badly written, “slop,” and the like. This view seems to be common among professional writers, artists, and musicians.

  • Many developers see AI code as needing large scale refactoring before it is “good.”

  • Differently, the Authors’ Guild — verified published authors, of which I’m a member — recently launched a “human authored” certification program to guarantee “authentic” works.

• Worries that AI proliferation will drown out good work, rewrite aesthetic preferences, or generally lead people to miss the important aspects of creation while focusing on mere output and “productivity”. A new book, More than Words, by John Warner discusses the latter angle in depth.

• A more generalized feeling that AI diminishes the appreciation of human creativity and human experiences, and that it devalues the many years and thousands of hours of effort that it takes to master a craft such as writing, drawing, painting, music, or programming.

Overall, some professional writers, developers, artists, and musicians experience AI output with a sense of revulsion. Although such an aesthetic response is emotional, I would note that it is not only emotional.

In my view, there are good reasons to treat aesthetic truth as being on par with other kinds of truths such as propositional (classically logical) truth. However, that discussion goes deep into philosophy and related areas such as cultural studies, and that is out of scope here (for more, a starting philosopher, which is not to say an easy philosopher, is Adorno).

How does this relate to the category of utility above? Answer: aesthetics and utility are separate. An interaction between product team members might go like this:

• Person A: This LLM letter-writing feature will lower the bar substantially for new writers [a utility claim]

• Person B: But its output is awful to read, so bland and tasteless [an aesthetic claim]

By sorting evaluations into these two categories, as well as the subcategories noted, we can better evaluate competing claims (and identify when they are not competing but are just different).

Theme 3: Social Context & Reaction

A third set of responses to AI depend on the interplay of any particular UXR’s own social situation and personality.

As we all know, AI has had widespread pressure for institutional adoption, and has also suffered from tremendous hype. At times this has been paired with claims — such as those about AGI — that are somewhere between fraudulent and incredibly ignorant (I’ve written about AGI and AI “IQ” as examples).

Each UXR responds to the social context of AI according to their own needs and personal style. While one UXR may wish to go along with demands, another may rebel against authority or hype. One may emphasize the future promise of AI, while another will see LLMs as a tool of corporate subjugation.

The impacts of AI in social contexts are not only evolving but also are more complex than any single person could observe. As in the old story about blind men and an elephant, different aspects appear with different people.

On the negative side, I’ve observed particularly hostile reactions prompted by the following:

• Hype in general. Some people detest hype and react against it (the Mystery AI Hype podcast is an example)

• Illogic. Some UXRs have a particular dislike of illogical claims made about AI, as they also might in other domains (nb: I personally have this tendency!)

• Management demands. Some UXRs expect the worse from management (perhaps because of past trauma) and may view AI as an extension of managerial or corporate domination.

In the specific UXR context, the pro & con trends for social context show up in claims such as:

• [pro] "AI is the future and UXers have to learn it”

• [con] “Management doesn’t understand what they’re talking about, and we need to stop the harm to users”

• [pro & con] “Excessive hype is destroying opportunities to talk about the real value of AI”

• [pro] “AI is an area where I can show thought leadership” or “By using AI, I can get ahead”

Again — as we saw above for claims of utility vs. aesthetics — the set of issues regarding the social positioning of AI is logically separate from the other areas outlined here. If someone says, “We need to learn AI to get ahead,” that doesn’t imply anything in particular, positively or negatively, about whether AI is useful, aesthetic, or as I discuss next, ethical.

Theme 4: Ethics & Externalities

Preliminary Points

This is the longest section, because ethical issues are perhaps the most contentious claims about AI. Extreme positive arguments claim that AI is a force for good that will solve cancer, mitigate climate change, and eliminate human drudgery. On the extreme negative side, AI is said to be a short term tool of theft, oppression, corporatism, and militarism, and a long term existential risk to human and the world ecosystem.

Before outlining my framework, I have two overall observations. First, my impression is that ethical takes are often derived post hoc. They may be used to reinforce stances that one has already adopted for reasons of aesthetics, emotions, self interest, etc. In other words, I believe UXRs often either like AI or not, for various reasons, and then adopt “ethical” reasons to justify a stance retroactively. I’m not saying that is always the case, but often occurs (and that is perfectly OK, in my view — there is no reason why ethics should be one’s first consideration, as long as they are an eventual consideration).

Second, and closely related, I observe that ethical takes are generally unpersuasive to anyone who doesn’t already hold them. The proponents of ethical arguments appear to hope that they will be persuasive to others — whether that is “AI will cure cancer!” or that “AI is theft!” — but I have seen no good evidence that such arguments are persuasive to anyone (in the sense of changing opinions).

Now, my claim that ethical arguments are unpersuasive does not imply that ethical arguments are wrong, nor that they are useless. Ethical positions may be useful for providing a backstop and support for views held otherwise, for giving a “record” for posterity, and for deepening one’s own thinking and moral actions. Persuasion is not the only salient outcome. (Full disclosure: I think about ethics a lot because I was a philosophy graduate student prior to psychology, and I wrote a philosophy Master’s thesis in ethics. I also taught the mandatory Research Ethics class to over 1500 employees when I worked at Google.)

With those preliminaries in place, the following points outline how I see ethical issues lining up. (n.b., this area is vast and I don’t pretend that these points are complete).

Ethics in AI Training

• Positive claims:

  • LLMs are a vastly scaled-up version of fair use, making massive information available to everyone

  • Training creates jobs in economically disadvantaged areas like Africa and Southeast Asia

• Negative claims:

  • LLMs are only possible because they use stolen intellectual property (example)

  • The workers who train LLMs are underpaid and treated abusively, undergoing substantial trauma

Ethics in AI Applications and Other Short-Term Results

• Positive claims:

  • LLMs deliver various features and productivity enhancement (refer to the “utility” section above)

  • LLMs help people such as non-native speakers or people with disabilities achieve parity with others (example)

• Negative claims:

  • LLM features have generally low marginal value (example), and may worsen products they go into

  • LLM features may exist not to deliver value but to capture personal data for corporate purposes

  • LLMs contribute to declining skills in reading, writing, and critical thinking

  • The push for LLMs brings an opportunity cost of less product attention to real user needs

  • LLMs push all outputs towards a style of white, western, English language, educated, affluent, and heteronormative modes because that style dominates their training data

  • LLM data centers capture water and electricity needed by homes, businesses, and agriculture

  • Contrary to claims that they achieve parity, LLMs perpetuate stereotypes and biases about disabled people and other groups

Externalities and Structural Issues (current, as opposed to hypothetical)

• Positive claims:

  • AI productivity (see “utility” above) will help companies achieve higher profits

  • AI is helpful for non-native speakers (by which, proponents often mean “of English”) and helps them write better, grammatically, etc.

• Negative claims:

  • LLMs contribute to the increasing domination and centering of white, male, affluent English as a structural norm (similar to the point in “applications” above). (Examples: 1, 2, 3, 4.)

  • Promises (or realities) of AI productivity are contributing to job losses

  • LLMs devalue workers and replace creators, leading to a race to the bottom for artistic creation

  • LLMs increase the concentration of power among a few companies and their billionaires, achieving higher profit and more power with fewer workers and worker protections

  • LLM data centers pollute the planet and contribute to acceleration of climate change

  • LLMs differentially hurt marginal communities (non-white, low-wage, global south, etc.)

Ethics & Hypothetical Future Results

• Positive claims (speculative):

  • AI speeds up scientific discovery, and new discoveries could mitigate many scientific problems (medical research, climate change, etc.)

  • AI could usher in an era of productivity such that people are freed to work less, more creatively

• Negative claims(speculative):

  • Bugs or malicious usage of AI pose a range of threats, ranging from modest scale problems (fraud, impersonation, theft) to socially catastrophic (financial collapse) to existential (engineered viruses, killer machines, nuclear war)

  • AI may lead to humans being massively unemployed as jobs are replaced by AI, and largely confined to “lower skilled” labor that AI cannot replace

  • LLMs change expectations about truth and the veracity of both information and work products

Research-Specific Ethical Issues

• Positive claims:

  • LLMs democratize and decrease the cost of research by providing synthetic training data and by analyzing data faster (example claim)

  • LLMs can discover new things, faster, by exploring research domains in new ways

• Negative claims:

  • Research applications depend on the uncredited work of other researchers

  • “Democratizing research” as machine output devalues research. Normalizing research-as-production is to abandon the value of researchers and their contributions (and jobs).

  • Synthetic data is misleading and biased, over-representing dominant social groups (as above) and specifically downplaying and systematically under-estimating marginalized groups

  • Subjecting respondent and user data to LLM systems violates privacy and informed consent (1, 2)

  • LLM research results amplify what is obvious in a data set and/or is “centroid” in its training data, rather than discovering new things or representing actual data (related)

  • Using a system with ethical problems (if one believes that) is itself a research ethics violation

In some discussions, folks go farther about their concerns with LLMs. In those cases, some UXRs and managers actively question the ethics of pressure that they feel to research or use AI.

That ethical questioning takes two forms. One form is whether it is ethical to do UX research about AI that is required to say something they don’t believe — namely, that LLMs add value. I often hear researchers complaining that they are being “forced” to find reasons that AI will add product value. In those cases, they feel that the users and customers they represent are being ignored or even actively harmed. They question whether there is any point in doing research that is forced to “validate” preordained executive decisions.

Side note: research usually is not directly ordered to support a decision (although that sometimes happens). Rather, results are demanded through organizational pressure, such as: having negative results challenged; being asked to do A/B testing between options that both assume some need (AI, for example); being promoted or praised for work that extolls the desired outcome; etc. Such mechanisms bias results but preserve management ability to believe that research is “honestly sharing data”.

The second form of UXR ethical questioning is at a higher level: whether it is ethical to use LLMs at all or to work for a company that promotes them or requires one to use them. That arises if one holds some of the negative ethical views above. Similarly — but as far as I can tell, much less frequently — a UXR who is an AI proponent may have ethical concerns about NOT using AI tools. This may occur when they are in an organization that is not using LLMs because of concerns about privacy, data protection, and the like.

How the Themes Help: a Brief Example

Here’s a small example of how I personally find this framework to be helpful. Yesterday I was reading a LinkedIn post and comment that went like this (I’m not going to call out anyone, so I paraphrase):

• Post: AI tools seem so potentially useful for research! What are people doing with them?

• Comment: I use them all the time. Anyone not using them is a dinosaur and will be left behind.

In the past, I would have read this sort of exchange as annoyingly lacking in information, as being hype, and as name-calling (“dinosaur”) that uses belittlement instead of informed discourse (perhaps to cover up the commenter’s own anxiety).

However, using the 4 themes outlined here, I now decode it as follows:

• Utility: there is a implicit claim of utility but no evidence

• Aesthetics: ignored

• Social context: an explicit claim that AI can help one get ahead, again without evidence

• Ethics: ignored

In short, the exchange boils down to an explicit claim that AI can help someone advance, and it ignores (or somewhat denigrates) the other potential positions. I imagine this reflects anxiety more than anything else.

With the 4 elements of the framework, the argument can be deconstructed to see (a) where it is making claims and not making claims; and — just as importantly — (b) partially defused of emotion, anger, confusion, and the like, because it is clearer what is going on.

Conclusion

If you made it this far, congratulations :) As I noted at the beginning, my goal here has been to organize some areas of discourse and debate for my own thinking.

Reviewing that framework, a few themes and observations stand out:

• The four themes here — utility, aesthetics, social function, and ethics — are mostly incommensurate and not directly comparable. That leads to two things:

  • The various sides in debates tend to miss each others’ points by responding across domains, such as answering a utility claim with an ethical rebuttal or vice versa. That convinces no one.

  • People simply ignore themes where the prevailing argument don’t agree with them. For example, proponents of utility dismiss may concerns about aesthetics and ethics.

• After outlining the themes here, I observe that most of the discussion about AI reflects a strongly productivist orientation — in society as well as in product teams and among UXRs. LLMs are discussed in terms of how much they help people produce, how fast one can write reports or code or create images, how much data one analyze, how many articles summarized, and so forth.

• The UXRs I talk with are increasingly frustrated at the situation because it leads to untenable demands on research. I believe much of that results from two things:

  • The incommensurability of the themes as outlined. Researchers have increasing cognitive dissonance and dissatisfaction as the thematic areas collide. For example, they may have ethical concerns competing with pressure to be productive; or pressure to support an AI feature competing with user data that suggests dissatisfaction. That leads to sharply decreased researcher morale.

  • They feel particular tension around the productivist orientation, that research is to be judged primarily on the basis of faster and more output (as opposed, say, to user benefit, or truth).

• None of this implies a both-sides equivalence. I outline the pro and con arguments not to equate them but instead to reflect that different people hold different views. Any given UXR will — and IMO should — take a principled stance that aligns with their own perspective. To reiterate, my goal in this article was to clarify the issues, so I can better understand the positions I take.

Finally, I would note that none of these tensions are new with AI. The tensions between research demands, ethics, productivity, and so forth have always existed in product research. However, the confusions and tensions appear to be worsening rapidly. I think that reflects two things:

1. With its expectations, hype, and extremes, AI brings the preexisting tensions into sharper and more intense focus.

2. This is happening at a time of organizational and “cultural” turmoil within tech, where UXRs already feel acutely anxious. With that baseline anxiety and uncertainty, the intensified tensions of AI are more difficult to understand, process, or resolve..

What should we do about it? I don’t have specific recommendations for now. Instead, what I believe is that awareness of the situation is the first step towards finding whatever is next.

I hope these reflections and an outline of the themes will contribute to such awareness. Cheers!

I recently read comments from politicians that “college should prepare students for high paying jobs.” Without getting into politics, it left me wondering: what were the high-paying jobs when I went to college?

In this post, I detail how I found government statistics from 1985 — the year I went to college at age 17 — and used R to extract data from a PDF file and make a simple “opportunity plot.”

I’m not claiming this is a complete (nor even a good) answer to the question about what jobs someone should choose. Instead, it’s a fun way to get data from a difficult source and then look back at jobs data from 40 years ago — and develop some R skills!

As always, I share R code along the way, and I compile it again at the end.

Getting & Cleaning the Jobs Data

The data I used come from the US Bureau of Labor Statistics in a PDF here (as of this writing), published in October 1985 (Prieser, 1985). This PDF file appears to be converted from a scanned document with optical character recognition (OCR).

The tables of interest show job counts and salary, on Page 2. Here is an excerpt, which among other things shows a slight slant to the page reflecting its presumed origin as a scanned document:

What I will do in R is this:

1. Read the job names, counts, and salaries from the PDF

2. Clean up the data

3. Group the rows into higher level categories (“Accountants”, “Chemists”, etc.)

4. Plot a 2×2 showing job counts vs salaries

To get the data from the PDF, I’ll use the tabulapdf package in R (Sepulveda, 2024).

Installation note: tabulapdf uses a Java library, and thus requires a local installation of Java Runtime Engine (JRE) and the Java Development Kit (JDK). That installation process is somewhat complex, so I will skip it here; but there are notes in the Appendix below on how I did it on my Macbook.

First, I download the PDF document from the BLS site and save it locally to ~/Downloads. Using tabulapdf, I extract “Table 2” from that file as two separate tables (one for each column). If you download the PDF and update the filename variable for wherever you put it and name it, then you should be able to follow along.

Here is the code:

filename <- "~/Downloads/bls-report-1985.pdf"  # update for your folder and filename
library(tabulapdf)                             # see Appendix for installation notes
tables <- extract_areas(filename, pages=c(2, 2))
tables[[1]]

In tabulapdf, there are options either to extract tables automatically (extract_tables()) or to tell it exactly where you are interested by selecting rectangular areas with extract_areas(). Our PDF has several tables with complex formatting, so I used extract_areas() to tell it exactly where to look.

In extract_areas(), the option pages=c(2, 2) tells it to look on page 2 and then to let me select 2 areas for extraction. When I run that, page 2 appears in the RStudio viewer. I click and drag to select each of the two areas in turn as shown here:

The initially-converted first column (tables[[1]]) looks like this:

Success! However, there is a lot of junk in the data such as ellipses, extra spaces, a “$” symbol, commas, etc. Next, we’ll clean that up.

To extract data from a different PDF, simply change the filename as above and then alter the pages= argument as needed for your file. Or see the help pages for tabulapdf for more options.

Cleaning the Data

tabulapdf gave us 2 tables, one for each “column” in the PDF. To proceed, we’ll first combine those into one data set and then clean up messy data that comes from 1980s table formatting and OCR scanning.

To combine the two column sets, I select only the columns with data we want — namely, columns 1, 11, and 12 from the left hand part of the table and 1, 7, and 8 from the right hand part. Then I set friendly names and bind them into a single data frame:

df1 <- data.frame(tables[[1]])[ , c(1, 11, 12)]
df2 <- data.frame(tables[[2]])[ , c(1,  7,  8)]
names(df1) <- names(df2) <- c("Job", "N", "Salary")
job.data <- na.omit(rbind(df1, df2))   # removes header rows from the tables

In this case, na.omit() removes rows without complete data (such as within-table header lines).

Next, I remove junk from the data and convert the numeric columns to numbers:

# clean up ellipses in the job names
job.data[ , 1]       <- gsub(" \\.", "", job.data[ , 1])  
# remove nuisance spaces, etc., and convert numeric columns to numbers
job.data[ , c(2, 3)] <- lapply(job.data[ , c(2, 3)],      
                                 function(x) as.numeric(gsub("[^0-9.-]", "", x)))

In this code, the first line uses gsub() to remove all instances of “ .” (the ellipses in job titles). The second line uses gsub() to remove all non-numeric characters from the 2 number columns, and then converts them to numeric data type using as.numeric(). (If you’re wondering, one could use tidyverse functions and/or base R pipes. I default to “very” base R, but everything in R has multiple solutions!)

So far, our data look like this:

Next we’ll group all of the Accountants together, Chemists together, and so forth. For the first step, we split each job title where there is a spaces + parenthesis (“ (“) to separate the parenthetically noted GS scales, and keep only the text that appears before that point, such as “Accountants I”. To do that, I apply str_split() from the stringr package to each job title, using an anonymous function that retains only the first part ([1]) of each split string:

library(stringr)
# Split on "(" [job levels] and only keep the descriptive part before that
job.data$JobGroup <- sapply(job.data$Job, 
                              function(x) str_split(x, " \\(", simplify = TRUE)[1])

Next I remove the Roman numerals (“I”, “IV” etc.) where they appear, and apply trimws() to strip off any left over white space (thanks to Ben Bolker for the regex in this post).

job.data$JobGroup <- trimws(gsub('([IVXLCM]+)\\.?$','', job.data$JobGroup))

While inspecting the data, I see 3 rows where OCR problems made problems for the JobGroup:

So I fix those manually:

job.data[c(96, 106, 107), "JobGroup"] <- c("Purchasing assistants", "Typists", "Typists")

Finally, I convert JobGroup to a factor (nominal) variable and review the result:

# Make the result into a factor variable
job.data$JobGroup <- factor(job.data$JobGroup)
# check the data
summary(job.data)
head(job.data, 8)

It’s working and we see that all of the Accountants are grouped (and Chief accountants are a different group — which is OK for now), and so forth:

Next we’ll compute the total employment and average salary per JobGroup.

For an important project, I’d do a deeper review to make sure I caught all of the data issues. For instance, I see some potential OCR errors like “GS-2” for "Accountants II”. For purposes of this post, I’ll simply go ahead and use the data as extracted to this point.

Grouping the Data

Now that we have assigned job groups, we need to aggregate them for total employment (sum of the N column per group) and average salary (taking the weighted average of Salary x N).

The tricky part is how to apply the weighted.mean() function in R across multiple groups. As always, there are multiple options (tidyverse, by(), a custom function, etc.) but a simple base R solution is to use an index column inside aggregate(). Here’s the code:

# tip from https://stackoverflow.com/questions/33692439/using-aggregate-to-compute-monthly-weighted-average 
job.data$row  <- 1:nrow(job.data)
# get the weighted mean salaries per group
job.sum       <- aggregate(row ~ JobGroup, job.data, 
                           function(i) weighted.mean(job.data$Salary[i], job.data$N[i])) 
# rename the "row" column
names(job.sum)[2] <- "AvgSalary"

In this code, the current row(s) that correspond to each JobGroup are passed by aggregate() to the anonymous function, which then returns a weighted.mean() for the variables of interest in those rows.

The next step is to sum up the number of jobs in each JobGroup, which is a simple aggregate(…, sum):

# find the total number of jobs per group
job.sum$Total <- aggregate(N ~ JobGroup, job.data, sum)$N

That’s it! Now we have data aggregated by job group with counts and weighted average salaries:

Next I’ll plot those.

Making an Opportunity Plot

A simple thought is that the “best jobs” are those that have the highest combination of availability and salary. (I’ll leave aside the question of future-looking availability and salary!) A 2×2 interpretation of a scatter plot is one way to examine that.

I plot average salary vs. job count in the BLS data as follows:

library(ggplot2)
library(ggrepel)
library(scales)
p <- ggplot(data=job.sum, 
            aes(x=Total, y=AvgSalary, label=JobGroup)) +
  geom_point(color="red") +
  geom_text_repel() +
  scale_x_log10(labels = label_number()) +
  xlab("Total Employment (log scale)") +
  ylab("Average Salary, weighted across levels") +
  theme_minimal() +
  ggtitle("Salary vs. Employment, US Statistics for 1985")

p

Here are a couple of notes on that code. First, it uses ggrepel (Slowikowski, 2024) to place the text labels (job groups) in readable positions. Second, it puts the X axis on a log scale for compactness; otherwise the engineers group would distort it substantially. Third, it uses scales (Wickham et al, 2023) to label the X axis more legibly. Finally, theme_minimal() removes some chart junk.

Here’s the result:

We see two particularly interesting groups. In the upper right hand side, there is a group of relatively to very common professions that are also relatively highly paid in 1985: Engineers, Systems analysts, and Chemists. In the upper left, there is a group of less common professions that are even more highly paid: Chief accountants, Directors of personnel, and Attorneys. (Those are likely all true today, as well!)

What would we do for further analysis? First, I’d want to include additional data that (presumably) is in other BLS data sets. For example, health care and education professionals, among others, are not included here. Second, we might use their data (page 1 in the PDF) on trends and changes to project forward.

Third, within the data set, we could consider some sort of implied transition among career levels to look at more of a career-spanning set of expectations. For example, the movement from one level to a higher one, within a job group, might be modeled as a Markov chain (somehow combining data across years, and/or within a data set, and/or using the change metrics — problems one would want to consider). Fourth, we could adjust salary levels to make them comparable across years, using inflation-adjusted values or the like.

Back to 1985

What about my choices in 1985? My personal college major was not influenced at all by expectation of salary or availability! I completed a double major in Psychology and Comparative Religion, and went (initially) to graduate school in Philosophy. That all reflected personal interest and a planned academic career. Eventually I changed to graduate school in Clinical Psychology. Data such as these would have been of no use or concern to 17 year old me (except as a matter of curiosity — which I still have).

What I can say, 40 years later, is this: I ended up in an uncommon profession (Quant UX Research) that is also highly paid — and was nowhere to be found in the world or data of 1985! My completely non-job-related education was perfect preparation in my case. I’ll write more about that another time.

I would, at a first approximation, do exactly the same thing again. But the data are still interesting!

Conclusion

This post demonstrated how to get data from a PDF — even a messy, slanted, older, erratically-OCR’d, scanned PDF — into R. And that let us have some fun (at least in my opinion) to look at jobs from way back when I started college.

Cheers and I hope you find it useful!

Appendix: Notes on installing tabulapdf and its Java dependencies

As I noted above, tabulapdf requires Java — both the runtime engine (aka “Java”, aka “JRE”) and the developer kit (JDK) — and the integration of those with R using rJava (Urbanek 2024). Installing those may be tricky depending on the details of your system.

On my M2 Macbook, here’s what I did. The rJavaEnv package (Kotov, 2024) was a particularly nice & helpful touch to install the JDK and get everything set up inside R. The steps:

### Appendix: setting up rJava on Mac OS X machine

# This is the sequence I used on MacBook with M2, may vary with other systems

# A1. Install Java runtime (if needed; usually is)
# ... https://www.java.com/en/download/
# ... installed relevant JRE (Java 8 Update 441, for Mac 64-bit ARM)

# A2. install the rJava library in R
install.packages("rJava")
# check that rJava is working:
library("rJava")

# A3. install Java development kit using rJavaEnv in R
#    (rJavaEnv helps get everything working correctly)
install.packages("rJavaEnv")
library(rJavaEnv)
java_quick_install()        # gets and installs the JDK, then sets system pointers
java_check_version_rjava()  # should show "21" + "21.0.6" or some similar/later version

# A4. install Tabula PDF library and dependencies needed for interactive usage in R
install.packages("tabulapdf")
install.packages(c("shiny", "miniUI"))

# A5. quit R / RStudio, reboot Mac to make sure everything is squared away and reloaded

# A6. test installation using tabulapdf data set
library(rJava)
library(tabulapdf)
# the following code is updated from tabulapdf vignette
f   <- system.file("examples", "mtcars.pdf", package = "tabulapdf")
out <- extract_tables(f)
str(out)
out[[1]]   # the "mtcars" data

If you run into issues, my main recommendations are (1) to update everything, (2) to make sure all installers (Java, R, and RStudio) are using matching CPU builds (e.g., 64-bit ARM), (3) reinstall, reboot, and (4) search for answers online.

I will note that there is not a lot of help online for debugging Java integration with R. It’s a bit of an edge case, although a highly useful one as we’ve seen.

All the Code

As always, I share all of my R code in one place for simplicity. Here it is, including the Appendix:

# Using R to parse 1985 jobs stats in a PDF
# Chris Chapman, March 2025
# for quantuxblog.com

# NOTE: the tabulapdf package requires Java; see appendix for setup notes

# 1. load BLS data for 1985
# get job data from US Bureau of Labor Statistics
# note: was working as of March 5, 2025
filename <- "~/Downloads/bls-report-1985.pdf"   # update for your folder and filename
library(tabulapdf)                              # see Appendix for installation notes
tables <- extract_areas(filename, pages=c(2, 2))
tables[[1]]

# 2. convert those to usable R format as a single data set
#    get the 3 main columns; bind them to 1 data frame
df1 <- data.frame(tables[[1]])[ , c(1, 11, 12)]
df2 <- data.frame(tables[[2]])[ , c(1,  7,  8)]
names(df1) <- names(df2) <- c("Job", "N", "Salary")
job.data <- na.omit(rbind(df1, df2))   # removes header rows from the tables

# 2.1 clean up the data to remove spaces, etc.
# clean up ellipses in the job names
job.data[ , 1]       <- gsub(" \\.", "", job.data[ , 1])  
# remove nuisance spaces, etc., and convert numeric columns to numbers
job.data[ , c(2, 3)] <- lapply(job.data[ , c(2, 3)],      
                                 function(x) as.numeric(gsub("[^0-9.-]", "", x)))
job.data

# 2.2 create job group factor that collapses the levels
library(stringr)
# Split on "(" [job levels] and only keep the descriptive part before that
job.data$JobGroup <- sapply(job.data$Job, 
                              function(x) str_split(x, " \\(", simplify = TRUE)[1])
# Remove the Roman numerals from the descriptions (another part of job levels)
job.data$JobGroup <- trimws(gsub('([IVXLCM]+)\\.?$','', job.data$JobGroup))

# And finally, fix 3 extraction errors manually
job.data[c(96, 106, 107), ]
job.data[c(96, 106, 107), "JobGroup"] <- c("Purchasing assistants", "Typists", "Typists")
# Make the result into a factor variable
job.data$JobGroup <- factor(job.data$JobGroup)
# check the data
summary(job.data)
head(job.data, 8)

# 3. get the total employment and weighted salaries per group
# create a row counter we can use to index the data when aggregating by group
# thanks for this tip, https://stackoverflow.com/questions/33692439/using-aggregate-to-compute-monthly-weighted-average 
job.data$row  <- 1:nrow(job.data)
# get the weighted mean salaries per group
job.sum       <- aggregate(row ~ JobGroup, job.data, 
                           function(i) weighted.mean(job.data$Salary[i], job.data$N[i])) 
# rename the "row" column
names(job.sum)[2] <- "AvgSalary"
# find the total number of jobs per group
job.sum$Total <- aggregate(N   ~ JobGroup, job.data, sum)$N

head(job.sum)


# 4. plot "opportunity" as # jobs vs average salary
library(ggplot2)
library(ggrepel)
library(scales)
p <- ggplot(data=job.sum, 
            aes(x=Total, y=AvgSalary, label=JobGroup)) +
  geom_point(color="red") +
  geom_text_repel() +
  scale_x_log10(labels = label_number()) +
  xlab("Total Employment (log scale)") +
  ylab("Average Salary, weighted across levels") +
  theme_minimal() +
  ggtitle("Salary vs. Employment, US Statistics for 1985")

p

### Appendix: setting up rJava on Mac OS X machine

# This is the sequence I used on MacBook with M2, may vary with other systems

# A1. Install Java runtime (if needed; usually is)
# ... https://www.java.com/en/download/
# ... installed relevant JRE (Java 8 Update 441, for Mac 64-bit ARM)

# A2. install the rJava library in R
install.packages("rJava")
# check that rJava is working:
library("rJava")

# A3. install Java development kit using rJavaEnv in R
#    (rJavaEnv helps get everything working correctly)
install.packages("rJavaEnv")
library(rJavaEnv)
java_quick_install()        # gets and installs the JDK, then sets system pointers
java_check_version_rjava()  # should show "21" + "21.0.6" or some similar/later version

# A4. install Tabula PDF library and dependencies needed for interactive usage in R
install.packages("tabulapdf")
install.packages(c("shiny", "miniUI"))

# A5. quit R / RStudio, reboot Mac to make sure everything is squared away and reloaded

# A6. test installation using tabulapdf data set
library(rJava)
library(tabulapdf)
# the following code is updated from tabulapdf vignette
f   <- system.file("examples", "mtcars.pdf", package = "tabulapdf")
out <- extract_tables(f)
str(out)
out[[1]]   # the "mtcars" data

References

Chang W, Cheng J, Allaire J, Sievert C, Schloerke B, Xie Y, Allen J, McPherson J, Dipert A, Borges B (2024). shiny: Web Application Framework for R. R package version 1.10.0, https://CRAN.R-project.org/package=shiny.

Cheng J (2018). miniUI: Shiny UI Widgets for Small Screens. R package version 0.1.1.1, https://CRAN.R-project.org/package=miniUI.

Kotov E (2024). rJavaEnv: Java Environments for R Projects. doi:10.32614/CRAN.package.rJavaEnv https://doi.org/10.32614/CRAN.package.rJavaEnv, https://github.com/e-kotov/rJavaEnv.

Prieser C (1985). “Occupational salary levels for white-collar workers, 1985”. Monthly Labor Review, Bureau of Labor Statistics, October 1985. At https://www.bls.gov/opub/mlr/1985/10/rpt1full.pdf, retrieved March 5, 2025.

R Core Team (2025). R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria. https://www.R-project.org/.

Sepulveda MV (2024). tabulapdf: Extract Tables from PDF Documents. https://github.com/ropensci/tabulapdf.

Slowikowski K (2024). ggrepel: Automatically Position Non-Overlapping Text Labels with 'ggplot2'. R package version 0.9.5, https://CRAN.R-project.org/package=ggrepel.

Urbanek S (2024). rJava: Low-Level R to Java Interface. R package version 1.0-11, https://CRAN.R-project.org/package=rJava.

Wickham H (2016). ggplot2: Elegant Graphics for Data Analysis. Springer-Verlag New York.

Wickham H, Pedersen T, Seidel D (2023). scales: Scale Functions for Visualization. R package version 1.3.0, https://CRAN.R-project.org/package=scales.

Wickham H (2023). stringr: Simple, Consistent Wrappers for Common String Operations. R package version 1.5.1, https://CRAN.R-project.org/package=stringr.

• Post 1 discussed the data — real data on UXRs’ preferences among Quant classes on a MaxDiff survey — and it reviewed the stack rank of preferences and the individual distribution of preferences.

• Post 2 examined patterns of correlation among UXRs’ class preferences.

• Post 3 discussed one process to look for useful Segments among respondents.

In this 4th post, I examine the Root Likelihood (RLH) fit measure, which summarizes how well MaxDiff (or Conjoint) utilities fit the observed data from survey respondents. It’s a long post, partly because the concept needs explanation, but also because I want to share closely-related ideas along the way.

This is only a partial discussion of quality assessment overall, because there are many ways to review data quality and model fit. You should add RLH inspection to other methods you might already use, such as identifying speeders. (However, I have warnings below about filtering respondents!)

As always, I share R code along the way. You can follow along live with the actual data — thanks to the generous sharing of anonymized data by the Quant UX Association!

Get the Data to Follow Along

The data here are individual estimates of interest in Quant UX training classes for N=308 mostly UXR respondents, as assessed on a MaxDiff survey. You can find the details in Post 1.

The following code loads the data needed for this post, and renames the variables to be shorter & friendly:

# get the data; repeating here for blog post 4, see post 1 for details
library(openxlsx)   # install if needed, as with all package calls
md.dat <- read.xlsx("https://quantuxbook.com/misc/QUX%20Survey%202024%20-%20Future%20Classes%20-%20MaxDiff%20Individual%20raw%20scores.xlsx")   # 
md.dat$Anchor <- NULL
names(md.dat)[3:16] <- c("Choice Models",  "Surveys",       "Log Sequences",    "Psychometrics", 
                         "R Programming",  "Pricing",       "UX Metrics",       "Bayes Stats",
                         "Text Analytics", "Causal Models", "Interviewer-ing",  "Advanced Choice", 
                         "Segmentation",   "Metrics Sprints")
classCols <- 3:ncol(md.dat)    # generally, Sawtooth exported utilities start in column 3

Model Fit Background: What is RLH?

[This section is long because I explain RLH conceptually, using simple examples. If you already understand RLH (or don’t care!), skip to the next section.]

RLH stands for “Root Likelihood” and it expresses the degree to which a fitted model matches observed data from respondents. RLH is usually calculated for observations in the training data (those for whom a model was fit). That’s because those are the observations for which we have individual level estimates.

Side note: in principle we could calculate RLH for any set of observations given a set of utility scores. For instance, we might calculate how well new observations match the utilities for a particular segment or an overall sample. That is rarely done in my experience, so I’ll just note it and not explore further.

Let’s break down the term “root likelihood” to see what is going on. First: "likelihood”. For a single observation, the likelihood is the odds that the single observation would have occurred, according to our model. For example, suppose we have a MaxDiff set of items A, B, C, and D. Further imagine that we have the following utility scores: Item A is preferred for a particular individual, with a utility of +1.0, while items B, C, and D have utilities +0.5, -0.8, and -1.2 respectively.

Given those utilities, and under the MNL (multinomial logit) share-of-preference formula, the preference share for item A for this respondent, as calculated in R syntax, is exp(1.0) / sum(exp(c(1.0, 0.5, -0.8, -1.2))) or 0.53. That means that on a task showing A, B, C, and D, the respondent is estimated to have a 53% chance of choosing their most preferred item, Item A. There leaves an estimated 47% chance they would choose one of the other options (most likely Item B, but potentially C or D).

If the respondent did in fact choose Item A, then the likelihood value of that observation is 0.53 (the relevant share of preference for that choice). In this case, the model is doing pretty well because it correctly predicted the most likely choice.

On the other hand, if they chose Item D, then the likelihood — according to the estimated utilities — would be exp(-1.2) / sum(exp(c(1.0, 0.5, -0.8, -1.2))) or a 5.9% chance and a likelihood value of of 0.059. In this case, unlike the odds for Item A above, the model is not doing well because its “agreement” with the actual choice was only 0.059.

In short, the “likelihood” part of the term calculates the odds we should have seen each individual response, according to the model. When the value is higher, it means that the data we saw was more “likely” according to the model.

Next: “root”. Given multiple observations, we can calculate their joint likelihood by multiplying the separate odds. For example, suppose that we ask the exact same two tasks in a row as we listed above, and that our respondent chooses A on one, but D on the other task. The likelihood of those two observations together is 0.53 * 0.059 or 0.031. But that number is across two trials. To get a single estimate, we take its square root, or sqrt(0.53 * 0.059) which is 0.177. That means that the root likelihood (RLH) of our two observations, given the estimated model, is RLH = 0.177.

Side note: you might notice that taking the simple product of multiple likelihood scores assumes that they are probabilities of independent events … but are the events here “independent”? Isn’t it the case that choosing A on one task is highly correlated with the odds of choosing A on another task?

The answer, conceptually speaking, is that dependence across tasks is already included in the model when it estimates the utility scores. Given the fact that we have estimates for A, B, C, D, and so forth, we can now ask, in effect, “what happens when we draw random samples of comparisons?” (the tasks that were given to the respondents). There is no dependence in how those tasks were drawn — they are independent of the model — so the probabilities can be considered independent for this purpose.

To be sure, one might pose higher-order questions about interactions, such as the degree to which the exact task order may influence responses, such that they interact with the model in a non-independent way. However, such questions rapidly become unanswerable — and most likely would have little effect on a model fit score. A general assumption of independence between tasks seems to work well.

If we have more than two observations, as we usually would in a MaxDiff survey, we just take an appropriate fractional exponent instead of the square root. If we have 10 trials, we would multiply 10 separate odds and then take the 10th root (exponentiate to 1/10) of the product.

In R, such a calculation looks like this:

(individualOdds <- seq(0.01, 0.99, length=12))  # placeholder for the likelihoods
(rlh <- prod(individualOdds) ^ (1/length(individualOdds)))

In the first line of this code, I create a set of fake odds between 0.01 and 0.99, so we will have values to work with. In real data the odds would be calculated from utilities and observations as noted above. The second line is the key point: it computes the root likelihood. First it multiplies all the odds (prod()) and then takes the Xth root of them (^(1/length(individualOdds))).

Side note about that calculation using prod() for repeated multiplication: this is the didactic & conceptual way to calculate RLH. See the code below for a better computational version.

In case you’re wondering … no, you don’t have to calculate RLH on your own from the data. A platform like Sawtooth Software Discover or Lighthouse Studio will calculate RLH and include it automatically in the individual level data. I’m showing the calculations only for one reason: so you will know how it works!

By the way, RLH works exactly the same way for conjoint analysis. After estimating a conjoint utility model, we can calculate the odds of each observed choice based on those utilities, and then take the Xth root to calculate RLH.

What You Can Do with RLH

In general, RLH can be used in two ways:

• As a high level signal of quality, namely whether your data are markedly suspect

• As a way to filter out potentially non-informative respondents

As for quality, I don’t say that RLH indicates whether a data set or a respondent is “good.” No single metric can assess that. However, if RLH is markedly low — as we explore below — then it indicates potentially serious problems. In particular, if the median RLH in a sample falls below a reasonable threshold (keep reading) then you have a problem such as inconsistent respondents, a poor survey, or a poor model.

Beyond the high level diagnostic value, some analysts use RLH to filter out so-called “bad” respondents. Typically this is done by setting a minimum threshold for an acceptable RLH (again, keep reading!), removing respondents who are below that value, and then estimating the model again without them.

In practice, I almost never remove respondents due to RLH for four reasons:

1. RLH does not identify whether responses are “bad”. It only says how likely they are, conditional on the model that was estimated from them. I discuss this below at some length, near the end of this post.

2. Instead of removing respondents, I try instead to use high quality data sources. I don’t believe analytics can “rescue” bad data sources.

3. Choice models are robust to random answers, straight lining, speeding, and similar “bad” response patterns. So, if even a modestly large number of respondents are answering in those ways, it is unlikely to seriously affect the ultimate estimates (at least at the level of sample averages).

4. Any method to remove “bad” respondents introduces other sources of bias. As one minor example: why not remove excessively high RLH respondents who are “too good”? Again, I say more below.

A counterargument to my position is the following: if a respondent’s RLH is near the value of “random” observations, then we are not learning much from them, so we might as well remove them from the model to be more precise and efficient. For reasons I discuss below, that is not the complete story and there is more to consider. We’ll get to that!

Meanwhile, before acting on RLH, we need to inspect it. We’ll do that next.

Examining RLH in the UXR MaxDiff Data

Now that you know how RLH works and what you can do with it, let’s look at the values from the Quant UX Association MaxDiff for N=308 respondents. Sawtooth’s data file helpfully includes the individual RLH values when it exports individual level data, so you don’t have to calculate anything.

Using the data set loaded above, we can set a friendly column name and start exploring:

names(md.dat)[2] <- "RLH"
summary(md.dat$RLH)

In our data, the MaxDiff RLH values range from 0.32 to 0.80, with a median of 0.60:

As always, it’s important to plot the values. The ggridges package makes nice density plots with the geom_density_ridges() function, including an option to fill in the individual points (jittered so they don’t overlap under the curve):

# density plot of the RLH values
library(ggplot2)
library(ggridges)
set.seed(98101)    # jittered points are slightly randomized
p <- ggplot(data=md.dat, aes(x=RLH, y=1)) +
  geom_density_ridges(jittered_points=TRUE, 
                      alpha=0.5, colour="darkblue") +
  xlab("Model fit score (RLH; > 0.3 is good)") +
  ylab("Relative prevalence (density)") +
  theme_minimal()

p

In the code above, I set y=1 because ggridges generally assumes that you have multiple series to plot overlapping density curves, identified with a nominal y variable. In this case, we have only one series, so I set a single dummy y value.

Here’s the result:

How do we interpret this chart? Well, I’ve already labelled the X axis with “> 0.3 is good” … but the reason for that label will have to wait until the next section.

Accepting that cutoff value for a moment, we see a few things in the chart:

• All of the individual fit values are “good” according to the purported cutoff value of 0.3.

• The curve is nicely Gaussian (normal) in shape, which is a good thing … for reasons I won’t go deeply into. (Hint: it relates to assumptions of the HB model as well as random tasks, and assumptions of random variation across people.)

• There is a small elevated tail on the lower end with RLH < 0.4. Those come from 2% of respondents for whom the model didn’t fit as well. We don’t know exactly why not, but perhaps they sped through it, or were distracted, or didn’t find the items or topic as relevant as other respondents did.

So we’ve seen the results and understood how they are calculated. But what is a “good” value for RLH? Is it OK that our median is 0.60? Why not 0.30 or 0.90?

In the next section, I discuss the threshold levels that make RLH “good” or not.

Are Those Values Good or Bad? What should we Expect?

In this section, I briefly discuss 2 heuristics that build intuition around RLH expectations. In following sections, I’ll look at a code-based analysis in R; and then mention another approach suggested by Sawtooth that uses random data.

For the first heuristic, a “good” value for RLH will be substantially better than random chance. For a MaxDiff (or Conjoint) task, one way to consider random chance would be if every item had the same likelihood of being chosen. For example, if a MaxDiff task has C=5 items on each screen, then we might say that each one has a random likelihood of 1/C = 1/5 = 0.20.

Put differently, our model should get 20% of the choices right merely by picking an item from 1-5 randomly. We want to do substantially better than that … let’s say we want to do “50% better.” That’s a heuristic cutoff but agrees with a lot of experience. Then we could set a minimum likelihood of RLH = 0.30 to be better than that form of random chance.

For the second heuristic, we note that the one above is a somewhat simplistic version of random chance. We might instead say, “we know that items will vary in preference, so there’s no way they will all have 1/C odds.” A different way to look at random chance incorporates that concept. This will help us look not at the minimum acceptable RLH (as above) but at what we might expect for a good RLH value.

Suppose we assume that 5 items will have relative preference of (1, 2, 3, 5, 8). I’m making that up (using the Fibonacci sequence) just as a heuristic example. However, it’s not an unreasonable set because real world results often find 1-3 items that are “winners” plus a long tail of less-preferred items.

In R, we could calculate the RLH expected given those odds. We will assume that the model knows those odds that they perfectly match the item chosen on the tasks (i.e., that the respondent chose the item that aligns with the most-likely item). Here’s the code in R:

# back of the envelope
# pick some preference distribution and calculate odds
itemOdds <- c(1, 2, 3, 5, 8)           # relative preference for 5 items
itemOdds <- itemOdds / sum(itemOdds)   # rescale so they sum to 1.0
max(itemOdds)          # relative odds for the most-preferred item

That is a value of itemOdds = 0.42. So — under this one set of preferences, a model that correctly predicted the choice of the most-preferred item — would have a likelihood or RLH of 0.42. That suggests heuristically that an RLH of approximately 2/C (2/5 = 0.40) may be pretty good.

BTW, you might note that this also means that a perfect model, given those preferences, would be wrong in its predictions ~60% of the time. We can’t expect our models to be extremely accurate about point predictions exactly because people’s preferences are tendencies — they are not 0/1 values. For example, someone who would register for one class may also have a high chance of registering for another one. Choice models are helpful because they tell us what is more or less likely, on average, and how options compare to one another … not because they exactly predict individual events.

Putting this value together with the previous minimum heuristic, where 1.5/C may be a lower cutoff, we could form a range bounded like this:

• Rough cutoff: < approx 1.5/C. For example, with 5 items shown at a time: 1.5/5 = 0.30. For 4 items shown at a time: 1.5/4 = approx 0.375 as a lower minimum.

• Clearly good: = approx 2/C. For example, with 5 items: 2/5 = approx 0.40.

In other words, RLH is heuristically “good” when it is in or above the range of (0.30 — 0.40) when we test 5 items at a time.

Are you thinking, “Wait! That’s a pretty low value. I would have expected a good model to have RLH of 0.80 or 0.90 or 0.99!”

You’re right — it is not a high value. We’ll see more about that below. Meanwhile, it is always important to remember that respondents usually find multiple items to be interesting, and might prefer one or another more or less randomly. For example, I like Dr. Pepper Zero soda … but a model of my preference is not 100% Dr. Pepper! I often drink other things, especially water, but also coffee, Gatorade, tea, Zevia soda, and so forth. That means we should not expect a model to do any better than to capture my most likely choice, at the level of its relative share — a likelihood much lower than 1.0.

Key point: these heuristic results should help you build intuition about why RLH will not be especially high in real data.

You might wonder next, “Can we quantify that better, not just using a thought exercise?” We’ll do that systematically in the next section.

Deep Dive: R Code to Simulate RLH Values

This section is a moderately deep dive into R simulation! If you are satisfied with the heuristics in the previous section, you can skip it … unless you like R as much as I do.

The previous section described why RLH in the range of RLH > roughly 1.5/C — which is to say, 0.30 for a MaxDiff with 5 items shown at a time — may be a reasonable minimum cutoff for a good respondent using heuristic logic. We also saw that RLH ~ 2/C — or 0.40 for 5 items — may indicate a particularly good fit between the observations and model. Now we’ll examine the expectation more systematically using code.

Here’s what I’m going to do. The short version is that I’ll do many, many random iterations of the kind of heuristic I showed above, for different sets of preference values. That will inform us as to the generally expected range of RLH values when the observations and data match perfectly.

More specifically, the code will simulate “respondent utilities and choice” as follows:

• Draw utilities as random normal values with mean=0 and some standard deviation (I’ll come back to that). This mirrors the assumptions of the HB models used to estimate MaxDiff utilities, so the values will be realistic.

• Draw random sets of “items” from those, 5 at a time, to mimic the utilities that might apply to a particular MaxDiff task.

• For each set of 5 item utilities, calculate the share of preference for the single most-preferred item. That gives us the highest possible likelihood value for that task (if the respondent always chose the most-preferred item, according to the model)

• Do the above using K [“nItems”] = 14 items (because that matches the number of items on the Quant UX survey that provided our data)

• For each simulated “respondent”, do this for C [“nShown”] = 5 items at a time on each screen, for S [“nScreens”] = 6 total screens (because that matches the length of the Quant UX survey).

• Simulate all of the above a total N [“nIter”]=10000 times (“respondents”)

Here is code that sets up the initial conditions for all of that. It also sets a random number seed to make the analysis repeatable:

nIter    <- 10000            # how many times to sample nItem MaxDiff simulations
nItems   <- 14               # number of items in a MaxDiff set
nShown   <- 5                # number of items shown on a single screen
nScreens <- 6                # number of screens in our survey
rlhDraws <- rep(NA, nIter)   # hold the results for each iteration
set.seed(98250)              # make it repeatable

Technical note: although MaxDiff utilities are always zero-centered (in their “raw” form estimated by an HB model), utilities vary from study to study and item to item in their standard deviation (SD). To draw random zero-centered, simulated utilities, we need to specify that variance. To get an SD value for this simulation, I look empirically at the Quant UX data and use the median SD (drawSD) for the items in that study:

summary(unlist(lapply(qux.md[ , classCols], sd)))   # answer: somewhere around sd ==(2.2, 2.6)
(drawSD  <- median(unlist(lapply(qux.md[ , classCols], sd))))

Now we’re all set up and ready to run the simulation. Following is the code. First I’ll give the whole chunk, and then I’ll discuss it below.

for (i in 1:nIter) {
  # set up 1 respondent "trial"
  pwsResp     <- rnorm(nItems, mean=0, sd=drawSD) # "nItems" random normal, zero-centered simulated part worths
  pwsResp     <- scale(pwsResp, scale=FALSE) # recenter to make sure they're zero-sum

  # iterate over nScreen tasks per respondent
  drawMax <- rep(NA, nScreens*2)             # hold the results across the screens for one respondent
  for (j in 1:nScreens) {
    pwsDrawn    <- sample(pwsResp, nShown)   # get the part worths for 1 simulated task  
    # "best" choice
    pwsExp      <- exp(pwsDrawn)             # exponentiate those part worths
    pwsMaxShare <- max(pwsExp) / sum(pwsExp) # our best (most likely) prediction would be the max utility item
    drawMax[j*2 - 1]  <- pwsMaxShare
    # repeat for the "worst" choice
    pwsExp        <- exp(-1 * pwsDrawn)      # exponentiate part worths for the "worst" choice direction
    pwsMaxShare   <- max(pwsExp) / sum(pwsExp)   
    drawMax[j*2]  <- pwsMaxShare
  }
  # calculate RLH from those observations
  rlh         <- exp(sum(log(drawMax[drawMax > 0]), na.rm=TRUE) / length(drawMax))    # safer equivalent of "prod(drawMax) ^ (1 / length(drawMax))"
  rlhDraws[i] <- rlh
  # add a bit of error checking, just in case of an off by one error etc :-/
  if (length(drawMax) != nScreens * 2) warning("The vector of partworths is off somewhere!", i, ":", drawmax)
}

This code has five main parts:

1. An outer loop iterates over the 10000 (nIters) simulated “respondents” (Line 1). For each of those iterations:

   1. It draws zero-centered utilities for the 14 items (Lines 3-4) for that 1 simulated respondent

   2. An inner loop iterates over 6 MaxDiff choice “trials” (Line 8), and inside that:

      1. It makes a trial with 5 randomly chosen items to be compared on 1 choice task (Line 9)

      2. It chooses the most likely Best & Worst on that random trial and calculates the likelihoods for those choices (Lines 11-17).

      3. When it’s done with the 6 tasks, it calculates RLH for the set of 12 observed Best & Worst “choices” for that respondent (6 Best + 6 Worst), and saves that result (Lines 20-21)

The error check line at the end was just some defensive coding that occurred to me along the way (nothing in particular prompted it; I just like to code defensively). It never triggered.

Side note: this code assumes that the overall, upper level means of all items are also zero. One could instead structure those; that’s a longer discussion. A more general approach would be to iterate over different sets of part worth scores; different survey lengths; and different sizes of choice sets. The code above could serve as the basis for a function that is refactored for use inside a larger and more general set of loops and/or parameter variations. That would be a good exercise!

BTW, you will see that RLH here is computed with a different formula than above (but one that is mathematically equivalent). Why? Because, when many odds are multiplied together, they will quickly underflow real number calculation engines. By taking the log() of them and then adding the logs rather than multiplying, we avoid that problem.

Side note: R can handle multiplying 12 probabilities — but if the survey were longer, then we could have problems. It’s always good to anticipate such things and proactively avoid them when coding!

Results from the Simulation

After running this code, we have RLH scores for 10000 simulated “respondents”. As a reminder, the code has assumed that all of the “observed” choices aligned perfectly with the most likely, most preferred items, according to the simulated individual level utilities. Thus it sets an upper bound for expectations of RLH. (Caveat: this assumes there is no “none of the above” option. I discuss that below.)

Here’s code to plot the results, slightly adapting the density chart code from above:

# density plot
set.seed(98195)    # jittered points are slightly randomized
p <- ggplot(data=as.data.frame(rlhDraws), aes(x=rlhDraws, y=1)) +
  geom_density_ridges(jittered_points=TRUE, 
                      alpha=0.1, colour="darkblue") +
  xlab("Model fit score (RLH; > 0.3 is good)") +
  ylab("Relative prevalence (density)") +
  ggtitle("RLH Simulation, Perfectly Made Choices (N=10000)") +
  theme_minimal()

p

And the resulting chart:

This tells us that when respondents make choices perfectly in alignment with their estimated preferences — with other assumptions about utility scores and survey parameters, as above — we can expect those upper-bound RLH values to fall between roughly 0.4 and 0.9, with most falling between 0.5 and 0.8.

In other words, if we have no other information except that a respondent shows RLH=0.5 or RLH=0.4, it could in fact be a maximum, perfect RLH score for them. In general, we should not expect — and instead, should be surprised, because it is near “perfect” — to see RLH >= 0.6 or so.

As we saw above, our actual UXR data had median RLH = 0.60. That suggests that our respondents are not too far below “perfect” in their response patterns.

Put differently, UXRs are great respondents when a survey is relevant and motivating for them! (In a consumer sample, I would expect substantially lower RLH, more like a median of 0.40.)

We can compare the two sets of results — actual vs. UXR — using the same kind of density plot. Here’s the code:

# combined density plot (using random subset of simulated data)
compare.df <- data.frame(RLH    = c(md.dat$RLH, rlhDraws),
                         Source = rep(c("UXR", "Sim"), 
                                      times=c(length(md.dat$RLH), 
                                              length(rlhDraws))))

# density plot
set.seed(98102)    # jittered points are slightly randomized
p <- ggplot(data=compare.df, aes(x=RLH, y=Source, color=Source)) +
  geom_density_ridges(jittered_points=TRUE, 
                      alpha=0.1) +
  xlab("RLH") +
  ylab("Relative prevalence (density)") +
  ggtitle("RLH Comparison: UXR Data vs. 'Perfect' Simulation") +
  theme_minimal()

p

In this code, I first combine the two data sources into a single data set. Then I use the same density plot functions, adding the “Source” of each data point (UXR data vs. simulation) as the y variable, which will stack the results as two density curves.

Here’s the chart:

As you can see, the UXR data is pretty close to “perfect.” Again, I would not expect this in most data sets! Usually I would be happy with median RLH > 0.4 or so. But this comparison highlights once more what we might expect as the upper bound of reasonable RLH.

Looking a bit further, we see an interesting suggestion of tri-modality in the UXR distribution of RLH. On the right hand side (peaking ~0.65), there is a suggestion of a group that looks exactly like the “perfect” data. Then there is a middle peak suggesting a group (peaking ~0.60) that looks slightly less than perfect. Finally, there is a left hand peak (peaking ~ 0.52) that shows lower RLH for some reasons (there are many) — which is not at all “bad” by any means, but contrasts the two higher modal peaks.

Overall I wouldn’t make much of that observation; I’m just highlighting it as a signal of potential additional questions we might have about respondent style, sources, quality, etc., in a sample.

Another Approach: RLH Cutoffs via Random Data

Another approach to using RLH as a cutoff comes from the choice modeling experts at Sawtooth (Orme, 2019 PDF). They noted two things:

1. RLH for “random” responses can vary on a given survey due to the details of the survey’s experimental design (e.g., some tasks are “harder” or “easier” relative to the model overall)

2. Their platform allows creation of random “respondents” who take the actual survey.

Putting those together, they suggested using the random respondent method in their platform to empirically determine a point at which the RLH for a particular respondent is clearly below the expectation for a random respondent. Specifically, identify a percentile cutoff, such as the 95th percentile (highest 5%) of RLH in the simulated random respondents. After identifying that point, filter out respondents below that; and then run the overall model again without them.

For step by step details, see the excellent white paper (Orme, 2019 PDF). Unlike the heuristic R code above, this method exactly fits your own survey. (BTW, that method could be used in any platform that offers random responses plus individual RLH calculation.)

However, I will also note that one should be cautious about filtering — as Orme notes in the white paper and as I will discuss next.

Caveats: RLH Fit Values and Filtering Responses for “Data Quality”

Before we finish, I want to clarify two things: that RLH is not about respondents but is about the fit between observations and model; and that filtering respondents with any method is risky.

RLH is not about respondent quality

Throughout the post, I’ve described how RLH relates to the mutual likelihood of a model and observations. I want to be very clear that this does not necessarily say anything about respondent quality.

Why not? Consider an edge case (actually somewhat common) when a respondent accurately reports that they do not like any of the items on a MaxDiff. Perhaps they are completely uninterested in the features or project … and they truthfully and diligently report that disinterest. Suppose that the model accurately says that their utilities are ~0 (±error) for every item.

In that case, the RLH for this diligent respondent on a C=5 items-at-a-time survey would approximate exp(0) / (5*exp(0)) = 1/C … indistinguishable from the expectation for a completely random respondent!

That means that low RLH is only a signal that something is wrong. It may be that the respondents are answering randomly. Yet it may be that your items are unappealing, or that you are targeting the wrong audience, and respondents are accurately communicating that. In this case, filtering out low RLH respondents could be the exact wrong thing to do, because their disinterest may be important to know.

That doesn’t mean that RLH is useless. Rather, RLH is one diagnostic signal, and needs to be placed into context with other signals (such as observations from an in-person pilot study, which I always advocate).

BTW, RLH can also be exceptionally high for similar reasons — which may happen in particular when a conjoint survey or MaxDiff has a “none” option. If a respondent answers “no, I don’t want it” on every task, then their “none” utility will have a very large value.

Suppose a respondent’s “none” utility = 5.0, while 10 other items each have utility = -0.5. Then, on a C=5 task including “none”, their expected RLH = exp(5) / (4*exp(-0.5) + exp(5)) = 0.98.

That might indicate a “bad” respondent who simplifies the survey by picking “none” every time. OTOH, it might be a good respondent who accurately and importantly says that they don’t want the product. Again, the point is not that RLH identifies whether any respondent is good or bad. Rather, it must be used diagnostically with other knowledge and signals.

Filtering respondents is risky

I say this every chance I get, and won’t belabor it: anytime we filter respondents — whether that is from a screener, a model fit estimate, or whatever — we are introducing bias into the sample and its results.

The worst offenders that I see in this are are surveys that use multiple screening items to identify the “right” audience. For instance, they might want to find “intenders in the next 3 months in our target audience who are familiar with the product category and already use at least one product in the category.”

If a survey does that, stop and reconsider! Each of the 5 conditions in that hypothetical screener adds error. Unless the items have been psychometrically evaluated for prevalence, validity, and relation to the constructs of interest, then I would assume that filtering is adding a large and unknown degree of error, not reducing it!

The same considerations apply to filtering due to RLH as I described above. Filtering by RLH might reduce non-informative respondents … yet it also might remove respondents who are telling you something vitally important about their own disinterest. It’s important to know when RLH is low, because you can follow up by exploring why it is low.

Conclusion

I’ll finish with my recommended sequence of considerations for RLH with a choice modeling survey (MaxDiff or Conjoint Analysis):

1. Always examine — through descriptive stats and plotting — the individual-level RLH estimates for your dataset in a choice model survey. Not just the overall average, but the individuals.

2. If the curve is approximately normal, and only a small % are below a cutoff of 1.5/C (e.g., 1.5 / 5 items = 0.3 for a 5-item-at-a-time MaxDiff) … then I’d say RLH looks good, and don’t worry further about it. The UXR data above is an example of this. (Yay! UXRs are excellent respondents.)

3. OTOH, if (say) >10% of respondents have RLH < 1.5/C, then you need to identify why that is occurring. The reasons might reflect any or all of these:

   1. Accurate responses of disinterest that are important to know

   2. A problem with the survey’s content, concept phrasing, programming, etc.

   3. The wrong target audience, perhaps indicating a need for new sample

   4. Low quality respondents due to response styles or panel characteristics

   5. Poor model fit for other reasons, such as inappropriate constraints on estimation

4. There is no way to distinguish those five possibilities for low RLH without additional information, such as other signals, qualitative pre-testing of the survey, reconsideration of any unusual model choices, and so forth.

5. Similarly, but less commonly, if your survey includes a “none” option, look out for an unusually high RLH peak (a moderate or high % of respondents with RLH > 0.8 or so).

In short, don’t use RLH as an automatic filter! Use it as a way to understand more about your data and to identify potential questions about respondents’ response styles.

I hope this discussion and the R code above will help you to use RLH with your data. Cheers!

All the R Code

As always, following is the R code for all of the blocks above, in one place. [BTW, the references to “12.x” are there because this code is continuous with code from the previous 3 posts of this series.]

##### 12. Data Quality / RLH

# 12.1
# get the data; repeating here for blog post 4, see post 1 for details
library(openxlsx)   # install if needed, as with all package calls
md.dat <- read.xlsx("https://quantuxbook.com/misc/QUX%20Survey%202024%20-%20Future%20Classes%20-%20MaxDiff%20Individual%20raw%20scores.xlsx")   # 
md.dat$Anchor <- NULL
names(md.dat)[3:16] <- c("Choice Models",  "Surveys",       "Log Sequences",    "Psychometrics", 
                         "R Programming",  "Pricing",       "UX Metrics",       "Bayes Stats",
                         "Text Analytics", "Causal Models", "Interviewer-ing",  "Advanced Choice", 
                         "Segmentation",   "Metrics Sprints")
classCols <- 3:ncol(md.dat)    # generally, Sawtooth exported utilities start in column 3

# simple calculations as described in the post
exp(1.0) / sum(exp(c(1.0, 0.5, -0.8, -1.2)))  # item A
exp(-1.2) / sum(exp(c(1.0, 0.5, -0.8, -1.2))) # item D       
0.53 * 0.059
sqrt(0.53 * 0.059)

(individualOdds <- seq(0.01, 0.99, length=12))  # placeholder for the likelihoods
(rlh <- prod(individualOdds) ^ (1/length(individualOdds)))

# 12.2 RLH
names(md.dat)[2] <- "RLH"
summary(md.dat$RLH)

# let's look at our UXR data
# density plot of the RLH values
library(ggplot2)
library(ggridges)
set.seed(98101)    # jittered points are slightly randomized
p <- ggplot(data=md.dat, aes(x=RLH, y=1)) +
  geom_density_ridges(jittered_points=TRUE, 
                      alpha=0.5, colour="darkblue") +
  xlab("Model fit score (RLH; > 0.3 is good)") +
  ylab("Relative prevalence (density)") +
  theme_minimal()

p

# 12.3
# how well might we *expect* to predict, if respondents answers agree *perfectly* with an estimated model?

# back of the envelope
# pick some preference distribution and calculate odds
itemOdds <- c(1, 2, 3, 5, 8)           # relative preference for 5 items
itemOdds <- itemOdds / sum(itemOdds)   # rescale so they sum to 1.0
max(itemOdds)          # relative odds for the most-preferred item

# code version
# we can do much better with a simulation model!
nIter    <- 10000            # how many times to sample nItem MaxDiff simulations
nItems   <- 14               # number of items in a MaxDiff set
nShown   <- 5                # number of items shown on a single screen
nScreens <- 6                # number of screens in our survey
rlhDraws <- rep(NA, nIter)   # hold the results for each iteration
set.seed(98250)              # make it repeatable

# we have to set distribution parameters for our simulated part worths
# in HB model, they are random normal, mean=0, ... but what sd should we use?
# look at empirical data to pick a reasonable sd
summary(unlist(lapply(qux.md[ , classCols], sd)))   # answer: somewhere around sd ==(2.2, 2.6)
(drawSD  <- median(unlist(lapply(qux.md[ , classCols], sd))))

for (i in 1:nIter) {
  # set up 1 respondent "trial"
  pwsResp     <- rnorm(nItems, mean=0, sd=drawSD) # "nItems" random normal, zero-centered simulated part worths
  pwsResp     <- scale(pwsResp, scale=FALSE) # recenter to make sure they're zero-sum

  # iterate over nScreen tasks per respondent
  drawMax <- rep(NA, nScreens*2)             # hold the results across the screens for one respondent
  for (j in 1:nScreens) {
    pwsDrawn    <- sample(pwsResp, nShown)   # get the part worths for 1 simulated task  
    # "best" choice
    pwsExp      <- exp(pwsDrawn)             # exponentiate those part worths
    pwsMaxShare <- max(pwsExp) / sum(pwsExp) # our best (most likely) prediction would be the max utility item
    drawMax[j*2 - 1]  <- pwsMaxShare
    # repeat for the "worst" choice
    pwsExp        <- exp(-1 * pwsDrawn)      # exponentiate part worths for the "worst" choice direction
    pwsMaxShare   <- max(pwsExp) / sum(pwsExp)   
    drawMax[j*2]  <- pwsMaxShare
  }
  # calculate RLH from those observations
  rlh         <- exp(sum(log(drawMax[drawMax > 0]), na.rm=TRUE) / length(drawMax))    # safer equivalent of "prod(drawMax) ^ (1 / length(drawMax))"
  rlhDraws[i] <- rlh
  # add a bit of error checking, just in case of an off by one error etc :-/
  if (length(drawMax) != nScreens * 2) warning("The vector of partworths is off somewhere!", i, ":", drawmax)
}

# density plot
set.seed(98195)    # jittered points are slightly randomized
p <- ggplot(data=as.data.frame(rlhDraws), aes(x=rlhDraws, y=1)) +
  geom_density_ridges(jittered_points=TRUE, 
                      alpha=0.1, colour="darkblue") +
  xlab("Model fit score (RLH; > 0.3 is good)") +
  ylab("Relative prevalence (density)") +
  ggtitle("RLH Simulation, Perfectly Made Choices (N=10000)") +
  theme_minimal()

p

# where does our observed data fall in the distribution?
ecdf(rlhDraws)(median(md.dat$RLH))               

# combined density plot (using random subset of simulated data)
compare.df <- data.frame(RLH    = c(md.dat$RLH, rlhDraws),
                         Source = rep(c("UXR", "Sim"), 
                                      times=c(length(md.dat$RLH), 
                                              length(rlhDraws))))

# density plot
set.seed(98102)    # jittered points are slightly randomized
p <- ggplot(data=compare.df, aes(x=RLH, y=Source, color=Source)) +
  geom_density_ridges(jittered_points=TRUE, 
                      alpha=0.1) +
  xlab("RLH") +
  ylab("Relative prevalence (density)") +
  ggtitle("RLH Comparison: UXR Data vs. 'Perfect' Simulation") +
  theme_minimal()

p

Citations

I’m reminding myself (and others) to systematically cite the important work others have done and that make R and other tools so valuable. Following are key citations for today’s code!

BTW, you can find citations for almost everything in R with the citation() command, such as citation("ggridges").

• Orme, B (2019). Consistency Cutoffs to Identify "Bad" Respondents in CBC, ACBC, and MaxDiff. Sawtooth Software technical paper, available at https://content.sawtoothsoftware.com/assets/48af48f3-c01e-42ff-8447-6c8551a6d94f

• R Core Team (2024). R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria. https://www.R-project.org/. Version 4.4.1.

• Schauberger P, Walker A (2024). openxlsx: Read, Write and Edit xlsx Files. R package version 4.2.6.1, https://CRAN.R-project.org/package=openxlsx.

• Wickham, H (2016). ggplot2: Elegant Graphics for Data Analysis. Springer-Verlag New York.

• Wilke C (2024). ggridges: Ridgeline Plots in 'ggplot2'. R package version 0.5.6, https://CRAN.R-project.org/package=ggridges.

1. What is quantitative user experience research at Pinterest?

2. You’re a STEM major? Your dream job might be in Design

3. From brain science to quantitative UX

Today, I want to tell a different story, one that involves doing qualitative UX research (a.k.a., qual UX) and how it made me a better Quant UXR.

For clarification, I’m not talking about partnering with a qual UXR. That’s often beneficial, especially if you have the privilege of working with multiple researchers (Lesson #4 here)! What I'm additionally arguing is that Quant UXRs can develop and execute qual UX research, and this will reciprocally improve their Quant UX research capabilities.

If you consider yourself someone who always defaults to quant research methods and you are skeptical about whether building up some qual research experience in your portfolio will actually make you a better Quant UXR — or if you're just too intimidated to execute research outside your methodological comfort zone — read on ...

The limitation of technical expertise

In the mid-2010s, Quant UX research was still defining its professional identity. When I entered the field, I thought what differentiated Quant UXRs from their qual counterparts was technical expertise — particularly the ability to design surveys, query log data, and program in languages like R or Python to format, manipulate, and analyze large datasets, and to run statistical modeling and create data visualizations. Back then, I believed that advancing these technical capabilities was the key to becoming a better Quant UXR.

Early in my career, my work reflected this mindset. I wanted to be an "numbers expert," showcasing advanced statistical models, intricate SQL queries, and meticulously crafted surveys. I built strength in transforming survey responses and behavioral log data into actionable user insights at scale.

However, technical mastery alone doesn't guarantee research impact. As my career progressed, I faced increasingly complex and ambiguous challenges that couldn't be solved with complex data analysis alone. Such research required a deep understanding of nuanced spaces and the ability to identify opportunities for meaningful product improvement, balancing both user experience and business needs.

At the same time, I came to appreciate a hierarchy of research excellence:

• A good researcher answers a question effectively.

• A great researcher selects the most appropriate tools to answer a question effectively.

• An amazing researcher ensures they are tackling the right question, then selects the most appropriate tools to answer it.

• An exceptional researcher ensures they are tackling the right question, then selects the most appropriate tools to answer it, and crafts compelling narratives that inspire action.

Modifiers used to describe a researcher shall not be used for a survey scale question :).

My work evolved from technical execution to uncovering deeper user insights that could shape business strategy. Technical skills remained critical, but they became tools in service of a more fundamental goal: understanding user experience at its deepest, most nuanced level.

The true measure of research impact, I realized, isn't the complexity of the analysis — it's the potential for meaningful change in both user experience and business outcomes.

First steps into qualitative research

“A fundamental principle of innovation or creative thinking is to start with empathy.” – IDEO

People SAY, DO, THINK and FEEL, and these don’t always align. Quant UX research focuses on understanding what (and how much) people DO and SAY they feel about a product or experience at scale by collecting and analyzing empirical evidence. In contrast, qual UX research delves deeper, exploring why people THINK and FEEL the way they do, uncovering nuanced motivations and underlying emotions. Combining them gives me a richer data set to analyze patterns, synthesize the meaning, identify opportunities, and build a compelling narrative.

My initial experience with qual research came through collaborations with skilled qual UXRs. This was done in two ways: (a) observing and leveraging the work of my qual UXR colleagues, and (b) designing and commissioning qual studies done by external research vendors. Through projects ranging from consumer shopping journeys and advertiser trust to teen safety and user segmentation, I developed a deeper appreciation for the nuanced and layered insights that qual research provides, extending far beyond what my data charts and dashboards alone can reveal.

These early collaborations were eye-opening. While I wasn't conducting the user interviews directly, I learned to:

• Leverage qual findings to complement and enhance my quant analysis.

• Triangulate quant and qual insights to craft narratives that are both data-driven and human-centered, combining breadth with depth.

• Recognize and value contextual details from the periphery of observations and interviews, uncovering hidden insights that often prove invaluable during research synthesis.

I thought I had gained a solid understanding of qual research through these collaborations, but conducting my own qual research projects taught me far more. Over the past couple years, I led several international research initiatives, employing both remote and in-person methodologies. The remote studies consisted of in-depth interviews (IDIs) conducted over video calls, working across multiple time zones and languages with the help of simultaneous interpreters.

For field research, I sourced and partnered with local research agency partners to operationalize the study, and facilitate on-site market deep-dives. I brought cross-functional teammates across the world to meet and interact with users in their own local context. My travel-mates spanned exec leadership, product managers (PMs), engineers, designers, analysts, and marketers. These studies were conducted across China, South Korea and Turkey.

For a Quant UXR who revels in the structure and predictability of quant methods, the unstructured nature of qual research initially felt messy, disorienting and deeply uncomfortable. Yet, this discomfort was a catalyst that allowed me to grow.

I was fortunate to learn from veteran qual UXRs who provided invaluable mentorship throughout my growth. They shared essential resources like recruiting screeners and interview protocols, which gave me a strong foundation to build upon. By shadowing their interview sessions, I gained firsthand experience with effective interviewing techniques and learned to handle unexpected challenges, from technical issues to participant no-shows. Their guidance on working with international research vendors proved especially valuable, helping me navigate cross-cultural research complexities.

With their support and encouragement, I developed confidence in leading end-to-end research processes, developing skills in:

• Research design and methodology: Crafting nuanced interview protocols and designing diverse research methodologies, including IDIs, focus groups, expert interviews, co-creation workshops, and dinner mixers that facilitate authentic conversations between my cross-functional teammates and research participants (who are English language learners in our target markets).

• Global research operations: Sourcing, vetting, and relationship managing of local qual UX agencies and facilities in target markets.

• Participant recruitment: Developing and localizing screeners to recruit research participants across complex cultural landscapes, languages, and timezones. (This one was particularly hard for me, not going to sugar coat it.)

• Moderation and facilitation: Conducting IDIs through simultaneous interpreters across multiple languages. In one challenging case, I conducted an interview in both English and Mandarin simultaneously to accommodate both the participant and stakeholders observing the session.

• Qualitative data analysis and synthesis: Synthesizing qual data, including individual stories, observations, and unstructured data, into a cohesive narrative.

• Leadership in cross-functional collaboration: Guiding large cross-functional teams during intensive international research trips, and facilitating cross-functional team debriefs and synthesis sessions.

These projects offered far more than just data collection. By conducting qual research myself, I learned to step back, to shift my focus from being the primary driver of the research to truly listen — not just to what the participants said, but also what they didn't say, and the nuances between the lines.

This shift in perspective taught me to set aside my own preconceptions and embrace genuine curiosity about what I might discover through these intimate exchanges across languages and cultures. The resulting data, synthesized insights, and the stories we crafted felt more human and more personally meaningful, because they were grounded in real human experiences.

The power of qualitative research for Quant UXRs

Qual research has been instrumental in elevating my work as a Quant UXR. By delving into the nuances of user experience, it has enabled me to reframe research questions, develop holistic research strategies, and channel user empathy for greater outcomes, empowers me to deliver more impactful insights, influencing stakeholders to prioritize the most critical opportunities with conviction.

Reframing research questions

In quant research, I often started with preset questions. However, qual research taught me the importance of understanding the user's world first. By listening to users' unfiltered experiences and identifying pain points they might not articulate in a survey, I am able to challenge assumptions and uncover deeper issues.

For example, when I worked on an educational product that connects English language learners with native English-speaking tutors through 1-on-1 online lessons, one data analysis revealed that despite offering free trial lessons to all new users, free trial lesson usage was low. The growth product team initially focused on optimizing the free trial's visibility on the homepage. However during an international research trip, we observed new users struggling to understand how to use the product. This led us to reframe the problem from "why aren't new users using free trial lessons?" to "how can we improve new user comprehension during onboarding?" The team shifted its focus to educating new users on how to navigate the product, and take key actions earlier in their learning journey, which resulted in a more significant impact on free trial lesson adoption, and ultimately increased the conversion rate.

Developing holistic research strategies

As a Quant UXR leading qual studies, it allowed me to strategically design research that investigated the "why" behind my statistical findings. Rather than participating in other qual UXRs’ work and following someone else's research protocol, I could craft interview questions that directly probed into patterns I'd identified in my data, creating a powerful feedback loop between methods.

For example, while my quant research report suggested that English language learners highly value access to native English-speaking tutors, the product team, based in Silicon Valley, initially prioritized AI technology in our new user acquisition strategy. When I took the team on an international research trip and immersed ourselves in non-English-speaking contexts, the team realized that many learners faced significant barriers to accessing native speakers, far greater than the team had initially imagined. One learner poignantly expressed, "Before discovering your product, learning English meant waiting 45 minutes in a classroom for a chance to speak one sentence with my English teacher."

Armed with both the statistical evidence from the quant report and these powerful firsthand experiences, my team was compelled to reevaluate our strategy and re-emphasize the value of native speaker access in our user acquisition efforts.

Channeling user empathy for greater outcomes

Perhaps, most significantly, qualitative research is pivotal in cultivating a shared, profound empathy for users among myself and stakeholders. By directly engaging with the people we design for, we collectively gain a deeper understanding of their motivations and frustrations — and the often unspoken anxieties that drive their behavior.

For example, in my survey study, we discovered that many English language learners in the APAC region highly valued the curriculum during lessons with native English-speaking tutors. Initially, our takeaway was that APAC learners have a tendency to prefer structured lessons over free-form speaking. However, after conducting interviews with learners, we uncovered a deeper insight: structured lessons with a pre-selected curriculum served as an avoidance mechanism for these learners’ anxiety, alleviating the dreaded fear of running out of topics to discuss with their tutor — a fear that significantly impacted their learning experience! This insight helped our product team not only build more experiences that allowed structured English lessons, but also explore other ways they could address the more impactful pain point and opportunity — helping reduce learner anxiety.

Tips for getting started with qual work

For Quant UXRs hesitant to dive into qual work, if you’ve read this far, I hope you are feeling more ready to take on the new challenge! Here are a few tips to get you started:

• Get exposure

  • Start small: observe qualitative interviews

  • Find at least one qual researcher as your mentor (I got really lucky on this one!)

• Change mindset/attitude

  • Approach individuals with curiosity and an open mind, recognizing that even small samples can offer valuable insights and unexpected discoveries

  • View it as diversifying the tools in your toolkit with new ones, not replacing them

• Take action

  • Practice active listening

  • Ask open-ended questions

  • Anytime you write a survey, before going live, pre-test it. Do this with a few users or coworkers live (even friends or family!), using a think aloud protocol

  • Ask for opportunities to conduct lightweight user interviews (for more references on how to conduct user interviews, Interviewing Users: How to Uncover Compelling Insights (2nd Edition), by Steve Portigal, is a great book)

UX managers?

Give your Quant UXRs room for qualitative work! It’s not something they do as a distraction from Quant UX research. Rather, it is a core part of great research, regardless of role.

The evolving research identity

My research identity is no longer about being "quant" or "qual”, because these are tools. It's about being a T-shaped, entrepreneurial-minded UXR who specializes in quant research methods — the vertical stroke of the “T”, the depth of my quant tools – but also is experienced in crafting and executing qual research – the horizontal stroke of the “T”, the breath of my research toolkit, AND is willing to do whatever it takes to get the insight needed to understand complex users and businesses, and explore opportunities at their intersection.

This essay was inspired by the many many long conversations I had with Gabe Trinofi, Altay Sendil, Cassandra Rowe, Chris Chapman, Scott Tong and Omar Seyal, and is dedicated to all the qualitative UXRs whose work inspired me to be a better researcher.

Special shout-out to Altay Sendil for his meticulous edits, and Jeff Miao for his creative illustration!

If you haven’t read Posts 1 and 2, you’ll want to catch up on the background and basic data in Post 1, and on the question of correlational patterns of interest in Post 2. As usual, I illustrate everything with R code and will discuss the code as we go, with all of the code compiled at the end.

The topic of this Part 3 is this: are there clusters (segments) of respondents who have similar patterns in their interest in classes? As my coauthors and I discuss in the R book and the Python book, the process of finding segments in real data is complex and there is no magic answer. Segmentation is an iterative process of finding some number of segments and seeing whether those segments are useful and interpretable for your business need.

In this post, I discuss one way to find the number of segments and to visualize the results … with a big caveat: there is much more to say! This is one illustration and is not a prescription. No blog post can give a complete recipe for clustering. For comprehensive discussion and learning, see “Learning More” at the end of this post. Also stay tuned for a future “Segmentation FAQ” blog post soon!

Warning: long post! I like talking about segmentation (both pro and con) :) Like many posts in this blog, this is intended to complement other writings. It complements the R book and Python book for conjoint, and the Quant UX book for MaxDiff.

First, Get the Data

You can read Posts 1 and 2 for background. But if you’re jumping in here, the following R code will load the data. These are N=308 responses estimating researchers’ levels of interest in Quant UX classes, from a MaxDiff survey.

You can get the data with:

# get the data; repeating here for blog post 3, see post 1 for details
library(openxlsx) # install if needed
md.dat <- read.xlsx("https://quantuxbook.com/misc/QUX%20Survey%202024%20-%20Future%20Classes%20-%20MaxDiff%20Individual%20raw%20scores.xlsx")   # 
md.dat$Anchor <- NULL
names(md.dat)[3:16] <- c("Choice Models",  "Surveys",       "Log Sequences",    "Psychometrics", 
                         "R Programming",  "Pricing",       "UX Metrics",       "Bayes Stats",
                         "Text Analytics", "Causal Models", "Interviewer-ing",  "Advanced Choice", 
                         "Segmentation",   "Metrics Sprints")
classCols <- 3:ncol(md.dat)    # generally, Sawtooth exported utilities start in column 3

I recommend to follow along live in R, installing packages as needed. All the R code is given as I go, and also compiled at the end of the post.

Background: General Clustering vs Latent Class Analysis

At a high level, there are two approaches to clustering/segmenting data from choice surveys.

One approach is to estimate respondents' preferences first as a whole sample, and then apply general clustering methods later to those estimates to find segments. My experience suggests that a majority of analysts use this approach for several reasons. First, it is easiest because it divides analysis into tidy stages. Second, you can use the widest range of tools and methods. Third, it does not require specialized software for the clustering portion, but uses general methods from R, Python, or wherever. Fourth, the approach can be used with any kind of data and with mixed types of data.

The second approach is to model segments at the same time that you estimate individuals’ preferences. This is the approach taken by latent class analysis (LCA), as applied to choice surveys. The primary advantage of LCA is that, by doing both steps together (estimation + segmentation), it is more robust to sparse data and noise in the data. On the other hand, LCA requires more specific assumptions about the data, along with specialized tools.

In the choice modeling community, the most common tools used for choice model LCA are Statistical Innovation’s Latent Gold, Sawtooth Software’s Lighthouse Studio, and proprietary code bases within analytics firms. These may add cost and complexity to analyses, while limiting the generality of what you can do. By contrast, general clustering methods are widely available in R and other platforms. (Note that choice model LCA is possible in R … but the effort is beyond reasonable levels for general practitioners. FWIW, I personally avoid doing choice model LCA in R.)

In this article, I demonstrate use the general clustering approach … with one note: if you frequently need to cluster choice model data, you should check out Sawtooth Software’s Lighthouse Studio and/or Statistical Innovations’s Latent Gold. Their tailored functionality can be helpful, efficient, and more precise for practitioners. For my part, I often use LCA in Sawtooth’s Lighthouse Studio. However, I also often use general clustering as this post illustrates.

In short, although I use choice survey data to demonstrate the methods here, the methods here are general. You can apply the tools and code in this post to a wide variety of data sources.

Finding a Number of Segments: Part 1

The first question in segmentation is this: are there segments at all? And if so, how many?

A reasonable assumption — especially in high dimensional space, i.e., in data with many variables — is that people do not cluster together, and that instead, they differ from one another in unstructured (“random”) ways. Some jargon for that is that the data exhibit a multivariate normal distribution (MVN) with one component. In other words, there is a single “segment” and every respondent is a member!

My take — see the “clustering” chapters in the R book and the Python book — is that MVN data with 1 cluster is quite often the “best” statistical answer … but the number of segments is not primarily a statistical question. There are hundreds of methods to cluster data and they typically give different answers. The question is whether we can find segments that are useful and informative for a business question, while secondarily optimizing statistically.

There are several ways we might inspect the data to see whether it appears to have clusters. One way is a model-based approach to determines how many clusters may underlie the data. The mclust library is one of my favorites. In R we can fit a clustering (segmentation) model to the MaxDiff data as follows:

# mclust uses model-based fit estimates to determine best number of latent clusters
library(mclust)    # install if needed
md.mc <- Mclust(md.dat[ , classCols])
summary(md.mc)

The result is:

It found a multivariate normal distribution with one component. In other words, the best fit is 1 segment.

Another approach is visual. We could use a dimensional reduction approach to represent the data in a visually suitable number of dimensions — 2 or 3 dimensions — and then inspect it to see whether there is some obvious clustering.

There are many dimensional reduction models but a modern starting point is UMAP (uniform manifold approximation and projection). UMAP attempts to “project” (i.e., re-estimate) data in a lower number of nonlinear dimensions (“uniform manifold”) while also preserving (“approximating”) the structure of the complete data. (For an introduction, see Coenen and Pearce on “Understanding UMAP” here.)

We can visualize the MaxDiff class interest data with UMAP as follows. First we fit a UMAP projection to the data, setting a random seed to make it reproducible:

library(umap)
# get umap representation of the individual interest data
umap.config <- umap.defaults
umap.config$random_state <- 98101

umap.sol <- umap(md.dat[ , classCols], config = umap.config)
umap.dat <- data.frame(umap.sol$layout)
# review the kind of data we have
str(umap.dat)

Using str() we see that the data now has 308 observations reduced to 2 dimensions, X1 and X2. The particular values don’t really matter in themselves; it is the relationships among the values that reproduce the relative structure of the observations vs. one another:

Now we can plot the UMAP points just like any other X / Y data using a simple scatterplot of the points:

# plot it
ggplot(umap.dat, aes(x = X1, y = X2)) +
  geom_point() +
  xlab("High order dimension 1") + ylab("High order dimension 2") +
  theme_minimal()

Here’s the chart:

This chart shows no obvious “clusters” of respondents. It is slightly denser in some areas than others but overall has an appearance of being smoothly round without separation among groups. This appearance is not a statistical test, but it is congruent with the idea that there are not multiple obvious segments but 1 segment of everyone.

So far we are striking out on finding segments. The data set strongly appears to come from a single multivariate normal population (i.e., from one segment).

But remember I mentioned that there are hundreds of clustering methods? Let’s keep prodding and see if we can find something else that is useful!

Finding a Number of Segments: Part 2

Another R library that suggests the number of clusters in data is NbClust. It implements 8 clustering methods with a choice of 6 different distance (dissimilarity) metrics applicable to each one. After you select a distance metric and clustering method, it reports 30 heuristics (sometimes fewer, for reasons I’ll skip) that each suggest the number of clusters.

That’s easier to understand from an example:

library(NbClust)
md.nbc <- NbClust(md.dat[ , classCols],
                  min.nc = 2, max.nc = 10,
                  method = "ward.D", 
                  distance = "euclidean")

In this code, I ask it to consider anywhere from 2 to 10 clusters (min.nc, max.nc). For a distance metric, I use the classic Euclidean metric (i.e., triangular hypotenuse, except extended to many dimensions), and cluster them with the “Ward’s D” method (see ?NbClust for details).

Here are the results, first for 24 methods that give textual results, and then for 2 methods that give graphical results:

The textual results say that a “winner takes all” vote by the methods indicates 2 clusters (as chosen by 9 out of 24 methods). However, almost as many methods (7 out of 24) suggest 5 clusters. In the graphical results, both charts suggest 2 clusters as the best fit; and one of them suggests 5 clusters as a secondary possibility.

So, should we choose 2 clusters? In practice, I believe it is rarely useful or desired (by stakeholders) to discuss only 2 clusters — unless we are strictly speaking of likelihood of a binary outcome (such as purchase behavior). For purposes of understanding customers and users, binary options are rarely illuminating.

Thus, instead of 2 clusters, let’s look at the “second best” option of 5 clusters. What would that solution tell us?

To begin with, we’ll impose a 5 cluster solution. There are many ways to do that, but in this case we can simply run NbClust() again, setting the minimum number to 5:

# we'll try a 5 cluster solution instead of 2
md.nbc <- NbClust(md.dat[ , classCols],
                  min.nc = 5, max.nc = 10,
                  method = "ward.D", 
                  distance = "euclidean")

md.segs <- md.nbc$Best.partition
table(md.segs)
proportions(table(md.segs))

This code runs NbClust again, with a minimum solution of 5 clusters (which is then found to be the best option with that limitation). We extract the segment assignment for each respondents ($Best.partition) and save them as md.segs. The final two lines show the counts of segment assignments and their relative proportions:

The smallest segment (segment 5) has 10% of respondents, while the largest (segment 1) has 28% of respondents. That is a reasonable spread. It does not suffer the common problem where one or two segments have a very small proportion of respondents such as only 2% or 5% of respondents.

Next we’ll see whether the segments differ in interpretable ways.

Visualizing Segments in Dimensional Space

Remember the UMAP chart above? Now that we have segment assignments, we can add those to the chart. Are there visual patterns when the segments are plotted in the higher order dimensions found by UMAP?

Here’s the code. It’s the same as above, except it now colors the points according to which segment each respondent is in (md.segs):

# add the 5-segment membership to the UMAP data
umap.dat$Segment <- factor(paste0("S", md.segs))

# plot it with segments
ggplot(umap.dat, aes(x = X1, y = X2, colour = Segment)) +
  geom_point() +
  xlab("High order dimension 1") + ylab("High order dimension 2") +
  theme_minimal() +
  ggtitle("Dimensional map with 5 segments")

Here’s the result:

Whoa! This is a chart to make a Quant UXer very happy! In it, there are five extremely clear slices of the plot. Almost all of the members of each segment are close to others in the same segment — which is to say that the segments map very cleanly to the higher order dimensions of the data. That suggests that we may see real differences when we examine those dimensions (interest in different classes).

But first, I hope you’re thinking to yourself, “Wait, am I fooling myself? Maybe the colors being close together doesn’t mean much.” Good question!

Let’s double-check ourselves by comparing what the UMAP chart would look like if the segments were randomly assigned. To do that, we shuffle the same set of assigned segment memberships (using sample()) such that they are randomly assigned to different rows of the UMAP data, and plot them again:

# make sure we're not fooling ourselves ...
# what would it look like if the segments were random?
set.seed(98107)
# the same segments but in random order
umap.dat$RndSeg <- sample(umap.dat$Segment)
# plot those
ggplot(umap.dat, aes(x = X1, y = X2, colour = RndSeg)) +
  geom_point() +
  xlab("High order dimension 1") + ylab("High order dimension 2") +
  theme_minimal()+
  ggtitle("Dimensional map with RANDOM segments")

The result:

In this chart, there is no pattern to the colors for segment assignment. This tells us that the extremely clear pattern shown in the previous chart reflects real structure of the segments, relative to the data; the appearance was not simply an accident.

In short, the five segments align strongly with real properties of the data. Now the question becomes: is that alignment interpretable and useful? I look at that question next.

Visualize Segments with a Heat Map

In the above analyses, we found — or, if you prefer, “imposed” — segment assignments that show statistical differentiation (the assessment heuristics in NbClust) and that align with the high dimensional data (the segment visual differences in UMAP). But are those results merely statistical or are they strong enough to be meaningful as a practical matter?

To find out, I’ll plot a heat map for class interest, broken out by segment. And, as you’ve probably guessed from my previous posts, I’ll make it a function :) With nothing more than minor tweaks (at most), this function will work for segmentation cross-tabs from many different kinds of data sets.

Here’s the function, with comments following below:

seg.heat <- function(dat, segs) {
  library(superheat)   # install if needed
  # aggregate mean value by segment
  heat.sum <- data.frame(t(aggregate(. ~ segs, data = dat, mean))[-1, ])
  # make the column names reflect the segment number and size (%)
  names(heat.sum) <- paste0("S", 1:max(segs), " (", 
                            round(prop.table(table(segs)), 2)*100, "%)")
  # draw it with superheat
  superheat(heat.sum, 
            heat.pal = c("red3", "white", "green3"),
            grid.hline.col = "white", grid.vline.col = "white",
            pretty.order.rows = TRUE,
            clustering.method = "hierarchical",
            X.text = round(as.matrix(heat.sum), 2),  
            X.text.size = 4, 
            left.label.size = 0.3,
            left.label.text.size = 4,
            bottom.label.text.size = 4,
            legend = FALSE)
}

In this function, I use the superheat package to draw a heat map. There are many alternatives but superheat is straightforward while offering many options. The first step is to use aggregate() to get the cross-tabs of item means broken out by segment. That single line of code also converts the result to a data frame as superheat expects. (FYI, I drop the initial row using [-1, ] in the conversion because that row contains column names rather than data.)

In the next line, I change the names() of the data frame so they start with “S” for segment, followed by the nominal segment “number”. I add a label for the relative percentage of each. That gives nice labels in superheat with the segment name and its size, such as “S1 (28%)”. BTW, if you had friendly names for the segments — such as “R Pirates” or “UX Mavens” — you could add them here.

After that, the final step is to call superheat to draw the heat map. I won’t explain all of the options in that call, but will mention the three most important ones: (1) I set a color palette that avoids extreme color intensity. (2) I use “pretty rows” and clustering to reorder the results to be grouped by similarity, making it easier to read and interpret the segments. (3) I add the mean values themselves — rounded off to 2 decimal places — to the cells using X.text. Without that, it would draw the heat colors but not label the values (which is also a perfectly appropriate choice).

We call the seg.heat() function with the MaxDiff estimates and assigned segments:

seg.heat(md.dat[ , classCols], md.segs)

Here’s the result:

What do we see? The five segments are strongly differentiated. I suggest you take a minute to interpret the results … before reading further!

What did you find? Here are a few reflections on each segment from my perspective. I tend to interpret segments according to strongest positive interest first.

• Segment 4 stands out strongly for its interest in R and statistical/data analyses. With roughly 25% of the sample, we can consider it to be a “hardcore Quant” or “data science in UX” segment.

• After that, Segment 5 stands out as a “Core UX” segment, strongly interested in classic mixed-methods UX issues, and not very interested in the stats topics.

• Segments 1 and 3 are quite similar, where both are interested in survey methods and choice models. The difference is that Segment 3 is also interested in core UX topics whereas Segment 1 is interested in pricing methods and more advanced choice models. Without looking at the data, my guess is that Segment 1 reflects quant researchers working in marketing, whereas Segment 3 has folks with similar skills and interests but working in UX.

• Finally, Segment 2 is not interested in much … except a segmentation class. That’s fine and is a common pattern — respondents who take the chance to communicate strongly about the one thing they want the most.

I will comment about one thing I almost always see in segmentation projects … but do NOT see in these data: there is often a segment that shows almost no differentiation of interest in anything. Such a segment is common in consumer data, and quite likely reflects low quality respondents, speeders, and/or those for whom the survey is inappropriate. It often gives a segment with 30% or more of the sample (I’ve seen as high as 60%). However, the present data come from an especially engaged and high quality panel (Quant UX conference attendees!) so they do not have that problem.

Now, is it actionable? What would we do from a business perspective with these data? From the point of view of the Quant UX Association offering classes, here are some thoughts:

• The results reinforce the idea (from Post 1) that — although the average interest in (for example) an R class is low, there is a core group of people (Segment 4) who are strongly interested. Now we have a good estimate for that interest, as a segment: 24%

• The results also reinforce the interest in a Segmentation class. It has strong interest for every segment except Segment 5 … adding up to 90% of the sample.

• After that, there is clear interest in Choice Models and related topics — in roughly 2nd or 3rd place for three segments — adding up to more than 70% of the sample.

• Although the size of Segment 5 is small (10%), its interest in UX metrics topics is extremely clear and crisp. That makes a nice possibility for a clear offering … if the Quant UX Association is able to reach the smallish group in a reasonable way.

Visualizing Individual Distributions within a Segment

Did you read post 1 in this series? Do you recall the chart of the individual distribution of interest for each class? (That was a density chart for each item.)

We can reuse that plot function now, and ask a similar question: within a particular segment, what is the distribution of interest? Is everyone in the segment alike, or are there strong differences?

Why would we ask that? For two reasons: to understand the details of the segment better, and to gain additional insight before we take action. For example, suppose we want to offer an R class and we believe that would target the 24% of respondents who fall into Segment 4. We might design the class or message it using insight about the other classes of interest to that segment (such as choice models and psychometrics).

However, if we design or message the class in that way, we would be well-advised not to choose something polarizing. It could be that some topic scores high on average with the segment and yet that is driven by a small group. We might be served better by choosing something that is lower on average but is more broadly acceptable. The distribution plot can help us to evaluate such questions.

As presented in Post #1 in this series, here is the function to make an individual-level distribution chart. This is just a repeat from that post (see there for explanation):

cbc.plot <- function(dat, itemCols=3:ncol(dat), 
                     title = "Preference estimates: Overall + Individual level", 
                     meanOrder=TRUE) {

  # get the mean points so we can plot those over the density plot
  mean.df <- lapply(dat[ , itemCols], mean)

  # melt the data for ggplot
  library(reshape2)
  #                       vvvv  assumes Sawtooth order;       vvv  (ID in col 1, remove RLH in col 2)
  plot.df <- melt(dat[, c(1, itemCols)], id.vars=names(dat)[1])

  # get the N of respondents so we can set an appropriate level of point transparency
  p.resp  <- length(unique(plot.df[ , 1]))

  # optionally and by default order the results not by column but by mean value
  # because ggplot builds from the bottom, we'll reverse them to put max value at the top
  # we could use fct_reorder but manually setting the order is straightforward in this case
  if (meanOrder) {
    plot.df$variable <- factor(plot.df$variable, levels = rev(names(mean.df)[order(unlist(mean.df))]))
  }

  #### Now : Build the plot
  # set.seed(ran.seed)   # optional; points are jittered; setting a seed would make them exactly reproducible
  library(ggplot2)
  library(ggridges)

  # build the first layer with the individual distributions
  p <- ggplot(data=plot.df, aes(x=value, y=variable, group=variable)) +
    geom_density_ridges(scale=0.9, alpha=0, jittered_points=TRUE,
                        rel_min_height=0.005,
                        position="points_sina",
                        # set individual point alphas in inverse proportion to sample size
                        point_color = "blue", point_alpha=1/sqrt(p.resp),
                        point_size=2.5) +
    # reverse y axis to match attribute order from top
    scale_y_discrete(limits=rev) +                                                   
    ylab("Level") + xlab("Relative preference (blue=individuals, red=average)") +
    ggtitle(title) +
    theme_minimal()

  # now add second layer to plot with the means of each item distribution
  for (i in 1:length(mean.df)) {
    if (meanOrder) {
      # if we're drawing them in mean order, get the right one same as above
      p <- p + geom_point(x=mean.df[[rev(order(unlist(mean.df)))[i]]], 
                          y=length(mean.df)-i+1, colour="tomato",      # adjust y axis because axis is reversed above
                          alpha=0.5, size=2.0, shape=0, inherit.aes=FALSE)

    } else {
      p <- p + geom_point(x=mean.df[[i]], 
                          y=length(mean.df)-i+1, colour="tomato",      # adjust y axis because axis is reversed above
                          alpha=0.5, size=2.0, shape=0, inherit.aes=FALSE)
    }
  }
  p
}

We can reuse that function exactly as is, except we plot only the respondents in Segment 4 by subsetting the data appropriately (md.segs == 4):

cbc.plot(md.dat[md.segs == 4, ], itemCols = classCols) + 
  ylab("Quant Course Offering") +
  ggtitle("Class interest: Segment 4 Only (R)")

The result:

In this chart we see the strong cluster for the R programming interest again, not only as the #1 item on average but showing a strong group of individuals with exceptional interest on the right hand side of the distribution.

After that, Choice Modeling appears to be a good secondary emphasis — not only is it #2 in position, but the distribution is tightly grouped and non-polarizing. Every respondent had interest greater than 0 (the “anchor” position for positive interest). Psychometrics, Causal Models, and Advanced Choice are all similar.

Farther down, we see that Metrics Sprints, Surveys, and Pricing are fairly highly polarizing within this segment: there are a few people with strong interest but also those with strong disinterest.

From a business perspective, that reinforces the idea that we might offer an R class with a secondary emphasis on choice modeling (like this post!) or psychometrics. And it suggests to avoid a few topics that are polarizing; they would lack appeal to too many participants (assuming our strategy is to go after the general audience).

Again, the key point of such deep dives is to understand the preference structure in more detail, we we can consider actions in design and/or promotion of our offerings. In this case, it could help with constructing the content of a class, with targeting respondents for interest in it, and describing it.

This is a good time to add one important caveat to all of this, an obvious point to researchers and yet one that is easy to forget: such results come from a particular sample, and we have to limit our interpretation to that sample (or its implied population). In this case, the results come from attendees of Quant UX Con … and Quant UX Con promoted many of the same topics (R, choice models, metrics, and the like). So to some extent it is no surprise that this sample would have these patterns of interest! Yet, at the same time, the data deepen our understanding of the sample, and our confidence in potential strategic actions.

But Again: There is no Magic Segmentation

Let’s take a look again at a fact I mentioned earlier: that there are hundreds of clustering methods — multiplied by dozens or even hundreds of ways to apply each of them. For example, with each one we might select among multiple distance metrics for observation similarity or multiple way to choose a number of clusters.

To take one example, we have already seen the mclust package that finds model-based multivariate normal mixtures (i.e., clusters). As we saw above, it suggests that the data represent 1 cluster. However, we can force it to fit a 5 cluster model, by setting the parameter G:

# we could force a 5-segment solution (for example) in Mclust
library(mclust)
md.mc <- Mclust(md.dat[ , classCols], G=5)

Now we can compare the solution from mclust to the solution from NbClust above:

# compare the mclust solution to the previous NbClust solution
table(md.mc$classification, md.segs)

Here is the comparison table, which presents NbClust segment assignments in the columns, cross-tabbed by mclust assignments in the rows:

We can’t compare the segment numbers — such as Segment 1 in each solution — because the assignment labels are nominal and have no order or meaning in themselves. For example, most of the respondents (26/31) who are labeled as “Segment 5” in NbClust fall into the group called “Segment 4” in the mclust solution.

However, if it were the case that only the labels were different, then all of the respondents in one column would fall into a single row. When they don’t, then it says that respondents assigned to one group in NbClust fall into multiple groups in mclust. In other words, the two methods find different groupings of respondents.

With that perspective, we can easily see signs of substantial disagreement between the two 5-cluster solutions. In Segments 1, 2, and 3 from NbClust — and segments 2, 3, and 5 from mclust — fewer than 50% of the observations from any segment are assigned to any single group in the other solution. In other words, the solutions disagree on a majority of segment assignments.

We might compute agreement by hand. Here’s one way. Column 4 maps best to the label of “3” in the other solution, capturing 55 respondents. Column 5 maps to “4” (26 respondents). Column 3 would map to “3” but that is already taken by Column 4, so we’ll map it to the second best assignment, “2” (N=21). That leaves Column 1 mapping to “5” (N=24) and Column 2 mapping to “1” (N=23). Adding them up, (55+26+21+24+23) / 308 = 48% raw agreement. That tells us something, but it would be better to adjust that with respect to chance (for example, if one cluster is large, then agreement could be “high” just by assigning all 100% of respondents to it!)

The adjusted Rand index is one method to determine congruence between cluster solutions, relative to base rate. We check that:

# Rand index for the degree of agreement between them
adjustedRandIndex(md.mc$classification, md.segs)

The answer is ARI = 0.186 … which means that the 5-segment solutions from NbClust and mclust agree only slightly better than random chance (about “18% better than random” to use a heuristic explanation of the index). Put differently, the two statistical models often assign respondents to quite different segments.

We can also draw a UMAP plot with the new mclust segment assignments:

# UMAP plot of the mclust assignments
set.seed(98107)
# the same segments but in random order
umap.dat$mclustSeg <- factor(md.mc$classification)
# plot those
ggplot(umap.dat, aes(x = X1, y = X2, colour = mclustSeg)) +
  geom_point() +
  xlab("High order dimension 1") + ylab("High order dimension 2") +
  theme_minimal()+
  ggtitle("Dimensional map with mclust segments")

Here’s the new plot:

The result shows segments that are much less cleanly separated than the NbClust segments, with respect to the UMAP dimensions that we plotted above. It has clear clusters, but not as clear as the chart above. Segment 3 is relatively separate from the others, but segments 1, 2, 4, and 5 have a lot of mixing with near neighbors assigned to other segments.

That doesn’t mean that NbClust is better than mclust … because UMAP dimensions are not the only method to assess segment clarity or utility. It only shows that the two solutions are different … because there is no single, canonical, or “best” segmentation method.

That’s not the end of the story, however! We need to look at the implications.

And Yet It May Be Useful

Beyond individual respondent assignment, there is a more important question: do the strategic implications differ between the two solutions? Let’s look at a heat map for the new mclust segments; our reusable function above makes it easy; we just pass those segment assignments to it (md.mc$classification):

# plot the 5-segment mclust solution as a heat map
seg.heat(md.dat[ , classCols], md.mc$classification)

Here’s the plot:

To interpret it, I have copied below the comments I made above about the NbClust segments … and added commentary that compares each segment to the new, and — on an individual level — substantially different mclust segment:

• Segment 4 [NbClust] stands out strongly for its interest in R and statistical/data analyses. With roughly 25% of the sample, we can consider it to be a “hardcore Quant” or “data science in UX” segment. \===> That aligns with Segment 3 in the new solution, although the segment is somewhat larger (34%)

• After that, Segment 5 [NbClust] stands out as a “Core UX” segment, strongly interested in classic mixed-methods UX issues, and not very interested in the stats topics. \===> That aligns with Segment 4 in the new solution, similarly sized.

• Segments 1 and 3 [NbClust] are quite similar, where both are interested in survey methods and choice models. Segment 3 is interested in core UX topics whereas Segment 1 is interested in pricing methods and more advanced choice models. \===> These are generally similar to Segments 2 and 5 in the new solution.

• Finally, Segment 2 [NbClust] is not interested in much … except a segmentation class. That’s fine and is a common pattern — respondents who take the chance to communicate strongly about the one thing they want the most. \===> That looks a lot like Segment 1 in the new solution.

In short, although the segmentation solutions differ substantially according to which respondents they places into each segment, the two methods ended up finding quite similar segments from a strategic point of view! The business implications — e.g., for R/stats classes, high interest in segmentation, a core UX segment, and so forth — appear to be nearly identical between the two solutions. That happy result reassures us about the plausibility and statistical stability of the solutions.

Put differently, although the assignment of each respondent to a segment is uncertain here and often disagrees, the segment averages end up being quite similar between the two methods. That is a general observation in choice model data (and TBH it also elates to a bedrock principle of statistics: we do stats to understand a population better by sampling and averaging! [among other ways])

However, I will add a caution: the strong similarity of the results here is an N=1 occurrence. There is no guarantee that two segmentation methods will give such similar results. In fact, given the low agreement we see for individual respondent assignment, my general presumption is that any two clustering solutions are somewhat likely to disagree in their strategic implications. We must do work to get beyond that, through careful interpretation and iteration, and by confirming and triangulating results!

Conclusion and a Final Warning

One point I have not yet made is this: an extremely common problem in cluster analysis is using far too many variables. In this post, I used 14 variables and I think that’s a perfectly reasonable size. It helps that they come from the same kind of observation (choice model) and estimation method (HB utilities). Yet many analysts attempt to cluster data sets with 50, 100, or 300 variables per respondent.

The problem with that is that clustering works by evaluating how similar respondents are to one another … and with so many variables, no one is similar! It is crucial to prune down the set of “basis variables” to be both few and of known importance. That’s a topic for more extended discussion.

Meanwhile, I hope this post demonstrates that:

1. It is quite possible to find useful segments in data

2. … even when a statistical model claims there are “no segments”

3. … while using general methods that work across many kinds of data

4. … and that provide good opportunities to cross-check our work as we go

5. … while building reusable functions along the way

6. … as long as we are cautious and avoid common pitfalls.

I hope you’ve found this interesting and helpful for your own analyses!

Learning More

If you’d like to learn more about segmentation, here are some places to start:

• For general segmentation, see the chapters on clustering and classification in the R book and the Python book.

• … or sign up for a class! In January 2025 there will be an online Segmentation Class from the Quant UX Association taught by surveys expert Keith Chrzan

• To learn about choice models in general, see discussion of conjoint analysis in the R book and the Python book, and of MaxDiff in the Quant UX book.

• … or join me for an upcoming offering of the Choice Modeling Master Class!

• For latent class models specific to conjoint analysis and MaxDiff, check out many papers over the years in the archives of the Sawtooth Software Conference (scroll down for the PDF archives by year).

All the Code

As always, here is all of the R code from this post compiled in one place. You can grab it with the copy icon and paste into your favorite R code editor.

##### Post 3 - Segmentation (clustering)

# get the data; repeating here for blog post 3, see post 1 for details
library(openxlsx)
md.dat <- read.xlsx("https://quantuxbook.com/misc/QUX%20Survey%202024%20-%20Future%20Classes%20-%20MaxDiff%20Individual%20raw%20scores.xlsx")   # 
md.dat$Anchor <- NULL
names(md.dat)[3:16] <- c("Choice Models",  "Surveys",       "Log Sequences",    "Psychometrics", 
                         "R Programming",  "Pricing",       "UX Metrics",       "Bayes Stats",
                         "Text Analytics", "Causal Models", "Interviewer-ing",  "Advanced Choice", 
                         "Segmentation",   "Metrics Sprints")
classCols <- 3:ncol(md.dat)    # generally, Sawtooth exported utilities start in column 3


##### 5. Number of segments?

# mclust uses model-based fit estimates to determine best number of latent clusters
library(mclust)
md.mc <- Mclust(md.dat[ , classCols])
summary(md.mc)     # best answer is only 1 "segment", multivariate normal within it

# initial visualization with UMAP

library(umap)
# get umap representation of the individual interest data
umap.config <- umap.defaults
umap.config$random_state <- 98101

umap.sol <- umap(md.dat[ , classCols], config = umap.config)
umap.dat <- data.frame(umap.sol$layout)
# review the kind of data we have
str(umap.dat)

# plot it
ggplot(umap.dat, aes(x = X1, y = X2)) +
  geom_point() +
  xlab("High order dimension 1") + ylab("High order dimension 2") +
  theme_minimal()


##### 6. Find number of clusters and initial solution

library(NbClust)
md.nbc <- NbClust(md.dat[ , classCols],
                  min.nc = 2, max.nc = 10,
                  method = "ward.D", 
                  distance = "euclidean")

# we'll try a 5 cluster solution instead of 2
md.nbc <- NbClust(md.dat[ , classCols],
                  min.nc = 5, max.nc = 10,
                  method = "ward.D", 
                  distance = "euclidean")

md.segs <- md.nbc$Best.partition
table(md.segs)
proportions(table(md.segs))


##### 7. Plot umap with segment membership coded

# add the 5-segment membership to the UMAP data
umap.dat$Segment <- factor(paste0("S", md.segs))

# plot it with segments
ggplot(umap.dat, aes(x = X1, y = X2, colour = Segment)) +
  geom_point() +
  xlab("High order dimension 1") + ylab("High order dimension 2") +
  theme_minimal() +
  ggtitle("Dimensional map with 5 segments")

# make sure we're not fooling ourselves ...
# what would it look like if the segments were random?
set.seed(98107)
# the same segments but in random order
umap.dat$RndSeg <- sample(umap.dat$Segment)
# plot those
ggplot(umap.dat, aes(x = X1, y = X2, colour = RndSeg)) +
  geom_point() +
  xlab("High order dimension 1") + ylab("High order dimension 2") +
  theme_minimal()+
  ggtitle("Dimensional map with RANDOM segments")


##### 8. Plot heat map for segments

# function to plot average values by segment
seg.heat <- function(dat, segs) {
  library(superheat)
  # aggregate mean value by segment
  heat.sum <- data.frame(t(aggregate(. ~ segs, data=dat, mean))[-1, ])
  # make the column names reflect the segment number and size (%)
  names(heat.sum) <- paste0("S", 1:max(segs), " (", 
                            round(prop.table(table(segs)), 2)*100, "%)")
  # draw it with superheat
  superheat(heat.sum, 
            heat.pal = c("red3", "white", "green3"),
            grid.hline.col = "white", grid.vline.col = "white",
            pretty.order.rows = TRUE,
            clustering.method = "hierarchical",
            X.text = round(as.matrix(heat.sum), 2),  
            X.text.size = 4, 
            left.label.size = 0.3,
            left.label.text.size = 4,
            bottom.label.text.size = 4,
            legend = FALSE)
}

# plot the segmentation solution
seg.heat(md.dat[ , classCols], md.segs)


##### 9. Plot distribution of interest *within* a segment
# check the data briefly ; not discussed in the blog post
summary(md.dat[md.segs==4, ])

# reuse our plot function -- copied here from the part 1 post
# see the part 1 blog post for an explanation of this function
cbc.plot <- function(dat, itemCols=3:ncol(dat), 
                     title = "Preference estimates: Overall + Individual level", 
                     meanOrder=TRUE) {

  # get the mean points so we can plot those over the density plot
  mean.df <- lapply(dat[ , itemCols], mean)

  # melt the data for ggplot
  library(reshape2)
  #                       vvvv  assumes Sawtooth order;       vvv  (ID in col 1, remove RLH in col 2)
  plot.df <- melt(dat[, c(1, itemCols)], id.vars=names(dat)[1])

  # get the N of respondents so we can set an appropriate level of point transparency
  p.resp  <- length(unique(plot.df[ , 1]))

  # optionally and by default order the results not by column but by mean value
  # because ggplot builds from the bottom, we'll reverse them to put max value at the top
  # we could use fct_reorder but manually setting the order is straightforward in this case
  if (meanOrder) {
    plot.df$variable <- factor(plot.df$variable, levels = rev(names(mean.df)[order(unlist(mean.df))]))
  }

  #### Now : Build the plot
  # set.seed(ran.seed)   # optional; points are jittered; setting a seed would make them exactly reproducible
  library(ggplot2)
  library(ggridges)

  # build the first layer with the individual distributions
  p <- ggplot(data=plot.df, aes(x=value, y=variable, group=variable)) +
    geom_density_ridges(scale=0.9, alpha=0, jittered_points=TRUE,
                        rel_min_height=0.005,
                        position="points_sina",
                        # set individual point alphas in inverse proportion to sample size
                        point_color = "blue", point_alpha=1/sqrt(p.resp),
                        point_size=2.5) +
    # reverse y axis to match attribute order from top
    scale_y_discrete(limits=rev) +                                                   
    ylab("Level") + xlab("Relative preference (blue=individuals, red=average)") +
    ggtitle(title) +
    theme_minimal()

  # now add second layer to plot with the means of each item distribution
  for (i in 1:length(mean.df)) {
    if (meanOrder) {
      # if we're drawing them in mean order, get the right one same as above
      p <- p + geom_point(x=mean.df[[rev(order(unlist(mean.df)))[i]]], 
                          y=length(mean.df)-i+1, colour="tomato",      # adjust y axis because axis is reversed above
                          alpha=0.5, size=2.0, shape=0, inherit.aes=FALSE)

    } else {
      p <- p + geom_point(x=mean.df[[i]], 
                          y=length(mean.df)-i+1, colour="tomato",      # adjust y axis because axis is reversed above
                          alpha=0.5, size=2.0, shape=0, inherit.aes=FALSE)
    }
  }
  p
}

# use our plot function with only the Segment 4 data
cbc.plot(md.dat[md.segs==4, ], itemCols=classCols) + 
  ylab("Quant Course Offering") +
  ggtitle("Class interest: Segment 4 Only (R)")

##### 10. There is no magic answer in clustering

# we could force a 5-segment solution (for example) in Mclust
library(mclust)
md.mc <- Mclust(md.dat[ , classCols], G=5)

# compare the mclust solution to the previous NbClust solution
table(md.mc$classification, md.segs)
# Rand index for the degree of agreement between them
adjustedRandIndex(md.mc$classification, md.segs)

# UMAP plot of the mclust assignments
set.seed(98107)
# the same segments but in random order
umap.dat$mclustSeg <- factor(md.mc$classification)
# plot those
ggplot(umap.dat, aes(x = X1, y = X2, colour = mclustSeg)) +
  geom_point() +
  xlab("High order dimension 1") + ylab("High order dimension 2") +
  theme_minimal()+
  ggtitle("Dimensional map with mclust segments")

# plot the 5-segment mclust solution as a heat map
seg.heat(md.dat[ , classCols], md.mc$classification)

# and plot the individual distributions for the "R" segment, similar to above
# not discussed in the post; just illustrating the reusable plot function :)
cbc.plot(md.dat[md.mc$classification==3, ], itemCols=classCols) + 
  ylab("Quant Course Offering") +
  ggtitle("Class interest: Mclust R Segment Only")

In this post, we’ll examine and visualize correlation patterns among the items. This also sets up another opportunity (as if I needed one!) to discuss why “statistical significance” in such data is usually unimportant. We can ignore “significance” with the data and yet still learn much from them.

If you haven’t seen it, you could review Part 1 of this series. It describes the data and presents basic data visualization code. Here I pick up where that post ended and discuss additional analyses. As usual, I share and discuss R code along the way; and compile all the code at the end.

First: Get the Data

If you ran the code in Part 1 then you may have the data. You can refer there for explanation; meanwhile, the following code obtains the data needed for this post.

Briefly, this code downloads MaxDiff scores in Excel format from a URL, and does minor clean up. This real data set estimates N=308 individuals’ interest in taking various Quant UX classes, such classes on R, Choice Models, and Segmentation.

# get the data; repeating here for blog post 2, see post 1 for details
library(openxlsx)
md.dat <- read.xlsx("https://quantuxbook.com/misc/QUX%20Survey%202024%20-%20Future%20Classes%20-%20MaxDiff%20Individual%20raw%20scores.xlsx")   # 
md.dat$Anchor <- NULL          # remove the anchor item that is a fixed value of 0
names(md.dat)[3:16] <- c("Choice Models",  "Surveys",       "Log Sequences",    "Psychometrics", 
                         "R Programming",  "Pricing",       "UX Metrics",       "Bayes Stats",
                         "Text Analytics", "Causal Models", "Interviewer-ing",  "Advanced Choice", 
                         "Segmentation",   "Metrics Sprints")
classCols <- 3:ncol(md.dat)    # generally, Sawtooth exported utilities start in column 3

Correlation Analysis: Basic

After reviewing and comparing the preferences for each class, as we did in the previous Post 1, a next question might be whether there are patterns among the potential class offerings.

Specifically, when respondents are interested in one class, what does that say about their interest in other classes? Are there combinations of classes that are associated with one another?

There are several ways to examine such patterns (see, for example, Chapters 4, 11, and 12 in the R book; or Chapters 4, 9, and 10 in the Python book). Here I’ll demonstrate visualization of a Pearson’s r correlation matrix.

Background: In a nutshell, Pearson’s r varies between -1.0 and +1.0. A value of 0.0 indicates no relationship between a pair of variables — in this case, it would mean there is no relation between interest in one class and interest in another. As r approaches +1.0 there is a stronger and stronger pattern of interest in the same direction for both classes together. As r approaches -1.0, it means that interest in one class is increasingly associated with disinterest in the other class.

The R code to visualize a general correlation matrix is not complex. First, we calculate the correlation matrix using cor(). Then we plot it with the corrplot library:

md.cor <- cor(md.dat[ , classCols])

# basic plot
library(corrplot)
corrplot(md.cor)

Here’s the result:

Wow! That’s a great chart for 3 lines of code. We immediately see some interesting combinations. For example, interest in Choice Models and Advanced Choice Models go together; as does interest in UX Metrics and Metrics Sprints. On the other hand, interest in Choice Models is negatively associated with Quant Interviewer training.

If you’ve read any of my R posts before, you know what’s next: we can do better!

Correlation Analysis: Improving the Visualization

One problem with the correlation plot above is that it is unordered with respect to the estimates themselves; items are given merely in the order as listed in the data.

A more useful correlation plot reorders the items in terms of similarity (i.e., clustering). Then we can see whether there are groups of items with similar patterns. Even better, we might draw boxes around those groups to highlight them.

That’s also straightforward in corrplot. Here’s the code:

# correlations with clustering
corrplot(md.cor, method = "ellipse",
         order = "hclust", addrect = 4)

To explain the options, method=”ellipse” converts the circles into ellipses that indicate strength of association (by narrowness) and direction (by slope). By using order=”hclust”, we ask corrplot to cluster the items for similarity. The addrect=4 option draws 4 boxes to highlight the clusters. Generally you pick that number according to what makes sense with some trial and error (but see below for more discussion of finding a “good” number of clusters!)

Here’s the result:

In this chart we see 4 groups of classes: Text Analytics, which is relatively unrelated to interest in any other particular class; a group for “core UX” topics (metrics, surveys, and interviewing); a group for “statistics” (R, Bayes, etc.); and a group for more “marketing” type classes (pricing, choice models, segmentation).

All of that makes complete sense!

I use clustered correlation plots routinely. They are extremely helpful and are a fast way to identify patterns of interest. I often add the correlation coefficients (but rarely show those to stakeholders!) corrplot can do that:

corrplot.mixed(md.cor, upper = "ellipse",
               order = "hclust", 
               tl.pos = "lt", number.cex = 0.75)

This uses the corrplot.mixed() function to visualize the correlations as ellipses and also show their values. The final options tell it put labels on the left and top (tl.pos=”lt”) and shrink the coefficient text to fit better (number.cex=0.75).

Here’s the result:

Statistically Significant? Don’t Ask!

Because the chart above shows correlation coefficients (Pearson’s r coefficient, as noted above), you might wonder, “are they statistically significant?”

My recommendation is to avoid the questions of “significance”. We might discuss that theoretically, which is a long discussion — see the topic as discussed in any of my books listed above. However, I will give a few short answers here … plus a visual demonstration.

First, the correlations here are post-hoc examinations of many coefficients. They do not come from any specific hypothesis test, and thus the usual assumptions of significance testing do not apply. Second, in practice we care much more about the absolute strength of association rather than the binary question of “significance”. Third, with enough sample, almost everything will be “significant” … and that doesn’t help us decide anything. I’ll say more about that in a moment.

Fourth, and most importantly, significance testing compares associations in the data to a null hypothesis of “no association” … and the idea of “no association” is a completely unreasonable hypothesis for such data. How could we possibly think — or be interested to test — that interest in one class by default is completely unrelated to interest in another class? That notion is easily dispelled simply by talking with people who take classes … or, as we’ll see in moment, by data!

You might be thinking, “show me the code!” Well, here it is!

We can test the association between a single pair of items with the cor.test() function. For example, the association between Text Analytics and Metrics Sprints is one of the smaller r values in the chart above. We can test it with:

# correlation testing (not recommended in this case, just demonstrating)
cor.test(md.dat[, "Text Analytics"], 
         md.dat[, "Metrics Sprints"])

R tells us that the Pearson’s r value is 0.1234 with a CI of (0.0119, 0.2320) … which is “significant” under classical assumptions (or mis-assumptions, as I mention above).

That’s just one pair of items, among the 14 * 13 / 2 = 91 total pairs. How many of those 91 are significant? We do a pairwise test for all pairs with cor.mtest() (“m” for “matrix” or “multiple” as you prefer). Then we compare the p-value to the alpha level of interest — let’s say 0.05 — and find the proportion (using mean() on the boolean comparison) that are “significant”. That’s easier than it might sound:

# first get the CIs for each pair
ptest = cor.mtest(md.dat[ , classCols], conf.level = 0.95)

# what proportion of those are "significant" ?
mean(ptest$p[upper.tri(ptest$p)] < 0.05)

Code details. The second line there should be unpacked slightly. First, we want to compare the p-values (saved in the object ptest as ptest$p) to a threshold value for “significance”, in this case 0.05. After making those comparisons, we take the mean() in order to find often that comparison is TRUE. However, the ptest object includes all pairs of comparisons, including comparison of each item with itself, which always has correlation of r\=1.0. In order to exclude those comparison, we select only the “upper triangle” of the matrix of comparisons, indexing the results with upper.tri().

The answer: 71% of all the associations in these data are significant.

One might think, “wow, that’s great!” … but it’s not. First of all, it merely says that people who are interested in some classes tend to be systematically interested or disinterested in other, related classes. That is not much of a discovery!

Second, it’s not great because — when treated as a binary, “significant” or not — it makes the situation less actionable in practice, rather than more actionable.

To see why the binary question of significance leads to less actionable insight, we can visualize what’s happening. That’s next.

Visualizing the Number of “Significant” Pairs

With corrplot, we can add annotation to charts based on on statistical significance (or, in fact, any matrix that has the same dimensions as the correlation matrix). The steps to do that are (1) to get a matrix with the p-values, as we already did above (ptest), and (2) add that to the correlation plot, indicating how to annotate it.

Given that we already have the matrix ptest with results of significance testing, we can annotate the correlation plot like this:

corrplot(md.cor, method = 'ellipse', 
         order = "hclust", addrect = 4, 
         p.mat = ptest$p,
         sig.level = c(0.001, 0.01, 0.05), pch.cex = 0.8,
         insig = "label_sig")

The last 3 lines are new here. First, we use p.mat=ptest$p to point to the p-values of interest. Next we set cutoff points for annotating the p-values, where sig.level=c(0.001, 0.01, 0.05) says to use 3 asterisks for p<.001, 2 asterisks for p<.01, and 1 asterisk for p<.05. You could change those cutoffs. pch.cex=0.8 shrinks the asterisks to fit better in the cells. Finally insig="label_sig" plots increasing levels (as set by sig.level), as opposed to a single binary flag for significance.

Here’s the result:

The key takeaway of this chart — compared to the ones above — is that almost everything is “significant” (under the [mis-]conception of significance discussed above). There are a few exceptions, especially for the Pricing class, but the general pattern is this: if we know that someone is interested or disinterested in one class, we can make pretty good guesses about what other classes might interest them.

But that is not a surprise! Of course we expect people to share interest — and knowing that their interests go together “significantly” adds nothing actionable to our understanding.

Put differently, when more than 70% of the boxes on the chart are “significant", there’s nothing in particular we can do with that information.

Instead of “Significance”

So what should we do instead?

First, if a stakeholder answers whether your results are statistically significant, the answer is NOT to dive into any discussion, and certainly not into any coefficients, p-values, or charts with asterisks. Here’s how I recommend to answer:

Q: Is this statistically significant?

A: Yes! And the actions we should take are …

How do we arrive at the actions to take? Consider evidence such as:

1. What are the overall patterns? You might, for example, consider the clusters of classes that we saw above.

2. How strong are the patterns? We could consider the strength of association — not just whether it is “significant” but how strong it is. Any association above r=0.5 or thereabouts is a strong relationship! (For more, try a search for “cohen correlation analysis pearson r strength”. Or see “Learning More” below.)

3. Are they actionable? This relates to our business goal. For example, consider in the results here, that Segmentation is closely associated with both Text Analytics and Logs Analysis. Now, suppose we’re going to teach segmentation — because it was the #1 ask as we saw in the previous blog post #1 — and that it will be offered as a half day class. The results suggest that we might add Text Analytics or Logs Analysis as a second half day, in order to maximize appeal to the students taking Segmentation. Or instead — depending on our strategy — it might suggest not teaching one of those, if we want to reach different sets of students between the two half-day sessions.

The key point is this: is you have enough data to feel good about your sample, and it is quality data, then you generally should not worry about “significance”. Worry instead about how to make strategic recommendations and how to help your stakeholders focus on those!

Final Note: More on the Item Clusters

There is one thing I should emphasize about the clusters here. They are clusters of items — which classes go together in their patterns of interest. They are not clusters of respondents!

Thus, for example, although the patterns clearly identify a “core UX” group of classes (among others), it doesn’t tell us how many people would be interested in that group, nor how strongly they would be interested (or disinterested).

To examine whether there are groups of people who share interest patterns, we need to look at clustering (segmenting) the respondents. That will be the topic of Post #3 in this series — stay tuned!

Learning More

As mentioned, stay tuned for Post #3 that will examine clusters of people (i.e., segmentation). That’s a different question than the clusters of items that we considered here.

To learn more about correlation analysis and related methods, check out Chapters 4, 11, and 12 in the R book, and Chapters 4, 9, and 10 in the Python book. A canonical, classic text for such analyses is Cohen, et al, Applied Multiple Regression / Correlation Analysis for the Behavioral Sciences (link).

And if you’d like to develop hands-on skills in Quant methods — including choice modeling surveys like the one that yielded today’s data — check out the classes from the Quant UX Association (full disclosure, I teach some of them but there are several other instructors lined up, too!)

All the R Code

As always, I compile the complete R code in one place. Here it is:

# get the data; repeating here for blog post 2, see post 1 for details
library(openxlsx)
md.dat <- read.xlsx("https://quantuxbook.com/misc/QUX%20Survey%202024%20-%20Future%20Classes%20-%20MaxDiff%20Individual%20raw%20scores.xlsx")   # 
md.dat$Anchor <- NULL
names(md.dat)[3:16] <- c("Choice Models",  "Surveys",       "Log Sequences",    "Psychometrics", 
                         "R Programming",  "Pricing",       "UX Metrics",       "Bayes Stats",
                         "Text Analytics", "Causal Models", "Interviewer-ing",  "Advanced Choice", 
                         "Segmentation",   "Metrics Sprints")
classCols <- 3:ncol(md.dat)    # generally, Sawtooth exported utilities start in column 3

##### 4. Correlation
md.cor <- cor(md.dat[ , classCols])

# basic plot
library(corrplot)
corrplot(md.cor)

# correlations with clustering
corrplot(md.cor, method = "ellipse",
         order = "hclust", addrect = 4) 

# add correlation coefficients
corrplot.mixed(md.cor, upper = "ellipse",
               order = "hclust", 
               tl.pos = "lt", number.cex = 0.75)

# correlation testing (not recommended in this case, just demonstrating)
cor.test(md.dat[, "Text Analytics"], 
         md.dat[, "Metrics Sprints"])

# plot with significance highlighted
# first get the CIs for each pair
ptest = cor.mtest(md.dat[ , classCols], conf.level = 0.95)

# what proportion of those are "significant" ?
mean(ptest$p < 0.05)

# add asterisks to the plot for "significant" at p=0.05 level
corrplot(md.cor, method = 'ellipse', 
         order = "hclust", addrect = 4, 
         p.mat = ptest$p,
         sig.level = c(0.001, 0.01, 0.05), pch.cex = 0.8,
         insig = "label_sig")

Before jumping into today’s topic, I will highlight the previous post in case you missed it: guest author — and coauthor of the Quant UX book — Kerry Rodden discusses HEART metrics. It updates Kerry’s classic description of HEART with new reflections on how to make UX metrics succeed in practice. Read it here!

Today I begin a series of posts that demonstrate working in R with individual level estimates from choice models such as MaxDiff and Conjoint Analysis. Those are the best estimates of preference for each respondent to a choice survey. I hope they will inspire you both to run choice surveys and to learn more from them!

Background and Assumptions

Assumptions: I assume that you know what a choice model survey is, and — at a high level — what individual scores are. If not, check out this post about MaxDiff surveys. Or take one of the upcoming Choice Modeling Master Classes from the Quant Association! Or for even more, find sections on MaxDiff in the Quant UX book and discussions of Conjoint Analysis in the R book and Python book.

However, to recap briefly, individual scores are the best estimates for each respondent who completed a choice survey. For example, they report each respondents’ interest — as estimated by the model — for every one of the items tested in the survey.

Data. Here I discuss data from N=308 UX Researchers who took a MaxDiff survey about potential classes from the Quant UX Association. As with most Quant Association surveys, the respondents agreed that we could use the (anonymous) data publicly. Yay! It is great to discuss realistic choice model data. The survey included 14 potential classes. Here is an example MaxDiff screen:

Data Format. The estimates here were estimated by Sawtooth Software’s Discover product. The code here begins with an Excel file exactly as exported by Discover. Each row is represents one respondent’s interest in each of the potential classes.

Note: Without going into the details of hierarchical Bayes (HB) models, I’ll note that the HB process gives 1000s of estimates for each respondent. The values here — the ones that most analysts would use — are the mean (average) of those for each person. Although the estimates are somewhat uncertain for one person, the overall set of estimates are an excellent representation for the group.

R code. I assume you can generally follow R. A key goal of this post is to share R code that will help you get accelerate your own work. (More in the R book!)

You should be able to follow along live in R — all of the data and code are shared here. In each block of R code, you can use the copy icon in the upper right to copy the code and paste it into R to follow along. Or see the end of the post for all of the code in one big chunk.

Packages. You may need to install extra R packages along the way. If you get an error running a “library()” command, you probably need to install that package.

Load the Data

Sawtooth Discover exports individual estimates as Excel (.xlsx) files. I’ve uploaded that file, and can download it directly from its URL using the openxlsx package:

# Read Excel data as saved by Sawtooth
library(openxlsx)   # install if needed
# online location (assuming you are able to read it)
md.dat <- read.xlsx("https://quantuxbook.com/misc/QUX%20Survey%202024%20-%20Future%20Classes%20-%20MaxDiff%20Individual%20raw%20scores.xlsx")   #

It’s helpful to do a bit of minor cleanup. First of all, in Anchored MaxDiff, each item is compared to an “anchor” of whether there is positive interest or not. The anchor estimate is always exported as “0”, so we can remove that as non-informative.

Second, the file sets column (variable) names that are the entire text of each item. It’s helpful to replace those with shorter versions that are easy to read.

Those steps are:

# Some minor clean up
# remove the "Anchor" item that is always 0 after individual calibration
md.dat$Anchor <- NULL
# Assign friendly names instead of the long names, so we can plot them better
names(md.dat)[3:16]  # check these to make sure we're renaming correctly
names(md.dat)[3:16] <- c("Choice Models",  "Surveys",       "Log Sequences",    "Psychometrics", 
                         "R Programming",  "Pricing",       "UX Metrics",       "Bayes Stats",
                         "Text Analytics", "Causal Models", "Interviewer-ing",  "Advanced Choice", 
                         "Segmentation",   "Metrics Sprints")

Next, I use str() and summary() to do some basic data checks:

# Basic data check
str(md.dat)
summary(md.dat)

Finally, I set a variable for the item columns (columns 3:[end]) so I can refer to them easily, without a magic number “3” showing up multiple times later:

# Index the columns for the classes, since we'll use that repeatedly
classCols <- 3:ncol(md.dat)    # generally, Sawtooth exported utilities start in column 3

With those lines, we have data! It’s time to look at the results.

Initial Chart: Overall Estimates

The first thing we might want to know — or that stakeholders will ask — is which classes are most desired. In the spreadsheet world, that might be done by finding the average level of interest and then creating a bar chart. We’ll do the same thing here: find the average interest and then generate a bar chart.

In R, there are many different ways to obtain column averages. I’ll use lapply( , mean) … just because it occurred to me first. I use barplot() to plot them:

# First, get the average values for the classes
mean.dat <- lapply(md.dat[ , classCols], mean)

# Plot a simple bar chart of those averages
# first set the chart borders so everything will fit [this is trial and error]
par(mar = c(3, 8, 2, 2))
# then draw the chart
barplot(height = unlist(mean.dat), names.arg = names(mean.dat),    # column values and labels
        horiz = TRUE, col ="darkblue",                           # make it horizontal and blue
        cex.names = 0.75, las = 1,                                   # shrink the labels and rotate them
        main = "Interest Level by Quant Course")

In the plot, I set margins for the graphics window with par(mar=…) to make it fit. That is a trial and error process for base R plots. I set the barplot() to be horiz[ontal] and slightly shrink (cex.names=0.75) and rotate the labels (las=1) to be readable.

Here’s the result:

It is very, well, spreadsheet-ish. We see the winner (segmentation) and loser (interviewer training) … but we can do much better!

Better Chart: Estimates with Error Bars

In the chart above, we can see the averages but have no insight into the distribution. Are the averages strongly different? Or are they very close in comparison to the underlying distributions? Put differently, is the “winner” (segmentation) really much stronger than the next three options (psychometrics, etc.) or is it only slightly better?

A better chart would show error bars for the means, so that we can tell whether differences are — to use the common but somewhat misleading term (for reasons I’ll set aside) — “significant”. We’ll do that by using the geom_errorbar() visualization option in ggplot2.

I do lots of choice models and have to make plots like this all the time. I often make similar plots repeatedly, plotting different subsets of data, different samples, and the like. For that, it is useful to make a function for such plots. I can simply reuse one function and not rely on error-prone copying and repetition of code.

Here’s a relatively basic function to plot average estimates from an anchored MaxDiff with error bars. (For other estimates such as general MaxDiff or conjoint, it will also work; just change the xlab() label, either in the function or by adding it afterward.)

plot.md.mean <- function(dat, itemCols) {
  library(ggplot2)
  library(reshape2)
  # warning, next line assumes we're using Sawtooth formatted data!
  md.m <- melt(dat[ , c(1, itemCols)])   # add column 1 for the respondent ID
  # put them in mean order
  library(forcats)
  md.m$variable <- fct_reorder(md.m$variable, md.m$value, .fun = mean)

  p <- ggplot(data = md.m, aes(x = value, y = variable)) +
    # error bars according to bootstrap estimation ("width" is of the lines, not the CIs)
    geom_errorbar(stat = "summary", fun.data = mean_cl_boot, width = 0.4,) +
    # add points for the mean value estimates
    geom_point(size = 4, stat = "summary", fun = mean, shape = 20) +
    # clean up the chart
    theme_minimal() +
    xlab("Average interest & CI (0=Anchor)") +
    ylab("Quant Course") 

  p 
}

There’s not a lot to say about this function. First, it melts the data to fit typical ggplot patterns. It adds the identifier column (1) for melt(); that would need adjusting if you have differently formatted data. Then it calls fct_reorder() from the forcats library to put the labels into order — in this case, to order them by the mean value of the grouped data. The error bars are plotted by geom_errorbar(), and that uses the mean_cl_boot option to find the confidence intervals by bootstrapping. (That function is in Hmisc, another potential package to install). Finally, after plotting the error bars, it adds the actual mean points with geom_point().

Now that we have a function, it is a simple command to plot the data:

plot.md.mean(md.dat, classCols)

Here’s the result:

Now we can see that the segmentation class is a fairly strong #1, while the next 3 classes (psychometrics, choice, surveys) are essentially tied. Among the 14 classes, 13 have average interest greater than zero — the MaxDiff anchor — while interviewer training falls below the anchor on average.

As a final note, because the function returns a ggplot object “p”, we could add other ggplot2 options. For instance, we might add “+ ggtitle(“My title!”)” to add a title or change the y axis label with “+ ylab(“a different label”)”

The drawback of this plot is the following: it assumes that we want to know how classes compare in average values, according to statistical significance. In actual practice, that is not usually the case.

Why not? We are usually uninterested in averages and their confidence intervals? Because most often, practitioners need to know how many respondents are interested in something, and how many of them are strongly interested or disinterested. We do not reach any “average” customer — we reach individuals.

So although an average chart with error bars is a good high-level view, there is more to learn.That brings us to my favorite chart: individual distributions!

Another Great Chart: Individual Distributions

There is one important question that an average chart — such as the ones above — cannot answer. That question is: “OK, this is the best item on average … but which item has people who are the very most interested in it?”

For that, it is helpful to examine the individual distributions — not only where respondents are on average but whether there are groups who differ strongly in interest above or below the average.

As you probably guessed, I’ll plot it with a reusable function! Here’s the function. It’s long but I’ll break it down below.

cbc.plot <- function(dat, itemCols = 3:ncol(dat), 
                     title = "Preference estimates: Overall + Individual level", 
                     meanOrder = TRUE) {

    # get the mean points so we can plot those over the density plot
  mean.df <- lapply(dat[ , itemCols], mean)

  # melt the data for ggplot
  library(reshape2)
  #                       vvvv  assumes Sawtooth order;       vvv  (ID in col 1, remove RLH in col 2)
  plot.df <- melt(dat[, c(1, itemCols)], id.vars = names(dat)[1])

  # get the N of respondents so we can set an appropriate level of point transparency
  p.resp  <- length(unique(plot.df[ , 1]))

  # optionally and by default order the results not by column but by mean value
  # because ggplot builds from the bottom, we'll reverse them to put max value at the top
  # we could use fct_reorder but manually setting the order is straightforward in this case
  if (meanOrder) {
    plot.df$variable <- factor(plot.df$variable, levels = rev(names(mean.df)[order(unlist(mean.df))]))
  }

  #### Now : Build the plot
  # set.seed(ran.seed)   # optional; points are jittered; setting a seed would make them exactly reproducible
  library(ggplot2)
  library(ggridges)

  # build the first layer with the individual distributions
  p <- ggplot(data=plot.df, aes(x=value, y=variable, group=variable)) +
    geom_density_ridges(scale=0.9, alpha=0, jittered_points=TRUE,
                        rel_min_height=0.005,
                        position="points_sina",
                        # set individual point alphas in inverse proportion to sample size
                        point_color="blue", point_alpha=1/sqrt(p.resp),
                        point_size=2.5) +
    # reverse y axis to match attribute order from top
    scale_y_discrete(limits=rev) +                                                   
    ylab("Level") + xlab("Relative preference (blue=individuals, red=average)") +
    ggtitle(title) +
    theme_minimal()

  # now add second layer to plot with the means of each item distribution
  for (i in 1:length(mean.df)) {
    if (meanOrder) {
      # if we're drawing them in mean order, get the right one same as above
      p <- p + geom_point(x=mean.df[[rev(order(unlist(mean.df)))[i]]], 
                          y=length(mean.df)-i+1, colour="tomato",      # adjust y axis because axis is reversed above
                          alpha=0.5, size=2.0, shape=0, inherit.aes=FALSE)

    } else {
      p <- p + geom_point(x=mean.df[[i]], 
                          y=length(mean.df)-i+1, colour="tomato",      # adjust y axis because axis is reversed above
                          alpha=0.5, size=2.0, shape=0, inherit.aes=FALSE)
    }
  }
  p
}

In the first couple of lines of the function, it finds the average value for each item using lapply(), the same as we already saw above. That’s so we can add those as a separate layer on the plot later. Then it melts the data, again just like saw above.

Next, it finds the total N of respondents and saves that as p.resp. Why? Because when we plot the individuals, we want to set a transparency alpha value. Setting alpha in inverse proportion to the (square root of the) number of respondents makes those points more legible.

By default, it puts the labels into their mean order, using the averages we calculated instead of fct_reorder() as above (everything in R has multiple good options!)

The next two big chunks build the plot in two states. The first big chunk uses the ggridges package to plot geom_density_ridges() density curves for the individual distributions. I won’t try to explain those; just look at the chart below! Its options add individual points to the curves, and sets a transparency alpha as I described above.

The second big chunk adds points to the chart, overlaying the density ridges with the average values for each item. To do that, it iterates over the items with a for() loop, and then adds the point in the proper place according to whether the items are displayed in sorted order or not.

We call the plot with a simple command, adding a custom y axis label:

cbc.plot(md.dat, itemCols=classCols) + ylab("Quant Course Offering")

Here’s the result:

Wow! This chart has a lot of great information.

I won’t interpret it in complete detail but will note a couple of interesting features. First, it reinforces that Segmentation is a strong #1 option — not only does it have the highest average, more than 90% of respondents show positive interest greater than the anchor value of 0. We also see at the upper end of interest — the right hand side — that Segmentation has many more respondents with particularly strong interest (greater than a value of 5.0, to choose an arbitrary point) than any other class.

However, we see some other things with subsets of respondents with high interest. For example, although the R Programming course is a weak #11 out of 14 in average interest, it has a small number of respondents showing the strongest interest of anyone in any class.

When we consider that hands-on classes are small, and reach only the people interested in them, these results suggest that an R class could be an good offering, even if the average is lower. We don’t care how people are uninterested — we only care whether we can reach enough people who are interested!

With that, I’ll leave you to inspect the chart and find other interesting ideas.

Coming up in Post 2

In the next post, I’ll go farther with these data and examine:

• Correlations among interests: if they like X, what else do they like or dislike?

• Finding clusters of classes that go together (item clusters — we’ll look at respondent clusters in post 3)

• … and later posts will look at respondent segmentation and (briefly) data quality

Stay tuned for Post #2 in a few days!

Meanwhile, if you’re interested in more about choice models and/or R, check out this post about MaxDiff surveys; and upcoming Choice Modeling Master Classes from the Quant UX Association; and sections on MaxDiff in the Quant UX book and Conjoint Analysis in the R book or Python book. Also, for more experienced choice modelers, I recently shared this post about misunderstandings of Conjoint Analysis.

All the Code

As promised, following is all of the R code from this post. You can use the copy icon in the upper right to grab it all at once, and paste into RStudio or wherever you code.

Cheers!

# blog scripts for analysis of individual level MaxDiff data
# Chris Chapman, October 2024

##### 1. Get individual-level mean beta estimates as exported by Sawtooth Software

# 1a. read Excel data as saved by Sawtooth
library(openxlsx)
# online location (assuming you are able to read it)
md.dat <- read.xlsx("https://quantuxbook.com/misc/QUX%20Survey%202024%20-%20Future%20Classes%20-%20MaxDiff%20Individual%20raw%20scores.xlsx")   # 

# 1b. Some minor clean up
# remove the "Anchor" item that is always 0 after individual calibration
md.dat$Anchor <- NULL
# Assign friendly names instead of the long names, so we can plot them better
names(md.dat)[3:16]  # check these to make sure we're renaming correctly
names(md.dat)[3:16] <- c("Choice Models",  "Surveys",       "Log Sequences",    "Psychometrics", 
                         "R Programming",  "Pricing",       "UX Metrics",       "Bayes Stats",
                         "Text Analytics", "Causal Models", "Interviewer-ing",  "Advanced Choice", 
                         "Segmentation",   "Metrics Sprints")

# 1c. Basic data check
str(md.dat)
summary(md.dat)

# 1d. Index the columns for the classes, since we'll use that repeatedly
classCols <- 3:ncol(md.dat)    # generally, Sawtooth exported utilities start in column 3


##### 2. Plot the overall means

# 2a. The easy way (but not so good)
# First, get the average values for the classes
mean.dat <- lapply(md.dat[ , classCols], mean)

# Plot a simple bar chart of those averages
# first set the chart borders so everything will fit [this is trial and error]
par(mar=c(3, 8, 2, 2))
# then draw the chart
barplot(height=unlist(mean.dat), names.arg = names(mean.dat),    # column values and labels
        horiz = TRUE, col ="darkblue",                           # make it horizontal and blue
        cex.names=0.75, las=1,                                   # shrink the labels and rotate them
        main ="Interest Level by Quant Course")


# 2b. A somewhat more complex (but much better) way
#     we'll make this a function ... it's often good to make anything long into a function :)
plot.md.mean <- function(dat, itemCols) {
  library(ggplot2)
  library(reshape2)
  # warning, next line assumes we're using Sawtooth formatted data!
  md.m <- melt(dat[ , c(1, itemCols)])   # add column 1 for the respondent ID
  # put them in mean order
  library(forcats)
  md.m$variable <- fct_reorder(md.m$variable, md.m$value, .fun=mean)

  p <- ggplot(data=md.m, aes(x=value, y=variable)) +
    # error bars according to bootstrap estimation ("width" is of the lines, not the CIs)
    geom_errorbar(stat = "summary", fun.data = mean_cl_boot, width = 0.4,) +
    # add points for the mean value estimates
    geom_point(size = 4, stat = "summary", fun = mean, shape = 20) +
    # clean up the chart
    theme_minimal() +
    xlab("Average interest & CI (0=Anchor)") +
    ylab("Quant Course") 

  p 
}
# call our plot
plot.md.mean(md.dat, classCols)



##### 3. Even better: Plot the distributions (individual estimates)

cbc.plot <- function(dat, itemCols=3:ncol(dat), 
                     title = "Preference estimates: Overall + Individual level", 
                     meanOrder=TRUE) {

    # get the mean points so we can plot those over the density plot
  mean.df <- lapply(dat[ , itemCols], mean)

  # melt the data for ggplot
  library(reshape2)
  #                       vvvv  assumes Sawtooth order;       vvv  (ID in col 1, remove RLH in col 2)
  plot.df <- melt(dat[, c(1, itemCols)], id.vars=names(dat)[1])

  # get the N of respondents so we can set an appropriate level of point transparency
  p.resp  <- length(unique(plot.df[ , 1]))

  # optionally and by default order the results not by column but by mean value
  # because ggplot builds from the bottom, we'll reverse them to put max value at the top
  # we could use fct_reorder but manually setting the order is straightforward in this case
  if (meanOrder) {
    plot.df$variable <- factor(plot.df$variable, levels = rev(names(mean.df)[order(unlist(mean.df))]))
  }

  #### Now : Build the plot
  # set.seed(ran.seed)   # optional; points are jittered; setting a seed would make them exactly reproducible
  library(ggplot2)
  library(ggridges)

  # build the first layer with the individual distributions
  p <- ggplot(data=plot.df, aes(x=value, y=variable, group=variable)) +
    geom_density_ridges(scale=0.9, alpha=0, jittered_points=TRUE,
                        rel_min_height=0.005,
                        position="points_sina",
                        # set individual point alphas in inverse proportion to sample size
                        point_color = "blue", point_alpha=1/sqrt(p.resp),
                        point_size=2.5) +
    # reverse y axis to match attribute order from top
    scale_y_discrete(limits=rev) +                                                   
    ylab("Level") + xlab("Relative preference (blue=individuals, red=average)") +
    ggtitle(title) +
    theme_minimal()

  # now add second layer to plot with the means of each item distribution
  for (i in 1:length(mean.df)) {
    if (meanOrder) {
      # if we're drawing them in mean order, get the right one same as above
      p <- p + geom_point(x=mean.df[[rev(order(unlist(mean.df)))[i]]], 
                          y=length(mean.df)-i+1, colour="tomato",      # adjust y axis because axis is reversed above
                          alpha=0.5, size=2.0, shape=0, inherit.aes=FALSE)

    } else {
      p <- p + geom_point(x=mean.df[[i]], 
                          y=length(mean.df)-i+1, colour="tomato",      # adjust y axis because axis is reversed above
                        alpha=0.5, size=2.0, shape=0, inherit.aes=FALSE)
    }
  }
  p
}

cbc.plot(md.dat, itemCols=classCols) + ylab("Quant Course Offering")

I led the early Quant UX Research team at Google, where we came up with the HEART framework to help teams define metrics of user experience. More than a decade ago, I wrote this blog post about UX metrics that introduced HEART to a wider audience outside of Google.

Since then, it has been applied by teams across the tech industry, and has found its way into many introductory resources about user experience and product management, including the popular book Escaping the Build Trap. Those resources generally cover similar ground to my original blog post, and there is little published material about what happens when teams actually apply HEART to real projects.

In this post, I'll go beyond those basic concepts and share some new perspectives on how to apply HEART in practice, based on material from Chapter 7 of the Quantitative UX Research book.

It's divided into four sections:

• What is the HEART framework?
• Don't skip the Goals-Signals-Metrics process
• Avoid individual pitfalls
• Look out for organizational challenges

With this knowledge, you'll have a better chance of helping your team reach a successful implementation of user experience metrics.

What is the HEART Framework?

HEART helps teams break down the broad concept of "user experience" into more specific, measurable outcomes. It also encourages teams to consider multiple aspects of the user experience when defining metrics, although it does not cover every possible aspect.

The acronym stands for:

1. Happiness: Measures of user attitudes, often collected via surveys. This might include satisfaction or perceived ease of use.

2. Engagement: The level of user involvement with a product, typically measured by frequency, intensity, or depth of interaction. For example, the number of visits during a certain time period, or the usage of key features.

3. Adoption: How many new users start using a product or feature. Making an explicit distinction between new and existing users helps a team to understand growth.

4. Retention: The rate at which existing users return to the product. This can be thought of as a long-term version of engagement. Some teams focus more specifically on failure to retain, which is known as "churn".

5. Task success: The efficiency, effectiveness, and error rate of user actions. This category often yields the most useful metrics for UX changes, provided that task-specific data is available.

Don't Skip the Goals-Signals-Metrics Process

HEART 💖 has a fun acronym that makes it easy to remember. Teams often get enthusiastic about it and jump straight to brainstorming metric ideas, because they want to get started on building a dashboard.

However, this is very unlikely to lead to a successful outcome. Metrics are not useful unless they are aligned closely with the team's high-level goals... and many teams are surprisingly unclear about what those are, especially in terms of user experience.

The Goals-Signals-Metrics process is designed to help with this problem by encouraging teams to start by thinking at a higher level.

1. Goals: Using the HEART framework as inspiration, define the overarching objectives for your product or feature. This step involves team discussions to align on priorities and user experience goals, including explicitly addressing the inevitable disagreements. Omit HEART categories that are less relevant to your project.

2. Signals: For each goal, identify possible signals — ways that success or failure might manifest in user behavior or attitudes. Map the goals to the data that you are (or could be) collecting about user experience. Consider both the ease of tracking these signals and their likely sensitivity to design changes.

3. Metrics: Develop specific, quantifiable measurements based on the signals. This involves steps like figuring out how to analyze the low-level signals (e.g., using averages or percentages) and deciding on appropriate time periods for aggregation.

By following this process, teams can create meaningful metrics that align closely with their product goals and user experience priorities. To give a simple example, a goal of "make the upload process easier" might map to signals relating to completion of each stage of the process, or, alternatively, to responses to an inline survey question about ease of use. The signal about completion could translate to a specific metric like "the percentage of times a user finishes the upload flow successfully, having started it in the past 7 days".

Two important aspects of the process that I want to emphasize:

• You Must Prioritize: Focus on implementing metrics related to your top goals. It's better to have a few well-chosen metrics than an overwhelming dashboard.

• You Must Iterate: When you've gone through the process the first time, you are not done. As you collect data and gain insights, be prepared to refine your choices over time.

Avoid These Individual Pitfalls

Certain issues come up over and over again for individual quantitative UX researchers when they try to apply HEART and Goals-Signals-Metrics. I still make some of these mistakes myself!

1. Neglecting Team Involvement

While it might seem efficient to develop metrics independently, this approach can backfire. Engaging your team throughout the process:

• Increases ownership and buy-in
• Improves the quality of metric ideas
• Enhances the impact of the eventual assessments

I suggest scheduling a collaborative session to work through the first part of the Goals-Signals-Metrics process with key members of your team: agree on goals, and brainstorm possible signals.

2. Starting Too Ambitiously

The HEART framework is most useful when applied to specific projects with engaged teams. Avoid the temptation to create organization-wide dashboards immediately. Instead:

• Begin with a single, receptive team
• Focus on details and learn from the experience
• Use this initial project as a case study to inspire other teams

3. Underestimating the Rest of the Process

HEART and Goals-Signals-Metrics are only the beginning of a long process. Just because you used them to come up with a metric idea, that doesn't mean that it's a good or a useful metric, or that it has any correlation with quality of user experience.

Be prepared for:

• Data analysis to refine your signals and metrics
• Iterating on your chosen metrics as you learn more
• Implementation challenges, including instrumentation and dashboard creation

Allocate sufficient time and resources for these crucial next steps.

4. Metric Overload

While HEART can help you generate numerous ideas, implementing too many metrics can be counterproductive. To avoid overwhelming your team and other stakeholders:

• Clearly prioritize your most important metrics
• Consider implementing secondary metrics on a separate dashboard
• Remember that you don't need to use all HEART categories – focus on what's most relevant to your project

Look Out For These Organizational Challenges

Every project takes place in context, and different organizations and teams have different challenges that you can attempt to proactively address, or keep in mind when selecting the right projects to apply HEART.

1. Fear of Evaluation

Introducing targeted metrics can expose project shortcomings, which may create anxiety within teams. To address this:

• If you are in a leadership role, foster a culture of learning from failures (e.g., blameless post-mortems); otherwise, prioritize working with teams who already have this kind of culture
• Involve stakeholders in the metric definition process to build trust and ownership
• Emphasize the value of being informed by data, even when results are unexpected

2. Single Metric Tunnel Vision

While having too many metrics is problematic, focusing on a single metric is also detrimental, but leaders are often drawn to that in an attempt to create clarity and focus. Chris has a separate post about the problems with "North Star" metrics. Remember:

• No metric can be a perfect representation of user experience
• Additional key metrics provide essential checks and balances
• Optimizing for one metric can render it useless for evaluation (Goodhart's Law)

3. Not Considering Ethical Implications

UX metrics are only proxies for user experiences. No metric of user engagement can actually identify how truly engaged a user is, or whether that engagement represents a positive experience for them. For example, time spent in a product is often used as a default metric of engagement, but this may not be appropriate for a given product, especially if unhealthy overuse is a possibility.

When implementing metrics:

• Consider long-term outcomes and potential negative consequences
• Use qualitative research to gain deeper insights into actual user experiences
• Be willing to iterate on metrics as you learn more about what constitutes a positive user experience

Key Takeaways

HEART is a useful tool to help teams focus on the user experience when defining metrics. Here's a summary of the main things to keep in mind when applying it in practice:

• Use HEART as a starting point to consider various aspects of user experience, but don't use all of the categories or force goals to fit into them
• Follow the Goals-Signals-Metrics process to create meaningful, prioritized metrics
• Collaborate with your team to define goals, signals, and metrics
• Start small and grow your approach over time
• Be prepared for the work that comes after initial metric definition
• Prioritize your most important metrics to avoid overload, but don't narrow down to a single metric
• Look out for the organizational challenges that come with negative results
• Consider ethical implications and long-term outcomes

For more detail on all of this, and a case study of a Gmail project, see Chapter 7 of the Quantitative UX Research book. I'm also available for consulting or speaking on these topics 😎

Have you applied HEART with your team? If you have, I encourage you to write about your experiences, so that the rest of the community can build on what you've learned.

This post is mostly an announcement — it's not a retrospective, which I'll save for another time. However, I share a couple of updates on what's next.

Why Now?

Basically, I feel ready. There is no particular external reason or pressure. I'm 56 years old, and am fortunate to be in good health, with a job that was going well.

On the other hand, after 24 years, plus more years of pre-Tech work, I have accomplished "enough" in the corporate world. The challenges in UX looking forward are no longer as exciting to me as they once were. (That's not a commentary on the UX world — only a comment about my perception.)

BTW, I'll give particular thanks to my most recent colleagues at Amazon Lab 126, where I was very happy. This decision is not a reflection on them or that position. (I can say the same for my 11 years each at Google and Microsoft.)

And I have opportunities apart from corporate work, as I mention next!

What's Next?

Many friends have asked me variations of, "What will you do? Don't just quit and watch TV!"

My answer: thanks and don't worry! For one thing, I have not turned on a TV set in 5 days (maybe longer, I don't even recall).

More to the point, I will have 4 primary activities keeping me busy:

• Running the Quant UX Association — I'll lead both the annual conference, Quant UX Con online, and our new in-person training classes.

• Other community: I'm an ordained Zen priest, and last year I received teaching authorization. With corporate retirement, I can also devote more time to that.
  \==> Update April 2025: I am starting the Tech Community Zen group for online practice. More here!

• Hobbies: I have several hobbies. Hobby #1 is boating in the Seattle region and north to British Columbia. We have plans to take our boat all the way to Alaska.

• Catching up on writing. I've overdue on projects such as new editions of my quant book or R book, and/or new books and articles. (You'll notice I list writing after the three "fun" activities above.)

I'm not going anywhere ... but keep in touch!

If you haven't already, follow the Quant UX Association and me on LinkedIn or Bluesky. Also, subscribe to this blog (hit the "follow" button above).

I hope to hear from you ... or even better to see you at Quant UX Con or a Quant class! Cheers,

-- Chris

P.S. The cover photo shows me in Shinjuku National Garden in Tokyo, in winter time. Photo credit to my Google colleague, Katie Tzanidou.

I'm talking about research presentations that are often confused with "portfolios". They are different from actual "portfolios" that compile many projects and that live online. I doubt whether those are valuable for general tech job applicants (who would look at them?) and in any case, they are different.

However, a presentation as described here could be a key building block in a complete portfolio (if you decide you need one).

Over the years as a hiring manager and interviewer, I've sat through perhaps 150 interview candidate presentations. I've also observed 1000s of industry research presentations — and of course I've been on the other side as a job candidate. Here are my reflections.

The short version is this: a UX research presentation should demonstrate how you have defined and answered an interesting research question. I emphasize "presentation", "defined", "answered", and "interesting". The presentation is about research impact and engagement, not about technical proficiency. In this post, I describe the pieces to deliver that.

An event shorter version is this: are you genuinely enthusiastic about the research you're presenting? If so, that is the best presentation!

It's a Presentation, not a "Portfolio"

The term "portfolio" is sometimes misleadingly used to refer to a presentation. It implies that a company wants you to compile an array of materials that describe the range of one's expertise and experience.

That has never been the case anywhere I have worked and hired candidates. Instead, what the hiring team really wants is a clear presentation of research. It should be an example of what they might expect from you, if you're hired.

My recommendation is to plan to present 1 case study ... and to have a 2nd case study as a backup. Almost always I find that the 1st case leads to plenty of questions and discussion, but occasionally it doesn't. I have never — not once in 20+ years — observed a need for more than 2 case studies.

I illustrate my points using slides from my own most recent job presentation (in early 2022, when I applied for my current job). I don't claim that my slides are awesome or that they are a model ... all I claim is that (a) they illustrate my points, and (b) they helped me land my job (and a quite similar approach helped land my previous job).

Throughout, I encourage you to focus on UX — user / customer experience — more than any "Quant" aspect. Impress the audience with what you have to contribute to product research, which is built on, but goes beyond, quant technical expertise.

The Audience has Broad, not Specific Expertise

Approach a research presentation just like you would in a typical stakeholder meeting with intelligent and interested colleagues who are not researchers — or, if they are researchers, who are not experts in your research area.

At companies like Google, Microsoft, and Amazon, candidates often assume that the interviewers are super-intelligent beings who will know and probe everything. That is not the case!

A better assumption is that interviewers are very smart colleagues who have a different specialization than yours. They will catch on quickly and ask great questions — and they will immediately sense when you are bluffing — but they also need to be given grounding in your area. If you're an academic, think of them as quick-minded professors from different but related departments.

Explain the basics briefly and prepare to take questions, but don't talk down to the audience. Don't assume that they are experts in your methods or that they want detailed technical explanations (they'll ask if they do). Minimize assumptions and jargon.

Here's how I introduced the topic in my own portfolio with two slides. You'll notice that they are very simple — they highlight key points I made verbally.

With this kind of framing, the audience knows what I'm going to talk about, and they are immediately focused on a problem statement ... as I discuss in the next section.

Have a Narrative Story with a Dilemma

This is the most important point: your case should have a narrative arc that explains why you did the research — from a business or product point of view — and how it influenced some action.

A good way to frame the arc is with a dilemma that gets resolved. For example:

• We needed to change X, but users wanted Y. Here's how research led to a good compromise.

• The team was pushing A and yet initial tests suggested that users hated A. Here's how research cleared up the situation.

• Our business wanted to expand in B, but the direction was unclear. Here's how research recommended a direction and how it turned out.

Remember that the audience are not specialists in your method. That means that the dilemma should involve the business and customers, not just research. For instance, it is not interesting to present, "Technical method A says X but technical method B says Y. I developed technical method C to resolve them." That is uninteresting to anyone except you and a small number of fellow researchers ... unless it connects very clearly to a customer or business problem.

Following the 2 slides above, here's how I laid out a specific dilemma:

Notice that I focus on a tradeoff involving both customers and the business — and again, it is simple and approachable for any intelligent audience.

A research presentation is a job sample and — as shown in my slide above — it should demonstrate the following two crucial skills for UX researchers:

• Focusing on the customer / user

• Presenting research clearly to colleagues

To be clear, I do not advocate "story telling" in the senses of inventing anything, including artificial drama, sprucing up a presentation with extraneous content or images, or anything like that.

Rather, a presentation should be clear and decision-focused. Clearly focusing ona customer & business dilemma helps to accomplish that.

Minimize What You Say About Yourself

Too many candidates present multiple slides that detail their academic background, previous work experience, projects they've done, and sometimes things like their hobbies, places of residence, and families.

I suggest having only 1 screen about yourself and omit anything that is not particularly relevant or that appears on a resume (such academic background), or is extraneous to the job (such as family and hobbies).

In my most recent portfolio, I did not mention my previous jobs (see resume!) or academic credentials (again, see resume!). Instead, I had one slide about myself ... and it framed how I view the landscape of user research, along with some things I've done:

Why did I use the 1 slide about "me" in this way? Because:

• This framing lets me share my "philosophy" of end-to-end research

• It shows that my focus is not on "me" but on how I approach research

• But is also highlights specific experience and competence

• It lets me link my research philosophy to the case studies I'll present

I then promised a deep dive into two areas ... which shows how and why the framing of this slide "about me" is more directly relevant to my research as such:

It also implicitly communicates, "I'm showing 2 things but I do much more."

Don't Include Confidential Information

A common — and in my opinion, serious — mistake is to include information that is confidential to your current or previous employer.

I've repeatedly seen candidates who present information including:

• Screen mockups and images of unreleased products

• Photos of customers and/or team members

• Names of customers and/or team members

• Statistical analyses of user needs, product behaviors, purchases, willingness to pay, etc., that come from internal, confidential data

• Internal data about finances, sales, staffing, or other business metrics

There are two problems with presenting such confidential information. First, it violates your employment agreement ... and the interviewing company will wonder whether you will do the same thing with them! Second, it demonstrates a lack of attention or competence with a core professional skill for UX researchers: ethical research.

But wait?! What can you show if you can't show actual results? Your audience will understand and appreciate the need for confidentiality. It is OK to have blurred images, hypothetical screen mock ups, similar but disguised products, stock photos of users, and altered statistics that communicate a parallel result.

Even better, if you get approval to publish research periodically in your career, you'll have a collection that is perfectly OK to share. That's not required but is nice when it occurs. Following is an example from my presentation (a similar example is shown in the Case 1 intro above). This slide also shows how to credit colleagues.

Resolve the Narrative Arc with a Decision and Impact

Every good story needs a clear resolution. Your case study should start with a clear dilemma or decision to be made, as noted above. Then you should end with a resolution of that dilemma.

I'm not going to recap my entire presentation, but I will skip ahead to the key research result for Case #1. I used game theory + conjoint analysis to answer the question, and presented the final result like this (after walking through the pieces):

You'll notice that this slide is technical but also quite approachable (especially after building up some of the components, which I did in previous slides). At the same time it is not particularly complex. The goal is to be clear and convincing, including only what is needed to get the point across. That has the nice side effect of making the audience feel smarter (which will make them like you!)

That slide directly answers the dilemma that I started with — "yes, we should invest in developing the new feature" — and it set up the next slide about research impact:

Notice again that I focused clearly on the business and the customers. And I also worked in a not-so-subtle plug for listening to users through research!

For More

Do you want to see my complete job presentation? I presented it live at Quant UX Con 2022 (after my interview) and the complete PDF is here.

Does something here worry you, because you didn't "get it right" in a recent presentation? No one ever gets these things completely right for one simple reason: a portfolio review is an interactive presentation ... and the other side is 50% of the equation. But that side is unknowable in advance.

Sometimes the interviewers are awesome (cue: the colleagues who interviewed me at Microsoft, Google, and Amazon). But sometimes interviewers are jerks (stories I've heard) — and really, it's better to find that out before taking a job. Don't worry about it! Just update your expectations and improve your presentation for the next round.

I hope something here will be useful for you in putting together a Quant UX research presentation. Thanks for reading!