We love Max-Diff! It is the industry gold standard for feature prioritization, and with good reason. It has been documented in journals, articles and white papers countless times how it is superior to typical Likert rating scales. The nature of the task forces respondents to make a trade-off among subsets of items, choosing the “best” and “worst” item within each group. After some modeling, the items are typically scored on a relative scale from 0-100, where both the rank order and the distance from one item to another is observed. And unlike rating scales which tend to have scores clustering on the high end, Max-Diff results in a nice spread of scores clearly indicating which items are relatively superior.
But, how do we know that the winning items are actually appealing to respondents, and not just the best of a set of bad options? Max-Diff scores are relative, meaning they only compare the items to each other. But we don’t have any information about an item’s absolute preference.
Luckily, we have a couple options.
Suppose a potato chip manufacturer wants to test out 10 new flavors and we run a Max-Diff exercise to get the order of preference. From the figure below, we see flavor A is leading the pack, with flavors B & C not far behind, and the rest further down.
We’ve been on the GRIT list of most innovative research companies for five years now. I’m proud of that achievement and of the fact that we’ve moved up 10 places in those five years (many much larger firms rank lower or not at all). I think the key point for me to share though is that we don’t innovate to make the GRIT list, but rather GRIT simply recognizes what is a way of life at TRC.
TRC was founded in 1987 at a time when more than half of all phone interviews were done using hard copy paper and pencil forms and almost no one had a PC on their desk. From the start, every TRC employee had a PC on their desk from interviewers through our top executives. To do this we installed what was, at the time, the largest PC network in the world (PC World Magazine wrote an article on us). From there we adopted digital recording technology so we could quantify quality, and then went on to become very early adopters of using the internet to do surveys.
Beyond data collection, we innovated in techniques. Over the years we created techniques like asymmetrical key driver analysis (which doesn’t assume that all features will have the same positive and negative impact) and Bracket (a more efficient way of doing ranking exercises). We also applied things that we learned from our many academic partners such as Smart Incentives (a gamified incentive aligned method for ideation within quantitative surveys).
We continue to come up with new ways of driving insights. Some are improvements on existing methods (such as better ways to do Discrete Choice) and some are applying new tools to better understand what drives consumer behavior (such as text analytics)....
Folks isolating at home during the COVID-19 pandemic are looking for inexpensive family-friendly ways to entertain themselves. Jigsaw puzzles seem to be fitting that bill, and my family has been doing them since before the shut-down began.
At my house, as we’ve gotten better at doing them, we’ve also gotten more particular about which puzzles to buy. Subject matter, the size and number of the pieces, the construction material, border type and repetitiveness of the patterns all factor into our decision for which puzzles to tackle. We won’t attempt something all in one color palette nor one with rounded edges (that grayscale Moon puzzle circulating social media is a definite NO). But we also don’t want to waste our time on something that is too easy or with juvenile subject matter.
As I’m dreaming of the perfect puzzle, I can easily see how a manufacturer could utilize conjoint to help determine the types of puzzles to design. Puzzle-buying consumers could trade-off puzzle features and price, perhaps even bundling some puzzles together. Suggestions for puzzle subject matter could be generated through a crowdsourcing-style research exercise, such as our Idea Mill™ agile product. The 6 to 36 designs with the most promise could then be winnowed down in an Idea Magnet™ feature prioritization exercise.
So now that I have the entire research program laid out, I just need a jigsaw puzzle company to embrace my research plan and quickly – before I run out of puzzles!
The current Covid-19 crisis has gotten me thinking about all of the big challenges that we have faced at TRC. These include:
• A flood that destroyed our Headquarters
• A fire that destroyed our biggest call center
• Three recessions of varying depths
• Transformational industry changes such as the move from phone to web
We’ve persevered for over 30 years through these and other challenges. I thought it might be helpful to share some of what I learned as a result. Hopefully it will help others navigate the very challenging current business environment.
Of course if you don’t have a plan you’ll more or less have to make it up as you go along. But let this be the last crisis that you haven’t planned for. With each challenge our plans have gotten better. You have to think through even unlikely scenarios. For example, you might have had a plan for people to work from home but did you think through whether your systems can handle that kind of load or whether it can be sustained over a long period of time?
My business’ most important asset is our staff. I’m lucky to work with really talented and smart people. Thing is, they are smart enough to recognize trouble and talented enough to have other options on where to work. Providing them with honest communication on where things stand and what you are going to do is critical. Don’t make promises you can’t keep. Your credibility will carry you far. I’m proud to say that more than half my staff has been with us for 20+ years. That doesn’t happen if you lose credibility....
I’ve been really impressed with the way our company and our clients have reacted to this crisis. First and foremost, we’re taking precautions to slow the spread. We, as well as our clients, moved quickly on things like suspending travel, replacing in-person qualitative with on-line approaches, and working from home. So far this has kept our staff and our clients free of the disease and allowed us to tirelessly meet our clients’ rapidly evolving needs and deadlines.
I’ve also been impressed by the way our clients have reacted from a business standpoint. With all the scary news and panic buying, it is impressive that our clients have reacted to the crisis seriously and thoughtfully. While it might seem insensitive to talk about business issues in a crisis like this, the reality is that we need a thriving economy for the well-being of everyone.
Every recession is different and this one is perhaps the most different of all. It came on more suddenly and was driven not by economic factors like inflation, commodity prices or financial institutions collapsing. With that said, such downturns are not unprecedented. They are referred to as “event driven” recessions. Unlike normal recessions, the event driven variety tend to be short lived with fast recoveries. According to the Goldman Sachs article the stock market typically takes just 15 months to fully recover from an event driven recession as compared to over 100 months for a structural one (such as the financial crisis of 2008). It is important to understand that while it takes 15 months to get back to where the market was pre-bear market, the recovery starts much earlier than that; it might have already started.
Simply put, even as the crisis grows you need to plan for the upsurge to come. In the short term, you need to understand what consumers are thinking and when that thinking will change. Most critically, you need to understand how to connect and the tone your audience will accept from you so you don’t alienate them. Longer term you need to understand how this recession will change behavior. We do so many discrete choice conjoint studies that I tend to think in terms of “features”…so what product and service features will change forever due to behavioral changes? For example, more on-line usage is a no-brainer given the success of this extensive national experiment in remote working....
On a day when I am packing up my office to work from home as a measure to combat the spread of Covid-19, I am faced with a dilemma that we frequently see in research – the need to differentiate the must-haves from the nice-to-haves. We have no idea how long our work-from-home mandate will last, so I’m planning for the long haul.
Technically, I could take home all of my personal possessions, but it’s a lot. I do not work light. My work office is more crammed with stuff than my home office is: an array of plants, pictures, mementos, décor and awards. Colored highlighters, pens and markers. None are mission-critical to my job, so they’ll stay behind (with the exception of one plant that I can’t bear to part with).
But then I have to make the decision as to which work-related items I must bring with me and which ones will enhance my work-from-home experience but aren’t absolutely required. Do I need my pencil sharpener? Note pads? Date book? Spare mouse? Speakers? What about my folders? All of them? Which ones?
I only have so much room in my vehicle, and only so many trips to the parking lot that I’m willing to make. So I need to determine the nature of each of these items and then decide: if it’s critical it goes in the “take” pile. If it’s nice to have, then I need to weigh it (literally) against the other nice-to-have items and take some and leave the rest.
In feature prioritization research, we ask participants to tell us what features they would like to see in a product, whether that’s a consumer good, a service, an app, or another type of product. The problem with just asking it that way is that the sky’s the limit – there’s no space issue, no concern for how many trips they have to make to the parking lot. So we place limiters on the exercise, such as having each item add cost, or telling them they can only have 5 features. This helps separate the critical features from the rest of the pack. But it still doesn’t tell us which are the nice-to-haves; items that could provide differentiation from what else is on the market, but aren’t considered table stakes....
Many of our clients ask for our help with feature prioritization as part of the product development cycle. This typically involves using a choice-based research method, such as Max-Diff, our proprietary Bracket™ or Conjoint analysis. We ask consumers to choose which features are the most meaningful to them by pitting the features (or in the case of Conjoint, pitting products containing various combinations of these features) against each other.
In most cases we focus on the development aspect of product development, which generally means adding features or enhancing those that already exist.
But sometimes, it’s not a matter of adding to, but rather subtracting from, what’s already there.
Consider the case of today’s mobile phones. The phone part of my mobile phone is actually pretty low on the list of the features I use – e-mail, text, and Internet access are way more important to me than making or receiving calls. While phone capability is still a must-have, I wonder how long it will be before the “phone” part of “mobile phone” will no longer be table-stakes in smartphone design....
For the third time, we've been recognized by GRIT as one of the 50 most innovative market research firms in the world. I'm humbled by the endorsement that our clients and peers have given us. It is especially gratifying since we are smaller than most of these firms and don't have the budget that they do for promoting themselves. It got me to thinking about what it is about TRC that overcomes these disadvantages.
Obviously, I could point to innovations we've developed like Bracket™ (a better way of ranking lists of items), Smart Incentives™ (using gamification to drive respondent engagement), advances in conjoint (too long to list here) and our agile suite (rigorous quantitative economical and fast solutions). I know the amount of time we put into developing these and the value they have delivered, but what I hear from clients is that this is only part of what makes TRC special.
Clients consistently tell us that they appreciate working with senior researchers who seek to find the solutions not to a research problem, but to a business problem. When their problem demands a solution such as the ones above, they value it, but they know that our goal is not to dazzle them with the latest tools we have come up with, but rather to do so by getting them the answers they need.
For example, in the last year we have had two clients in very different industries (CPG, Healthcare) come to us with very different problems (sorry I can't share that). To solve both of them we used an old technique (Multi-Dimensional Scaling or MDS). This tool was not designed to solve the problems they had, but by breaking the problem down it was clear to us that it was the right tool to do so. In both cases this solved their business challenge and in one it allowed us to do so at a fraction of the anticipated budget....
I’m a big fan of Russ Roberts’ podcast Econtalk. While technically about economics he covers a variety of subjects with his guests and it always gives me something to think about. Recently he welcomed Mary Hirschfeld of Villanova University to talk about her book “Aquinas and the Market”. She challenges some of the fundamentals of modern Economics.
Economists typically see “rational” behavior as one in which a person attempts to maximize their wealth. This leads to the behavioral economic principles that see choices that don’t maximize wealth as “irrational”. For example, an unemployed baseball fan catches a multi-millionaire player’s 500th home run ball. The baseball is likely worth tens of thousands of dollars so if the fan gives it back to the player that is deemed “irrational”.
Dr. Hirschfeld might see this differently. She feels that economics itself is irrational…it fails to recognize the value of non-monetary aspects of a transaction. The fan might think the player actually deserves the ball more than the fan and thus puts value in doing the right thing…returning the ball to its rightful owner.
She repeatedly makes the point that wealth is a means to an end and that focusing on it will not maximize your value as a human being. The rational human thinks about what they really need materially, spiritually and emotionally. They can then make the trade-offs necessary to maximize their overall well-being. They might decide that teaching math in a disadvantaged area, even though they will make less, is a better fit for them than say taking a job on Wall Street....
Technology is simply brilliant! If I didn’t already embrace that fact, “60 Minutes” reinforced that upon me with their article about Kai-Fu Lee, the “Oracle of Artificial Intelligence” recently.
A company of Mr. Lee’s, Face ++, has deep-learning facial response programs that can tell educators which students are engaged, bored or confused during classroom lectures. Teachers can see at what point during lecture these responses happened, and can follow up with the individuals.
And, modern kitchens can re-order supplies for you, such as your fridge noting the milk or eggs are empty, or when you have used any of the food you purchased. The fridge can automatically contact your supplier and new food arrive before you even knew it was needed.
But, the practical success of technology has its limitations. The teachers can be alerted as to which students were excited/confused by the lecture, but the software does not know “why” it happened, and doesn’t know anything more than the facial response identified. “60 Minutes” notes that “a typical AI system can do one thing well, but can’t adapt what it knows to any other task.”...
Every winter one of my co-workers at TRC reminds us that Girl Scout Cookie Season is upon us. GSCS (my abbreviation) is a time to call attention to, celebrate and support the agency dedicated to building “girls of courage, confidence, and character, who make the world a better place.” All well and good, but when it comes down to it, I just want the cookies.
Their best sellers year after year – Thin Mints, Peanut Butter Patties/Tagalogs and Carael Delites/Samoas – contain chocolate. But I don’t eat chocolate (doctor’s orders). Which means every year I hold my breath until I learn whether my favorite GSC – the Lemonade – is still on the list. And thankfully, it is being offered once again in 2019. I ordered 5 boxes from my co-worker’s daughter.
My no-chocolate policy makes me a “niche” consumer of not only GSCs but of snacks and candies in general. When Hershey came out with Hershey Gold in 2017 (essentially a chocolate bar without the chocolate), I thought they had developed it just for me! I hadn’t eaten a candy bar in years until I tried that one. Now it’s my go-to when I need a little something sweet.
But the problem with niche markets is they are small by definition. I worry that with a limited market, eventually Lemonades and Hershey Gold will be dropped in favor of more popular products....
I was reading Russell Perkins blog (Russell and his firm Infocommerce group help clients develop product strategy and new product development) about the use of Customer Lifetime Value (CLV) in Relationship Scoring. He takes note of a new trend to apply CLV across all customer touchpoints.
As researchers the concept of CLV is something we are quite familiar with. It seeks to take into account everything known about a customer (these might include factors ranging from past purchase and payment behavior to things like credit score, income or level of education) in order to determine the value that customer is likely to bring over their lifetime.
Relationship scoring then uses this to determine how the customer should be treated. High value customers are given opportunities from better customer service to special offers. Is this idea really all that new?
In many respects it is not. Long before advanced algorithms firms recognized that some customers were more valuable than others. For example a good butcher knew how important each customer was and provided perks to them (like setting aside the best cuts of meat).
I spent a very long time on Amazon and Staples.com recently trying to find a replacement for the little hand-held gadget that I use to clip crossword puzzles out of my newspapers. Typing “Paper cutter” in the search box didn’t get me there. And neither did “Scissors”. I eventually gave up. It wasn’t until I saw it in a good old-fashioned mail-order catalog that I learned it is called a “Gift wrap cutter.” And so I was finally able to go back online and order one.
As researchers, we are always interested in understanding consumer choices. We ask respondents to rate importance, rank priorities, and trade-off among complex configuration scenarios. We discuss and make our own trade-offs in terms of design simplicity, project cost and informational objectives. And sometimes we try new approaches.
Anyone who reads the news is aware that there is a whole new consumer product category on the horizon: marijuana is now legal for medical purposes in over 30 states and for recreational use in at least 9 of those states (as of this writing). In partnership with NJ Cannabis Media (www.njcannabismedia.com), we took the opportunity presented by this new consideration dynamic to test a new choice evaluation strategy.
Essentially, we were interested in understanding the roles of 4 factors in adult recreational-use marijuana purchase decisions. A constant sum exercise (allocation of 100 “importance points”) among self-reported current and potential users yielded the following priority distribution: