Nate Silver’s much anticipated (at least by some of us) new venture, launched recently. In his manifesto he describes it as a “data journalism” effort, and for those of us who have followed his work over the last five years – from the use of sabermetrics in baseball analysis through the predictions of presidential politics – there is plenty to look forward to. Apart from the above topics, his website is focusing on other interesting areas such as science, economics and lifestyle, bringing data-driven rigor and simple explanation to the understanding of all these fields. It follows the template of the blog he ran for the New York Times as well his bestselling book, The Signal and the Noise: Why So Many Predictions Fail, But Some Don’t. As a market researcher, I found much to like in the basic framework he has laid out for his effort.
In critiquing traditional journalism, Nate describes a quadrant using two axes – Qualitative versus Quantitative, and Rigorous & Empirical versus Anecdotal & Ad-hoc.
He is looking to occupy the mostly open top left quadrant, while arguing that opinion columnists too often occupy the bottom right quadrant and traditional journalism generally occupies the bottom left quadrant. For someone with such a quantitative background he is not dismissing the qualitative side at all. On the contrary, he argues that it is possible to be qualitative and rigorous and empirical, if one is careful about the observations made (and cites examples of journalists such as Ezra Klein, who occupy the top right quadrant).
For those of us in market research the qualitative versus quantitative dimension is, of course, very familiar. Somewhat less so is the second dimension – rigorous and empirical versus anecdotal and ad-hoc. But this second dimension is especially important to consider because it directly affects our ability to appropriately generalize the insights we develop. As practicing researchers, we know that qualitative research is excellent for discovery and quantitative is great for generalizations. But we also know that is not always the way things are done in practice.
Too often qualitative is used to generalize, a phenomenon that becomes especially tempting when a variety of cool new approaches are becoming easily available for widespread use. This puts us in the danger category where Nate puts traditional journalism. Finding a few vivid examples, extrapolating from that and writing a story about a new trend that the journalist has “discovered”. As he points out, this is a big problem for two reasons. One, the vividness of the anecdotal examples makes them very emotional and powerful. Secondly, the story telling inherent in journalism provides a compelling context that makes it easy to generalize from the anecdote to the population. The Nobel winner Daniel Kahneman has explained this phenomenon in his bestselling book, Thinking, Fast and Slow. He talks about the inability of people to prevent themselves from generalizing from powerful anecdotal data. Nate argues that traditional journalism needs to overcome this and he is helping that process with his approach. Could we say that in market research the same thing is not happening? That sometimes anecdotal information from qualitative research is not treated like quantitative data and generalized?
Unlike in journalism, where a rigorous data-driven approach is emerging, in market research we have a well established tradition of rigorous and quantitative efforts. Our problem lies more in the inappropriate use of that approach than in its underuse. When we make survey questionnaires tedious and don’t think about the appropriate way to ask questions, we are encouraging responses that really don’t generalize well to the population. Ideally we want the respondent to answer as though they are behaving in a real situation. We know that is rarely, if ever, going to happen. But why would we not do the best we can to encourage that behavior?
For example, providing trade-off questions to a respondent is a good way to elicit a more thoughtful and realistic response because that mirrors the real world of decisions. But what happens when we overload our conjoint studies with too many attributes and levels? Respondents (who are not buying anything for real in a survey) start to ignore what we are asking and answer in a way that does not properly reflect their real world behavior. So, even though we are ostensibly using a quantitative and rigorous approach, by using it inappropriately we end up with compromised data that affect generalization.
Nate’s manifesto is aimed at bringing quantitative rigor to the world of traditional journalism. But we researchers can take lessons from that to reform the way we work. We can make qualitative more rigorous, by using multiple approaches to ensure that the insights discovered have the potential to be generalized, while also not putting the burden of generalization on the qualitative research. We can make quantitative research better by ensuring that respondents are given the best opportunity to provide answers that mirror real world decisions and using a vivid storytelling approach to explain those results and thus making them more actionable.
Let’s listen to the fox.