A former colleague of mine used to tell us to “torture the data until it confessed”. In other words, don’t just stop your investigation at the first finding. But rather, keep poking, prodding, flipping and coercing until you feel you’ve uncovered all the data has to give. Ah…images of Jack Bauer doing his thing flash through my mind just thinking about our own data “torture” sessions.
All kidding aside, what my colleague was really trying to say was spot on. I’m sure we’ve all known researchers who habitually stop at the first find. They rarely take the time to consider different ways of looking at data, of considering the message within.
On a good day this means they’ll never find the more important gem that lies inside. On a bad day they will completely miss that finding that totally counters, and trumps, their initial find. Good researchers are curious. Good researchers look deeper, and often times find hidden pockets of value, leading to poignant insights. I think a straight-forward example that we deal with all the time would be applying Discrete Choice to develop or modify an offering. The easiest thing to do would be to just take all the features with the highest utilities and say “we’ve got a winner”. A better approach would be to use optimization routines to identify the bundles that provide the most revenue or profitability across various segments.”
But how much is enough? The reality and nature of statistical analysis is such that there is some interpretation involved. So if you keep pushing – I’m sorry, torturing – it is likely that you will eventually get a “confession” that lines up pretty nicely with what you and/or your client were hoping to learn. But that doesn’t make it right.
As researchers, we are charged with helping our clients get answers to questions that, presumably, will help their business succeed. As a rule, sometimes those answers hurt. I recently did research where the data, even after some prodding, pretty clearly told my client “your program is not working…investing in X was not a good idea”. Some clients are very open-minded about the research process, and in fact fully understand the value in even the worst news. Other clients, well, not so much.
As professionals, we have an obligation to be curious, and persistent, and also honest in our assessment of what the data are saying. The simply reality is that none of us really knows what answers we are going to get prior to doing the research. No less an authority than Albert Einstein once said, “If we know what it was that we were doing, it would not be called research, would it?”
So, torture – when it comes to data anyway – really is a good idea, but only to a point. What is more important is to keep an open mind to what you might learn.