I read an interesting story about a survey done to determine if people are honest with pollsters. Of course such a study is flawed by definition (how can we be sure those who say they always tell the truth, are not lying?). Still, the results do back up what I’ve long suspected…getting at the truth in a survey is hard.
The study indicates that most people claim to be honest, even about very personal things (like financing). Younger people, however, are less likely to be honest with survey takers than others. As noted above, I suspect that if anything, the results understate the potential problem.
To be clear, I don’t think that people are just being dishonest for the sake of being dishonest….I think it flows from a few factors.
First, some questions are too personal to answer, even on a web survey. With all the stories of personal financial data being stolen or compromising pictures being hacked, it shouldn’t surprise us that some people might not want to answer some kinds of questions. We should really think about that as we design questions. For example, while it might be easy to ask for a lot of detail, we might not always need it (income ranges for example). To the extent we do need it, finding ways to build credibility with the respondent are critical.
Second, some questions might create a conflict between what people want to believe about themselves and the truth. People might want to think of themselves as being “outgoing” and so if you ask them they might say they are. But their behavior might not line up with reality. The simple solution is to ask questions related to behavior without ascribing a term like “outgoing”. Of course, it is always worth asking it directly as well (knowing the self image AND behavior could make for interesting segmentations variables for example).
Third, some questions are simply beyond a respondent’s ability to answer. If you read my blogs you know I often rail against the overuse of rating scale questions…especially those that ask respondents to rate features that they don’t think about in the real world. A solution here is to use choice questions in place of rating scales wherever possible and to allow a response for people who honestly don’t know the answer.
In fact, we’ve found that choice questions are often a great solution for all of the above issues. Discrete Choice, for example, allows us to ask complex questions in a way that allows respondents to do what they do in everyday life…make choices! The complexity of the question, however, also makes it harder for them to mislead us. With multiple features and levels to juggle, they will tip their hands whether they want to or not.
So, while quantifying something like the truthfulness of respondents is always going to be hard to do, work such as the study cited here tells us we are wise to be concerned about it.
Rich brings a passion for quantitative data and the use of choice to understand consumer behavior to his blog entries. His unique perspective has allowed him to muse on subjects as far afield as Dinosaurs and advanced technology with insight into what each can teach us about doing better research.