My favorite feature of Quirk's Marketing Research e-newsletter is Research War Stories. In one issue this spring, Arnie Fishman reported that he had an unexpectedly high result when he asked research participants whether they eat dog food "all the time." He framed the question by asking how often they ate each of a variety of "exotic foods," including rattlesnake meat and frog kidneys, among others.

This got us thinking that maybe you'd get a different result if you asked just about dog food rather than about dog food amongst other crazy types of foods. So, being the researchers that we are, we designed a monadic design experiment to see what would happen.

Using Arnie's same framework of exotic foods, we asked one group of our online research panelists how frequently they eat dog food. On the next screen we asked the same question about rattlesnake meat. They always saw dog food first, so they had no other stimulus when they answered the dog food question.

We asked another group of panelists about dog food, rattlesnake meat, frog kidneys, gopher brains, and chocolate covered ants all on the same screen. We hypothesized that this group would be more open to admitting to eat dog food when grouped with these other items rather than just being asked directly about dog food.

Well, we were wrong about that – none of the folks asked about dog food alone admitted to eating dog food all the time, and 1% of those asked about dog food amongst the other exotic items did so (not a statistically significant difference). The percent of folks in both groups saying that they "never" ate dog food was the same as well (96%). So in our experiment, the "framing" of the question had no bearing on the response.

But we also twisted things around a bit, and asked a third group how frequently they feel other people in the US eat dog food. The "all the time" category rose slightly, but not significantly, to 2%. But the big change was in the "never" category – only 44% of the panelists in this group said that people in the US never eat dog food. We saw the same pattern for the other foods as well. In this case they're answering for a group, not for a single person, so clearly there has to be an allowance that some people are eating these types of foods some of the time.

As researchers, we are sometimes asked to locate and interview people who engage in socially unpopular activities. Understanding whether people will admit to engaging in them is critical not only to recruiting research participants, but also to interpreting the data. This is especially important where complex methods are used, such as discrete choice in pricing research, where assumptions about the target market size are built into the analysis, and distortion can have huge impacts.