Sometimes as researchers we get too hung up on knowing everything.   We get frustrated by interesting findings that can't be explained with the available data and this can cause us to miss important insights. I suspect that the proliferation of available data will do little to help fill in the blanks...in fact, it might make the problem worse. A simple exercise in text analytics highlights this point.

There are now an array of tools available to help quantify and understand massive amounts of text.  For example,  at one of our conferences last year, Oded Netzer of Columbia University presented an amazing tool that analyses message boards and other online forums to learn about specific markets (slides can be found at:  http://www.trchome.com/research-knowledge/conferences/437). Tools like these provide a rich and valuable source of data, but insight can also be gleaned from far more simple approaches.

There was a fascinating news item today on NPR about the use of text mining to understand something about a person’s private life. Ian Lancashire, Professor of English at the University of Toronto used text mining to study Agatha Christie’s novels. While he has done this with other authors before he came cross something particularly interesting in Agatha Christie’s 73^rd novel.