knowing enough to make the right decisionSometimes as researchers we get too hung up on knowing everything.   We get frustrated by interesting findings that can't be explained with the available data and this can cause us to miss important insights. I suspect that the proliferation of available data will do little to help fill in the fact, it might make the problem worse. A simple exercise in text analytics highlights this point.

There are now an array of tools available to help quantify and understand massive amounts of text.  For example,  at one of our conferences last year, Oded Netzer of Columbia University presented an amazing tool that analyses message boards and other online forums to learn about specific markets (slides can be found at: Tools like these provide a rich and valuable source of data, but insight can also be gleaned from far more simple approaches.

Aaron Gerow (Trinity College) and Mark Keane (University College Dublin) conducted a simple, yet insightful analysis of four years worth of articles in three publications that dealt with the stock market. Unlike many text analytic exercises, they didn’t attempt to figure out what was being said. Instead they simply counted up how often various words were being used over time. This provided some insight into the diversity of opinions being shared…namely, if they found less different words being used (but on average with more frequency) it probably indicated that opinions were more similar than if a greater number of different words were being used.

They then plotted these changes in diversity against stock market performance and found something very interesting. As markets went up, the diversity of opinion went down. On the other hand, as markets dropped, diversity went up. They can’t explain why this is the case. It may be that there are only a few explanations for a good market and many for a bad one.

Another interesting point is that both the steady drop in the market (that started in 2007) and the recovery (that started in 2009) were preceded by changes in the diversity of the words being used by financial reporters. The data can’t tell us if this change was brought about by reporters sensing the market direction was about to change (and that in turn changing their writing) or if the change in their writing caused the change in market direction.

While both the reason for the changes in diversity of words used and understanding which was the cause and which was the effect are interesting questions, they are not essential to making decisions on market timing. As an investor what really matters is that you’d be wise to be wary when you note financial reporters start to have more diverse opinions and be ready to invest when those opinions start to coalesce again.

In other words, sometimes we don’t need all the answers as long as you have the most useful ones.

Finally, it is worth noting that this is based essentially on two data points and that my record as an amateur financial advisor is far worse than my record as a professional researcher so probably best not to risk your nest egg on it.