Question of Causality

Here a question Steven Levitt raised in his book Freakonomics: Why is it the cities with the most police officers are often the cities with the highest crime rates?

Could it be that the police officers are the cause of the crime? By extension, would crime go down, then, if there were fewer police officers on the streets?

Such logic sounds silly, but this issue highlights the problems of observational studies -- statistical mining of medical data to find correlations. Although it worked well in finding the link between smoking and lung cancer, observational studies have serious disadvantages compared with the more conclusive, much more expensive "randomly controlled experimental study."

Gautam Naik's article in the May 4th Wall Street Journal explained:

In the late 1980s and early 1990s, a raft of observational studies consistently suggested that hormone-replacement therapy, or HRT, could protect postmenopausal women against heart disease. Tens of thousands of women were given the drugs on that basis.

It was a bad call. Many of the studies were eventually undermined because women who used the drugs were healthier than those who didn't, and thus had lower rates of heart disease anyway. Later controlled trials suggested that not only did HRT fail to protect against heart disease, but it might have increased the risk. [Source]

So if observational studies are so flawed, why has the number of these studies grown exponentially over the past twenty years?

Because they're cheap. Observational studies are based on past-oriented, already-collected data. Doing a new experiment requires a lot more funding than just crunching a bunch of numbers looking for trends.

The hazard is that these cheap, dime-a-dozen studies can be misinterpreted by "experts" and lead to a sensational news story. Since the whole purpose of these studies is to find some deep Holy Grail of insight, the authors can be counted on to trumpet their own findings.

But as the late Richard P. Feynman stated, "The first principle is that you must not fool yourself and you are the easiest person to fool."