Nothing makes a sensational news story quite like one centred on a medical drama. Sex sells insalubrious newspapers, as well as everything from ice cream to motor cars, but it does not have the same impact as a story that pricks the mortality of the reader, or someone dear to them.

Stories about recent question marks over published evidence relating to statins – drugs that lower cholesterol concentrations and can therefore prevent certain types of heart disease – have done just this. A variety of news organisations have reported the story, including more reputable ones aiming at a general audience (BBC reports here and here), but also aimed at the scientific community, ensuring exposure to both trained and probably untrained audiences.

This broad presentation of the story led to my receiving worried e-mails asking whether I think the respondent should continue to take these medications. Needless to say, I cannot possibly advise, but it did set me thinking about what the evidence was behind this story and why there was a debate at all. Why was the evidence about statins not clear enough for senior physicians to make a decision on whom should take them or not?

The root of the problem was that two scientific papers about the side effects of statins. In these papers, whether or not it is worth patients who have a low risk of heart disease taking statins was questioned. In other words, if someone in the lowest risk group with respect to contracting heart disease takes statins, side effects like throat, joint and muscular pain, nosebleeds, and digestive upsets, may be palpable but the benefit(s) of the drugs are not. Because of the newsworthy status of this topic, the papers were discussed in British Medical Journal articles, such as one by Aseem Malhotra.

From a scientific standpoint, the striking thing about these articles and the subsequent commentaries is two-fold. First, the data was not analysed correctly and neither was this oversight noticed by its authors, reviewers or editors. Inevitably, this reflects poorly on all parties involved and taken together, these seem a devastating failure for a high-profile journal, especially one that is also read by readers who are not medically or scientifically trained (if only journalists).

Second, the evidence that was mis-analysed and possibly misinterpreted was data from an uncontrolled observational study. The phrase ‘uncontrolled observational study’ is lumpy and I suspect rather opaque to non-scientists. Clarity about its meaning has been missing from the general debate, so I provide an insight here:

Observational means that the data has been collected by simple measurement from a given cohort, with calculations made thereafter. This cohort is likely to be an homogenous group, and one hopes a large one, so that the differences between individuals that occur ordinarily do not skew the overall result. For example, if you wanted to test a sun cream, you might opt for a group of people that represents the spectrum of those who are very susceptible to sunburn through to those who are much less so, with everything in between. But, you would also want to know that all the individuals spent about the same length of time in the sun, so the recorded values were comparable and made sense when an average was calculated. This is where ‘observational’ is a devastating compromise, as the latter point is not always met, and so the data between individuals varies considerably and in different ways. The point that someone is more susceptible to sunburn than someone else, is therefore lost. Even enormous groups are insufficient for providing a clear answer. The more serious problem with observational studies however is that no test has been carried out. A series of measurements have been taken, but no scientific conclusion can be drawn, because no causal link can be made. Only a correlation exists, because a direct link has not been established.

Uncontrolled is also a serious compromise with respect to the quality of the data and what it means. A controlled study is one in which there are adequate comparison measurements made, in order to show the significance of the data from the test subjects. Using the sun cream example, controls in that study would be people who did not use sun cream at all (or one without the active ingredient) but also ones in which sunburn is known to occur. The latter are called positive controls and are a way of determining what the thing one is looking for actually looks like. There are, or course, ethical concerns about such things, but scientifically, in order to determine whether a given occurrence is the causal result of something, clear comparisons are required in order to assess the significance of the data from the test subjects.

Observational studies, controlled or not, do not have this facility. It is a real shame as many things are not easy to test causally, and the ethical considerations for many others are insurmountable. However, observational studies are just that: they see, they do not determine scientifically, and they should never be given greater credence than that. A great emphasis is placed on ethically soundness in life sciences studies, but we lack the same attention to scientific principles.