Unmanned satellite-aircraft devices can now stay in orbit more than a year, then
return to earth and make a soft landing at a military base! Just imagine the accuracy and precision of the measurements and analysis needed to achieve this. This accomplishment deserves more than a hat-tip to the scientific method.
Yet when it comes to medical research, arguably as important to the well-being of citizens, accuracy and precision are nowhere in sight. Results of studies are all over the map. Some researchers best distinguish themselves by the clever ways they can transform any set of results into support for whatever drug, device or vaccine their sponsor wants to sell. Imagine if they were rocket scientists!
I have
previously discussed about ten methods that have been used to squeeze an ill-fitting conclusion from a clinical dataset. One such method: the handling of subjects "Lost to Followup" has now been shown to skew the results of as many as one third of clinical trials published in top medical journals, according to an open-access
article in the May 18, 2012 BMJ by Akl EA et al. Akl's group examined 235 papers in 5 top medical journals for whether losses to followup were mentioned, and how the losses were handled. Over half the authors of the 235 papers (54%) did not respond to questions from Akl's group regarding details of the research, which is troubling.
Nineteen per cent of the trials did not mention how losses to followup were being handled. Of those papers that did mention losses to followup, generally 2-15% of total subjects were lost.
When reasonable assumptions about these losses were made, it was found that the conclusions of the trials could become nonsignificant in about 1/3 of cases. In other words, the positive result would be lost.
From the SUNY Buffalo press release:
“We found that in up to a third of trials, the results that
were reported as positive – in other words, statistically significant –
would become negative – not statistically significant, if the
investigators had appropriately taken into consideration those
participants who were lost to follow-up,” says Elie A. Akl, MD, MPH,
PhD, lead author, and associate professor of medicine, family medicine
and social and preventive medicine at the University at Buffalo School
of Medicine and Biomedical Sciences and School of Public Health and
Health Professions. He also has an appointment at McMaster University
“In
other words, one of three claims of effectiveness of interventions made
in top general medical journals might be wrong,” he says.
In
one example, a study that compared two surgical techniques for treating
stress urinary incontinence found that one was superior. But in the
analysis published this month, it was found that 21 percent of
participants were lost to follow-up. “When we reanalyzed that study by
taking into account those drop-outs, we found that the trial might have
overestimated the superiority of one procedure over the other,” Akl
says.
According to Akl, it has always been suspected, but never
proven, that loss to follow-up introduces bias into the results of
clinical trials. “The methodology we developed allowed us to provide
that proof,” he says.
The methodology that he and his coauthors
developed consists of sensitivity analyses, a statistical approach to
test the robustness of the results of an analysis in the face of
specific assumptions, in this case, assumptions about the outcomes of
patients lost to follow-up.
