How to Manipulate Statistics in Research – A Short Guide to Bad Science

How to Manipulate Statistics in Research - A Short Guide to Bad Science

Introduction

Today I’m not gonna teach you how to deceive. But, I’m gonna show you how statistics can be deployed for wrongdoing. In particular, I’m going to discuss a few methods that are often applied to research studies so that desired results are more easily attained.

The untrained eye (at least 90% of us) is unaware of such practices, mostly because of the lack of education in critical thinking, logical fallacies, math, and statistics. Hopefully, this will help.

I have learned about these strategies while reading Bad Science, a sobering and, at the same time, funny work by Ben Goldacre [1].

Goldacre shoots at the field of ‘nutritionism’ (or: over-complication of simple and sensible dietary advice) and at those who package sophisticated strategies – such as complex diets, detoxes, light therapies, the worship of one food or another, promotion of, so called, super foods, etc.

If you want to improve your health, things should be fairly simple: eat clean, exercise, sleep. This will get you at least 80% closer to your end goal. Then, you can use other fancy strategies to get even closer.

To be balanced in his approach, Goldacre not only goes after nutritionism and alternative medical practitioners (naturopaths, homeopaths, chiropractors, acupuncturists, etc.), but he also takes on the conventional medical field, the pharmaceutical industry, and the way celebrities endorse products.

Overall, the idea of the book is that he is after those who take advantage of the gullibility (to be read: lack of education) of the public at large (me and you), for deceiving purposes. Here, we’ll only focus on how manipulation of statistics can be used in research studies.

So, in his words, this is the “dark side of statistics or if I may, how pharmaceutical people can use statistics to manipulate the public when publishing studies. [1]

Manipulating the Data

  1. Ignore the protocol entirely

“Always assume that any correlation proves causation. Throw all your data into a spreadsheet programme and report – as significant – any relationship between anything and everything if it helps your case. If you measure enough, some things are bound to be positive just by sheer luck.” [1]

This is probably one of the most common mistakes when looking into data. It’s not only done for manipulative purposes, it’s also done by those who interpret studies (often, even without being aware of it). Humans are creatures of bias; we like to see patterns where they are none. This is unromantically called pareidolia.

As a kid growing up with my grandparents in a village, I would often stare at the clouds for minutes at end, entertained by the different familiar objects, shapes, and faces I would observe.

How to Manipulate Statistics in Research - A Short Guide to Bad Science - 2

Of course, once you know this, you should be able to guard yourself against your own correlation/causation bias and also against the deceiving tactics of other parties.

  1. Play with the baseline

“Sometimes, when you start a trial, quite by chance the treatment group is already doing better than the placebo group. If so, then leave it like that. If, on the other hand, the placebo is already doing better than the treatment group at the start, then adjust for the baseline in your analysis.” [1]

When doing controlled experiments, researchers often start with an end goal in mind; they do the experiment to prove their hypotheses instead of seeking/testing evidence that would disprove them. Knowing that you’re after some kind of result (any) and forcing your way to it (stubbornly, on-purpose) stands against good research practices.

  1. Ignore dropouts

“People who drop out of trials are statistically much more likely to have done badly, and much more likely to have had side-effects. They will only make your drug look bad. So ignore them, make no attempt to chase them up, do not include them in your analysis.” [1]

Goldacre mentions a few studies of pharmaceuticals that apply to such methods. Peter Gotzsche also talks about this, extensively, in his book [2].

  1. Clean up the data

“Look at your graphs. There will be some anomalous ‘outliers’, or points which lie a long way from the others. If they are making your drug look bad, just delete them. But if they are helping your drug look good, even if they seem to be spurious results, leave them in.” [1]

Since you are in control of your data, there’s no bad data. Any information can be engineered to serve a certain purpose. It’s all about the light in which you put your data (context and perspective).

  1. The best of five…no…seven…no…nine!

“If the difference between your drug and placebo becomes significant four and a half months into a six-month trial, stop the trial immediately and start writing up the results: things might get less impressive if you carry on. Alternatively, if at six months the results are ‘nearly significant’, extend the trial by another three months.” [1]

After reading this, I reminded myself that I often read studies making claims along these lines: that for some purpose the study ended sooner or later than it was initially proposed. Moreover, if you do not specify your timeline before you begin the study, you can prolong the study for as long as you want (to be read: until you reach your desired outcome).

  1. Torture the data

“If your results are bad, ask the computer to go back and see if any particular subgroup behaved differently. You might find that your drug works very well in Chinese women aged fifty-two to sixty-one. ‘Torture the data and it will confess to anything’ as they say at Guantanamo Bay.” [1]

The major requirement is to be in possession of a large dataset with many features. This is where you can do wonders with the p-value and with relative vs. absolute risk – to name a few schemes.

  1. Try every button on the computer

“If you’re really desperate, and analysing your data the way you planned, does not give you the results you wanted, just run the figures through a wide selection of other statistical tests, even if they are entirely inappropriate, at random.” [1]

Ending is everything. Goldacre says that publishing your study should be done wisely. If your study ended up exactly as you wanted, shoot for big publications.

For positive trials with unfair tests:

“put it in an obscure journal (published, written and edited entirely by the industry): remember, the tricks we have just described hide nothing, and will be obvious to anyone who reads your paper, but only if they read it very attentively, so it’s in your interest to make sure it isn’t red beyond the abstract.” [1]

For trials with embarrassing findings:

“hide it away somewhere and cite ‘data on file’. Nobody will know the methods, and it will only be noticed if someone comes pestering you for the data to do a systematic review. Hopefully, they won’t be for ages.” [1]

Conclusion

Anyone can find and read studies nowadays. Information is at the tip of your fingers. Yet, few are able to properly interpret them.

Two barriers stand in the way: personal biases (logical fallacies) and researchers’ biases (their own logical fallacies, conflicts of interest, hidden agendas, etc). And as Goldacre points out, the untrained, gullible, person is not only at the mercy of the practices mentioned above, but also at the mercy of health gurus, quacks, alternative medical practitioners, celebrities, etc.

To stay away from that:

You have to know how to make sense of and filter through the flood of information that you are exposed to through your senses.
Find out about your inborn logical fallacies (your brain’s factory faults). I wrote about a few of them here.
Learn statistics and math (yep, I know most people hate this).
Train yourself in critical thinking. Become more literate; self-education is your ally.

Resources:

  1. Ben Goldacre – Bad Science: Quacks, Hacks, and Big Pharma Hacks
  1. Peter Gotzsche – Deadly Medicines and Organised Crime: How Big Pharma Has Corrupted Health Care
  1. Steven Novella – Your Deceptive Mind: A Scientific Guide to Critical Thinking Skills

Images: here and here

Get on my list of friends
More about my book Persistent Fat Loss
More about my book Ketone Power
More about my book T-(Rx)
More about my book Periodic Fasting

Related posts:

Comments

comments

Leave a Reply

Your email address will not be published. Required fields are marked *