An example may be warranted. Let’s say it’s 1940 and you’re a scientist interested in the question does smoking cause lung cancer? Your hypothesis is probably along the lines of “I hypothesize that smoking leads to lung cancer.” If your dearest grandmother and father died of lung cancer, you may already think the hypothesis is correct; if you happen to collect a paycheck from Phillip-Morris and smoke yourself, you may think the hypothesis is wrong.
Now having a pre-established bias is not inherently problematic, it’s inherently human. What you do with this bias is the issue. In the above example, it is fairly clear what the biases are and why they exist and these biases are likely to be identified by a scientist who’s worth her salt. One of our jobs as good scientists is to identify sources of bias. In the above example there are strong personal sources of bias, but in the day-to-day workings of a research laboratory there are more subtle biases. The I-think-this-is-a-cool-idea bias (probably the most common), or the this-result-could-result-in-a-glamour-mag-publication bias, or my-competitor-has-another-idea bias., the list goes on. So, it is important to identify your biases. But this begs the question WHY?
One reason I want to address here, because it goes well beyond the laboratory, is the issue of confirmation bias. Confirmation bias is what happens when you conflate positive data (results that affirm your preconceived notions) and/or diminish negative data (results that contradict your preconceived notions). Confirmation bias helps the casinos make several billion dollars in profit every year. Confirmation bias leads to the retraction of some high profile publications every year. Confirmation bias makes pundits and those that parrot them look like idiots.
So, using our smoking/lung cancer hypothesis above scientist #2 might use a few 2 pack/day smokers without cancer as her sample population. Whereas scientist #1 might recruit her study population in the cancer wards. Although my examples are over the top, the choices made by a scientist that are a product of confirmation bias could be much more subtle and not apparent to other scientists who may be reviewing the work for a publication. This is why good scientists are, or at least try to be, vigilant about their biases and take steps to ensure they aren’t screwing up.
Say for instance you think there is a link between Chronic Fatigue Syndrome and XMRV or that there is a link between the MMR vaccine and autism. Do you find the one poorly designed study that supports your position while simultaneously excluding the plethora of other studies that don’t? If so, you may be a confirmationally biased redneck.
This goes well beyond the sciency stuff. Do you go to Vegas and proclaim the $250 jackpot you won, while failing to point out the 32 $20 bets you lost? The casinos aren’t making billions in profits every year on paying out more than they take in.
Do you have a problem with illegal aliens in Arizona as a recent threat to our well being? Maybe you note whenever an illegal is arrested for a crime as proof of a problem while ignoring fact that crime is lower* in Phoenix in 2009** than in 2002**?
Do you think atheists have split into the old pleasant docile type and some New type that is the root cause of most problems in America? Maybe you’ll take the word of a sockpuppet identity thief because they agree with what you already think over many real voices that disagree.
Hot today...global warming is true. Cold tomorrow...global warming is a myth.
*This is a 10 year comparison for Phoenix specifically.
**Compare pages 21 and 14 in the 2009 and 2002 years for a simplified breakdown.
4 comments:
Confirmation bias:- very interesting, logical comments. Unfortunate choice of example with XMRV and CFS/ME. In the clinical community we are astounded that the many go no further than count headlines. 1 for, 3 against. As if an outcome can be determined by some sort of gut-feel vote. Never mind reconciling the differences.
A question for you:- What do you call it when public health officials direct the withdrawal of a scientific paper, already accepted for publication because it goes against their intuition [FDA XMRV paper by Harvey Alter] while allowing a study that agrees with their intuition to be released [CDC Switzer study]? See Vincent Racianello's coments here http://www.virology.ws/2010/06/30/publication-of-xmrv-papers-should-not-be-blocked/
Could it possibly be called "Confirmation Bias"?
What do you call it when a purportedly medical research paper finds patients for a serious illness by phone survey rather than clinical diagnosis? Would this be accepted for say cancer? [The CDC paper "found" patients by phone interview. In contrast with Lombardi and Alter papers that accepted the work of their scientific peers for the last 25 years that had established clear clinical diagnostic criteria]
Could this possibly be called "Confirmation Bias" - looking for something where you know it will not be found? Perhaps that accusation is unfair. But why ignore the work of many bio-medical researchers before you, as though your insight was "special". [CDC ignoring Canadian consensus criteria and a large body of scientific papers establishing neurological abnormalities in CFS/ME].
The scientific process needs defending. Unfortunately in this instance it is being left to clinicians and patients to express that defence. Voices not generally accepted. Fortunately a few more "acceptable" voices have stepped up to the plate.
Whether there is a link or not is not known. But please, a bit of anger at the interference to the scientific process. Let science take it's course.
Peter Wachtel,
Melbourne Australia
Funny, I don't have a horse in the XMRV CFS pseudodebate and it was ~4 words of a ~650 word post. However, it prompted a ~300 word response from you.
While I understand the desire for quick fixes and definitive explanations for our ills, that is a problem not a feature.
Personally, I agree that the paper submitted to PNAS should be published, unless there are additional factors you are leaving out. The peer review process is flawed, but should be respected.
Regardless, you, Dr. Racianello, the WPI etc all have horses in this debate yet you only point out the biases of those who disagree with you. One might call that poisoning the well. I suggest you check out ERV's postings on XMRV and CFS (I know you already have), she has some pretty damning technical problems with the XMRV CFS link data, which as far as I can tell have not been adequately addressed.
Thankyoufor taking the time to reply. My apologies if my expansion of a 4 word comment into an essay seemed confrontational. I accept that my response was actually at a tangent to your original discussion of human bias. And that you were making a broader point with a few examples.
As to "horses", patients and clinicians do have a horse. But not the one you are suggesting. eg., the technical issues ERV raised. They could be 100% correct. Or maybe not. I'm not sure that I am entitled to care too much. "We" outsiders expect this technical issue to be sorted out by the scientific process. We can only very generally judge the credibility of those who speak. Listening to Ila Singh's discussion of techniques what I hear is a very circumspect discussion of the detection difficulties by a hands-on, experienced researcher. I hear that yes, actually, the technology that ERV derides as "1970's technology" is indeed the most reliable way of detecting this virus. ERV's comment on this issue read like a fundamentalist approach - "it is written". 21stC PCR amplification technology is the most reliable, anything else is backward. I'm guessing that when "you guys" (the scientific community) finish assessing this, the scientifically circumspect viewpoint of those like Ms Singh will be the correct one. But maybe not. At least we'll have an answer.
The horse for patients/clinicians is actually whether this illness should be investigated from a medical perspective or from a social/public health/ psychological viewpoint. The approx 2,500 peer-reviewed papers of the last 25 years that overwhelmingly support a medical approach have not resulted in anything meaningful like clinical treatment trials. Why? We have treatments for many diseases of unknown etiology. The xmrv "horse" just happens to be one in the public eye that might influence policy.
Wanting quick fixes and explanations is a bias that should be questioned as you say. It is also rather transparent and easy to discuss. Various groups have challenged the original paper of the National Cancer Council, Cleveland Clinic and the WPI. And done so formally in published commentaries. What I don't see is the other side of the coin.
One example:- the (inappropriate) name "CFS" was coined by the CDC in the 1980's to classify outbreaks of a "flu from which no one recovered". In 2010 the CDC conducts a study. The case definition it uses excludes those after whom it itself named the disease. (Because of the presence of neurological abnormalities). What is going on? Why has the CDC altered it's research definition so dramatically? Why is this definition so at odds with clinical experience? Is this warranted or not? I do not see scientific commentary discussing this.
Another: it is a matter of public record that the CDC was at the receiving end of congressional investigations for mis-appropriating money earmarked for CFS research. Money was allegedly diverted for "more worthy research". Many of the same people are involved in the recent research. Is not the onus on the CDC to keep this issue in the open? The potential confirmation bias of those producing the Sep 2009 paper has been formally discussed. But none of those 3 labs have been charged with finds mis-appropriation. Where are the questions to the CDC from the scientific community requesting affirmation that past history has not biased the recent study?
Anyway, enough of this. Thanks again for your previous response.
Damn right! Confirmation bias affects our everyday thinking in the lab. But even in if we are able to over come it (which is very hard), it can play bigger roles in the process ahead. For example, in our current peer-review system, if you want to get a paper selected you suggest a few names which are people who do similar sort of work as you or who you think will buy your arguments. When the paper arrives at these reviewers desk they will suffer from a confirmation bias because the paper they are reading is helping them support their own hypothesis. Thus, it might happen that the reviewer might overlook some details and allow the publication of a paper which did not deserve to be published.
Post a Comment