Is Peer Review Broken?

No.

Oh, you wanted more than that? Maybe some nuanced reflection on the issue of peer review? Ok, I'll give it a go, but I won't do nuance. Nuance died when Trump was elected, actually before that, but talk about nail in a coffin.


There have been many stories floating around for years on the failings of peer review (the vetting of scientific studies by other expert scientists in the field prior to publication). These stories usually follow the publication of some study that is fundamentally flawed or unacceptable for any number of reasons. Several examples come to mind Arsenic bacteria, Cold fusion, Caterpillar hybridization, etc. So questions 'Is Peer Review Broken? and If so, How Do We Fix It? come up.

For those who live in the scientific universe you can skip the blue paragraphs, otherwise if you want a short breakdown of the process feel free to read them.

Once a group of scientists have made observations and gathered data, they write a story (aka scientific manuscript). I want to note this manuscript is a story, not in a fiction story sense, but in a narrative sense. The authors may not describe the experiments in the order they were conducted, because it makes a more logical narrative to describe things out of order. The authors may use their 20-20 vision to redescribe why an experiment was carried out, because at the time of writing the manuscript the original reason may not make sense in light of the narrative. Again, for those just looking to find reasons to disparage science,  I'm not suggesting authors are manipulating data or trying to obscure their findings (although there are cases of this), I'm talking about making a compelling argument to convince a skeptical audience of experts that their interpretations of the data (aka conclusions) are correct.

Ok, once the manuscript is written revised and edited until most if not all the authors hate the thing, it is submitted to a scientific journal for publication. The journal then assigns the paper to an editor who then decides if the paper is of sufficient rigor and interest to the readers to actually get peer reviewed. If the paper passes this hurdle, the editor sends a number of requests to other scientists asking them to evaluate the manuscript, including the experimental approaches and interpretations of the conclusions. (There are some variations on this model, but most journals follow this model. Some have additional levels of scrutiny, but this tends to be early in the process and not by current scientists.) Many potential reviewers decline and the editor repeats sending out requests until, usually, at least 3 agree. The 3 reviewers then read and critique the manuscript and provide feedback to the authors and editor on the pros and cons of the manuscript, this is the PEER REVIEW component. At this point the editor makes a decision on the manuscript which ranges from (rarely) acceptance, editorial revisions, more experiments needed revisions, to outright rejection. The most common response is some kind of revision, either with or without more experiments, and the authors deal with those critiques and resubmit a revised manuscript. (In the case of outright rejection, the authors usually revise the manuscript based on the review comments and send the manuscript to another journal (They revise because the same reviewers are likely to see it.)) The manuscript can either be accepted/rejected by the editor or (most commonly) sent out for re-review

Peer review, much like everything humans do, is a human endeavor. So it is subject to human limitations. This is not new or particular special, it's simply a fact that humans are not robots, have biases, and some even have ulterior motives. In a simplistic sense we can say peer review is broken, because people will and do make mistakes at all the levels described above. However, by this criteria peer review has always been broken and always will be broken. But this is a stupid criteria. Let's go through how this process can fail, because once we know how it can fail, we can make recommendations for how to fix it.

1. Poor editors

This works for and against a manuscript. For you: The editor thinks your shit don't stink. They can send out your manuscript, when they wouldn't send out the same manuscript from a different group. They can pick reviewers they know to be 'easy' or can write the invitation letter in such a way to encourage a positive review. How can this latter event happen? An editor, who is an established scientist can send to one of their former graduate students/post-docs who is now an independent scientist the following invitation,

Dear Prof X, I have this manuscript I think would be a perfect fit for journal Y, do you have time to review it? Abstract attached below.

Do you see how the letter could effect the review? You have extreme cases like Lynn Margulis obtaining numerous reviews until she had 3 she could use to accept an atrocious paper in PNAS (ignoring all the reviews that noted the fundamental errors in the paper).

2. Poor reviewers

I've been an editor for several journals. I can give you two easy reasons why you might get a poor review. First, you, as an editor, may not be an expert in the area the manuscript addresses. You might be generally aware of the area, but are certainly no expert, which means you likely do not know who the experts are in that field (because its not your field). You can do PubMed searches to identify people who have published in specific areas, but you don't know them or their research. So an editor may not be obtaining 3 rigorous expert reviewers.
Second, even in an area of expertise, the researchers you know are experts often say no when asked. This is particularly true if the journal isn't one of the top journals out there. There is little prestige saying you reviewed papers for a general journal like PLoS ONE in your annual reviews compared to Science or Nature.

3. Poor journals

There are also two versions of this. First, there are journals that will publish anything if you pay, its their business model. You can check out Beall's predatory journal list to identify many of these. Second, there are top-tiered journals that care about mass media dissemination of the work published there, also their business model (Science, Nature, I'm looking at you). The Arsenic bacteria was published in one of these journals, as were the ENCODE papers. This is not simply a journal issue as there were problems at all levels, but the journals actively advertised this work.

So how do we fix peer review?

Most of the discussion I've seen has been dealing simply with reviewers, which I think is the least broken aspect of the peer review process. (It's like blaming teachers for poor student performance and ignoring income inequality and poverty.) The solutions I've seen generally revolve around identifying reviewers, which is stone cold fucking stupid. I've obtained jackass reviews on publications and wish I knew who the dumbass was, but that's kind of the point of anonymous reviewers. If I disagree with a reviewer I cannot subconsciously or consciously screw them over on one of their papers or (more importantly) grant proposals in the future. Non-anonymous reviewers means no early career scientists will review papers for fear of career suicide or early career scientists will review papers and be favorable in the hope for favorable reviews in turn going forward. Non-anonymous peer review would essentially end all the good things of peer review and solve exactly 0 problems of peer review.

One idea I have seen floated around is publishing the reviews, which I actually support. The reviewer remains anonymous, but also has to take some ownership of their review. This could reduce what I think have been some bullshit critiques. While the reviewer would remain anonymous, the community could see what the issues were and decide if those were reasonable (and reasonably dealt with by the authors) or unreasonable and the community could actually comment on it (because the age of social media has changed things profoundly).

How can we fix things? I like the publishing reviews along with the articles (make them available online). BioRxiv may help with this as authors can post their original manuscripts that the world can see to compare reviewer critiques to. I personally like the idea of paying reviewers. $50 a review, not enough to cover the cost of the review but to provide some incentive. (I expect I spend 4 hours on every paper I review rigorously, because I check the literature, so not even minimum wage in some states. Some papers are so bad, they can be reviewed in an hour or two, but I wonder why the editor sent the paper out (see below).) You review 6 papers a year, which is pretty low in my experience, you make an extra $300 bucks, which is not nothing. If you suck at reviewing, editors stop asking you which has a financial consequence. I can see the argument that the money could allow systemic abuse, where reviewers want to appease the editors so they get more assignments, but this is essentially the amount I could make mowing a couple of lawns on a summer evening, probably in less time. (FYI, many journals make good money on the backs of free reviewers and free editors.)

One idea I have not seen floated around is to make editors more accountable. Too often in my submitted manuscripts (and those manuscript I have reviewed), the editor simply defers to the reviews and takes no responsibility. If a reviewer asks for an additional experiment, it must be done even if it has no effect on the conclusions made in the manuscript. In too many cases the editors simply pass information between the reviewers and authors.

How can we fix this? One idea is make editors more accountable. First, pay them. Say $3000 a year, this is essentially the cost of one article. FYI the authors pay to publish their work in the journal. If an editor is not doing a good job, boot them and take on another one. Screw it, hire professional editors, PhD scientists, for $90,000 a year and have them cover a research area. 30 articles covers their salary (another 10-15 for benefits). How many Nature papers are biology related every week?!?! Maybe the CEOs make a little less in order to support high quality science?

What about the journals? Well, authors should stop fighting to get their shit in glamour mags. I know scientists are under immense pressure to publish in C/N/S journals (Cell, Nature, Science), but do these journals really publish the best of the best? Don't know, what I do know is that any good study published in an open access journal is available to everyone, every-fucking-one-with-an-internet-connection!!! How many people actually peruse journals anymore as opposed to PubMed searches? I still subscribe to Science and/or Nature, but primarily for the news, reviews, and opinion pieces as well as to support their policy and outreach initiatives. If you are doing quality work, it will be read, because google. If I can find a decent Chinese restaurant in Rome online, I can find interesting articles on phenotypic diversity in microorganisms online. I would point out the Noble Prize winning research on B-cells (the antibody producing cells of the body) was published in the fucking Journal of Poultry Science.

In summary, peer review is a human endeavor and subject to human foibles. Is it perfect? No. Can it be improved? Marginally. Is it the best we have? Absolutely, but with the caveat that minor improvements can be made and the acceptance that there is no such thing as perfection, simply the ongoing striving for perfection.

The Changing Climate of Science in the USA (not a post on climate change)

UPDATE: If you are coming from uncommondescent.com please substitute the letter o for all letter u's to avoid 'the vapors'. Also, substitute Darwinius masillae anytime you see the word ENCODE so you won't miss the fucking point (to make things easy, you'll only have to make this substitution once).

One of my heroes: from here

I have seen a shift in the way science is being conducted in the United States. This shift still reflects of minority of the science being done, but it also represents the majority of the science being reported or disseminated to the public. In short, it appears to me that the pendulum has swung from favoring rigorous science to favoring and rewarding what I will call 'splash' science. To be clear this struggle between rigorous and splash science is not new nor different than in previous generations. Nor is all rigorous science not splash and vice versa. However, I think in the US the pendulum has swung dramatically to the splash at the expense of the rigorous. This change in trajectory is not surprising as funding has constricted immediately following a massive expansion. There are too many mouths at the trough and they are competing for those few morsels of grain.

More and more, scientific research is being sold on its revolutionary impact and not on its scientific merit. Of course 'impact' sounds much more important than 'merit'. Hell, important and impact both begin with the letter 'i' so there must be something to that. It seems much more science is being sold as 'paradigm shifting,' 'completely unexpected,' 'novel' (the only one that is true, but only in the trivial sense), or 'needing to rewrite the textbooks.' In these cases, it's also 99.99999999% bullshit (e.g. ENCODE).

2nd edition, 2011

Now admittedly and importantly, there are many studies that reveal unexpected results that lead to interesting and a variety of unexpected questions, which can themselves lead to new insights. For example, I sat in the audience at an American Society of Microbiology conference on Candida and Candidiasis where the phenomenon of white-opaque switching (a well known but poorly understood phenotype of certain Candida albicans strains) was directly and elegantly linked to mating (a process that, at the time, had recently been described but the biology nor the relevance was not understood). This was one of those 'HOLY SHIT!' moments that was amazingly cool, but also neither paradigm shifting nor required the rewriting of textbooks. In almost every single case these types of studies will not shift a paradigm nor require the revision of any textbooks. The results may be unexpected, but at most they will lead to the addition or significant revision of chapters in specialized topic books, such as the Candida and Candidiasis book from ASM.

It could be argued that inflating the importance of a study does not undercut the underlying data. But this argument is generally wrong at several levels. First, in order to emphasize the ephemeral, the actual suffers. In order to emphasize the ability to grow in high levels of arsenic, Wolfe-Simon focused on the bacteria using As in place of P in DNA and other macromolecules. The ability of the isolated bacterium to grow in such high concentrations of arsenic is interesting, but this was ignored to focus on the rewriting of textbooks on the structure of nucleic acids, which was wrong. Second, to push your paradigm shifting results, you have to actively ignore or overlook the contradictory data, even that data contained within your own work. Third, you have to discount and/or disregard the data, usually mountains of data, that led to the current paradigm in the first place.

These issues are what concern me most. This is not how I was trained as a scientist and is philosophically opposed to my understanding of the scientific process. In science, at least at the core, we try to prove ourselves wrong. We do not try to prove that X causes Y, we try to prove that X does not cause Y. When we obtain data that undercuts a paradigm, we do not write a fucking press release, we first consider how we fucked up the damn experiment! We do not identify the next great anti-cancer therapeutic target, we identify a protein that is required for uncontrolled cellular replication in a certain cell line under certain growth conditions in the lab.

If we as scientists, have truly identified a paradigm shifting result or established that the textbooks need to be rewritten, this will come out in the end. If we hoist ourselves by our own petard, then we have a problem. Think about this, when we push these boundaries of science as I see happening too often in publications and manuscripts I review, are we any different than the snake oil salesmen of yesteryear, or the person at the other end of the psychic hotline, or the politician that assess every problem to some simplistic social issue we already agree with.

We're scientists. We're better than this.

Teaching Critical Thinking

One question I grapple with is 'how do we get students to ask questions about, or rather to question, peer reviewed research papers?' This is based on my experience that undergraduate students and even many introductory graduate students have difficulty grasping the concept that there may be issues or even important problems with peer reviewed research. Part of this is based on an inherent appeal to authority/self confidence issue, how could a lowly undergraduate find something problematic with a paper written by Ph.D., or equivalently trained, scientists.

Why We Care About Critical Thinking

However, the question I am grappling with is just a subset of the more important issue, how do we teach students how to ask the 'right' questions. The key here being 'right.' This is a fundamental aspect of critical thinking. Being able to identify the assumptions, biases, controls needed, discrepancies, etc. in an argument, and a peer reviewed research paper is nothing if not an argument. I find the most successful approach is to identify these, and other, points by asking questions. Again these question have to be the 'right' questions.

In my advanced undergraduate class, I can usually classify my students into 3 categories: the non-questioners, the trivial questioners, and the rare critical thinking questioner. By the end of my course, I want my students to find themselves generally in this latter category.

In the first category, the non-questioner, we find the shy students who are uncomfortable speaking up. This silence could be the result of inherent shyness, poor classroom experiences, or even cultural issues. In fact, this point of 'cultural issues' reminds me that it is important for me to remember that women and minorities are frequently ignored or blatantly omitted from discussions. In my experience, there are as many if not more women promoting the discussions in my course as men. Regardless, I try to address the issue of cultural differences early by calling on women and minorities during our 

The Shy Student

discussion sessions. Included in this category of students are those who are not confident with the material and thus do not want to speak up for fear of saying something stupid. I provide many resources and tools to help bring students up to speed if they are missing some background, so I tend to be less sympathetic with these students because they, by definition, must be aware of their deficiencies and choose not to address them. Of course, it takes awhile to separate these students from those who are shy, but it is disheartening to identify a student as being  intellectually lazy, lazy in general, or indifferent. To be clear, I have had students that lack some of the foundational material needed for my course that have worked hard to address these issues, and I help them as much as possible, having one on one meetings as much as they need to go over concepts, specific papers, etc. I love these students, because they have a drive that is infectious. Getting back to the shy students, how can I help get them engage in the course, such that they can move to category 3 and without having to change their personalities? For these students, all students actually, I have online components to the course. In addition to in class discussions, I have an online forum to initiate new discussions or continue discussions started in the classroom. This provides a place for those students who are inherently shy and students who are not comfortable thinking on their feet, which is what the classroom discussions entail. Students can use these forums to ask broad questions, initiating discussions beyond the minutiae of the papers they read. Students can also ask for help if there is something in the papers, a method, conclusion, etc. they do not understand, and students can help their colleagues by answering those requests for help. While I monitor the discussion boards, I refrain from commenting as much as possible, such that it quickly becomes a student-driven environment. While not perfect, there are mechanisms to promote moving students from category 1 to category 3.

Were you there? Only applies
to science not the New Testament.

The second category: the trivial questioner, is the place I work the most. Not that a specific student is a constant trivial questioner, but rather it is a constant place we come back to in class. This is not a problem because it does serve as a constant 'teachable moment.' The trivial questioner falls into the meme that there are no stupid questions. Of course there are stupid questions! In fact, the 'there are no stupid questions' comment is itself a stupid comment. I understand the 'there are no stupid questions' concept, but it is used with the tacit understanding that everyone is acting in good faith. This is seldom the case. For example we have Ken Hamm's 'Were you there?' question. This question is bullshit and not acting in good faith. The fact he encourages ten year olds to ask this question just serves to exemplify the moral vacuum in which Hamm resides. Hamm knows this question is a bullshit question, but it is a nice soundbite gotcha-sounding question to the masses. However, the ten year olds Hamm sends out to 'ask' questions do not know why this is bullshit, and that is his goal. However, we can use it as a teachable moment. The problem, in my opinion, is that Hamm knows many children would never ask the question, but will think the question and then answer it for themselves. In my course, there are many 'were you there?' type questions. Not necessarily from the Hamm perspective, but from the 10 year old perspective of 'this sounds good, I'll go with it' perspective.' These kind of questions are particularly present at the beginning of the course and I like to think I help move the students into the 3rd category. It's possible that I push these students into the 1st category, but I doubt that based on the quality of the discussions as the semester progresses. By way of example, every year when discussing a paper using a mouse model, the question will arise 'well I am concerned that the study only used female mice and I wonder what the data would be if male mice were used?'  This kind of question is relatively easy to come up with because we teach students 'black and white' thinking, everything is a binary decision. So when the student reads the methods and materials and sees '20 female C57/B6 mice were....' the student immediately thinks 'male' or vice versa in a 'tell me the word that pop into your head when I say...' kind of way.

So how do I encourage questions/comments of these 2nd category students without pushing them into the 1st category? What I have found works is to mimic Socrates, I ask questions. For example...

I am concerned that the study only used female mice and I wonder what the data would be if male mice were used?

Why do you think this might make a difference? I agree that there are important differences between females and males, I'm wondering how you think these differences apply to this study?

Well there could be differences due to hormones or something...

That's a great point, because that is clearly the case in certain instances like Paracoccidiodes infections. Is there anything in this system that makes you think there is would be a sex-based difference?

.....

Ok that's a good point at face value, but maybe needs further consideration, did anyone have additional issues with this study?

The point is to encourage/require the students to have a scientific justification for their concerns, questions, critiques. This, in my opinion, is the most difficult thing the students can learn and that I can teach. The point is that I need to teach the students how to question the studies, but also to question their own questions/concerns. But, I want to emphasize that it is ok to be wrong! We talk about well conducted studies that have generally solid conclusions and identify potential concerns. These concerns may not change the overall conclusions, but do raise concerns with sub-conclusions that may not be valid. In fact, the introductory paper we are discussing is the one I railed on previously. We will also be considering the press release. This is a change from the last couple of years when we discussed a well written, described, and assessed paper. I'm interested in how this approach works.

Own this book!!! 

Finally, we come to the third category: the critical thinker questioners. For these students, I can only refine their skills, improve their writing, and expose them to new and interesting areas. Almost uniformly, these students have research experience and likely significant experience. However, these students almost certainly have holes based on the areas they have been exposed to. These students know the potential issues in the area they are familiar with, but lack the similar approach/mindset in the areas they are not familiar with. This is one of the reasons it is essential for scientists to read well outside their fields. Breadth of knowledge promotes a better assessment for how your studies fit into the broader world of science. This increases the impact of your research not only from the study in question, but also in the questions you ask in the first place.

FYI. While I have categorized student comments/questions into 3 groups, no one student (nor the instructor) fits into a given group. Furthermore, I love my course because I learn so much from the students, even those that tend to cluster in a specific category. The lessons I learn may vary, but I learn important concepts, holes, insights from all three groups.  I thank the students from previous years for helping me develop these insights and to improve my courses.

Father-in-law of a Science Fair

Well another Twin Cities Creation Science Association science fair has come and gone. This ostensibly is a science fair for homeschooled children, but is more pointedly a “Christian-only science as described in the bible 'science' fair.”

from here

This is my 4th TCCSA science fair (you can see my write ups on the others here, here, and here). However, this represents the first time I was able to attend when the students were presenting their posters to the judges. Of the 23 (one for each human chromosome?) posters, I was able to talk with about half in some detail. Not surprisingly talking with the kids was a blast...well talking to some of the kids was a blast. Like other venues, some kids were truly enthusiastic about what they had done, whereas others not so much. One has to wonder how much the parents forced this latter group into participating and even going so far as to do the work.

I do want to comment on the venue. In previous years this religion-as-science fair was held at the Har Mar Mall in the large open space/hallway separating the stores that defines a mall. The positives of the Har Mar Mall was the fact I could buy a new book at Barnes&Noble and get a beer with friends at Old Chicago afterwards. Both of these abilities served to limit the soul crushing depression associated with kids being taught that religion = science. The negative with the Mall was the lack of emphasis on the work the kids did. There are elderly power-walkers doing laps, potential homeless folks trying to stay warm, peripheral security guards watching potential homeless folks, and shoppers scurrying from one store to the next. All of this detracted from the posters. However, one of the goals of the TCCSA religion-as-science fair is to bring Christianity to the masses:

Five things to remember:
1. Know your material. 
2. Be confident. 
3. Communicate well. 
4. Be thorough. 
5. Pray your exhibit will witness to non-Christian visitors. (emphasis mine)

from here

So from that perspective, the Mall is a useful venue even if, and maybe especially if, it detracts from the science. Regardless, this year the event was held at Northwestern 'College', which  (their words) "exists to provide Christ-centered higher education equipping students to grow intellectually and spiritually, to serve effectively in their professions, and to give God-honoring leadership in the home, church, community, and world." Thus, Northwestern College seems like the perfect place for the religion-as-science fair. The campus is adjacent to Little Lake Johanna and quite pleasant/quiet. I jogged around it a few times last summer and look forward to it again when winter is over. The plus is that the environment was more conducive to looking at the posters and engaging with the people there. Northwestern College is also close to a favorite watering hole for the post-fair recuperation drink.

So I arrive and started looking at posters. When a poster had a student standing by it with no one showing an immediate interest, I went there and asked the student if they would tell me about their work...and now we enter the realm of any K-12 science fair occurring at (a diminishing number of) schools all over the country. We have the good and the bad.

This fair represented age groups from around third grade to junior/high school I'm guessing, so the level was quite diverse. There were two categories: experimental and report. The reports, regardless of venue, tend to bore me because I find it difficult to engage the student in the process of science. There were two of this type that I recall: one on butterflies and one on cranberries (I learned that cranberries grow and are farmed in bogs!). Neither student for these posters was available when I was in the vicinity.

The experimental covered a range of science. We had the standard 'what freezes first? warm or cold water' Spoiler Alert: It's cold water, and a study on the 5 second rule (see below). Notably absent was the 'do plants grow better when music my parents like is played?' project.

My favorites:
1. the 5 second rule presented by a young woman (Im guessing 4-6th grade range). She conducted several experiments and even had some important controls included. She was familiar with the literature on the topic and was clearly well prepared. Most importantly was her enthusiasm, I was exhausted by the time she was done.  It was a lot over the top, but compared to so many student posters, including graduate students' at professional meetings, this was a breath of fresh air...albeit at ~100 mph so maybe a tornado of fresh air. 

2. One young man studied the parallax effect using trap shooting as a model. Basically he had people shoot clay pigeons with either one eye open or the other or both and graphed the success rate. He did replicates and used several different people termed expert and amateur at shooting. I wanted to ask him if he thought practice had more to do with it, but he was also not around when I there. This idea was supported by his data where the amateurs' success wasn't altered by a change in eye usage, whereas the experts' were. I liked this project because it tried to look at something interesting using readily available resources. Based on the pictures, he and his family appear to be trap shooting enthusiasts. This seemed liked a great way to combine something he enjoys with some science.

My least favorites: 
1. The unprepared. These were students who probably had little involvement with their research other than assembling the poster. There were several in this category. I worry about my own son's science projects, because I help him. I know I can come up with experiments/projects for him, walk him through the experiments, and prep him for the presentation. But how does that help him or teach him science? It just teaches him to do what he's told (how I wish for that some days though!). I find asking him questions works best. Ask him what he thinks, ask him about other possibilities to explain his data and then ask what we could do to rule them out. This approach seems to work based on feedback I've received from those who talked to him at his school's science fairs. 

2. The I'm too cool for school. There was one of these. The 'let's put a mentos in soda and see what happens' experiment. This young man was clearly one of the older participants. I asked him to walk me through his study and his immediate response was 'Well this is the top left corner, and over here is the top right corner.' Normally I would just be done, but I decided to play. I asked what he wanted to learn and why, we all know it foams up. *crickets* I asked him what causes the foam? *crickets* I asked him in what way he thought his work represented a good example of science in action? *crickets* I then suggested he check out the '5 second rule' poster.

3. The I totally outwitted all those experts. This was the one overtly creationist one I recall. (This is sold as a science fair for home schooled children and I expect some (many?) participants are not necessarily raised by creationists but simply the parents are using the forum to have their kids participate in a science fair.) This young woman, on the older side, was interested in stalactite formation, which some guides at a local cave said took millions of years to form. Well, using a solution of epsom salts she showed that crystals could form within a few days. She then concluded, and I quote, "I proved them wrong." I was polite and asked some methodology questions. I did ask if she thought the stalactites she saw were made from epsom salts and she of course said no. I then asked if she thought that might matter.

I spoke with a number of other students, who were great. It's fun talking science and seeing the joy of learning something new in a person's, especially a young person's, eye. The sense of accomplishment with being able to explain something new to someone. Of course at this religion-as-science fair the posters were tainted by the requirement of a bible verse relevant to the science project (which is part of the judging criteria). Prizes were awarded and you could not win based on the fact that the judges were not impressed with your bible verse. Luckily I met up with two friends Greg and Mike at the fair and we relocated to the local pub to decompress afterwards. It was good to end on a high note, especially one that lacked a required bible verse. 

Update: a reader noted that the original cartoon I inserted made fun of a creation science fair participant. My intention is not to disparage the kids who are the victims in these events. Thus, I replaced it with a more relevant cartoon. Thanks to the reader for bringing that issue to my attention.  

Of all the crazy.....Editor's edition

I serve as an editor for several scientific journals. Basically I assess submitted manuscripts at a general level to see if the work is appropriate for the journal. I also make sure the work appears valid at a most basic level. If the manuscripts meet these requirements, I recruit a number of expert reviewers to rigorously assess the data and conclusions of the paper using criteria specific to the journal. (There is significant overlap in criteria between journals, but each journal has some specific issues or requirements.) Once the reviews come in, I have to integrate the reviewer assessments as well as my own and reach a decision on the paper: accept, revise, reject, etc.

Today I was looking at my new assignments and saw a paper that based on the title and abstract was appropriate for the journal. I quickly skimmed through the paper, which looked a little light, but seemed to relate some useful new information. So I went looking for some reviewers (I try not to use the same reviewers, because we are all busy and it's important to spread the workload around). One of the easiest way to identify reviewers is through a simple pubmed search. Enter a couple of relevant keywords related to the paper and voila instant potential reviewers.

Except today. I enter two obvious keywords, hit return, and get ~100 articles. The first of which is on the same topic as the paper I'm handling (I can tell from the title). Well that sucks, but it happens. Separate groups work on similar problems. Hopefully there will be enough differences between the papers that the one I have can still move forward.

Oh wait, the published paper is by the same group! FUCKITALL. I looked through the just published paper and see that there is minor difference, but the papers are essentially the same. On the plus side, there was an additional data set included to a figure in the paper in my hand. On the negative side, there was an identical figure. (The word you are looking for is plagiarism, and yes you can plagiarize yourself.)

This pisses me off, because authors agree when they are submitting papers that their work is not being considered elsewhere. For the authors, they can maximize the chance of getting published. But the authors are wasting everyone else's time. Half of the reviewers are wasting their time assessing the rigor of a paper that is not and will not be publishable. Add in the waste of time on the editors and journal associated staff and it is bad practice.

Congratulations you earn the:

Evolution: Time is on Our Side

A friend of mine asked me the following ‘I just learned that mathematicians assert (not necessarily in an argument to support God) that there hasn't been enough time for the theory of evolution to be viable. I'm making an assumption you have been aware of this given your profession and scientific mind and I'm curious of your take on this.’

My initial response was along the lines of ‘I don’t have time for a substantive response at the moment but the short answer is that some mathematicians are dumbasses.’

This represents my more substantive response because I know she is interested in an actual answer. A problem comes from the nature of the question itself. I do not know the context with which these statements of mathematicians came up. Was the discussion with an actual mathematician or is this some second hand remark made by a stealth-creationist. (Although the conversation may not have been directly related to a god concept, I am unaware of any general concerns with the theory of evolution that is not steeped in religion.) What was the conversation that led up to this point? I do not know the answers to these questions, but I have a couple of educated guesses what the context was, but if I am far from the mark my friend can let me know and I’ll add another post if needed.

So my two thoughts on how a conversation related the issue of mathematics and time for evolution can be grouped into the following categories: 1. Historically Inspiring; 2. Tedious Probabilities.

Historically Inspiring (please be this one).

The theory of evolution is a dramatically different beast from that first outlined by Charles Darwin in the 1859 publication Origin of Species. (This is not a drawback of science, it’s a feature. Science progresses.) When Origin of Species was written, geologists had calculated the age of the Earth as 20 - 400 million years old. By geologists, I refer to Lord Kelvin of absolute zero fame. My friend may remember Lord Kelvin from Mr. Daigle’s chemistry class as the person who calculated absolute zero as -273°C or 0°K (the K stands for Kelvin). Anyway, Lord Kelvin calculated how long it would take the Earth to cool to its current temperature if it started as a molten ball and came up with a range of 20 - 400 million years in 1864, 5 years after Darwin published.

Kelvin ultimately settled on a more narrow range of 20 - 40 million years. Regardless, Kelvin believed this was too short for evolution to explain the diversity of life on Earth. You know what, Charles Darwin concurred. Darwin knew that his theory requires that the Earth to be extremely old. In the first edition of Origin of Species, Darwin argued that the time necessary for erosion to form the Weald in England is at least 300 million years (But we would also need to factor in the time to deposit all the material to be eroded among other things, which brings us to an age of billions of years). However, based on Kelvin’s calculations, Darwin removed these arguments from later editions of Origin of Speices and referred to the problem (or Kelvin) as an ‘odious spectre’ in letters. To be clear here, both scientists had data to back up their claims, but Darwin was quite cautious in his claims.

Smarter than me, but
still totally wrong

The interesting point is that Kelvin was flat out wrong. Based on what was known at the time, Kelvin’s approach was defensible. However, this was before we knew about radioactivity. The Earth did cool, but it is not simply a loss of heat issue, as assumed by Kelvin. Radioactive decay generates heat, and there is a lot of radioactive decay within the planet. Kelvin thought the Earth’s core was solid and that all heat transfer was by conduction (wrong on both counts).

By making a number of assumptions, which were defensible at the time, Lord Kelvin mathematically derived an age of the Earth that undercut the Theory of Evolution as well as most of geology. However, data was already in existence that suggested the Earth was much older than the age calculated by Kelvin. As more knowledge was gained, it became clear that Kelvin’s assumptions were invalid and thus his calculation wrong. Turns out the Earth is ~4.5 billion years old, which is plenty of time for geological formations to arise and giraffes to evolve. Interestingly, molecular geneticists have calculated that the last common ancestor of life lived ~3.6 billion years ago, which is not long (relatively speaking) after the planet formed. Supportive evidence for these dates comes from the earliest fossils, which are ~3.4 billion year old bacteria.

(Point of reference, Kelvin, a faithful Christian, calculated that only a moron can believe the Earth is a few thousand years old.)

Tedious Probabilities (probably this one)

Another common way to try and use math to disprove the theory of evolution is to misuse probability. The idea is to come up with a probability statement and then use that to show there is not enough time in the age of the universe for life to have occurred. If the odds against an event happening are so huge, it is impossible for the event to have occurred within the time frame the universe has existed. There are two problems with these approaches. First, it is trivial to come up with probabilities that sound impossible for events to occur even those for those events that have already happened. Second, those making the probability statements make assumptions that have nothing to do with biology or reality for that matter.

Making big numbers to impress those not used to big numbers.

Let’s start with a simple probability idea and work our way up. If we get a penny and flip it, the chance of it coming up heads is ½ or 50% (the other possibility being tails of course). The odds of getting heads twice in a row is ¼ (½ x ½) or 25%. Similarly the odds of flipping a coin and having it come up heads and then tails is also ¼. We can take this a little further, the chance of flipping a coin 10 times and having it come up heads each time is 1/1024. 1/1024 is the same as 0.0009765 or 9.77x10^-4 or ~1x10^-3_. It’s about 1 time in a 1000. The important thing here is that the odds of getting any specific combination of heads and tails in 10 flips is about 1 in a 1000. However, if you flip a coin 10 times, you will get a specific combination. I just flipped a nickel 10 times and got T(ails), T, H(eads), T, H, T, H, H, T, H. Was the chance of that happening 1 in a 1000? Well, it was before I flipped the nickel the first time, but now that it has happened the chance that it happened is 100% or 1/1.

When making these kind of arguments, the person picks a really big number to be awe inspiring, like the number of atoms in the entire universe. The number of atoms in the entire universe is ~1x10⁸¹, which is a 1 followed by 81 zeros. Everyone knows the universe is really fucking big (what’s bigger?) and atoms are really fucking small, so it makes intuitive sense that the number of atoms in the universe is probably the biggest big number of them all. So, if you can come up with a probability that is greater than the number of atoms in the universe, it must be impossible right?

Remember when we flipped the coin 10 times above? The odds were ~1/1000 (a 1 followed by 3 zeros) that any specific sequence would come up. Well if we flip that coin 270 times, the odds of it coming up heads every time, or any other specific sequence, is 1/1.9x10⁸¹. If I flip that coin every 15 seconds, it will take me just over an hour to get enough flips to get a sequence of heads and tails. If we calculate the odds of getting that sequence ahead of time, we get a number greater than the number of atoms in the universe! Using the creationist logic, then it was impossible to get the sequence of 270 Heads or Tails we just got.

Catnip for dumbasses

Maybe a more current analogy is in order. Powerball! The odds of a specific set of Powerball numbers coming up is 1/1.9x10⁸ (1 time in 190,000,000 tries. Face it, while someone will win occasionally, you won’t). The odds that the last 3 drawings would give rise to the numbers 13,28,49,51,59, 33; 2,24,46,52,56, 19; 4,19,33,41,59, 9 is 1 in 6.86x10²⁴. However, these numbers were in fact drawn. On Dec 9^th the odds were 1 in 6.86x10²⁴ now they are 1 in 1 or 100%. The difficulty with thinking about these types of calculations is that we forget that although any specific event may by improbable, a specific event will happen.

Just to get a number bigger than the number of atoms in the universe, the biggest big number, let’s go back 12 Powerball drawings, which would be drawings that occurred over roughly the last month (11/9/2011 - 12/17/2011). The odds of all the specific numbers being drawn were ~1/2.21x10⁹⁹ which is more than the number of atoms in the universe by 1,000,000,000,000,000,000 times (10⁹⁹ - 10⁸¹). Ergo, those numbers could not possibly have been drawn! But these odds were only true before the Nov. 9^th drawing. Again, the odds today are 100%, because this event already happened. It’s pretty easy generating really big numbers to sound impressive, but just because someone has a big number doesn’t mean he knows how to use it right.

Which leads us directly to the second problem here, the assumptions.

The assumptions used to make the calculations regarding evolution in the first place are suspect (wrong is a better word, fraudulent is the best word because those making these arguments have had it explained to them before). For example, the assumption in these types of calculations is that there were a bunch of chemicals and then ...

...these chemicals came together to form the first cell. First, no scientist worth her salt has ever made such a claim, although a lot of creationists have. I don’t want to get into a discussion about the origin of life in this post, but I do want to stress that I have never seen the absurd idea that cells just poofed into existence fully formed from scratch except by creationists. Irony alert: creationists think life zapped into being en masse by god, but ridicule biologists for thinking cells zapped into being en masse by evolution (even though biologists don’t think that).

The ‘calculations’ I have seen use assumptions like the following. Let’s assume that a cell needs 100 different proteins to survive and a protein is on average 100 amino acids long. (This is great because then the creationist can then honestly state that these are conservative estimates. As far as we  know a cell needs more than 100 types of protein to live and proteins average length is more like 300 amino acids. This serves to make the result of the creationists argument that much more impressive.) With these (wrong) assumptions, what are the odds that the 20 kinds of amino acids will randomly come together to form all these proteins simultaneously to allow for a viable cell.

Using these assumptions, which have nothing to do with reality, the following ‘calculations are made. The chance of a single protein randomly assembling is 1/1x20¹⁰⁰ the denominator, as you know, is a number bigger than the total number of atoms in the universe. Add in the factor that you need this to happen 100 times (for each protein). It ain’t never gonna happen! CHECKMATE biologists, the theory of evolution is impossible.

Of course, no biologist thinks or suggests that the above happens. But why let reality interfere with your delusions.

To summarize:

There was a legitimate concern about the age of the Earth and whether there was enough time available for evolution to explain the diversity of life (and geology to explain the geological formations around us). This concern was resolved soon after by physicists and geologists.

Probability is poorly understood and can be used quite effectively to convince people of erroneous ideas. This provides a chance to reinforce the importance of critical thinking and skepticism.

It is easy to demonstrate something cannot be true if you are comfortable being a liar. First, misrepresent the position you are arguing against. Second, make up a bunch of bullshit premises you state represent the position in question. Third, show why the position is wrong because the bullshit premises are bullshit.

If that does not address you question, please let me know.


Regardless, if it wasn't clear enough, here is what the Rolling Stones have to say on the matter.

Statistics ≠ Checking Your Brain at the Door

As part of an endeavor to improve undergraduate writing, I was involved in a day long session of reading senior level writing assignments.  Basically, a group of us had a list of criteria and ranked each assignment as meeting or not meeting the criteria. We were not grading or assessing the worth of the assignments simply whether a given criteria was met or not met. I learned a few things during the 7 hours of reading 16 assignments (~20-30 pages each), one of which I want to touch on here.

COLLEGE SENIORS DO NOT UNDERSTAND STATISTICS!!!

Now I am not a statistician nor do I have any real expertise in statistical analysis. In fact, I turn to statisticians when I need to do statistical analyses beyond student T-tests or analysis of variance. However, I think I know enough about statistics to not make the error of the p-value.

The p-value essentially tells you the probability that some event, data, occurrence is due to chance (generally referred to as the null hypothesis). So if you are hoping that the effect you are looking at is not due to chance you want a small p-value. The question, of course, then becomes 'how small'? The scientific community has generally agreed that a p-value of < 0.05 is a rigorous cut-off. A p-value > 0.05 is considered to be reasonable odds that your effect may be due to chance. However, this cut-off of 0.05 is arbitrary and indeed higher and lower cut-offs are used in some fields.

To be clear, p-values can range from 0 - 1, so you can consider a p-value of 0.05 to be analogous to a 5% chance that the effect you are looking at to be due to chance. That also means that your p-value of 0.06 means there is a 6% chance your data is due to chance and that is too high for most scientists to consider your data significant.

xkcd's take

Now we come to the error of the p-value. By way of example, you should check out xkcd's acute take on the problem (well several problems, but the one we care about is central). If we look at a lot of different data under a given condition, then we should expect a data set to show a p-value < 0.05 on average 1/20 times (5%) that is strictly due to chance. This does not mean that we should discount a result that comes with a p-value of 0.05. It means there is only a 5% chance the result is due to chance. However, if there is additional data to back this result up, we can increase our confidence even more. If there is not additional data, hopefully the scientists (aka senior undergraduate students) will at least acknowledge the limitation of the many data points. Sadly, both of these were lacking in a couple of cases that I observed, although I admit I do not know if this represents a statistically significant (p < 0.05) result.

Bride of the Creationist Science Fair

From all-creatures.org

Not sure if it was just me but it seemed like the fair was smaller this year, not a lot smaller, but smaller. Also, I noticed that the vast majority of posters were analogous to posters you would see at my son's  public school science fair. There were no posters that I saw using a motor to prove intelligent design, or sand to prove the Noahic flood made the Grand Canyon, etc. Mostly the posters were the book report demonstration variety posters you see at elementary school science fairs.


There were a few experimental posters, one was particularly striking. This student wanted to know if your dominant eye allowed for faster reflex responses than the non-dominant eye. I liked this one, because it was internally controlled (same person, two eyes).


Another had to do with reading versus color perception. This one had the names of colors written in the color, ie purple red blue, compared to the same word order in mismatched colors, ie purple red blue. The student measured how quickly a person read each page. Pretty clear cut differences in response times were shown. I wish I could have spoken to the student, because I really liked the approach, but once I saw the data and experimental set up I wondered if a person reading the first page realizes the words/colors match and then stop reading and just look at the colors. (This is not a critique of the student or poster, this is how science is done. We come up with a question/hypothesis, test it, and then based on the results revise/repeat. We also look at the experiment/results and see if other possibilities are suggested.)


Anyway, there is nothing new here from previous years. Same organizers, looking all cro-magnanish, as the last few years. So I don't have much to report, however I did come away with some insights that I thought I'ld share. Clearly, Jesus is lord and evolution is a lie foisted upon us by the liberal fascist Muslim commies. Wait that wasn't it.


This year my 8 year old joined me (so we could go swimming afterwards). Looking at a poster on birds, my son asked me to read some of the panels (they were in cursive...free hand cursive...young child free hand cursive, so he ask me to interpret). So I read the bible verse for the poster which was about the fifth day of creation old testament style: and on the fifth day God created the birds. Again, this poster was on birds. My son then asked me a question that kind of rocked my world.


Son  "What does that say, I can't read cursive?"
Papa  "On the fifth day God created the birds."
Son  "How do they know?"


That's it right there. A scientific world-view in a nutshell, an 8 year old nutshell. "How do they know?" That also demonstrates more clearly than I can explain the difference between science and religion. "How do they know?"

From here

How do creationists know birds were created on the fifth day by the God of the Jews? Because it says so in Genesis chapter 1 verses 22-23, which in my New American Standard bible reads, 'And God blessed them, saying, "Be fruitful and multiply, and fill the waters in the seas, and let birds multiply on the earth." And there was evening and there was morning, a fifth day.' In religion, at least the fundamentalist versions, you look at the text and there's your evidence. Actually, maybe it's better to say that in religion, at least the fundamentalist versions, you look at the text and there are your conclusions. Since you have a conclusion, not a hypothesis, there is really no need to get data to support it. Basically, you can say on the fifth day God created birds. Q.E.D.


With science it doesn't work this way. Yes, you can find statements written in books stating that birds evolved over millions of years likely from a dinosaur ancestor. So really its just the same as the bible right? You can place the Bible and The Origin and Evolution of Birds side by side and see that one says Day 5, ~6,000-10,000 years ago, and the other says that 150,000,000 years ago a lineage descended from the dinosaur gave rise to what would ultimately be modern birds. I mean both depend on the written word of some text right? Wrong.


See in the former case, the Genesis verse is the conclusion from the beginning. There is no question about the matter. There can be no evidence to the contrary because the matter is established fact. In the latter case, the book sentences are a conclusion based on several hundred years of data, analysis, and understanding. However, the former book (Bible) always said that birds were created several thousand years ago. Indeed the Bible has said that for ~3000 years. In the intervening time, which led up to the current textbook description, we have learned much about the universe, the planet, life, evolution, etc. For example, when the Genesis story was first established, we did not know anything about radioactive decay, or even radiation, or atoms, or molecules, or microscopic things. In fact, anything affecting a person's life that was due to a microscopic thing would be blamed on a macroscopic invisible ethereal entity, like a ghost, angel, demon, etc. When the Genesis story was established the diversity of life was not appreciated, because so little diversity was known. DNA was not known, fossils were ignored, in fact most of a person's time was spent trying to survive until tomorrow. When the modern textbook on bird origins was written, someone didn't wake up one morning and shout Eureka! and rewrite bird history. No, the latter book (textbook) was written over the course of two thousand years, with most of the writing occurring over the last couple hundred years. Words, sentences, paragraphs, and chapters have been written, edited, and deleted as more information and detail has been obtained (almost all of this information occurring in other areas of science!).


So when my son and I have this conversation:

Son  "What does that say, I can't read cursive?"

Papa  "Modern birds are descended from dinosaurs like the T-rex."

Son  "How do they know?"

I have answers, not answers like "Well on page 5 line 22 and 23 of 'The Origins and Evolution of Birds' says that". No, I can talk to him about how we know and the cool thing is that if I don't know 'how we know that', he and I can find out together. Personally, I find that much more fulfilling than 'Well this book says so'.