Wednesday, November 12, 2008

Re: Open Source != peer review

Andrew has an interesting thread on the content of a slide of a recent presentation. In the comments you can read the back and forth on things; indeed, there are very many aspects to things and he did ask a very complex question, of which he assumed that I understood what he was asking, and I indeed assumed too that I understood what he was asking:
    Some argue that doing good computational-based science requires open source. The argument is that scientists need to review the source code in order to verify that it works correctly. How, they argue, can you review someone else's paper if you can't review the source code used to make that paper?

    I like open source. (My talk goes into the philosophical differences between "open source" and "free software.") I think there should be support for peer review. But I don't understand why the ability to see the source code, in order to review it for scientific quality, requires the right to redistribute the source code to others.
So, I assumed he was interested in hearing why people thing open source benefits open source. Misinterpreting the last two words, I though access to the code and the ability to redistribute code I find bad in my peer review. There was another incorrect assumption on my side: I had open peer review in mind, as I like so much about open source projects, instead of the peer review as in paper peer review, prior to the preprint server age. Another thing I understood incorrectly, was that he was only referring to computational packages, not cheminformatics in general. My mistake. Being from a GCC meeting, I assumed the latter.

Therefore, a lot of miscommunication. I agree to a large extend with Andrews analysis: peer review is certainly possible without Open Source. Actually, this matches closely with the discussion between Cathedral versus Bazaar opensource projects (see my post earlier this week). He argues that current opensource (cheminformatics) do not have enough eyeballs, and indicates that money buys eyeballs. Indeed it does.

However, the original argument I wanted to make, but failed, is that Open Source (any kind of access to the source code) is a strict requirement for reviewing the implementation. We do not want black boxes.

How you organize this access to the source code is another thing, and topic of much of the discussion in Andrews blog. There are many solutions, but all include some sort of access to the source code. Redistribution is not a requirement, though, if the review is only send upstream, as is common in reviewing papers.

I feel that Open Source is a solution worth fighting for, but I do understand the argument that funding of this approach remains to be a problem. Open Source cheminformatics is the equivalent of a preprint server; one solution to peer review, a good one, I think, not the only one. The parallels are seemingly even stronger: you cannot review a paper by just reading the abstract and the conclusion: a paper is not a black box either.

Anyway... just a tip of the iceberg touched in the discussion. Feel free to join in.