Tuesday, July 19, 2011

Bad Science; Bad Journals; Nonsense articles

There is a rather interesting, and very important discussion going on on the Blue Obelisk eXchange. What should people do with articles that are clearly wrong, in their opinion. The question highlights a paper where homology modeling was incorrectly performed, identifying the wrong active side, and consequently rubbish QSAR model.

    Update: There was some confusion about what papers we are talking about. It's not about peer-review, or the minimal publishable unit. Please read the question on the Blue Obelisk eXchange, I linked to in the first paragraph. That question is the context. We're not talking about mediocre papers here; we're talking of papers where something is seriously wrong. Mapping this to metabolomics (in reply to Julio's blog reply): someone published a papers analyzing MSn data, and got the whole wrong chemical structures. For each of them. Worse, they used a known, intrinsically wrong method. That's the kind of paper we are talking about. Paper that if you had that on your CV, you could forget about that funding anyway.

The question is about how to take this to the community. Get a commentary published in the journal? Take it to the blogosphere? Others joined in with additional stories, confirming that cheminformatics is in no way other than other fields, with respect to bad papers. Retractions is very unlikely, but commentaries are very uncommon in our field indeed, despite we all have our top five of bad papers.

Peer-review is no longer the answer. As wdiwdi describes, it is all too easy to just submit the paper, unmodified, to a lesser journal. And honestly, nowadays, when reading the ToC of the top journal in our field, the JCIM, I skip more than 80% of the papers ("Oh no, not another 15 structure QSAR-docking paper"). I recently, very briefly though, heard Wendy Warr (I value her opinion very much, and try to read whatever from here for which I do not have to pay personally) say that cheminformatics representation is a done business (in slightly other words), but doubt that can be true (and tend to disagree), if we see so much literature where we question the usefulness, even in the JCIM.

Option 1: Write the journal with a commentary or letter.
It seems, however, this is not frequently done. Indeed, I chose not to do this either. The barrier is large: suspicion is not good enough, and you have to provide rock-solid proof. Ah, but that is hard, in a field where reproducibility itself is at a low level (see our Blue Obelisk paper), though there is increasing awareness (the Blue Obelisk paper is in the 5th most cited in the journal in 2010, see this paper). Actually checking the JCIM author guidelines the journal does not, in fact, have a manuscript type for this kind of community response. Neither does MolInf seem to have one. The JChemInf does have a commentary type, but note that these are normally commissioned by the editorial board. But, as Antony points out, editors may be objecting against community replies on articles.

Some newer journals allow you to just comment on a paper. This is a very simple but effective approach to making community feedback possible.

Option 2: Use the power of the blogosphere.
This is a tried-and-proven mechanism. It has shown to have major impact in chemistry, with the sodium hydride story as most prominent example. If properly crafted, and taking advantage of services like ResearchBlogging, this can have a major impact. But, it has the downside that long term visibility is not necessarily preserved. Short term visibility in the browser can be organized with userscripts (as we have shown in this paper), and PLoS journals make the blogosphere part of their article level metrics, with ResearchBlogging again as important intermediate.

Why should we not just ignore those papers (as we now do), but act on them? I think that a good, high-level communication in the field is important to make this field go forward. It is my personal feeling that cheminformatics has effectively come to a halt. New generation cheminformaticians have great trouble finding good education with most research groups and education facilities in Europe closed down (what is the situation in the USA?), resulting in a field where people decreasingly know how to properly evaluate research, reducing the level of peer-review, etc, etc.

So, I think a spark in the community re-enforcing good quality cheminformatics papers is critical to the future of the field. And I do hope the people who contributed to the Blue Obelisk eXchange question so far will start writing about the papers they did not like. I am going to closely follow the discussion, and looking forward to what comes out of it. I also like to invite the community to start blogging: that is, what do you feel is needed to improve the quality of cheminformatics literature?

I would say, yes, we clearly need a better alternative for the current peer-review.