Pages

Tuesday, July 19, 2011

Bad Science; Bad Journals; Nonsense articles

There is a rather interesting, and very important discussion going on on the Blue Obelisk eXchange. What should people do with articles that are clearly wrong, in their opinion. The question highlights a paper where homology modeling was incorrectly performed, identifying the wrong active side, and consequently rubbish QSAR model.

    Update: There was some confusion about what papers we are talking about. It's not about peer-review, or the minimal publishable unit. Please read the question on the Blue Obelisk eXchange, I linked to in the first paragraph. That question is the context. We're not talking about mediocre papers here; we're talking of papers where something is seriously wrong. Mapping this to metabolomics (in reply to Julio's blog reply): someone published a papers analyzing MSn data, and got the whole wrong chemical structures. For each of them. Worse, they used a known, intrinsically wrong method. That's the kind of paper we are talking about. Paper that if you had that on your CV, you could forget about that funding anyway.

The question is about how to take this to the community. Get a commentary published in the journal? Take it to the blogosphere? Others joined in with additional stories, confirming that cheminformatics is in no way other than other fields, with respect to bad papers. Retractions is very unlikely, but commentaries are very uncommon in our field indeed, despite we all have our top five of bad papers.

Peer-review is no longer the answer. As wdiwdi describes, it is all too easy to just submit the paper, unmodified, to a lesser journal. And honestly, nowadays, when reading the ToC of the top journal in our field, the JCIM, I skip more than 80% of the papers ("Oh no, not another 15 structure QSAR-docking paper"). I recently, very briefly though, heard Wendy Warr (I value her opinion very much, and try to read whatever from here for which I do not have to pay personally) say that cheminformatics representation is a done business (in slightly other words), but doubt that can be true (and tend to disagree), if we see so much literature where we question the usefulness, even in the JCIM.

Option 1: Write the journal with a commentary or letter.
It seems, however, this is not frequently done. Indeed, I chose not to do this either. The barrier is large: suspicion is not good enough, and you have to provide rock-solid proof. Ah, but that is hard, in a field where reproducibility itself is at a low level (see our Blue Obelisk paper), though there is increasing awareness (the Blue Obelisk paper is in the 5th most cited in the journal in 2010, see this paper). Actually checking the JCIM author guidelines the journal does not, in fact, have a manuscript type for this kind of community response. Neither does MolInf seem to have one. The JChemInf does have a commentary type, but note that these are normally commissioned by the editorial board. But, as Antony points out, editors may be objecting against community replies on articles.

Some newer journals allow you to just comment on a paper. This is a very simple but effective approach to making community feedback possible.

Option 2: Use the power of the blogosphere.
This is a tried-and-proven mechanism. It has shown to have major impact in chemistry, with the sodium hydride story as most prominent example. If properly crafted, and taking advantage of services like ResearchBlogging, this can have a major impact. But, it has the downside that long term visibility is not necessarily preserved. Short term visibility in the browser can be organized with userscripts (as we have shown in this paper), and PLoS journals make the blogosphere part of their article level metrics, with ResearchBlogging again as important intermediate.

Why?
Why should we not just ignore those papers (as we now do), but act on them? I think that a good, high-level communication in the field is important to make this field go forward. It is my personal feeling that cheminformatics has effectively come to a halt. New generation cheminformaticians have great trouble finding good education with most research groups and education facilities in Europe closed down (what is the situation in the USA?), resulting in a field where people decreasingly know how to properly evaluate research, reducing the level of peer-review, etc, etc.

So, I think a spark in the community re-enforcing good quality cheminformatics papers is critical to the future of the field. And I do hope the people who contributed to the Blue Obelisk eXchange question so far will start writing about the papers they did not like. I am going to closely follow the discussion, and looking forward to what comes out of it. I also like to invite the community to start blogging: that is, what do you feel is needed to improve the quality of cheminformatics literature?

I would say, yes, we clearly need a better alternative for the current peer-review.

11 comments:

  1. I really don't fill uncomfortable about the articles that have "docking + qsar + homology modeling", or did smthing that already have been done for many times (it's boring but that the live it is). But! If the authors ignore previous modeling papers, ignoring the wet-lab data, and god damn ignoring general sense e.g. docking blind docking for the protein with well known binding site, etc ... - it makes me crazy!

    ReplyDelete
  2. This comment has been removed by the author.

    ReplyDelete
  3. Egon, since there have not been formal mechanisms to correct errors in the literature we have an opportunity to create them. This is our approach with the melting point curation project. For example, a reason is given as to why the 97.5C value can't be correct for a 2008 paper reporting the mp of 4-benzyltoluene.. In this particular case I thought the story was interesting enough to blog about. I also contacted the author and he said the student who made the compound is no longer around but he would re-synthesize to understand why a compound that is clearly a liquid at rt was reported as yellow crystals melting at 97-98C. The more we discuss these problems with chemical information openly and provide multiple dissemination channels the more likely the errors will be discovered and not cause wasted time in the future.

    ReplyDelete
  4. Vladimir, I don't object to such papers in general :) It is an important approach; it's just that often overestimate the applicability domain beyond to what is reasonable, I think.

    ReplyDelete
  5. Jean-Claude, a single bad melting point might indicate something is wrong. The papers discussed on the Blue Obelisk eXchange, and those that triggered my blog post, have so much more wrong that the whole paper is wrong, and none of the conclusions are valid.

    I have updated my blog post to make that double clear.

    I keep forget that most people do not click on links in blog posts, and that I need to repeat the context in my own blog.

    Someday, yeah, someday, I will start use my blog is a publishing enterprise, rather than a personal notebook.

    ReplyDelete
  6. This comment has been removed by the author.

    ReplyDelete
  7. I sympathize with your frustrations and I am glad to see such issues have been openly discussed. Have you written to the first author of said problematic study about your concern? For anything with a clear critical flaw, far beyond vaguely questionable issues, journals should retrieve the problematic article.

    ReplyDelete
  8. I have removed my old comment in response to direct advice from the chief editor of an ACS journal.

    According to ACS policies, any and all information regarding the review process of submitted papers to a journal is confidential, even in most general terms (i.e. the fact that a manuscript was ever submitted to a specific journal), and that includes the words and content of one's own review contribution.

    ReplyDelete
  9. Wolf, thanx for the update!

    I assume there is no such thing as super-injunction, allow you to actually state that you cannot talk about it :)

    ReplyDelete
  10. There is now a page describing the incident at

    http://www.xemistry.com/Mestres_Critique

    ReplyDelete
  11. Dear Wolf, thank you for the update! I will read your critique with much interest!

    Yesterday I was pinged about a new website, PaperCritic:

    http://www.papercritic.com/

    Readers of this post might be interested in this initiative.

    ReplyDelete