Skip to content

Rigor in Academic Publishing

August 16, 2009

I opened this blog with a few paragraphs explaining why I believe it’s important for us to encourage our students to communicate and network in modern modes. Following up on that, I was sent a link to this recent journal article by Whitworth and Friedman. The somewhat bitter tone of the article aside, it’s a worthwhile read if you’re interested in the future of academic collaboration and publishing.

The authors’ most controversial conclusion is that we may need to sacrifice some rigor for progress. They argue that errors of omission (not publishing that which is later shown to be true) are as damning as printing that which is false, so we must accept the risk of some inaccuracies in order to publish articles that are relevant (timely) or which challenge accepted practice (controversial). I have some personal experience on the subject, and while that conclusion is difficult to accept (when I wrote my first draft of this entry, I opposed it), I think it’s correct.

My research group in graduate school went through a long publication drought (more than two years). We were building a computing system built on a non-traditional computing model, so our publications were judged irrelevant and/or difficult to compare to common benchmarks. We spent far longer gathering evidence to support our results than similar projects based on more traditional ideas. More importantly, during that time, we had difficulty finding venues to share our preliminary results, so we were unable to get critical feedback that would have helped us progress, and researchers on related projects were unable to learn from our mistakes and successes. Furthermore, I believe that in a less supporting environment, the project would not have been carried to completion. A multi-year drought can kill an academic career.

Rigor rightfully has a central position in science. We must demand accuracy and correctness. However, I agree that as a community, we must also provide opportunities — and credit — for publishing controversial and preliminary results, and we should have balance rigor with innovation and timeliness. To do otherwise will decrease the rate at which we generate knowledge.

One Comment leave one →
  1. Andrew Petersen permalink*
    August 17, 2009 6:21 pm

    Birman and Schneider recently published a viewpoint in CACM that describes a reviewing problem in the systems field. They report that the culture of the community has led to a rapid expansion in the number of papers submitted to top coferences, and they further note that these submissions are often relatively unpolished.

    The result is an increasingly arbitrary reviewing system, and they advocate a cultural shift to recognize multiple levels of publication: publication of initial, unpolished work at workshops or conferences specifically designed for them; small, innovative nuggets at top conferences that highlight incremental but significant advances; and fully reviewed, complete journal submissions as an ultimate goal. As noted in the other work we’ve discussed here, the fundamental component is social change. When evaluating a researcher’s quality, departments and tenure committees must consider traditional journal submissions as well as work published in more rapid, collaborative, and (occasionally) unpolished formats. Timeliness and rigor both have value.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: