Sunday, November 02, 2014

Reforming the “peer review” system. A proposal.

This blog post comes on the heals of the last. I read with interest Leithart’s interaction with Campbell’s Justification Theory, but my thoughts ended up in a rather different place. I began reflecting on the need to keep the reviewers reviewed, to keep them on their toes, responsible and accountable.

Certainly Leithart’s post energised my thoughts, as this is not the first time his interaction with scholars on First Things could have been more vigorous and responsible. But I do not mean to point fingers just at him – and for the record, I have greatly enjoyed his own work and scholarship for years! To be honest, I have posted a couple of book reviews here I wish were kinder or at least fairer. Rather, my concern is about the peer review system generally.

For the following reasons, I think the review system at present, especially as represented in journals, is in need of reform. Too many poor reviews are penned which do not do justice to the works with which they interact.

I think there are obvious reasons for this. People want to add to their “publications” list for their CV, and a review, however poor, can simply be added. Editors cannot possibly keep up-to-date with all areas of not just NT studies, but even specific areas such as the Synoptics and Paul. This means that individuals are not well positioned to evaluate all submissions. So almost anything could pass as a review, and because reviewers are not held duly accountable, suitable checks and balances are missing. To take a silly and less important example, one reviewer once chided something I had published for its lack of engagement with German scholarship, a claim so profoundly factually false I wondered whether he had read more than my book title.

Many have experienced something similar. It is indeed easy to forget how much work authors put into their works, and how much thought they usually require (at least the best of them). Then along comes a review and people are led astray by someone who has put in a few minutes (usually mistaken) thought.

The result of all of this for those of us who are cognisant of the problem is a profound distrust in the so-called peer review system. Though not quite as bad as Amazon reviews, or some comments on news item blogs, the academic review system is a community lacking accountability. Nor can one always rely on established voices, respected scholars and such like, as they are often defending their treasured views developed decades ago, and so they can sometimes automatically resist development, even if it be thoroughly legitimate (I think I speak from experience, here!)

So, how can things be improved? Can confidence in the peer review system be restored? I think so. My proposal is simple:

For journal reviews, the author of the reviewed book should be asked to write a couple of paragraphs in response to any given review. Nothing too long, or it would never happen, just a 300-500 words reaction, giving the author the final word. The book author would be given a relatively short period, say 3-5 weeks, to:

a) Note what issues the review has hit upon that will give the reviewed-book author pause for thought. What has the reviewer said that is helpful, whether positive or negative?

b) Offer a brief response to critical points.

That’s it, in a nutshell, hardly rocket science. And I think this would be enough to encourage more accountability and thus reform the entire peer review system, restoring confidence. Why would this work? Simple: if a reviewer writes knowing that his review will be judged by the book author immediately following his or her review, this will surely encourage more care and attention.

Indeed, I call on all journals to adopt such an author-response system. Academia deserves it. 

So “who reviews the reviewers?” is indeed a key question. Over at Syndicate, I think Christian and the team have provided an excellent model for future consideration, and I’m honoured to be involved with a project that provides a pathway into the future.


At 11/02/2014 6:37 PM, Blogger Richard Fellows said...

Great post, Chris. Many reviews seem written to display the reviewer's knowledge rather than to inform the reader. In the interests of accountability, why not name the reviewer who falsely accused you of failing to engage with German scholarship?

If journals feel that they don't have enough space to publish authors' responses to reviews (or even if they do), could they at least give a web address where the author could post a response and the conversation could continue? That is to say, each published piece should have an associated blog post where it can be discussed.

At 11/02/2014 9:20 PM, Blogger James Goetz said...

Chris, I mostly agree with you, but I propose that all journals could make it simpler by offering all book authors the opportunity to reply to the respective review within say 3 to 6 months while the reply could be in a later volume of the journal.

At 11/03/2014 8:33 AM, Blogger Terry Wright said...

To be honest, I'm not overly convinced by this. Inviting a book author to respond to a review could mean some people - alas, definitely not me! - having to spend far more time doing this than on, say, research. The other thing is that a review is simply that - a review - and if someone reviews a book poorly, then, in the age of blogging and social media, any poorly written review will soon be destroyed with fire.

If anything, I'd say that issues of quality control lie with the journal's editor(s), perhaps the reviews editor. If a review becomes known for its poorly written content, surely the review editor can invite a response from the author under critique.

At 11/03/2014 3:27 PM, Blogger Chris Tilling said...

That's a good idea, Richard, and given that all journals should have a developed online presence these days, makes all the more sense

James, possibly, but I think a brief response could be organised in a shorter time frame, and it helps the researcher to keep tabs on author responses, too. I would prefer it be in the same edition, but if that doesn't work, then perhaps your idea is best

Terry, Here's why I would push back. Your position effectively amounts to a counsel of despair for journals. I think we can do better. Plus, I really do not think it takes long for an author to write 300-500 words about their own work; I can't see that being a huge burden. As for blog responses, yes, all for them, but fact remains that (a) printed texts are referenced in monographs and articles more readily than blogs, (b) blog responses have not yet encouraged responsible book reviews. A direct approach will, I think. I could also say something about accessibility of response and the aura around printed texts to drive the point home.

At 11/03/2014 4:03 PM, Blogger Terry Wright said...

I'm not sure what you mean by a 'counsel of despair', Chris. Besides, it seems to me that reviews in journals have always run the risk of misrepresenting authors, unintentionally or not. That's not to say that this tradition shouldn't be done away with, of course. But I can't help but wonder if it just adds to academic administration.

About the word count: true, 300-500 words isn't a lot. But if you happen to publish a book that ends up being reviewed a lot, that could be a lot of 500-word-responses the author would need to write! And chances are that substantial and valid critiques of a book would probably require more than 500 words.

I also wonder how things might come across generally. I can appreciate that, if a review misrepresents an argument and is generally a bit arsey, you might want a formal printed opportunity to respond. But if a review misrepresents an argument but is positive, would the author be willing to forego a response, at least for the journal in which the review is published? What would be the motivation for any response? - the author's credibility and/or scholarship, or the potential sales, or what?

I think it's for considerations such as these that I'd still want there to be more initiative shown by the journal editor(s).

Interesting topic, by the way. :)

At 11/03/2014 7:06 PM, Blogger Chris Tilling said...

"About the word count: true, 300-500 words isn't a lot. But if you happen to publish a book that ends up being reviewed a lot, that could be a lot of 500-word-responses the author would need to write! And chances are that substantial and valid critiques of a book would probably require more than 500 words."

Fair enough. Perhaps the author should be expected to respond, invited to, but he or she may not choose to for whatever reason. That an author will be given sight and a chance to respond may be enough

By a 'counsel of despair' I mean the suggestion that we leave specifically journal reviews where they are. Change is needed as editors simply cannot do it all any more.


Post a Comment

<< Home