Reforming the “peer review” system. A proposal.

This blog post comes on the heals of the last. I read with interest Leithart’s interaction with Campbell’s Justification Theory, but my thoughts ended up in a rather different place. I began reflecting on the need to keep the reviewers reviewed, to keep them on their toes, responsible and accountable.

Certainly Leithart’s post energised my thoughts, as this is not the first time his interaction with scholars on First Things could have been more vigorous and responsible. But I do not mean to point fingers just at him – and for the record, I have greatly enjoyed his own work and scholarship for years! To be honest, I have posted a couple of book reviews here I wish were kinder or at least fairer. Rather, my concern is about the peer review system generally.

For the following reasons, I think the review system at present, especially as represented in journals, is in need of reform. Too many poor reviews are penned which do not do justice to the works with which they interact.

I think there are obvious reasons for this. People want to add to their “publications” list for their CV, and a review, however poor, can simply be added. Editors cannot possibly keep up-to-date with all areas of not just NT studies, but even specific areas such as the Synoptics and Paul. This means that individuals are not well positioned to evaluate all submissions. So almost anything could pass as a review, and because reviewers are not held duly accountable, suitable checks and balances are missing. To take a silly and less important example, one reviewer once chided something I had published for its lack of engagement with German scholarship, a claim so profoundly factually false I wondered whether he had read more than my book title.

Many have experienced something similar. It is indeed easy to forget how much work authors put into their works, and how much thought they usually require (at least the best of them). Then along comes a review and people are led astray by someone who has put in a few minutes (usually mistaken) thought.

The result of all of this for those of us who are cognisant of the problem is a profound distrust in the so-called peer review system. Though not quite as bad as Amazon reviews, or some comments on news item blogs, the academic review system is a community lacking accountability. Nor can one always rely on established voices, respected scholars and such like, as they are often defending their treasured views developed decades ago, and so they can sometimes automatically resist development, even if it be thoroughly legitimate (I think I speak from experience, here!)

So, how can things be improved? Can confidence in the peer review system be restored? I think so. My proposal is simple:

For journal reviews, the author of the reviewed book should be asked to write a couple of paragraphs in response to any given review. Nothing too long, or it would never happen, just a 300-500 words reaction, giving the author the final word. The book author would be given a relatively short period, say 3-5 weeks, to:

a) Note what issues the review has hit upon that will give the reviewed-book author pause for thought. What has the reviewer said that is helpful, whether positive or negative?

b) Offer a brief response to critical points.

That’s it, in a nutshell, hardly rocket science. And I think this would be enough to encourage more accountability and thus reform the entire peer review system, restoring confidence. Why would this work? Simple: if a reviewer writes knowing that his review will be judged by the book author immediately following his or her review, this will surely encourage more care and attention.

Indeed, I call on all journals to adopt such an author-response system. Academia deserves it. 

So “who reviews the reviewers?” is indeed a key question. Over at Syndicate, I think Christian and the team have provided an excellent model for future consideration, and I’m honoured to be involved with a project that provides a pathway into the future.