A Brand-new "Much Better" Review Scoring System!

Authors Note: I was told this artcile needs a sarcasm warning, so, be warned. :p

The moral imperative of a computer architecture researcher is that you should ABO: Always Be Optimizing. Let’s be honest, you can’t ABO unless you ABO all the time … for everything.

Time for a case study: One of the main tasks of an academic is reviewing other people’s papers, and that requires giving out scores. These scores are used, in part, to determine which papers get accepted into a conference or journal. The scoring system defines the possible scores and what they mean; here’s an example that has been used for quite some time in our field:

  1. Reject
  2. Weak reject
  3. Weak accept
  4. Accept
  5. Strong accept

There’s so much wrong here, let me break it down. First, look how simple it is, BORING! There’s no ambiguity over what a score means, or how a reviewer would vote, so that’s going to mean less time discussing whether to accept the paper – more time for boring research : (. Most importantly, it doesn’t even impose its own viewpoint about how reviewers should be deciding one way or another, that’s no fun! The key thing is that changing the scoring system with each conference has no downside because consistency and a common understanding over what a score means doesn’t matter!

So what should we do instead? Well, here’s the scoring system used by the latest ISCA conference:

  1. Accept as it is – High-quality paper that’s bound to move the needle in at least one major dimension (e.g., radically new insights, new research direction, methodological breakthrough). Must have in the program.
  2. Accept with minor changes– Solid paper with concerns that can be easily addressed in a revision/rebuttal. Should have it in the program.
  3. Minor revision – Reasonable paper with incremental improvement or with some deficiencies which could be addressed by revision.
  4. Major revision – Fair work but with concerns. Satisfactory revision is critical for acceptance in the program
  5. Reject – Serious problems that entirely compromise the paper. Revision will not help.

One word: beautiful. Off the bat, nice how the direction of the scores is flipped to shake things up – I’d love to see the expression on those students’ faces when they realize that all 5’s means unanimous reject .. ah classic. But the big plus here is the highly-specific explanations in each category! Notice that the only way to vote for accepting the paper without changes is to suggest that it has “radically new insights” – nice job setting the bar high, after all a lower acceptance rate == a more prestigious conference. Next, it’s great to have two categories that both advocate accepting the paper with minor changes – more arbitrary choices is always better! Finally, the only way to reject the paper is to say that it has “serious problems” that “entirely compromise” the paper. If you ask me, I’d say this language is necessary: those PhD student tears aren’t going to cry themselves.

Update 2/3: I just found out some good news about the ISCA reviewing system: only the papers with good scores are awarded a revision.
So for example, if all reviewers mark “Accept as is” the paper will be given the chance to have a revision. If all reviewers mark “Major Revisions”, then the paper will not be allowed to make a revision. I am in awe.

But why stop here, we can ABO the hell out of this system: more highly-specific choices and more blunt reviewer honesty please!

Without further ado, I present you the future of academic review scoring systems: