Scientific peer review compared to pull request peer review
@Migueldeicaza pointed to an article titled "The rise and fall of peer review", which argues that the current method of scientific peer review is relatively recent (it only became common in the 1960s) and hasn't worked out.
The problems are fairly well-known: peer review takes a long time, greatly delaying the publication of useful results; reviewers don't catch the really important problem because they don't look closely at the underlying data, the statistical methods, and so on; some reviewers (stereotypically "reviewer number 3") make unreasonable demands of authors; peer review (combined with using published papers as the measure of worth) funnels work into easily defensible topics and methods; and so on.
I think the author goes too far by saying peer review is worse than nothing. (Disclosure: I'm married to a scientist who, though retired, still gets papers to review and tries hard to do a good job.) But that's because I think a lot of the current problems with science are not due to peer review but are instead just exacerbated by peer review.
But that's quite possibly a distinction that doesn't make a practical difference. ("What difference would it practically make to anyone if this notion rather than that notion were true? If no practical difference whatever can be traced, then the alternatives mean practically the same thing, and all dispute is idle." – William James)
Let's take a different topic, reviews of pull requests. Many people (I'm one), loathe them. Why? They delay integration of changes, which causes all sorts of knock-on slowdowns; problems that could have been discovered (and fixed) early are instead discovered at the end; because the time reviewers spend reviewing is valued less than time they'd spend programming, they're incentivized to skimp on reviews, often just commenting on trivia rather than real design problems; the pickiest or most headstrong reviewer tends to wear everyone down, often leading to a sort of least-common-denominator programming; and so on.
Sound familiar? Like peer review?
Pair programming (or the newer practice of ensemble programming) works against at least some of those problems by: running designing and reviewing in parallel, so that you deal with one problem now rather than n problems all at the end; encouraging the social practice of fast-feedback conversation instead of the anti-social practice of flinging assertions into the ether and some time later receiving a rebuttal (or, too often, grudging acquiescence because it's not worth the effort and delay of another round-trip); and so on.
It seems like something similar would be better for science than peer review.* I imagine a person, designated a sounding board, being formally associated with an experiment from the very start. That wouldn't exactly be a pairing relationship – I think it's probably better if Professor Board is not at the same institution as Professor Experimenter – but it would be one of rather more frequent back-and-forth than happens after reviewers are sicced on (what Professor Experimenter fervently hopes will be) the final draft.
One way to incentivize more thorough reviewing would be to give the sounding board credit when the paper is published. In fact, someone really good at being a sounding board would be in high demand and would likely have their pick of promising experiments.** I could imagine really good sounding boards gaining academic promotion because they're so good at it. After all, they're helping to produce better science. I can even hope that a paper that has a well-known sounding board would thereby have its status elevated – be a marker of quality similar to the stature of the journal that finally publishes it.
No doubt a similar or better proposal would have problems with entrenched interests. (Elsevier make good money off the current system and has no interest in measures of merit other than journal impact factor and citation measures.) However, it could piggyback on top of current efforts to route around the damage that the current system imposes
* I am well aware that here I am, another computer programmer telling people in other fields how easy their problems are. That trick always works.
** I acknowledge there very well may be a Gresham's Law problem here: the sounding board who doesn't actually do any work might become very popular and so seem more useful to science. Content curation is a hard problem that we don't exactly seem to be getting better at.