Methodology & Definitions

Companion to The State of Code Review 2026

This page exists so every number in the report can be checked. If you find a hole, tell us — recomputations welcome.

The one table to read first: denominators

Different metrics use different denominators on purpose. Forcing one denominator everywhere would produce dishonest numbers, so each metric is measured against the population where it's meaningful:

When the report quotes a median instead (e.g. 66.5% silent approvals), that's the median of per-repo rates — each repository contributes one number regardless of size, so a few giant repos can't drag the figure. For the headline metric the two views land 4.3 points apart (62.2% pooled vs 66.5% median repo) and on the same side of 60% — the gap exists because large repos silently approve somewhat less than the median repo, and we report both so neither view hides the other.

"Rubber-stamping" — the word vs the measurement

The report uses "rubber-stamping" as its narrative term, so let's be precise about it.

The term is old and well-established. "Rubber-stamp" entered English from late-1800s factory bureaucracy: as paperwork grew, managers began marking documents approved with pre-inked stamps — fast, efficient, and increasingly disconnected from anyone actually checking anything. It became the standing metaphor for approval as routine rather than judgment, and it's been dictionary English for over a century. In software engineering it names a recognized code-review anti-pattern: approving pull requests with little or no actual review, usually under time pressure, overload, or oversized diffs.

But rubber-stamping describes behavior, and behavior isn't observable in API data. We cannot see whether a reviewer read the diff, thought hard, and simply had nothing to say — or scrolled past and clicked. What we can observe is the trace: an approval that carries no words, no questions, and no inline comments. That observable is what this study measures, and it has its own precise name:

Silent approval is the metric. Rubber-stamping is the interpretation. The study's claim is not "62% of reviewers didn't read the code" — it's "62% of approvals leave no evidence of engagement, and at that scale, the absence of evidence is itself the finding."

Some silent approvals were genuinely reviewed out-of-band (pairing, Slack, calls) — that's the study's biggest stated caveat, and it's why every silence figure is a floor on invisible review, not an accusation aimed at individuals.

One more distinction that matters. Some teams deliberately rubber-stamp trivial changes — a few even use an explicit "rubber stamp" PR label for typo fixes and formatting, and that's healthy process, not an anti-pattern. Our numbers can't be inflated by that legitimate case: every PR under 10 lines, every docs-only change, every dependency bump and release commit is excluded before any rate is computed (see the exclusion ledger below). What remains — and what silently gets approved 62% of the time — is substantive code.

Definitions, exactly

Silent approval. The unit is the PR: a PR counts as silently approved when every effective approval on it (state APPROVED, from a human who is not the author, not later dismissed) has an empty body and zero inline comments. Two edge cases the definition includes, stated so nobody has to discover them:

• A round-2 approval counts as silent. If a reviewer commented on an earlier round, the author pushed a fix, and the reviewer then approved without words, that PR is silent under the primary definition — the engagement happened, but not in the approval. 22.1% of 2026 silent-approval PRs match this pattern. Reclassifying them as engaged makes the era jump larger, not smaller (+9.7pp instead of +8.5pp), because this kind of participation itself declined across eras — see the robustness section.
• PR conversation-thread comments don't count as engagement. GitHub has a third comment surface — plain comments on the PR thread ("issue comments") — and we store only their count, not their authors. So a reviewer who debated in the thread and then clicked approve is silent under the primary definition. The robustness section bounds how much this can matter.

Engaged review. Stricter than "not silent": at least one approver wrote something in the review body or left at least one inline comment.

Merged with zero human approval. A merged PR with no APPROVED review from any human other than the author. Counted only in team repos (≥ 3 distinct human authors).

Self-merged. The PR author pressed merge themselves, and nobody else approved. Machine merges (merge queues, auto-merge) are excluded from this metric's denominator.

Below recommended pace. Approval faster than 500 lines per hour — the SmartBear/Cisco inspection-rate threshold above which defect detection measurably degrades in the published inspection literature. This measures pace, not comprehension.

Faster than humanly readable. Approval faster than 300 lines per minute, derived from the ~300 words-per-minute ceiling for natural-language reading. Code is denser than prose, so treating one line as one word is generous to the reviewer. Above this line, arithmetic alone says the diff wasn't read.

Trustworthy review clock. The speed tiers need a defensible "reading started" moment. We use the latest review-request event before the approval; if nobody was explicitly asked, the draft-exit (ready-for-review) timestamp; PR creation time is a last resort that the headline tiers refuse (its records are excluded and reported separately — 15.2% of approved PRs). An earlier version of this page said "last substantive push" — that was wrong; commit push timestamps are not collected, and the clock never uses them.

The clock has a known false-positive path, and the "faster than readable" tier inherits it: when a reviewer reads round 1, comments, and is re-requested after the author's fix, the clock restarts at the re-request — so a quick, legitimate approval of a small fixup can register as faster-than-reading even though the total reading time was long. Arithmetic proves nothing was read after the final clock anchor; it cannot see reading that happened before it. This is why speed appears in the composite floor metric only in conjunction with other conditions, and why we don't hang era verdicts on speed tiers.

Bug-fix linkage (used only in the withdrawn finding). A "fix" is detected from branch-name conventions (fix/, hotfix/, bugfix/ and variants); it links to the most recent earlier PR touching one of the same files within 14 days, always within a single repo. 20,033 fix PRs detected, 68.6% linked to a plausible culprit. This is a noisy proxy, which is one reason the pooled defect effect deserved distrust — and got it.

What went in, what stayed out

Of 77,949 collected PRs, 51,065 were analyzed. Every excluded PR is counted under a named reason, and the ledger balances exactly (77,949 = 51,065 + 26,884):

This matters: including trivial PRs would inflate every rate in the study. It's the first thing a critical reader would attack, so it's the first thing we removed.

Bot detection uses GitHub account types plus the [bot] login convention. A bot registered as a normal user account counts as human — a known, small leak in the human direction.

Sampling

Where candidates come from (the part of "auditable" that matters most): candidate pools are built from the GitHub Search API over a fixed grid of 11 primary languages × 6 star bands (100 to 50,000 stars), filtered to non-fork, non-archived, non-mirror repositories, then shuffled with a seeded RNG and eligibility-checked until each cohort quota fills. The company cohort additionally draws from a recorded seed-org list. Three discovery runs (seeds 42, 43, 45) are logged in the collection manifest with their quotas and acceptance counts. What this frame generalizes to: active, public, multi-author GitHub repositories in mainstream languages with 100+ stars — not private repos, not the long tail below 100 stars, and nothing outside GitHub.

505 candidate repositories checked → 262 eligible → 261 collected. Eligibility: ≥ 30 merged PRs in the window and ≥ 3 distinct human authors, which removes abandoned and single-person projects. Every rejected candidate is recorded with its rejection reason, so both selection and rejection are auditable end to end.

Cohorts: 121 company-backed, 130 community, 11 AI-tooling (this cohort stalled below its 60-repo target because GitHub's code-search API withholds results for our token type; it doesn't affect the AI analysis, which uses co-author trailers measured on every repo — 162 have a measurable ratio). Sampling used seeded shuffles across three recorded runs.

Window: PRs merged from 2026-01-01, capped at 500 per repository, most recent first.

Known thin spots, stated up front: only 12 repos have 40+ authors and only 12 have under 5 — the two bands carrying the team-size contrast. The contrast is significant (rank-sum p = 0.005), but each tail is one unlucky repo away from moving several points. Our stated next step is stratified sampling of those bands — and the published rule we hold ourselves to: sample by segment, report by segment, never let a stratified sample produce a single global headline number.

The era comparison, precisely

The same 262 repositories were re-probed for Jan–Jun 2022 and Jan–Jun 2019 (placebo), with the identical eligibility bar applied per window: 118 repos qualified for 2022, 53 for 2019. The 2026 side of every era comparison is cut to Jan–Jun so all windows are six months.

On calling 2022 "before AI": we don't, anymore. Copilot had been in technical preview since mid-2021 and went GA in June 2022; ChatGPT arrived November 2022. Active public GitHub repos are exactly the early-adopter population, so the Jan–Jun 2022 arm is best described as before mass AI code generation — before chat assistants and coding agents made generated code a volume phenomenon — not before AI-assisted autocomplete existed. If early-2022 autocomplete already depressed review attention, our era contrast understates the true change.

Collection coverage. The main 2026 walk collects newest-first with a 500-PR cap per repository, which would under-cover the Jan–Jun sub-window for high-volume repos. So the 96 repositories that hit the cap were re-collected in full for Jan–Jun 2026 in a dedicated pass, and the era arm reads from that collection. No repository enters an era comparison with partial window coverage — a high-volume repo gets the same six months as a small one.

Pairing rules:

• A repository enters a pairwise comparison only with 30+ eligible PRs in both eras. 92 repos pair across 2022–2026, 43 across 2019–2022. 41 repos qualify in all three windows — the constant panel. Any trajectory across all three years is quoted from that panel only; putting the two pairwise tables side by side would silently compare different repo subsets.
• Every pairwise test is a two-sided sign test on per-repo deltas — no pooled averages, for the same Simpson's-paradox reason that killed our defect claim — with a bootstrap 95% CI on the median delta (2,000 seeded resamples).
• α = 0.05. Eleven metrics are era-tested in parallel; the Bonferroni-adjusted bar is 0.0045. The two headline p-values (0.0006 and 0.00005) clear both bars; no verdict in this study changes under either threshold.
• AI signals are not attached to historical windows: they're measured from a repo's current state and would be anachronistic in 2019/2022.

The era verdict is a difference-in-differences test — not a comparison of two test outcomes. An earlier version of this page defined "AI-era shift" as significant in 2022–2026 and not significant in 2019–2022. That rule commits a textbook error — the difference between "significant" and "not significant" is not itself significant (Gelman & Stern) — and since the placebo arm has fewer repos, it also smuggles in a power asymmetry. The corrected rule: for each constant-panel repo, compute (its 2022→2026 delta) − (its 2019→2022 delta) and test that one distribution of acceleration values. A metric earns "era shift" only if the acceleration test itself passes. Here is what that stricter test says about the two metrics the report leads with:

Levels are panel medians; deltas are medians of per-repo deltas, which is why rows don't subtract exactly. The two rows say very different things:

• Silent approvals rose sharply in the AI-era window (+8.4pp on the panel, +8.5pp across all 89 pairs, p = 0.0006) and did not detectably move in the placebo window (−0.8pp, p = 0.75). But the acceleration itself — the difference between the two windows — has a wide interval that includes zero: 41 repos, each measured in three windows, is too noisy to certify it separately. The claim we stand behind is therefore: the rise is real, and it is confined to the AI-era window; that the window boundary is the cause is supported by every point estimate and proven by none.
• Changes requested fell at the same per-repo pace in both windows (−1.4 vs −1.3pp; acceleration ≈ 0, p = 1.0). An earlier version of this study called it an AI-era shift. It is not — it is a pre-existing decline that continued, and the corrections log below records the downgrade.

Confounds we report side-by-side rather than "correct for":

• Machine merging barely existed pre-2022 (0% of PRs in 2019 → 9.9% in 2022 → 12.4% in 2026). It shifts the self-merge denominator across eras; correcting for it would mean inventing a counterfactual, so instead the numbers sit next to each other, labeled.
• The review clock differs in kind across eras. Draft PRs and review-request events were rarer in 2022 and marginal in 2019 (trustworthy-clock coverage: 69.2% → 85.3% → 84.4%), so speed-tier denominators cover a different share of approvals per era. Speed metrics are accordingly not given era verdicts beyond "no shift."
• Historical labels are 2026 labels. Stars, cohort, language, and team size are sampled once (2026) and reused for past windows — fine as identifiers, wrong as era-specific measurements, so we never use them that way.
• Survivorship — and it cuts deepest in the placebo arm. Only repos alive and eligible in 2026 have a past to compare, and a repo that already had 30+ PRs and 3+ authors in 2019 is by construction one of the oldest, most established projects in the sample. Teams that stay together grow mutual trust, and trusting teams review more quietly — an aging effect that is correlated with the time axis itself. We can't remove it; we can only say the placebo window (same aging, no jump) bounds how fast that mechanism moves on its own.

And the honest frame for the whole arm: this is a natural experiment, not a randomized trial. The rise in silent approvals sits in the 2022→2026 window on the same repositories, with a flat placebo — but at least three things changed in that window with matching timing: mass AI code generation (chat assistants from late 2022, coding agents after), the remote-to-office reversal (peak remote in early 2022 pushed review into writing; the return to offices moved some of it back to desks, where it leaves no trace), and the 2022–23 layoffs (fewer reviewers, same queue). We tested the mechanism our own report proposed — repos whose PR volume or diff size grew most should have grown most silent — and found nothing (Spearman ρ = −0.13, p = 0.23 across the 89 paired repos; see robustness below). So we no longer call AI the "dominant candidate": the era is the finding, and separating those three candidates needs a fourth time point (a 2024 arm: gradual growth favors the AI-volume story, a step at 2023 favors return-to-office), which is on the roadmap.

Robustness: the checks that could have killed the headline

Every check here re-runs the silent-approval headline under a hostile assumption. Computed from the stored records (analysis/robustness.mjs → data/robustness.json), deterministic, recomputable.

Definition variants. The primary definition is unchanged; each variant reclassifies a class of silent approvals as engaged and re-measures the 2022→2026 paired delta:

Three readings, one per row:

• Round-2 approvals are not inflating the jump — hiding them enlarges it. The share of silent approvals where the approver had engaged on an earlier round fell across eras (30.6% in 2019 → 27.4% in 2022 → 22.1% in 2026), so reclassifying them as engaged makes the era delta bigger, not smaller.
• Bots are not the jump. In 2026, 33.9% of silent-approval PRs carry a content-bearing bot review before the human approval (0.1% in 2022 — the CodeRabbit era didn't exist yet). If the +8.5pp were humans delegating reading to bots, removing bot-reviewed PRs should shrink it. It grows to +12.3pp: where no bot reviewed, silence rose even more.
• The thread-comment bound is the one variant that absorbs the delta — and it is loose by construction. We store thread-comment counts without authors, so this variant counts the PR author's own "fixed, please re-check", every bot's posted summary (review bots post thread comments too), and drive-by comments as reviewer engagement. Both author activity and bot thread posts grew sharply in the AI era, which is exactly the direction that erases the delta. The honest statement: under the primary definition the jump is +8.5pp; under the loosest possible reclassification it is 0; the truth lies between, and pinning it needs comment authors — a planned re-collection, listed in the roadmap.

What's being reviewed also changed. Two descriptive trajectories on the constant panel, published because the report's load story depends on them (paired deltas use the same pairing rules as every other metric):

Typical PR size is the era-shaped one: near-flat levels through the placebo window, then a large, near-unanimous paired increase after 2022. PR count grew across the whole span, but total lines are heavy-tail dominated — a few giant PRs carry most of a window's volume, which is why count × median-size is not a valid volume estimate, and why total LOC could fall 2019→2022 while counts rose (2019's totals leaned on occasional monster PRs). Headcount grew too, which is why the load claim is stated per person: lines per author fell through the placebo window (hiring outpaced code) and rose about 40% in the AI window, back to 2019 levels — consistent with +69% total code against +6% authors 2022→2026. On the review side the bench grew faster (13 → 17 approvers), holding lines per average approver flat across the AI window — but the load concentrated: the busiest approver's line-volume grew ×1.47 in the paired median (Wilcoxon on log-ratios p = 0.032; the direction-only sign test sits at 0.056, and we report both), after a ×1.02 placebo window. Windows with zero approvers drop out of pairs for approver metrics, mirroring null-rate handling elsewhere. So the silent-approval jump is not explained by average per-reviewer line volume (consistent with the null dose-response test above); what grew is the load on the person already carrying the repo. Authors counted over all collected human PRs in the window; "frequent" means 3+ merged PRs in the half-year; approvers over eligible PRs. All reported as description, not as era verdicts — no diff-in-differences label is claimed for any row.

Threshold sensitivity. The floors and ceilings are judgment calls, so here is the headline at the neighbors:

The headline moves about a point per doubling of the trivial floor and never approaches a different conclusion. Two thresholds remain reported-but-untested: the 30-PR eligibility bar and the 14-day fix-linkage window (the latter only ever fed the withdrawn defect finding).

What we store (and refuse to)

Metadata only. Timestamps, line counts, file paths, labels, review states, reviewer logins, review body lengths, comment counts. No code, no diffs, no file contents, no PR titles, no review text, no commit messages are stored. Titles and review bodies cross the wire because GraphQL has no server-side length function — they're reduced to integers in memory and never written to disk. Verified mechanically: zero text-field occurrences across all 77,949 records.

One storage decision with an analytical cost, named plainly: PR conversation-thread comments are stored as a count only — no authors, no timestamps. That's why the thread-comment robustness variant above is an upper bound instead of an exact number, and why collecting comment authorship (still no text) is the first item on the re-collection roadmap, alongside closed-but-unmerged PRs (the sharpest review outcome — rejection — never merges and so never enters this sample) and a 2024 era arm.

The free analyzer runs under the same rule on your repo.

Reproducibility

The pipeline is three commands: discover, collect, analyze (the era arm adds recheck, backfill, compare; the robustness checks are a fourth offline pass). The analyzer, the era comparison, and the robustness suite are pure functions of the collected data — re-running them produces byte-identical output (bootstrap intervals included: resampling is seeded), and metric definitions can be revised and re-tested without re-fetching anything from GitHub. That property is what made this page's own corrections cheap to run. The analysis inputs (report.json, comparison.json, robustness.json) are the artifacts every number traces back to.

2026 data collected 2026-08-15; era data collected 2026-08-19/20.

Corrections log

We keep this section deliberately public. Four corrections so far — two findings that did not survive scale, one claim that did not survive a better test, and one testing rule we replaced:

1. "Unreviewed PRs are 2× more likely to need fixing" — a between-repo artifact (Simpson's paradox). Within repos: sign test p = 0.78. Withdrawn.
2. "Heavy-AI repos rubber-stamp less" — an artifact of unequal band sizes. True correlation: r ≈ 0.05. Replaced with the null.
3. "Pushback erosion is an AI-era shift" — wrong, and our own numbers said so. Changes-requested fell −1.4pp/window before the AI era and −1.3pp/window during it (diff-in-diff p = 1.0): the same decline, continuing. An early version of the report called this erosion "steep and unambiguous after 2022"; that sentence is withdrawn. What remains true: pushback keeps falling (3.4% of merged PRs in 2026) and the AI era did nothing to slow it.
4. The era-labeling rule itself. We originally labeled a metric an "AI-era shift" when it was significant in 2022–2026 and not in 2019–2022 — the significant-vs-nonsignificant fallacy (Gelman & Stern). The rule is now a paired difference-in-differences test on the constant panel. Under the stricter rule the silent-approval claim is stated as an era-window association with an unproven acceleration (details in the era section), and the pushback claim downgraded per entry 3.

The first two had tight confidence intervals at the smaller sample. That's the standing warning for this entire class of analysis — pooled rates over heterogeneous repositories produce confident-looking numbers that don't replicate — and it's why every claim was checked within repositories, and now within the same repos across eras, before publishing.

If you recompute something and get a different answer, we want to know.