An AI code review benchmark on verified bugs

crbench runs every reviewer on the same 127 cases: 91 real pull requests from Sentry, Grafana, Keycloak, Cal.com, Kubernetes and more, plus 36 built to test specific bug classes, across 8 languages. A bug only counts when something proves it.

The results

Reviewer Overall Real bugs caught Comments valid
Drape Deep Review 70% 89.1% 99.1%
Drape Review 61% 81.8% 98.9%
Cursor Bugbot 54% 74.5% 98.8%
Claude Code (Opus 5.5) 53% 79.3% 92.6%
Greptile 39% 62.8% 95.6%
Cubic 38% 62.4% 90.6%
Qodo 37% 56.1% 94.6%

crbench leaderboard, November 2026. Overall weighs bugs caught by severity and charges for false alarms. This table lists the reviewers you can buy; the full leaderboard also has research runs.

Big pull requests and cross-file bugs separate reviewers

80%
of bugs Deep Review caught on pull requests over 500 lines. The other reviewers on the same pull requests: 57%
87%
of cross-file bugs Deep Review caught. The next best commercial reviewer: 78%

Most AI reviewers look good on small pull requests. On large ones, and on bugs whose cause sits in a file the diff doesn’t show, they miss real bugs. See how Drape reviews a pull request →

How crbench is built

Real pull requests — and a bug only counts when something proves it.

crbench starts from real pull requests, many of them collected by Martian’s Code Review Bench — we’re grateful for the work that went into it. On top of those, crbench sets a higher bar for what counts as a bug.

Every defect needs proof: a fix or revert that landed after the change merged, independent corroboration, or an audit. Any valid finding earns credit, not only the ones on an answer key, so a reviewer that finds a real bug nobody listed isn’t scored as noise.

Every label we dropped is published with its reason, and anyone can submit a reviewer and re-run the scoring.

Read the methodology →

What doesn’t count as a bug

  • A call that compiles. If the merged PR builds, “won’t compile” isn’t a defect.
  • A change the language allows. If tsc --strict passes, it isn’t an interface break.
  • An import the same PR defines. The class exists, so the import resolves.
  • A null-safe comparison. Comparing a missing value with != can’t throw.
  • A style preference. “Consider extracting a constant” is advice, not a bug.

Catch the bug before it merges.

14-day free trial. No credit card required.