BenchReceipt

Who it's for

Maintainers and staff engineers at small teams and OSS projects now receiving agent-generated performance PRs (the wave kicked off by auto-research loops like today's 232x kernel post); today they cope by manually re-running hyperfine and eyeballing diffs, or trusting the PR description.

The problem

Time: honestly verifying one claimed speedup takes 3-6 hours of harness work (fresh inputs, variance control, equivalence checks) per PR, and with several agent perf PRs a week that is a lost workday; a gamed benchmark that slips through costs a production incident plus rollback.

How to build it

CLI + GitHub Action: runs both implementations Docker-isolated, emits a signed receipt (JSON + public HTML page + README badge), posts the verdict as a PR check; specialization scan via tree-sitter for C/C++/Rust/Go/Python.

How it makes money

OSS authors free (drives the badge network); teams pay ~$99/mo for the CI gate, private receipts, and a per-PR claim ledger, because verification runs burn compute on every PR and a 'we verified before merging' audit trail matters in incident reviews — hyperfine is free but does no equivalence fuzzing, no specialization scan, and no signed artifact.

Why it doesn't exist yet

Incumbents (CodSpeed, Bencher, hyperfine, CI perf gates) assume honest benchmarks — they catch regressions, not gaming, and none do differential output fuzzing or input-specialization detection; GitHub's review AI comments on diffs rather than empirically reproducing claims. Indie gap: a small trusted signed artifact ('show me the receipt') that becomes the social standard on optimization posts — too niche for GitHub to prioritize, perfectly sized for one builder.

First users

Authors of viral HN speedup posts who want pre-emptive credibility before the debunk thread arrives, plus maintainers of performance-critical OSS (image, parsing, DB libraries) already receiving waves of agent optimization PRs — the badge on the README is the viral loop.

Build size

1 person x 8 weeks. Included: Docker-isolated benchmark runner with fresh-input protocol, differential fuzzer with counterexample dump, tree-sitter specialization scanner, signed receipt generator, GH Action wrapper. Excluded: GPU/CUDA timing support, hardware perf-counter analysis, any web app beyond hosted receipt pages.

Biggest risk

GitHub/Copilot or CodSpeed ships native benchmark verification in CI, making a standalone tool redundant; secondarily, the agent-perf-PR wave could recede if model labs stop pushing optimization claims.

Conditions for a hit (all 3 required)

• Given a repo with a benchmark entrypoint and two implementations (base vs candidate), produces a signed receipt (JSON + public URL) stating median speedup across at least 30 runs with randomized fresh inputs, machine spec, and variance.
• Auto-generates at least 1,000 inputs outside the benchmark's original input set and reports any output mismatch (stdout/exit-code/hash) between the two implementations, with counterexample files attached to the receipt.
• Static scan of the candidate implementation flags any code specialized to the benchmark's test data (literal test values, input-size branches) with file:line citations in the receipt.

How it's judged (in 6 months)

GitHub repo matching the 3 features with >=1,000 stars, or a Product Hunt daily top-5 launch of such a tool(judgment date 2027-02-16)

AI self-confidence 45/100 — self-reported likelihood of meeting the criterion, not a business success rate