Same-runner benchmark comparison

Date: 2026-06-27 Area: ci / tooling Issue: #815 Pull Request: #886

The PR "Benchmark Results" comment previously classified benchmarks as 🟢 improved / 🔴 regressed by comparing the PR's freshly-measured run against a cached baseline measured in a different job, on a different GitHub-hosted runner, at a different time (whatever main last pushed). The baseline was written by ci.yml's benchmark job via actions/cache/save and restored in pr.yml's benchmark-comment job by restore-keys prefix. Cross-runner/cross-time variance is systematically larger than each run's own min/max spread, so the range-overlap classifier (improved iff PR min > baseline max) flipped large numbers of benchmarks to improved/regressed even when the diff could not possibly affect them — e.g. PR #805 (regex-decode only) reported arraybuffer.js and arrays.js, which use no regex, as broadly improved. The verdicts were confident but not actionable, and a real regression was indistinguishable from runner noise.

We now build the PR's base commit (github.event.pull_request.base.sha — the main the PR targets) in a dedicated build-main job, and the benchmark job benchmarks that main build and the PR build back-to-back on the same runner, after a discarded warm-up run that sheds runner cold-start (page faults, sub-turbo clocks). The range-overlap classifier is unchanged; it now compares two runs measured under the same conditions, so the systematic between-runner offset cancels and only within-run noise remains — which is exactly what range overlap is designed to absorb. The deterministic profile diff (opcode/allocation counts) is regenerated from the same main build instead of a cached profile, making the whole comparison self-contained; the cross-runner benchmark-*-baseline-* and profile-baseline-* caches in ci.yml and their restores in pr.yml are removed. The ~200-line comparison previously inlined in workflow YAML moved to scripts/benchmark-compare.js with unit tests in scripts/test-benchmark-compare.ts, so the classifier semantics — including the #815 false-positive case — are locked by a test rather than embedded in pr.yml.

The explicit trade-off is cost: building and benchmarking main inside every PR run roughly doubles the benchmark job's CI time (one extra --prod build of the benchmark runner plus a second benchmark pass per mode). That is the accepted price of a trustworthy comparison. The comparison stays advisory (comment-only), not a merge-blocking gate: even same-runner measurement has a noise floor (re-running the identical --prod binary showed ±11–15%, up to ±59%), VISION.md (line 43) deprioritizes raw throughput, and test262 remains the gating regression check. The range-overlap rule absorbs that floor as "unchanged overlap," and the comment now reports the measured median per-run variance so readers can judge whether a sub-noise delta is meaningful.

Alternatives considered and rejected: normalizing the cached cross-runner baseline to an in-run calibration reference — cheaper, but a single calibration benchmark cannot cancel per-workload-class variance (memory-bound vs CPU-bound runners scale differently), so false verdicts shrink rather than disappear; a consistent self-hosted runner — removes cross-runner variance structurally but adds infrastructure burden and runs untrusted PR code on owned hardware in a public repo; and demoting timing entirely in favour of the already-trustworthy deterministic profile diff — honest and cheap, but the deterministic diff only catches regressions that change instruction or allocation counts, not a pure wall-clock-per-op slowdown, which is part of what the gate exists to surface. Per-file interleaving of the two binaries (the issue's "ideally interleaved") is a possible future refinement if residual same-runner drift proves material; the reported noise floor makes such drift visible. See docs/build-system.md benchmark job reference and VISION.md.