0178 - AWFY is timed warm, engines alternate their order, and interpreter-class comparators run beside QuickJS
Date: 2026-11-05
Area: benchmarking, ci, website
Issue: #2066
Supersedes: in part ADR 0087 and ADR 0091
Related: ADR 0076, refines ADR 0176, #2102
Context
ADR 0176 judges the first performance target on the AWFY and JetStream QuickJS ratios of the Performance Barometer, and leaves two questions of the measurement protocol to #2066: how AWFY is timed, and in what order the engines run.
Under driver version 1 every AWFY sample timed one cold call of the benchmark's
innerBenchmarkLoop in a fresh process (two for CD). Several rows were below
ADR 0087's 0.5 ms floor: in CI run 41824992451 Mandelbrot took 0.02 ms in
Goccia and 0.01 ms in QuickJS. A cold sample mostly measures startup for the
short benchmarks, and JIT-enabled Node.js before its JIT has warmed up, so its
Node.js column was slower than Goccia on three rows. Both lanes also ran
Goccia before QuickJS in every repetition, so any effect of running first or
second fell on one engine. The only comparators were QuickJS and JIT-enabled
Node.js, which left the lane with no second interpreter-class point.
Decision
AWFY is timed warm (driver version 2). Each process creates one benchmark
instance and runs warmupIterations iterations, each timed on its own, then
measuredIterations iterations timed as one window. An iteration is one
innerBenchmarkLoop(innerIterations) call, verified as in upstream AWFY's
harness. A sample's headline durationMicros is the window divided by
measuredIterations: the steady-state time of one iteration. Because the
window has two clock reads however many iterations it holds, a short iteration
adds no timer cost, and the 0.5 ms floor applies to the window, not to one
iteration. The first warm-up iteration is kept as firstIterationMicros, the
cold figure, and each engine's stats carry its median as
medianFirstIterationMicros; no ratio uses it. The headline over processes
is the median of the samples, six in CI.
The counts live per benchmark under warmProtocol in perf/awfy/manifest.json.
They keep each benchmark's problem size (innerIterations 1, CD 2), five
warm-up iterations, and enough measured iterations for QuickJS's window to
last about 50 ms or more on a 2022 laptop core (100 times the floor). Havlak,
whose single iteration takes over 20 s in Goccia, runs one warm-up and one
measured iteration. The problem sizes are unchanged because Mandelbrot,
NBody and Havlak verify only at the sizes upstream lists, and the next
listed sizes (Mandelbrot 500, NBody 250000) cost Goccia 1.2 s and 8.3 s per
iteration. A change of any count changes the barometer's protocol key and
breaks the trend line.
Engines alternate their order (both lanes, driver version 2). The first
round runs the --engines order, and every next round runs the reverse of the
one before (unrecorded --warmup rounds count as rounds; CI runs none), so every pair of engines swaps places from one repetition to the
next. The order is deterministic, so a report can be rerun in the same order,
and over an even number of repetitions each engine of a pair runs first
equally often. CI runs six repetitions in both lanes, so the --engines order
and its reverse run three times each; with an odd number of engines the middle
one keeps its place, but still swaps order with every other engine. Random order was rejected: it balances only in expectation and
needs a recorded seed to be reproducible. Every sample records engine,
repetition and position (its place in that repetition's order), each
target records executionOrder, and the report's metadata names the order
mode and the base order. Target-1 evidence cites the order from those fields.
Interpreter-class comparators run beside QuickJS.
node-jitlessis Node.js with--jitless, V8's Ignition interpreter with no JIT tier, from the same Node.js binary asnode. It runs by default in both drivers and in CI, and the barometer publishes its ratio.jscis the JavaScriptCore shell with--useJIT=false(the same option asJSC_useJIT=false), which leaves the LLInt interpreter. CI runs it in both lanes: every AWFY and JetStream job onubuntu-latestinstalls the shell from the Ubuntu archive'slibjavascriptcoregtk-binpackage (.github/scripts/install-jsc.sh). The archive drops a package version once an update supersedes it, so the version is recorded rather than pinned: the driver writes the package and version into the report's engine metadata, the CI reports fail when a report that measured JSC records none, and the barometer breaks the trend line when it changes. The drivers' defaults leavejscout, because a local machine may have no shell; each manifest'sciReport.enginesadds it for CI, and a local run opts in with--engines ...,jsc --jsc <path>.- The JIT flags are arguments, not environment variables, so every sample's recorded command shows them.
Goccia gets a separate ratio against each comparator: geomeanRatios has one
pair per engine, and the PR comments and the barometer show a column per
comparator. QuickJS stays the only comparator target 1 counts. The
interpreter-class engines are context, and JIT-enabled Node.js stays on the
barometer as an outside reference.
Consequences
- Driver version 2 and the protocol key break the AWFY and JetStream trend lines, as ADR 0091 requires for a measurement change. A version 1 result does not count toward target 1 under version 2.
- The warm figures differ from the cold ones by benchmark. On one machine
(Ryzen 9 6900HX, QuickJS 2026-06-04), the QuickJS geomean ratio was 4.78
cold and 5.27 warm;
NBodywent from 5.83 to 9.18 andJsonfrom 6.92 to 8.85, because the cold rows had mostly measured startup. - The AWFY job takes longer. On the same machine the CI report took 292 s
with four engines, against 139 s for the cold protocol with three, most of
it
Havlak's two Goccia iterations per process. The PR lane runs two Goccia binaries, so its share ofHavlakdoubles. - CI runs six repetitions instead of five, and JSC joins
node-jitlessas a new engine, so the AWFY lane takes six samples per target and engine where it took five, from five engines (six in the PR lane), and the JetStream lane goes from three engines (four in the PR lane) to five (six). Both new engines are interpreters, but each costs far less than Goccia, whose samples dominate both lanes. Every AWFY and JetStream job also installs JSC. - Per-platform measurement and JavaScriptCore on macOS runners stay with #2102.
- What stays: ADR 0076's same-runner rule, ADR 0087's target, then repetition, then engine sampling and its raw samples, verification and metadata, and ADR 0091's lanes, frozen JetStream subset, ratio convention and segmentation. The JetStream lane's in-process protocol (three iterations with First/Worst/Average sub-scores) is unchanged.