Parenthesized-group re-lexing

Measurement record for issue #808 / ADR 0079. Point-in-time snapshot — not maintained.

Executive Summary

• The arrow-vs-parenthesized probe (IsArrowFunction) and two sibling probes truncated their look-ahead tokens on rollback, so the real parse re-lexed every (...) group it scanned: O(n²) on nested groups, ~2× constant on arrow/callback-dense code.
• Keeping the probe's tokens unless one is goal-sensitive (/ or template-tail }) eliminates the re-lexing for the common case.
• Nested-group lexing drops from O(n²) to linear: ≈287× faster at nesting depth 1600. Arrow-dense lex+parse is ≈2.1× faster on an 8k-declaration module — above the noise floor.
• The first attempt (goal-tagged re-derivation, as the issue proposed) was measured broken, not slow: it re-classified correct division / as regex and failed to parse the Date shim. The shipped design never re-derives an already-scanned token.
• Residual super-linear parse time on pathological nesting is the parser re-reading cached tokens (no character scanning); it is out of scope.

Method

All numbers are GocciaScriptLoader --output=json phase timing (the same vehicle as ADR 0037), median of 7 runs, macOS darwin/aarch64, FPC 3.2.2, development build. Two synthetic corpora:

• nested — one top-level expression const _ = ((( … 0 … ))); at parenthesis depth D. Each ( triggers a fresh IsArrowFunction probe over the whole inner span.
• arrow-dense — N declarations of const fK = (a, b, c) => (a + b) * (c - a);. Linear in N; every line has a parenthesized parameter list and parenthesized sub-expressions.

The benchmark runner is execution-only (it times the run callback, not parsing) and the dynamic Function constructor that would let a benchmark re-parse is disabled by default for sandbox safety, so a benchmark cannot gate parse cost — the loader's parse-phase JSON is the measurement vehicle, and parse-behaviour regressions are gated by the JavaScript regression tests under tests/language/.

Results

Nested groups — lexing goes from O(n²) (≈4× per depth doubling) to linear (≈2×):

Parse time stays super-linear after the fix because the parser still re-reads cached tokens once per enclosing probe — array reads, no lexing — so total front-end time at depth 1600 still falls from 292 ms to 32 ms (≈9×).

Arrow-dense — linear before and after, with a roughly halved lexing constant:

≈2.1× faster lex+parse at 8k declarations, consistent across sizes and well above the ±11–15% parser-benchmark noise floor.

What did not work — goal-tagged re-derivation

The issue proposed tagging each cached token with its scan goal and re-deriving it whenever the parser later requested a different goal. Implemented, it failed at engine start-up: the Date shim's Math.floor((year - 1969) / 4) raised “Unterminated regular expression literal.” The parser peeks the operator / under its default InputElementRegExp goal and consumes it under InputElementDiv; re-deriving on that benign difference re-lexed a correct division / as a runaway regex. The lesson: an already-scanned token is authoritative and must never be re-classified — the only safe choice is whether a probe's tokens are kept at all, decided once from whether any of them is goal-sensitive.