Testing

For contributors writing, running, or debugging tests.

Executive Summary

• Three testing layers — JavaScript end-to-end tests (primary), CLI behavior tests (secondary), Pascal unit tests (tertiary)
• Valid code → tests/; rejected code → scripts/test-cli-*.ts — JavaScript tests assert valid code runs and returns the right result; parser/lexer rejection (SyntaxError, caret, error envelope) belongs in scripts/test-cli-parser.ts / test-cli-lexer.ts, because eval is not installed and the Function constructor is off by default (--unsafe-function-constructor), so a parse error cannot be asserted from inside a JS test
• Built-in test framework — describe/test/expect with async support, mock functions, lifecycle hooks, and Vitest-compatible matchers
• One method per file — Each test file focuses on a single method; edge cases are co-located with happy-path tests
• Cover the contract, not only the example — Include boundaries, invalid inputs and receivers, coercion/order, state transitions, descriptors, and both execution modes when those perspectives apply
• Structure is checked in CI — Run bun run scripts/check-test-structure.ts before submitting test-suite organization changes
• Run with: ./build.pas testrunner, ./build/GocciaTestRunner -P tests, and ./build/GocciaTestRunner -P tests --mode=bytecode (see Testing requirements for -P and the one-time --trust alternative)

GocciaScript uses three testing layers in priority order:

1. JavaScript end-to-end tests (primary) -- .js tests in tests/ that exercise the full pipeline through the same public surface that users call. CI runs the full suite in both interpreter mode and bytecode mode. Every new feature or bug fix should include tests at this layer.
2. CLI behavior tests (CI integration) -- Standalone bun scripts under scripts/test-cli-*.ts (test-cli.ts, test-cli-lexer.ts, test-cli-parser.ts, test-cli-config.ts, test-cli-apps.ts, test-cli-permissions.ts, test-cli-embedded-resources.ts, test-cli-intl-taint.ts, test-cli-differential.ts, test-cli-coverage-parity.ts) that the PR and main workflows run via bun run in the cli job (test-cli-coverage-parity.ts on main only). They invoke GocciaRunner, GocciaTestRunner, and GocciaBenchmarkRunner as subprocesses and assert on exit codes, output structure, and error envelopes — above all parser/lexer rejection that a JS test cannot express (malformed source must fail with a SyntaxError, caret, suggestion, and JSON error envelope, in both modes), plus JSON output structure, coverage CLI, source maps, numeric separator rejection, timeout handling, global injection, and config loading. The matchers these scripts share live in scripts/test-cli/assertions.ts and are themselves unit-tested by scripts/test-cli-assertions.ts, which spawns no binaries — a defect in a shared matcher silently weakens every harness that uses it, so its contract is locked by its own test.
3. Pascal unit tests (tertiary) -- Native *.Test.pas coverage for low-level runtime and value system internals that are not reachable through a stable public API.

When choosing where to add coverage, prefer the most public entry point — and match the kind of check to the layer that can actually express it:

A JavaScript test that merely parses and runs a construct — e.g. (a / b) evaluating to 1, or (/x/).test(...) returning true — is a positive execution/conformance test. It confirms the construct works, but it does not prove that a different, malformed construct is rejected, and it is not the lexer/parser gate. When a change is about what the parser or lexer accepts or rejects (disambiguation, early errors, token classification), put the rejection cases in scripts/test-cli-parser.ts / test-cli-lexer.ts.

Do not use dynamic source execution (Function(...), eval(...), or equivalent runtime compilation) inside the JavaScript suite merely to smuggle parser failures into a runtime test. Keep dynamic-code assertions in tests/ only when the dynamic-code API itself is the feature under test, such as Function constructor behavior. Direct eval is host-gated, so its private GocciaTest262Runner coverage belongs in scripts/test-cli-apps.ts, outside the JavaScript suite.

Avoid tests that lock onto private helper functions or transient implementation structure when the same behavior can be validated through a documented user-facing command or script entry point. Where native Pascal tests are still appropriate, keep them as stateless and repeatable as possible so they behave like pure input/output checks rather than process-sensitive probes.

Keep suite titles, test names, and failure messages aligned with the layer under test. End-to-end JavaScript and CLI tests should use JavaScript/runtime terminology and describe behavior at that layer, while lower-level Pascal tests may use implementation terms when that is the behavior being exercised.

Test Organization

tests/
├── built-ins/              # Built-in object tests
│   ├── Array/              # Array constructor and prototype methods
│   │   ├── constructor.js
│   │   ├── from.js         # Array.from
│   │   ├── of.js           # Array.of
│   │   └── prototype/
│   │       ├── map.js, filter.js, reduce.js, forEach.js
│   │       ├── find.js, findIndex.js, indexOf.js, lastIndexOf.js
│   │       ├── sort.js, splice.js, shift.js, unshift.js
│   │       ├── fill.js, at.js, includes.js, concat.js, reverse.js
│   │       └── ...
│   ├── ArrayBuffer/        # ArrayBuffer constructor, static/prototype methods
│   │   ├── constructor.js, isView.js, toString-tag.js
│   │   └── prototype/
│   │       └── slice.js
│   ├── Error/
│   ├── JSON/
│   ├── Map/
│   ├── Math/
│   ├── Number/             # Number methods and constants
│   │   ├── parseInt.js, parseFloat.js, isNaN.js, isFinite.js, isInteger.js
│   │   ├── isSafeInteger.js, epsilon.js, maxSafeInteger.js
│   │   └── maxValue.js, minValue.js, nan.js
│   ├── Object/             # Object static and prototype methods
│   │   ├── keys.js, values.js, entries.js, assign.js, create.js, is.js
│   │   ├── defineProperty.js, defineProperties.js, getOwnPropertyDescriptor.js
│   │   ├── freeze.js, isFrozen.js
│   │   ├── getPrototypeOf.js, setPrototypeOf.js
│   │   ├── fromEntries.js, groupBy.js
│   │   ├── prototype/      # Object.prototype instance methods
│   │   │   ├── toString.js, hasOwnProperty.js, isPrototypeOf.js
│   │   │   ├── propertyIsEnumerable.js, toLocaleString.js, valueOf.js
│   │   └── ...
│   ├── Promise/             # Promise constructor, static methods, microtask ordering
│   │   ├── constructor.js, resolve.js, reject.js
│   │   ├── all.js, all-settled.js, race.js, any.js
│   │   ├── microtask-ordering.js, thenable-adoption.js
│   │   └── prototype/
│   │       ├── then.js, catch.js, finally.js
│   ├── Set/
│   ├── SharedArrayBuffer/  # SharedArrayBuffer constructor, prototype methods
│   │   ├── constructor.js, toString-tag.js
│   │   └── prototype/
│   │       └── slice.js
│   ├── String/
│   │   └── prototype/
│   ├── structuredClone/    # structuredClone tests (primitives, objects, collections, errors, arraybuffer)
│   ├── Symbol/
│   └── TypedArray/         # TypedArray constructors, prototype methods
│       ├── constructors.js, element-access.js, buffer-sharing.js
│       ├── from.js, of.js  # Static methods at top level
│       └── prototype/      # Instance methods (one file per method, edge cases included)
│           ├── fill.js, map.js, filter.js, reduce.js, reduceRight.js
│           ├── indexOf.js, lastIndexOf.js, includes.js
│           ├── find.js, findIndex.js, findLast.js, findLastIndex.js
│           ├── sort.js, reverse.js, slice.js, subarray.js
│           ├── set.js, copyWithin.js, at.js, with.js
│           ├── every.js, some.js, forEach.js, join.js
│           ├── toReversed.js, toSorted.js
│           └── values.js, keys.js, entries.js
│
└── language/               # Core language feature tests
    ├── classes/            # Class declarations, inheritance, private fields/methods/getters/setters
    ├── declarations/       # let, const
    ├── decorators/         # TC39 Stage 2.7 decorators and decorator metadata
    ├── expressions/        # Arithmetic, comparison, logical, destructuring, trailing commas, etc.
    │   ├── addition/       # Addition with ToPrimitive
    │   ├── arithmetic/     # Division (IEEE-754 signed zeros, Infinity), exponentiation (Infinity edge cases)
    │   ├── bitwise/        # Bitwise OR, AND, XOR, NOT, shifts
    │   ├── modulo/         # Floating-point modulo
    │   ├── conditional/    # Ternary precedence
    │   ├── optional-chaining/  # Optional chaining (?.) edge cases
    │   ├── trailing-commas.js  # Trailing commas in calls, parameters, constructors
    │   └── ...
    ├── functions/          # Arrow functions, closures, higher-order, recursion
    │   └── function-length-name.js  # Function.length and Function.name
    ├── identifiers/        # Unicode support, reserved words
    ├── modules/            # Import/export
    ├── objects/            # Object literals, methods, computed properties
    ├── statements/         # if/else, switch/case/break, try/catch/finally
    │   ├── try-catch/      # Try-catch-finally edge cases
    │   └── unsupported-features/    # Warning-mode recovery tests for disabled compatibility syntax
    └── unary-operators.js

Constructor requirements belong with the constructor they constrain. For example, the Map()-without-new rejection belongs in tests/built-ins/Map/constructor.js, not in a shared tests/built-ins/constructors/require-new.js file.

File Naming and Layout Conventions

Follow these rules when creating or organizing test files:

1. One method per file — Each built-in method or static function gets its own test file. Never bundle multiple methods into a single file like prototype-methods.js or static-methods.js.

# Correct — each method is its own file
tests/built-ins/TypedArray/prototype/fill.js
tests/built-ins/TypedArray/prototype/map.js
tests/built-ins/TypedArray/prototype/indexOf.js
​
# Wrong — multiple methods in one file
tests/built-ins/TypedArray/prototype-methods.js

2. Prototype methods go in a prototype/ subfolder — Instance methods (e.g., Array.prototype.map) live in BuiltIn/prototype/methodName.js. Static methods and constructor tests live directly in the BuiltIn/ folder (no separate static/ subfolder).

tests/built-ins/TypedArray/
├── constructors.js              # new TypedArray(...) constructor variants
├── buffer-sharing.js            # Buffer sharing behavior across views
├── from.js, of.js               # Static methods at top level
└── prototype/                   # Instance methods
    ├── at.js
    ├── fill.js
    ├── indexOf.js
    ├── map.js
    └── ...

3. Edge cases are co-located, not separate — Edge cases (NaN handling, Infinity, negative indices, empty arrays, clamping, etc.) belong in the same file as the happy-path tests for that method. Do not create standalone edge-cases.js files — they become disconnected from the feature they test and make it unclear which method the edge case validates.

// tests/built-ins/TypedArray/prototype/fill.js — CORRECT
describe("TypedArray.prototype.fill", () => {
  test("fills entire array", () => { ... });
  test("fills with start and end", () => { ... });
​
  // Edge cases are part of the same file
  test("negative start index counts from end", () => { ... });
  test("NaN fill value becomes 0 for integer types", () => { ... });
  test("Infinity clamps to 255 for Uint8ClampedArray", () => { ... });
});

# Wrong — edge cases in a separate catch-all file
tests/built-ins/TypedArray/edge-cases.js

4. Object prototype methods use prototype/ too — Object.prototype.toString, Object.prototype.hasOwnProperty, etc. follow the same convention as Array. Use Object/prototype/toString.js, not Object/prototype-toString.js.

tests/built-ins/Object/
├── keys.js                      # Object.keys (static method)
├── assign.js                    # Object.assign (static method)
├── prototype/                   # Instance methods on Object.prototype
│   ├── toString.js
│   ├── hasOwnProperty.js
│   ├── isPrototypeOf.js
│   ├── propertyIsEnumerable.js
│   ├── toLocaleString.js
│   └── valueOf.js

5. Test titles should match the test layer — describe(...) and test(...) strings should use the vocabulary of the layer they exercise. End-to-end JavaScript tests should avoid leaking Pascal implementation terms; lower-level Pascal tests do not need that restriction.

// Correct — end-to-end JavaScript test uses JavaScript/runtime terminology
test("JSON.stringify preserves round-trip precision for large fractional floating-point numbers", () => {
  // ...
});
​
// Wrong — end-to-end JavaScript test leaks Pascal implementation terms
test("JSON.stringify preserves round-trip precision for large fractional doubles", () => {
  // ...
});

6. Cross-surface files are narrow exceptions — A file may span several methods only when the behavior itself is one indivisible cross-surface contract, not merely because the methods are related or share an implementation. Examples include a proposal-version absence contract such as Temporal/removed-methods.js, or a constructor-family invariant such as Error/error-prototype-constructor.js. A shared receiver-branding matrix such as Date/prototype/receiver-branding.js is also appropriate when it verifies the same internal-slot requirement across methods without duplicating that matrix in every method file.

These exceptions must be:

• named after the single contract rather than methods.js, misc.js, or error-cases.js;
• explicitly allowlisted in scripts/check-test-structure.ts with a concrete reason;
• kept small enough that a failure still identifies the contract that broke.

Shared setup or table data is not by itself a reason to combine operations. Prefer a helper imported by focused files when several operations need the same fixtures.

Coverage Perspectives

A passing happy-path example proves only that one route works. For every method or operation, inspect its public contract and implementation branches, then add the perspectives that can change its observable result:

• ordinary behavior, including more than one representative input where the operation has distinct modes;
• boundaries and sentinels such as empty input, zero and signed zero, NaN, infinities, minimum/maximum values, and first/last valid indexes;
• invalid arguments and incompatible receivers, including the exact error type;
• argument coercion, throwing coercions, and observable evaluation order;
• repeated calls and state transitions for mutable or stateful APIs;
• property metadata and descriptors for exposed built-in properties;
• independent expected values for encoders/decoders and getters/setters rather than round trips that can let two matching defects cancel out;
• interpreter and bytecode parity, which the full JavaScript suite gate verifies.

Not every bullet applies to every operation. Each added test must protect an observable contract or a meaningful implementation branch; do not add combinatorial duplicates merely to satisfy a checklist.

Structural Check

Run the test-suite structure check after adding or reorganizing JavaScript tests:

bun run scripts/check-test-structure.ts

The check rejects catch-all filenames, known multi-operation files, reliably detectable prototype-method suites outside prototype/, and parser-rejection assertions hidden behind Function(...) or eval(...). Dynamic-code rejection tests are allowed only where the dynamic-code API itself is under test.

Cross-Platform Newline Rules for Data Format Parsers

When a parser implements a format with spec-defined newline semantics, test the format semantics directly instead of inheriting the host OS newline convention.

• Do not treat LineEnding / sLineBreak as the expected runtime result for parsed data just because the test is running on Windows.
• Prefer explicit \r\n or #13#10 fixtures when adding regression tests for multiline parsing, folding, or block-scalar behavior.
• Assert the format-defined canonical result. Example: TOML multiline strings normalize recognized newlines to LF (\n) even when the source text uses CRLF.
• For parser inputs that come from files, use Goccia.TextFiles.ReadUTF8FileText for file I/O and TextSemantics helpers (CreateFileTextLines, StringListToLFText, newline normalization) for parser-facing text. Avoid TStringList.LoadFromFile and string(UTF8String(...)). Keep file text as UTF8String until parser entry, canonicalize parser-facing source text to LF through the shared text semantics, and add at least one regression that hits the real file-loading path with non-ASCII data.
• When code needs a "10 characters" or "first identifier code point" style rule on UTF-8 text, do not use raw Length, Copy, or byte indexing on string/UTF8String; those operate on bytes under our FPC settings.
• Keep one public-surface regression in tests/ and, when the parser exposes a reusable native utility like TGocciaTOMLParser, add a focused Pascal regression alongside it as well.

Test Metadata (Optional)

Test files can include JSDoc-style metadata inspired by test262:

/*---
description: Tests for basic addition operations
features: [addition, arithmetic]
---*/

Running Tests

Build the GocciaTestRunner

./build.pas testrunner

This also builds the native FFI fixture library from fixtures/ffi/fixture.c, so the full JavaScript suite can run the folder-configured FFI tests locally.

Run All Tests

# Interpreter mode (default)
./build/GocciaTestRunner tests
​
# Bytecode mode
./build/GocciaTestRunner tests --mode=bytecode

Both execution modes must pass. Test subtrees that require opt-in parser or runtime behavior declare it with a local goccia.json (for example, tests/language/asi/goccia.json enables ASI). Subtrees that reach outside their own directory declare the grant in a permissions block, with scopes relative to that config file: tests/built-ins/FFI (allow-ffi for the repo's fixtures/ffi), tests/built-ins/fetch (allow-net for the loopback test server and example.com), tests/built-ins/ShadowRealm, tests/language/modules/**, and tests/language/source-type (allow-read for shared fixtures and helpers), and tests/language/modules/node-modules (also allow-import for its committed node_modules). Declaring the grant beside the tests that need it, rather than passing --allow-* to the whole run, keeps every other folder under the default profile; see Permissions. CI runs the full suite in interpreter mode and bytecode mode as separate matrix jobs.

Those permission blocks, and the unsafe-* keys some folders set, only take effect once the configs are accepted; without that, the runner refuses the whole run and lists each untrusted config. Testing requirements covers -P and --trust.

Run a Specific Test File

./build/GocciaTestRunner tests/language/expressions/addition/basic-addition.js

Run a Test Category

./build/GocciaTestRunner tests/built-ins/Math/

GocciaTestRunner Options

Run ./build/GocciaTestRunner --help for the full list, including engine, permission, limit, coverage, and config-trust options.

--exit-on-first-failure stopping behaviour

The flag stops the file queue, not the run in progress:

• Sequential (--jobs=1) — execution stops at the failing file; no file after it runs.
• Parallel (--jobs=N, N > 1) — the queue is cancelled, but files a worker already picked up run to completion, and a worker reaching for its next file before the cancellation is visible starts that one too. Files after the failure can therefore still execute: expect roughly the failing file's position plus about one file per worker. The exact count depends on scheduling and is not reproducible between runs — do not assert on it.

Files the queue never reached are omitted from the report rather than counted as failures, so reported totals shrink as workers are added. For a deterministic "everything up to the failure and nothing after" run — bisecting, or a reproducible failure list — use --jobs=1.

# CI-friendly: no progress, stop on first failure
./build/GocciaTestRunner tests --no-progress --exit-on-first-failure
​
# Deterministic stop: nothing after the failing file runs
./build/GocciaTestRunner tests --no-progress --exit-on-first-failure --jobs=1
​
# Silent mode: only show results, suppress script console output
./build/GocciaTestRunner tests --silent
​
# Run tests with 4 parallel workers; --jobs=1 forces sequential execution
./build/GocciaTestRunner tests --jobs=4

Engine-integrity faults abort the run

There is one stop the runner makes on its own, without being asked for it. An engine-integrity fault — a use-after-free, an invalid dereference, a heap whose own bookkeeping is destroyed — unwinds past every guest catch to the host (ADR 0109), and when one escapes a test file's execution or the end-of-run inline-snapshot write-back, the whole run stops instead of the file being recorded as one more failure. Every later file would be executing on state the engine has already lost track of, so whatever it reported would mean nothing. (Those are the paths that abort. A fault from the runner's own setup — argument parsing, path expansion, config discovery — still lands in the shared CLI error handler that every Goccia binary uses, and exits 1 with no Integrity fault: line.)

The runner names the faulting file and the exception class on stderr under a fixed prefix, stops dispatching files, and exits 70:

Integrity fault: EObjectCheck in tests/some/file.js: Object reference is Nil
Integrity fault: the engine can no longer vouch for its own state, so the run is aborted and the remaining files were not executed.

No summary and no JSON envelope follow — a run that stopped mid-way has no total worth reporting. Grep CI logs for Integrity fault: to find these. Unlike --exit-on-first-failure, this does not let in-flight files finish: under --jobs the process ends as soon as the faulting worker has written its diagnostic, because a worker the watchdog has abandoned outlives the run and could otherwise fault after the main thread has stopped listening. The exit code is distinct from the ordinary failure exit 1 on purpose, so a harness can tell a suite that failed from a suite that stopped being trustworthy; see CLI Conventions. A refused allocation under --max-memory is deliberately not one of these: it is a verdict on one file delivered by an intact heap, so it stays a per-file failure and the run carries on.

Snapshot Testing

toMatchSnapshot() writes external snapshots to __snapshots__/<test-file>.snap. toMatchInlineSnapshot() inserts or updates the snapshot in the test source. Both use Vitest-compatible formatting and are available in interpreter and bytecode modes. The runner refuses a .snap file or a __snapshots__ directory that is a symbolic link rather than read, write, or delete through it, so a link committed to a repository cannot redirect a snapshot update outside the project; a snapshot file is replaced by writing beside it and renaming.

test("user", () => {
  const user = { id: 42, name: "Ada" };
​
  expect(user).toMatchSnapshot("public user");
  expect(user).toMatchInlineSnapshot({
    id: expect.any(Number),
  });
});

The first object argument is a snapshot property shape. Matching happens before any write, and the stored value replaces shaped fields with their asymmetric matcher descriptions. Snapshot shapes support expect.anything, expect.any, expect.closeTo, expect.arrayContaining, expect.objectContaining, expect.stringContaining, expect.stringMatching, and expect.schemaMatching, plus expect.toBeOneOf, expect.toSatisfy, expect.toBeFasterThan, and expect.toBeSlowerThan, including their supported expect.not forms. Snapshot assertions themselves do not support .not.

Local runs create missing snapshots. Existing mismatches fail until the runner is invoked with -u, --update, or --update-snapshots; update mode also prunes obsolete entries. Unless an explicit update flag is supplied, environments detected as CI by Vitest's std-env provider conventions—including GitHub Actions, GitLab, TeamCity, Buildkite, CircleCI, CI, and CONTINUOUS_INTEGRATION—do not write snapshots and fail on missing, mismatched, or obsolete entries. External snapshots are unavailable for stdin input. Existing inline snapshots can be compared from stdin, but they cannot be created or updated there.

Register a Vitest/pretty-format-compatible serializer when a test value needs a domain-specific representation. The most recently registered serializer runs first and must provide test plus either serialize or legacy print:

expect.addSnapshotSerializer({
  test(value) {
    return value?.kind === "point";
  },
  serialize(value) {
    return `Point(${value.x}, ${value.y})`;
  },
});

Native embedders can replace snapshot text formatting through IGocciaSnapshotFormatter and persistence/source editing through IGocciaSnapshotHost. See ADR 0096 for the boundary.

Official YAML Suite

For YAML parse-validity checks against the official yaml-test-suite (pinned to a specific SHA in the script), run:

python3 scripts/run_yaml_test_suite.py
python3 scripts/run_yaml_test_suite.py --output=tmp/yaml-suite-results.json
python3 scripts/run_yaml_test_suite.py --suite-dir=/path/to/yaml-test-suite

This check is intentionally parse-only: it compares whether each suite case should parse or fail, not whether Goccia's runtime values exactly match the suite's JSON fixtures. That distinction matters because Goccia intentionally canonicalizes complex YAML mapping keys into strings.

Official TOML Suite

For TOML 1.1.0 checks against a prepared checkout of the official toml-test corpus, run:

./build.pas tomlcompliancerunner
./build/GocciaTOMLComplianceRunner --suite-dir=/path/to/toml-test --output=tmp/toml-suite-results.json

The runner verifies the checkout against tests/compliance/toml-test.pin without fetching or cloning. It launches its private worker mode once per case, applies a bounded --jobs limit (defaulting to the online processor count through the same detector as TestRunner), and classifies mismatches, false accepts, false rejects, timeouts, crashes, and infrastructure failures. Valid cases are compared with the official tagged JSON fixtures through TGocciaTOMLParser.ParseDocument(...), preserving scalar distinctions such as integer, float, datetime, datetime-local, date-local, and time-local.

The runner prints a human summary, optionally writes the shared compliance JSON envelope with --output, and exits non-zero for compliance or infrastructure failures.

The harness and any file-backed parser regression must read source text as UTF-8 bytes first and decode it strictly before handing UTF-16 text to the parser. Do not place encoded bytes in a Pascal string or rely on compiler code pages. The shared TextEncoding and FileUtils boundaries keep behavior identical on Windows and non-Windows hosts and prevent mojibake such as José -> José.

Official JSON5 Suite

The upstream JSON5 parser cases are committed as an ordinary generated JavaScript test suite. Run parser and stringify compliance directly through TestRunner:

./build.pas testrunner
./build/GocciaTestRunner tests/built-ins/JSON5/upstream-parse.js tests/built-ins/JSON5/stringify.js --output=tmp/json5-suite-results.json

The generated suite records the revision in tests/compliance/json5.pin. TestRunner owns concurrency, per-file timeouts, aggregation, exit status, and the JSON report. Invalid JSON5 remains input to JSON5.parse(...) inside valid outer JavaScript. Normal compliance execution requires neither an upstream checkout, Python, nor Node.js.

Maintainers regenerate the JavaScript suite from an already-prepared checkout:

node scripts/regenerate-json5-tests.js /path/to/json5 <pinned-sha> tests/built-ins/JSON5/upstream-parse.js

Regeneration uses Node.js to execute the upstream reference tests, verifies and records the checkout commit, and is intentionally separate from normal compliance runs.

Pinned es-toolkit library probes

The bounded es-toolkit lane runs named public-package probes in both execution modes and emits classified JSON without making a whole-suite percentage claim. See es-toolkit Validation for the pinned artifact, command, failure taxonomy, and sandbox/host-surface boundaries.

Run Pascal Unit Tests

./build.pas tests
for t in build/Goccia.*.Test; do "$t"; done

Writing Tests

See Test Framework API for the complete test authoring reference including assertions, mocks, lifecycle hooks, async patterns, and cross-runtime compatibility.

Test Principles

1. JavaScript tests are the source of truth — The JavaScript test suite is the primary mechanism for verifying correctness and ECMAScript compatibility. If a behavior isn't covered by a JavaScript test, it isn't guaranteed.

2. Isolation — Each test file is independent. No shared state between files. The GocciaTestRunner executes each file in a fresh TGocciaEngine, and each engine owns its own realm. Engine tear-down between files unpins the per-engine intrinsic prototypes (Array.prototype, Object.prototype, every error and Temporal prototype, …), so userland mutations in one file — including non-configurable property additions — cannot leak into the next file even when the same worker thread runs both. When multiple explicit input paths are passed, the runner also keeps folder configs per-file; use --config for a deliberately shared root config.

3. Grouped by feature — Tests are organized by the feature they validate, mirroring the structure of the language specification. Prototype methods live in prototype/ subfolders; static methods live in the built-in's root folder.

4. One method per file — Each test file focuses on a single method, operation, or language feature. fill.js tests TypedArray.prototype.fill; map.js tests Array.prototype.map. Never bundle multiple methods into a single file.

5. Edge cases are co-located — Edge cases are co-located with happy-path tests (see File Naming and Layout Conventions).

6. One scenario per test — Keep each test(...) focused on a single behavior branch or input class. If undefined, null, and missing input go through different validation paths, they should be three tests, not one combined test. Combining many assertions into one block makes failures harder to localize and silently reduces test-count granularity during refactors.

7. Readable as specification — Test names should describe the expected behavior. A passing test suite serves as living documentation of what GocciaScript supports.

8. Always pass an error constructor to .toThrow() — When testing that code throws, pass the expected error constructor (TypeError, RangeError, SyntaxError, Error, etc.) to .toThrow() rather than calling it bare. This ensures the test verifies the correct error type, not just that something throws. For async code, prefer await expect(promise).rejects.toThrow(TypeError) over manually checking err.name.

// Preferred — verifies the error type
expect(() => null.foo).toThrow(TypeError);
expect(() => new Array(-1)).toThrow(RangeError);
await expect(Promise.reject(new TypeError("boom"))).rejects.toThrow(TypeError);
​
// Acceptable for cases where the error type is implementation-defined
expect(() => riskyOp()).toThrow();
​
// Avoid — doesn't verify the error type
expect(() => null.foo).toThrow();  // passes even if the wrong error is thrown
​
// Avoid — bypasses the matcher and only inspects a stringly-typed property
promise.catch((err) => {
  expect(err.name).toBe("TypeError");
});

Testing Strategy Design Decision

JavaScript end-to-end tests are the primary testing mechanism. Every new feature or bug fix must include tests that validate the behavior through the full pipeline (lexer → parser → interpreter/VM execution) using the most public surface available. CI runs all tests in interpreter mode and bytecode mode.

• Specification by example -- Each test file is a runnable specification of expected behavior.
• End-to-end validation -- Tests exercise the full pipeline, catching integration issues that unit tests would miss.
• Readable specifications -- JavaScript test files are readable by anyone familiar with Jest/Vitest conventions.
• Source of truth -- If a behavior isn't covered by a JavaScript test, it isn't guaranteed.

CLI behavior tests — the bun scripts under scripts/test-cli-*.ts, run by the workflows' cli job — form the second layer, verifying that the command-line tools produce correct output structure, reject malformed source with the right error, and that options like --coverage, --output=json, --source-map, and --timeout work end-to-end. These tests run in CI on every pull request.

Pascal unit tests (*.Test.pas) exist as a tertiary layer for behavior that cannot be reached through script code or other documented user-facing entry points. Even there, prefer stateless, repeatable input/output checks over tests that are tightly coupled to incidental implementation structure.

How the GocciaTestRunner Works

The GocciaTestRunner program:

1. Scans the provided path for .js, .jsx, .ts, .tsx, .mjs, and .mts files. A directory named node_modules is never descended into: a committed node_modules tree is a module-resolution fixture the suites beside it import (see Module Resolution), not a suite of its own.
2. For each file, creates a fresh TGocciaEngine with the capability set its options and goccia.json resolve to, applies source type from CLI/config or .mjs/.mts inference, attaches TGocciaRuntimeCore, applies the test-runner runtime profile, and installs the FFI runtime extension when that set grants ffi (--allow-ffi or the permissions block of the file's goccia.json).
3. Loads the source. In both modes the runner calls runTests directly after evaluation, so a test function starts with the whole --max-stack limit. In bytecode mode the runner then waits for the work the tests left pending, such as a timer an afterAll set, and a rejection that work leaves unhandled fails the file.
4. Executes the script — describe/test blocks register themselves during execution. Nested describe blocks are supported; suite names are composed with > separators (e.g., "Outer > Inner"). Skip state is inherited by nested describes.
5. runTests() executes all registered tests, reconciles snapshots through the installed host, and collects results.
6. When the run has more than one file, GC.Collect runs after each file: on the main thread in a sequential run (--jobs=1) and on the worker that ran the file in a parallel one. This reclaims the file's garbage before the next file starts. It does not reclaim the file's parsed AST, which is never freed, so memory still grows by roughly the size of each file's AST. A one-file run does not collect after its file.
7. Aggregates pass/fail/skip counts across all files.
8. Prints a summary with total statistics.

CLI Behavior Tests (CI Integration)

The CLI behaviour tests are standalone bun scripts under scripts/test-cli-*.ts that the PR and main workflows run (via bun run) in the cli job on every pull request. They assert on command output, exit codes, and error envelopes rather than internal state, catching regressions in the user-facing interface. test-cli-parser.ts and test-cli-lexer.ts specifically own the parser/lexer rejection paths (malformed source must fail with the correct SyntaxError and envelope) that the JavaScript suite cannot express.

What the CLI tests cover:

This table is non-exhaustive — the scripts/test-cli-*.ts scripts that the cli job runs are the source of truth. The intent is to check all CLI options, all parser/lexer error paths, and all output-format correctness. To add a new check, add a case to the matching scripts/test-cli-*.ts (parser/lexer rejection → test-cli-parser.ts / test-cli-lexer.ts; CLI flags/output → test-cli.ts / test-cli-apps.ts); the cli job already invokes these scripts, so no workflow change is needed.

Pascal Unit Tests (Tertiary)

Pascal unit tests are the tertiary testing layer for low-level internals that are difficult to validate through JavaScript alone (e.g., value representation, function/scope internals, and test-framework state behavior).

These compile as standalone executables and run directly:

./build.pas tests
./build/Goccia.Values.Primitives.Test
./build/Goccia.Values.FunctionValue.Test
./build/Goccia.Values.ObjectValue.Test
./build/Goccia.Builtins.TestingLibrary.Test

Pascal Test Framework

Pascal tests use TestingPascalLibrary.pas which provides a TTestSuite base class and a generic Expect<T> fluent assertion API:

type
  TMyTests = class(TTestSuite)
    procedure TestSomething;
    procedure SetupTests; override;
  end;
​
procedure TMyTests.SetupTests;
begin
  Test('should do something', TestSomething);
end;
​
procedure TMyTests.TestSomething;
begin
  Expect<string>(Value.ToStringLiteral.Value).ToBe('hello');
  Expect<Double>(Value.ToNumberLiteral.Value).ToBe(42);
  Expect<Integer>(Length(Params)).ToBe(2);
  Expect<Boolean>(Value.ToBooleanLiteral.Value).ToBe(True);
  Expect<Boolean>(Value.ToNumberLiteral.IsNaN).ToBe(False);
end;

Generic Assertions

The primary assertion API uses a standalone generic Expect<T> function that returns a TExpect<T> record:

On failure, ToBe produces descriptive messages like Expected "hello" to be "world" or Expected 3.14 to be 2.71.

TTestSuite also provides a Fail(Message) method to force a test failure with a custom message.

FPC 3.2.2 AArch64 Compiler Bug

Expect<T> is intentionally a standalone generic function (not a method on TTestSuite) and TExpect<T> is a record (not a class). This design works around a known FPC 3.2.2 AArch64 compiler crash: when a class with a generic method is inherited across compilation units, the compiler raises an internal EAccessViolation. The standalone function + record pattern avoids this entirely while preserving the fluent Expect<T>(...).ToBe(...) syntax.

NaN Checks in Pascal Tests

When testing for NaN, use the IsNaN property on TGocciaNumberLiteralValue:

// Correct — uses the property accessor
Expect<Boolean>(Value.ToNumberLiteral.IsNaN).ToBe(True);
​
// Also correct — IsNaN delegates to Math.IsNaN(FValue) internally
Math.IsNaN(Value.ToNumberLiteral.Value)

TGocciaNumberLiteralValue stores a single Double in FValue using standard IEEE 754 bit patterns for NaN, Infinity, and -0. The IsNaN and IsNegativeZero property accessors delegate to Math.IsNaN and an endian-neutral sign-bit check; IsInfinity is true for +Infinity only, and IsNegativeInfinity covers -Infinity. Prefer the property accessors for readability.

Testing the Test Framework Itself

Goccia.Builtins.TestingLibrary.Test.pas validates the JS-level assertion matchers from Pascal. Since the matchers call AssertionPassed/AssertionFailed (which update TGocciaTestAssertions internal state) rather than raising Pascal exceptions, the native tests use two helpers to verify both the return value and the recorded pass/fail outcome:

• ExpectPass(result) — Asserts the matcher returned UndefinedValue and CurrentTestHasFailures is False
• ExpectFail(result) — Asserts the matcher returned UndefinedValue and CurrentTestHasFailures is True

Each test starts with a BeforeEach override that calls FAssertions.ResetCurrentTestState to clear the failure flag. The CurrentTestHasFailures property and ResetCurrentTestState method are public on TGocciaTestAssertions for this purpose.

When to write Pascal unit tests vs JavaScript tests:

• JavaScript tests (preferred) — For any behavior observable from script code. This is almost everything.
• Pascal unit tests — Only for internal implementation details not reachable from script code (e.g., the test framework's own pass/fail recording logic).

CI Integration

GitHub Actions CI (.github/workflows/ci.yml) runs on push to main and tags, with a post-build job fan-out plus release packaging. Its first job, supersede, cancels any in-flight main run whose commit is an ancestor of the pushed commit, or its own run when a newer main commit is already being tested; tag and dispatched runs are never cancelled. See Build System for the rules.

supersede → toolchain → build → test             → artifacts (main), release (tags)
                              → toml-compliance  →
                              → json5-compliance →
                              → awfy             →
                              → jetstream        →
                              → web-tooling      →
                              → benchmark        →
                              → cli              →
                              → test262 → test262-merge
test-structure

toolchain / build — toolchain restores (or builds) the cached FPC cross-compilation toolchain; build cross-compiles all binaries for every target and uploads them as intermediate artifacts. test-structure runs scripts/check-test-structure.ts and the report-publishing helper tests without a build.

test (needs build, all platforms) — Downloads pre-built binaries, runs all JavaScript tests and Pascal unit tests. Outputs JSON files via --output=<file> for CI timing comparison.

toml-compliance (all platforms) — Downloads the prebuilt GocciaTOMLComplianceRunner, prepares the exact pinned toml-test checkout, and relies on the runner exit status. CI performs only lightweight validation of the report envelope before uploading it.

json5-compliance (all platforms) — Downloads GocciaTestRunner, verifies that the committed generated parser suite names the pinned JSON5 revision, and runs it together with the local stringify suite. CI relies on TestRunner's exit status and performs only lightweight validation of its JSON report before upload.

test262 (needs build, ubuntu-latest x64 only, non-blocking) — Runs the official conformance suite in bytecode mode, in four shards, from the shared pin in scripts/test262-suite-sha.txt; test262-merge merges the shards, uploads the JSON report, and saves a main baseline cache for PR deltas. The run step is continue-on-error: true so known steady-state conformance failures do not block unrelated work; the downstream PR comment still gates regressions against the cached main baseline. See test262.md for the harness contract and Build System for the workflow wiring.

awfy (needs build, ubuntu-latest x64 only) — Runs the pinned AWFY JavaScript report set from perf/awfy/manifest.json under Goccia bytecode, QuickJS, and the latest Node Current release resolved at workflow time with five interleaved samples per engine. Uploads the normalized awfy-report artifact; on main, publishes the compressed report plus daily pointer to the awfy/ Vercel Blob namespace when BLOB_READ_WRITE_TOKEN is configured.

jetstream / web-tooling (ubuntu-latest x64 only) — A *-plan job reads the workload matrix from the lane's manifest, *-workload runs each workload against the built binaries, and the final job merges the reports. See JetStream 3 and Web Tooling.

benchmark (needs build, all platforms) — Downloads pre-built binaries, runs all benchmarks.

cli (needs build, all platforms) — Downloads pre-built binaries and runs CLI behavior smoke tests via Bun: test-cli-assertions.ts (the shared matchers), test-cli.ts (options across all apps), test-cli-lexer.ts (numeric-separator rejection), test-cli-parser.ts (error display), test-cli-config.ts (config-file loading and per-file inheritance), test-cli-permissions.ts (capability flags, config permissions blocks, limits, and removed options), and test-cli-apps.ts (app-specific features, including GocciaScriptLoaderBare stdin/file checks, CLI-local print, and runtime-global absence), test-cli-embedded-resources.ts, test-cli-intl-taint.ts (Intl robustness under lazy materialization), test-cli-differential.ts (differential suites against Vitest and bun), and test-cli-coverage-parity.ts (identical coverage counts in both modes; main only). The x86-64 Linux leg also runs the pinned es-toolkit compatibility probes and uploads their classified JSON report. Windows runs additionally assert that GocciaRunner does not link against OpenSSL DLLs.

artifacts (needs test + toml-compliance + json5-compliance + awfy + jetstream + web-tooling + benchmark + cli, main only) — Uploads release binaries after all checks pass. test262 is not a gating dependency — failing tests there cannot block a release.

nightly (.github/workflows/nightly.yml) — Runs on a schedule, every three hours. It takes the newest CI run that succeeded for a push to main and publishes that run's builds as the nightly prerelease when its commit is ahead of the one the last nightly was built from, or has diverged from it after a rewritten main; when nothing new has passed, or the run's one-day build artifacts have expired, it publishes nothing. "Run workflow" on main publishes the newest passing build at once. A missing target artifact fails the job and leaves the existing release in place. It needs the whole run to succeed, so test-structure and test262-merge gate it as well: failing test262 tests do not fail test262-merge, but a missing or timed-out shard does. See Build System.

release (same gates, tags only) — Packages and publishes release archives.

The test, awfy, jetstream, web-tooling, benchmark, cli, toml-compliance, json5-compliance, and test262 lanes run in parallel after build. Every lane reuses the already-built binaries from the matrix build artifacts instead of installing FPC again, so platform-specific issues are caught on the same Linux, macOS, and Windows targets as the main test lanes (AWFY, JetStream, Web Tooling, and test262 run on Linux only since those outcomes are platform-independent for this engine).

PR Workflow (.github/workflows/pr.yml)

Runs on every pull request, including stacked PRs whose base is not main, on ubuntu-latest x64 only:

build → test, boot-timing, benchmark*      → timing comment (benchmark* also → benchmark comment)
      → awfy-report*                        → AWFY Results comment
      → jetstream-workload* → jetstream-report → JetStream comment
      → web-tooling-workload → web-tooling-report → Web Tooling comment
      → test262 → test262-merge             → test262 comment
      → cli
docs, test-structure (no build)       * also needs build-main, a build of the PR's base commit

docs runs the Markdown lint and the scripts/check-doc-*.ts / check-conformance-claims.ts checks.

The PR workflow also runs the pinned es-toolkit compatibility probes in the Linux cli job and retains the classified JSON report. It posts a Suite Timing comment with expandable test-runner and benchmark summaries. Each summary shows timing, top-level GocciaScript GC metrics, and selected FreePascal heap allocation metrics for interpreter mode and bytecode mode. GC memory rows aggregate the main thread plus all worker thread-local GCs. The benchmark runner collects after every file. The test runner collects after every file of a run with more than one file, on the worker that ran it. The collection count includes those collections. Reclaiming a worker's heap at shutdown is not counted as a collection. The comment hides negative FreePascal heap free-space deltas because they are valid allocator diagnostics but are noisy in a PR summary. See benchmarks.md for details on the benchmark comparison format.

The AWFY Results comment posts median timings and geomean ratios for the pinned AWFY report set under Goccia bytecode, QuickJS, and the latest Node Current release. The underlying awfy-report artifact keeps the five raw interleaved samples per engine, min/max/CV, environment metadata, and first-class failure outcomes.

The test262 conformance comment posts a non-blocking summary using the actions/github-script@v7 pattern, marker <!-- test262-results -->. The markdown is generated by bun scripts/run_test262_suite.ts --comment <results.json> <baseline.json|->. It contains:

• A per-category breakdown table (run / passed / Δ pass / failed / timeouts / Δ timeout / wrapper-infra / pass-rate / Δ rate) for each of built-ins, harness, intl402, language, staging, plus a totals row.
• A wrapper-infra-failure top-line warning when non-zero (gates the run as untrustworthy).
• An Areas closest to 100% table listing the three test262 directories (keyed by the first two path components, e.g. built-ins/Math, language/expressions) with the highest pass rate, filtered to areas with at least 25 attempted tests and below 100%.
• When a main baseline is cached, Δ vs main columns show absolute count delta and percentage-point delta. Deltas below ±0.05pp render as ±0pp to keep the table readable.
• A collapsible Per-test deltas section listing newly-passing, newly-failing, newly-timed-out, and no-longer-timed-out test IDs.

The website compatibility dashboard at /compatibility reads main-branch test262 reports from Vercel Blob at request time with CDN caching. Main CI publishes future dashboard points directly, and cd website && bun run backfill-test262 can seed historical days from retained artifacts or fresh historical reruns. Website builds do not download artifact ZIPs or bake test262 data into the deployment. This keeps the public pass-rate timeline and group coverage tables tied to generated CI data rather than hand-maintained percentages.

Weekly crons open (or update) a single PR every Monday with the latest upstream SHA so pins don't go stale:

Each workflow reuses a fixed branch (chore/<suite>-bump), so an unmerged PR is updated in place rather than replaced. See test262.md for details on the test262 harness contract.

Running test262 locally

./build.pas test262runner
./build/GocciaTest262Runner \
  --suite-dir=/path/to/test262 \
  --filter="built-ins/Array/*" \
  --output=results.json

The native runner accepts an existing checkout via --suite-dir; CI checks out the SHA pinned in scripts/test262-suite-sha.txt. Run ./build/GocciaTest262Runner --help for execution options including --categories, --max-tests, --mode={interpreted,bytecode}, --jobs, --timeout (a duration such as 20s), and --max-memory (a size such as 512MiB). The TypeScript utility handles only cross-shard merge, profile aggregation, and comment rendering.

Coverage

The GocciaTestRunner and GocciaRunner support JavaScript source-level coverage reporting via the --coverage flag. Coverage tracks which lines, branches, and functions of JavaScript source code are executed at runtime.

Coverage implies --mode=bytecode

Enabling coverage — via --coverage, --coverage-format, or --coverage-output — switches the engine to bytecode mode automatically, overriding any --mode from the command line or a config file, exactly as --profile does. The implication is silent: --mode=interpreted --coverage is not an error, it simply produces the bytecode report, and no flag combination yields an interpreter-mode one. The reason is that the interpreter's coverage path is structurally incomplete and unmaintained — it instruments only the entry file (imported modules run with coverage disabled, so their lines, branches, and functions never appear) and counts a statement hit per executed AST node rather than per executed source line, so its counts are not comparable to bytecode's. Coverage therefore always reports through the bytecode path, over the entry file and every module it imports. The switch is applied in each application's Validate (TTestRunnerApp.Validate, TRunnerApp.Validate), after config-file application and before the tracker is initialized.

Usage

# Console summary (printed after execution)
./build/GocciaTestRunner tests --coverage
​
# lcov output (for Codecov, Coveralls, or genhtml)
./build/GocciaTestRunner tests --coverage --coverage-format=lcov --coverage-output=coverage.lcov
​
# JSON output (Istanbul-compatible)
./build/GocciaTestRunner tests --coverage --coverage-format=json --coverage-output=coverage.json
​
# GocciaRunner also supports coverage
./build/GocciaRunner example.js --coverage

What is Tracked

Line coverage: Which source lines were executed and how many times. Instrumented in the bytecode VM's main dispatch loop, with per-frame line-change deduplication so several instructions on one source line count as one hit.

Branch coverage: Which branch arms were taken at:

• if/else statements (branch 0 = consequent, branch 1 = alternate)
• Ternary expressions (? :)
• Short-circuit operators (&&, ||, ??)
• switch statement case clauses

Function coverage: Which user-defined functions were created and called. A function is registered when its definition is evaluated and is reported at its declaration line (what LCOV's FN: means), not at the first line of its body; created-but-uncalled functions are retained with zero hits, and repeated calls increment the function hit count. Definitions inside untaken control flow are never created and therefore do not appear.

Parallel runs: --jobs=N does not change any of these numbers. Each worker collects into its own tracker and the counts are added together at the end, so line, branch, and function hit counts for a given run are identical whatever --jobs is set to.

Output Formats

Report Path Keys

Every file gets exactly one record per run, under one canonical path, however it was reached. A file can enter a run by two routes at once — named on the command line, and imported by another file — and each route knows it by a different spelling: the command line supplies whatever the user typed, import resolution produces an absolute path. Keying by the incoming spelling used to split such a file into two records whose hits were never added together.

The canonical form, applied uniformly to the console summary, the lcov SF: records, and the JSON keys and "path" fields:

• repo-relative when the file is under a repository root — the nearest ancestor directory holding a .git entry (a directory in a normal clone, a file in a linked worktree or submodule);
• absolute when the file is outside any repository;
• / as the separator, on every platform, including Windows.

Repo-relative is what the consumers want: Codecov matches report paths against repository paths, so a build machine's absolute paths need a fixes mapping while repo-relative paths match directly, and genhtml resolves relative paths against its working directory. Forward slashes matter because imports resolve through FPC's ExpandFileName — on Windows the raw paths carry backslashes, which genhtml and Codecov commonly mishandle in an SF: record.

Canonicalization is textual (ExpandFileName plus the repo-root trim), so it does not resolve symlinks: two spellings that differ by a symlinked ancestor — macOS's /var → /private/var, say — still produce separate records. Reaching the same file by two genuinely different real paths is not something normal entry-plus-import runs do.

Paths with no on-disk backing are identities, not files — <stdin>, and multifile section names such as <stdin>[part1] — and are left verbatim. Report keys are never used to read source: the tracker remembers the native on-disk path behind each key, so reports still render source when the process runs from outside the repository.

JSX Source Map Integration

When coverage is used with .jsx/.tsx files, the JSX transformer produces a source map that maps transformed (post-JSX) coordinates back to the original source. The coverage system integrates this source map so that lcov and JSON reports reference original JSX source positions, not transformed positions. This ensures downstream tools (Codecov, genhtml, Istanbul) display accurate line and branch locations.

Architecture

Coverage is recorded on the instrumented bytecode dispatch loop (FCoverageEnabled on TGocciaVM). When --coverage is not passed, production dispatch omits line-hit recording; enabling coverage, opcode profiling, stop-IP, or an instruction limit selects the instrumented loop. No separate build is needed.

Data is collected by TGocciaCoverageTracker (Goccia.Coverage.pas), a per-thread tracker that follows the same Initialize/Shutdown pattern as TGarbageCollector and TGocciaCallStack. During parallel test runs each worker thread initializes its own thread-local instance; after all workers complete, TGocciaThreadPool.MergeCoverageInto merges their data into the main thread's tracker via TGocciaCoverageTracker.MergeFrom, which adds the source hit counts into the destination (via AddLineHits / AddBranchHits) rather than registering a single hit per covered entry. Output formatting is in Goccia.Coverage.Report.pas. A JSX source map is registered with TGocciaCoverageTracker during file registration and applied only at report generation time, so the recording hot path is unaffected: BuildTranslatedLineHits translates transformed line hits back to original coordinates, and branch positions are translated via TGocciaSourceMap.Translate during report emission.