Testing

For contributors writing, running, or debugging tests.

Executive Summary

• Three testing layers — JavaScript end-to-end tests (primary), CLI behavior tests (secondary), Pascal unit tests (tertiary)
• Valid code → tests/; rejected code → scripts/test-cli-*.ts — JavaScript tests assert valid code runs and returns the right result; parser/lexer rejection (SyntaxError, caret, error envelope) belongs in scripts/test-cli-parser.ts / test-cli-lexer.ts, because with no eval/Function a parse error cannot be asserted from inside a JS test
• Built-in test framework — describe/test/expect with async support, mock functions, lifecycle hooks, and Vitest-compatible matchers
• One method per file — Each test file focuses on a single method; edge cases are co-located with happy-path tests
• Cover the contract, not only the example — Include boundaries, invalid inputs and receivers, coercion/order, state transitions, descriptors, and both execution modes when those perspectives apply
• Structure is checked in CI — Run bun run scripts/check-test-structure.ts before submitting test-suite organization changes
• Run with: ./build.pas testrunner, ./build/GocciaTestRunner tests, and ./build/GocciaTestRunner tests --mode=bytecode

GocciaScript uses three testing layers in priority order:

1. JavaScript end-to-end tests (primary) -- .js tests in tests/ that exercise the full pipeline through the same public surface that users call. CI runs the full suite in both interpreter mode and bytecode mode. Every new feature or bug fix should include tests at this layer.
2. CLI behavior tests (CI integration) -- Standalone bun scripts under scripts/test-cli-*.ts (test-cli.ts, test-cli-lexer.ts, test-cli-parser.ts, test-cli-config.ts, test-cli-apps.ts, test-cli-embedded-resources.ts) that the PR and main workflows run via bun run in the cli job. They invoke GocciaScriptLoader, GocciaTestRunner, and GocciaBenchmarkRunner as subprocesses and assert on exit codes, output structure, and error envelopes — above all parser/lexer rejection that a JS test cannot express (malformed source must fail with a SyntaxError, caret, suggestion, and JSON error envelope, in both modes), plus JSON output structure, coverage CLI, source maps, numeric separator rejection, timeout handling, global injection, and config loading. The matchers these scripts share live in scripts/test-cli/assertions.ts and are themselves unit-tested by scripts/test-cli-assertions.ts, which spawns no binaries — a defect in a shared matcher silently weakens every harness that uses it, so its contract is locked by its own test.
3. Pascal unit tests (tertiary) -- Native *.Test.pas coverage for low-level runtime and value system internals that are not reachable through a stable public API.

When choosing where to add coverage, prefer the most public entry point — and match the kind of check to the layer that can actually express it:

A JavaScript test that merely parses and runs a construct — e.g. (a / b) evaluating to 1, or (/x/).test(...) returning true — is a positive execution/conformance test. It confirms the construct works, but it does not prove that a different, malformed construct is rejected, and it is not the lexer/parser gate. When a change is about what the parser or lexer accepts or rejects (disambiguation, early errors, token classification), put the rejection cases in scripts/test-cli-parser.ts / test-cli-lexer.ts.

Do not use dynamic source execution (Function(...), eval(...), or equivalent runtime compilation) inside the JavaScript suite merely to smuggle parser failures into a runtime test. Keep dynamic-code assertions in tests/ only when the dynamic-code API itself is the feature under test, such as Function constructor behavior. Direct eval is host-gated, so its private test262-host coverage belongs in scripts/test-cli-apps.ts, outside the JavaScript suite.

Avoid tests that lock onto private helper functions or transient implementation structure when the same behavior can be validated through a documented user-facing command or script entry point. Where native Pascal tests are still appropriate, keep them as stateless and repeatable as possible so they behave like pure input/output checks rather than process-sensitive probes.

Keep suite titles, test names, and failure messages aligned with the layer under test. End-to-end JavaScript and CLI tests should use JavaScript/runtime terminology and describe behavior at that layer, while lower-level Pascal tests may use implementation terms when that is the behavior being exercised.

Test Organization

tests/
├── built-ins/              # Built-in object tests
│   ├── Array/              # Array constructor and prototype methods
│   │   ├── constructor.js
│   │   ├── from.js         # Array.from
│   │   ├── of.js           # Array.of
│   │   └── prototype/
│   │       ├── map.js, filter.js, reduce.js, forEach.js
│   │       ├── find.js, findIndex.js, indexOf.js, lastIndexOf.js
│   │       ├── sort.js, splice.js, shift.js, unshift.js
│   │       ├── fill.js, at.js, includes.js, concat.js, reverse.js
│   │       └── ...
│   ├── ArrayBuffer/        # ArrayBuffer constructor, static/prototype methods
│   │   ├── constructor.js, isView.js, toString-tag.js
│   │   └── prototype/
│   │       └── slice.js
│   ├── Error/
│   ├── JSON/
│   ├── Map/
│   ├── Math/
│   ├── Number/             # Number methods and constants
│   │   ├── parseInt.js, parseFloat.js, isNaN.js, isFinite.js, isInteger.js
│   │   ├── isSafeInteger.js, epsilon.js, maxSafeInteger.js
│   │   └── maxValue.js, minValue.js, nan.js
│   ├── Object/             # Object static and prototype methods
│   │   ├── keys.js, values.js, entries.js, assign.js, create.js, is.js
│   │   ├── defineProperty.js, defineProperties.js, getOwnPropertyDescriptor.js
│   │   ├── freeze.js, isFrozen.js
│   │   ├── getPrototypeOf.js, setPrototypeOf.js
│   │   ├── fromEntries.js, groupBy.js
│   │   ├── prototype/      # Object.prototype instance methods
│   │   │   ├── toString.js, hasOwnProperty.js, isPrototypeOf.js
│   │   │   ├── propertyIsEnumerable.js, toLocaleString.js, valueOf.js
│   │   └── ...
│   ├── Promise/             # Promise constructor, static methods, microtask ordering
│   │   ├── constructor.js, resolve.js, reject.js
│   │   ├── all.js, all-settled.js, race.js, any.js
│   │   ├── microtask-ordering.js, thenable-adoption.js
│   │   └── prototype/
│   │       ├── then.js, catch.js, finally.js
│   ├── Set/
│   ├── SharedArrayBuffer/  # SharedArrayBuffer constructor, prototype methods
│   │   ├── constructor.js, toString-tag.js
│   │   └── prototype/
│   │       └── slice.js
│   ├── String/
│   │   └── prototype/
│   ├── structuredClone/    # structuredClone tests (primitives, objects, collections, errors, arraybuffer)
│   ├── Symbol/
│   └── TypedArray/         # TypedArray constructors, prototype methods
│       ├── constructors.js, element-access.js, buffer-sharing.js
│       ├── from.js, of.js  # Static methods at top level
│       └── prototype/      # Instance methods (one file per method, edge cases included)
│           ├── fill.js, map.js, filter.js, reduce.js, reduceRight.js
│           ├── indexOf.js, lastIndexOf.js, includes.js
│           ├── find.js, findIndex.js, findLast.js, findLastIndex.js
│           ├── sort.js, reverse.js, slice.js, subarray.js
│           ├── set.js, copyWithin.js, at.js, with.js
│           ├── every.js, some.js, forEach.js, join.js
│           ├── toReversed.js, toSorted.js
│           └── values.js, keys.js, entries.js
│
└── language/               # Core language feature tests
    ├── classes/            # Class declarations, inheritance, private fields/methods/getters/setters
    ├── declarations/       # let, const
    ├── decorators/         # TC39 Stage 2.7 decorators and decorator metadata
    ├── expressions/        # Arithmetic, comparison, logical, destructuring, trailing commas, etc.
    │   ├── addition/       # Addition with ToPrimitive
    │   ├── arithmetic/     # Division (IEEE-754 signed zeros, Infinity), exponentiation (Infinity edge cases)
    │   ├── bitwise/        # Bitwise OR, AND, XOR, NOT, shifts
    │   ├── modulo/         # Floating-point modulo
    │   ├── conditional/    # Ternary precedence
    │   ├── optional-chaining/  # Optional chaining (?.) edge cases
    │   ├── trailing-commas.js  # Trailing commas in calls, parameters, constructors
    │   └── ...
    ├── functions/          # Arrow functions, closures, higher-order, recursion
    │   └── function-length-name.js  # Function.length and Function.name
    ├── identifiers/        # Unicode support, reserved words
    ├── modules/            # Import/export
    ├── objects/            # Object literals, methods, computed properties
    ├── statements/         # if/else, switch/case/break, try/catch/finally
    │   ├── try-catch/      # Try-catch-finally edge cases
    │   └── unsupported-features/    # Warning-mode recovery tests for disabled compatibility syntax
    └── unary-operators.js

Constructor requirements belong with the constructor they constrain. For example, the Map()-without-new rejection belongs in tests/built-ins/Map/constructor.js, not in a shared tests/built-ins/constructors/require-new.js file.

File Naming and Layout Conventions

Follow these rules when creating or organizing test files:

1. One method per file — Each built-in method or static function gets its own test file. Never bundle multiple methods into a single file like prototype-methods.js or static-methods.js.

# Correct — each method is its own file
tests/built-ins/TypedArray/prototype/fill.js
tests/built-ins/TypedArray/prototype/map.js
tests/built-ins/TypedArray/prototype/indexOf.js
​
# Wrong — multiple methods in one file
tests/built-ins/TypedArray/prototype-methods.js

2. Prototype methods go in a prototype/ subfolder — Instance methods (e.g., Array.prototype.map) live in BuiltIn/prototype/methodName.js. Static methods and constructor tests live directly in the BuiltIn/ folder (no separate static/ subfolder).

tests/built-ins/TypedArray/
├── constructors.js              # new TypedArray(...) constructor variants
├── buffer-sharing.js            # Buffer sharing behavior across views
├── from.js, of.js               # Static methods at top level
└── prototype/                   # Instance methods
    ├── at.js
    ├── fill.js
    ├── indexOf.js
    ├── map.js
    └── ...

3. Edge cases are co-located, not separate — Edge cases (NaN handling, Infinity, negative indices, empty arrays, clamping, etc.) belong in the same file as the happy-path tests for that method. Do not create standalone edge-cases.js files — they become disconnected from the feature they test and make it unclear which method the edge case validates.

// tests/built-ins/TypedArray/prototype/fill.js — CORRECT
describe("TypedArray.prototype.fill", () => {
  test("fills entire array", () => { ... });
  test("fills with start and end", () => { ... });
​
  // Edge cases are part of the same file
  test("negative start index counts from end", () => { ... });
  test("NaN fill value becomes 0 for integer types", () => { ... });
  test("Infinity clamps to 255 for Uint8ClampedArray", () => { ... });
});

# Wrong — edge cases in a separate catch-all file
tests/built-ins/TypedArray/edge-cases.js

4. Object prototype methods use prototype/ too — Object.prototype.toString, Object.prototype.hasOwnProperty, etc. follow the same convention as Array. Use Object/prototype/toString.js, not Object/prototype-toString.js.

tests/built-ins/Object/
├── keys.js                      # Object.keys (static method)
├── assign.js                    # Object.assign (static method)
├── prototype/                   # Instance methods on Object.prototype
│   ├── toString.js
│   ├── hasOwnProperty.js
│   ├── isPrototypeOf.js
│   ├── propertyIsEnumerable.js
│   ├── toLocaleString.js
│   └── valueOf.js

5. Test titles should match the test layer — describe(...) and test(...) strings should use the vocabulary of the layer they exercise. End-to-end JavaScript tests should avoid leaking Pascal implementation terms; lower-level Pascal tests do not need that restriction.

// Correct — end-to-end JavaScript test uses JavaScript/runtime terminology
test("JSON.stringify preserves round-trip precision for large fractional floating-point numbers", () => {
  // ...
});
​
// Wrong — end-to-end JavaScript test leaks Pascal implementation terms
test("JSON.stringify preserves round-trip precision for large fractional doubles", () => {
  // ...
});

6. Cross-surface files are narrow exceptions — A file may span several methods only when the behavior itself is one indivisible cross-surface contract, not merely because the methods are related or share an implementation. Examples include a proposal-version absence contract such as Temporal/removed-methods.js, or a constructor-family invariant such as Error/error-prototype-constructor.js. A shared receiver-branding matrix such as Date/prototype/receiver-branding.js is also appropriate when it verifies the same internal-slot requirement across methods without duplicating that matrix in every method file.

These exceptions must be:

• named after the single contract rather than methods.js, misc.js, or error-cases.js;
• explicitly allowlisted in scripts/check-test-structure.ts with a concrete reason;
• kept small enough that a failure still identifies the contract that broke.

Shared setup or table data is not by itself a reason to combine operations. Prefer a helper imported by focused files when several operations need the same fixtures.

Coverage Perspectives

A passing happy-path example proves only that one route works. For every method or operation, inspect its public contract and implementation branches, then add the perspectives that can change its observable result:

• ordinary behavior, including more than one representative input where the operation has distinct modes;
• boundaries and sentinels such as empty input, zero and signed zero, NaN, infinities, minimum/maximum values, and first/last valid indexes;
• invalid arguments and incompatible receivers, including the exact error type;
• argument coercion, throwing coercions, and observable evaluation order;
• repeated calls and state transitions for mutable or stateful APIs;
• property metadata and descriptors for exposed built-in properties;
• independent expected values for encoders/decoders and getters/setters rather than round trips that can let two matching defects cancel out;
• interpreter and bytecode parity, which the full JavaScript suite gate verifies.

Not every bullet applies to every operation. Each added test must protect an observable contract or a meaningful implementation branch; do not add combinatorial duplicates merely to satisfy a checklist.

Structural Check

Run the test-suite structure check after adding or reorganizing JavaScript tests:

bun run scripts/check-test-structure.ts

The check rejects catch-all filenames, known multi-operation files, reliably detectable prototype-method suites outside prototype/, and parser-rejection assertions hidden behind Function(...) or eval(...). Dynamic-code rejection tests are allowed only where the dynamic-code API itself is under test.

Cross-Platform Newline Rules for Data Format Parsers

When a parser implements a format with spec-defined newline semantics, test the format semantics directly instead of inheriting the host OS newline convention.

• Do not treat LineEnding / sLineBreak as the expected runtime result for parsed data just because the test is running on Windows.
• Prefer explicit \r\n or #13#10 fixtures when adding regression tests for multiline parsing, folding, or block-scalar behavior.
• Assert the format-defined canonical result. Example: TOML multiline strings normalize recognized newlines to LF (\n) even when the source text uses CRLF.
• For parser inputs that come from files, use Goccia.TextFiles.ReadUTF8FileText for file I/O and TextSemantics helpers (CreateUTF8FileTextLines, StringListToLFText, newline normalization) for parser-facing text. Avoid TStringList.LoadFromFile and string(UTF8String(...)). Keep file text as UTF8String until parser entry, canonicalize parser-facing source text to LF through the shared text semantics, and add at least one regression that hits the real file-loading path with non-ASCII data.
• When code needs a "10 characters" or "first identifier code point" style rule on UTF-8 text, do not use raw Length, Copy, or byte indexing on string/UTF8String; those operate on bytes under our FPC settings.
• Keep one public-surface regression in tests/ and, when the parser exposes a reusable native utility like TGocciaTOMLParser, add a focused Pascal regression alongside it as well.

Test Metadata (Optional)

Test files can include JSDoc-style metadata inspired by test262:

/*---
description: Tests for basic addition operations
features: [addition, arithmetic]
---*/

Running Tests

Build the GocciaTestRunner

./build.pas testrunner

This also builds the native FFI fixture library from fixtures/ffi/fixture.c, so the full JavaScript suite can run the folder-configured FFI tests locally.

Run All Tests

# Interpreter mode (default)
./build/GocciaTestRunner tests
​
# Bytecode mode
./build/GocciaTestRunner tests --mode=bytecode

Both execution modes must pass. Test subtrees that require opt-in parser or runtime behavior declare it with a local goccia.json (for example, tests/language/asi/goccia.json enables ASI and tests/built-ins/FFI/goccia.json enables FFI). CI runs the full suite in interpreter mode and bytecode mode as separate matrix jobs.

Run a Specific Test File

./build/GocciaTestRunner tests/language/expressions/addition/basic-addition.js

Run a Test Category

./build/GocciaTestRunner tests/built-ins/Math/

GocciaTestRunner Options

--exit-on-first-failure stopping behaviour

The flag stops the file queue, not the run in progress:

• Sequential (--jobs=1) — execution stops at the failing file; no file after it runs.
• Parallel (--jobs=N, N > 1) — the queue is cancelled, but files a worker already picked up run to completion, and a worker reaching for its next file before the cancellation is visible starts that one too. Files after the failure can therefore still execute: expect roughly the failing file's position plus about one file per worker. The exact count depends on scheduling and is not reproducible between runs — do not assert on it.

Files the queue never reached are omitted from the report rather than counted as failures, so reported totals shrink as workers are added. For a deterministic "everything up to the failure and nothing after" run — bisecting, or a reproducible failure list — use --jobs=1.

# CI-friendly: no progress, stop on first failure
./build/GocciaTestRunner tests --no-progress --exit-on-first-failure
​
# Deterministic stop: nothing after the failing file runs
./build/GocciaTestRunner tests --no-progress --exit-on-first-failure --jobs=1
​
# Silent mode: only show results, suppress script console output
./build/GocciaTestRunner tests --silent
​
# Run tests with 4 parallel workers; --jobs=1 forces sequential execution
./build/GocciaTestRunner tests --jobs=4

Engine-integrity faults abort the run

There is one stop the runner makes on its own, without being asked for it. An engine-integrity fault — a use-after-free, an invalid dereference, a heap whose own bookkeeping is destroyed — unwinds past every guest catch to the host (ADR 0109), and when one escapes a test file's execution or the end-of-run inline-snapshot write-back, the whole run stops instead of the file being recorded as one more failure. Every later file would be executing on state the engine has already lost track of, so whatever it reported would mean nothing. (Those are the paths that abort. A fault from the runner's own setup — argument parsing, path expansion, config discovery — still lands in the shared CLI error handler that every Goccia binary uses, and exits 1 with no Integrity fault: line.)

The runner names the faulting file and the exception class on stderr under a fixed prefix, stops dispatching files, and exits 70:

Integrity fault: EObjectCheck in tests/some/file.js: Object reference is Nil
Integrity fault: the engine can no longer vouch for its own state, so the run is aborted and the remaining files were not executed.

No summary and no JSON envelope follow — a run that stopped mid-way has no total worth reporting. Grep CI logs for Integrity fault: to find these. Unlike --exit-on-first-failure, this does not let in-flight files finish: under --jobs the process ends as soon as the faulting worker has written its diagnostic, because a worker the watchdog has abandoned outlives the run and could otherwise fault after the main thread has stopped listening. The exit code is distinct from the ordinary failure exit 1 on purpose, so a harness can tell a suite that failed from a suite that stopped being trustworthy; see CLI Conventions. A refused allocation under --max-memory is deliberately not one of these: it is a verdict on one file delivered by an intact heap, so it stays a per-file failure and the run carries on.

Snapshot Testing

toMatchSnapshot() writes external snapshots to __snapshots__/<test-file>.snap. toMatchInlineSnapshot() inserts or updates the snapshot in the test source. Both use Vitest-compatible formatting and are available in interpreter and bytecode modes.

test("user", () => {
  const user = { id: 42, name: "Ada" };
​
  expect(user).toMatchSnapshot("public user");
  expect(user).toMatchInlineSnapshot({
    id: expect.any(Number),
  });
});

The first object argument is a snapshot property shape. Matching happens before any write, and the stored value replaces shaped fields with their asymmetric matcher descriptions. Snapshot shapes support expect.anything, expect.any, expect.closeTo, expect.arrayContaining, expect.objectContaining, expect.stringContaining, expect.stringMatching, and expect.schemaMatching, plus expect.toBeOneOf, expect.toSatisfy, expect.toBeFasterThan, and expect.toBeSlowerThan, including their supported expect.not forms. Snapshot assertions themselves do not support .not.

Local runs create missing snapshots. Existing mismatches fail until the runner is invoked with -u, --update, or --update-snapshots; update mode also prunes obsolete entries. Unless an explicit update flag is supplied, environments detected as CI by Vitest's std-env provider conventions—including GitHub Actions, GitLab, TeamCity, Buildkite, CircleCI, CI, and CONTINUOUS_INTEGRATION—do not write snapshots and fail on missing, mismatched, or obsolete entries. External snapshots are unavailable for stdin input. Existing inline snapshots can be compared from stdin, but they cannot be created or updated there.

Register a Vitest/pretty-format-compatible serializer when a test value needs a domain-specific representation. The most recently registered serializer runs first and must provide test plus either serialize or legacy print:

expect.addSnapshotSerializer({
  test(value) {
    return value?.kind === "point";
  },
  serialize(value) {
    return `Point(${value.x}, ${value.y})`;
  },
});

Native embedders can replace snapshot text formatting through IGocciaSnapshotFormatter and persistence/source editing through IGocciaSnapshotHost. See ADR 0096 for the boundary.

Official YAML Suite

For YAML parse-validity checks against the official yaml-test-suite (pinned to a specific SHA in the script), run:

python3 scripts/run_yaml_test_suite.py
python3 scripts/run_yaml_test_suite.py --output=tmp/yaml-suite-results.json
python3 scripts/run_yaml_test_suite.py --suite-dir=/path/to/yaml-test-suite

This check is intentionally parse-only: it compares whether each suite case should parse or fail, not whether Goccia's runtime values exactly match the suite's JSON fixtures. That distinction matters because Goccia intentionally canonicalizes complex YAML mapping keys into strings.

Official TOML Suite

For TOML 1.1.0 checks against a prepared checkout of the official toml-test corpus, run:

./build.pas tomlcompliancerunner
./build/GocciaTOMLComplianceRunner --suite-dir=/path/to/toml-test --output=tmp/toml-suite-results.json

The runner verifies the checkout against tests/compliance/toml-test.pin without fetching or cloning. It launches its private worker mode once per case, applies a bounded --jobs limit, and classifies mismatches, false accepts, false rejects, timeouts, crashes, and infrastructure failures. Valid cases are compared with the official tagged JSON fixtures through TGocciaTOMLParser.ParseDocument(...), preserving scalar distinctions such as integer, float, datetime, datetime-local, date-local, and time-local.

The runner prints a human summary, optionally writes the shared compliance JSON envelope with --output, and exits non-zero for compliance or infrastructure failures.

The harness and any file-backed parser regression must read source text as UTF-8 bytes first and decode it strictly before handing UTF-16 text to the parser. Do not place encoded bytes in a Pascal string or rely on compiler code pages. The shared TextEncoding and FileUtils boundaries keep behavior identical on Windows and non-Windows hosts and prevent mojibake such as José -> José.

Official JSON5 Suite

The upstream JSON5 parser cases are committed as an ordinary generated JavaScript test suite. Run parser and stringify compliance directly through TestRunner:

./build.pas testrunner
./build/GocciaTestRunner tests/built-ins/JSON5/upstream-parse.js tests/built-ins/JSON5/stringify.js --output=tmp/json5-suite-results.json

The generated suite records the revision in tests/compliance/json5.pin. TestRunner owns concurrency, per-file timeouts, aggregation, exit status, and the JSON report. Invalid JSON5 remains input to JSON5.parse(...) inside valid outer JavaScript. Normal compliance execution requires neither an upstream checkout, Python, nor Node.js.

Maintainers regenerate the JavaScript suite from an already-prepared checkout:

node scripts/regenerate-json5-tests.js /path/to/json5 <pinned-sha> tests/built-ins/JSON5/upstream-parse.js

Regeneration uses Node.js to execute the upstream reference tests, verifies and records the checkout commit, and is intentionally separate from normal compliance runs.

Pinned es-toolkit library probes

The bounded es-toolkit lane runs named public-package probes in both execution modes and emits classified JSON without making a whole-suite percentage claim. See es-toolkit Validation for the pinned artifact, command, failure taxonomy, and sandbox/host-surface boundaries.

Run Pascal Unit Tests

./build.pas tests
for t in build/Goccia.*.Test; do "$t"; done

Writing Tests

See Test Framework API for the complete test authoring reference including assertions, mocks, lifecycle hooks, async patterns, and cross-runtime compatibility.

Test Principles

1. JavaScript tests are the source of truth — The JavaScript test suite is the primary mechanism for verifying correctness and ECMAScript compatibility. If a behavior isn't covered by a JavaScript test, it isn't guaranteed.

2. Isolation — Each test file is independent. No shared state between files. The GocciaTestRunner executes each file in a fresh TGocciaEngine, and each engine owns its own realm. Engine tear-down between files unpins the per-engine intrinsic prototypes (Array.prototype, Object.prototype, every error and Temporal prototype, …), so userland mutations in one file — including non-configurable property additions — cannot leak into the next file even when the same worker thread runs both. When multiple explicit input paths are passed, the runner also keeps folder configs per-file; use --config for a deliberately shared root config.

3. Grouped by feature — Tests are organized by the feature they validate, mirroring the structure of the language specification. Prototype methods live in prototype/ subfolders; static methods live in the built-in's root folder.

4. One method per file — Each test file focuses on a single method, operation, or language feature. fill.js tests TypedArray.prototype.fill; map.js tests Array.prototype.map. Never bundle multiple methods into a single file.

5. Edge cases are co-located — Edge cases are co-located with happy-path tests (see File Naming and Layout Conventions).

6. One scenario per test — Keep each test(...) focused on a single behavior branch or input class. If undefined, null, and missing input go through different validation paths, they should be three tests, not one combined test. Combining many assertions into one block makes failures harder to localize and silently reduces test-count granularity during refactors.

7. Readable as specification — Test names should describe the expected behavior. A passing test suite serves as living documentation of what GocciaScript supports.

8. Always pass an error constructor to .toThrow() — When testing that code throws, pass the expected error constructor (TypeError, RangeError, SyntaxError, Error, etc.) to .toThrow() rather than calling it bare. This ensures the test verifies the correct error type, not just that something throws. For async code, prefer await expect(promise).rejects.toThrow(TypeError) over manually checking err.name.

// Preferred — verifies the error type
expect(() => null.foo).toThrow(TypeError);
expect(() => new Array(-1)).toThrow(RangeError);
await expect(Promise.reject(new TypeError("boom"))).rejects.toThrow(TypeError);
​
// Acceptable for cases where the error type is implementation-defined
expect(() => riskyOp()).toThrow();
​
// Avoid — doesn't verify the error type
expect(() => null.foo).toThrow();  // passes even if the wrong error is thrown
​
// Avoid — bypasses the matcher and only inspects a stringly-typed property
promise.catch((err) => {
  expect(err.name).toBe("TypeError");
});

Testing Strategy Design Decision

JavaScript end-to-end tests are the primary testing mechanism. Every new feature or bug fix must include tests that validate the behavior through the full pipeline (lexer → parser → interpreter/VM execution) using the most public surface available. CI runs all tests in interpreter mode and bytecode mode.

• Specification by example -- Each test file is a runnable specification of expected behavior.
• End-to-end validation -- Tests exercise the full pipeline, catching integration issues that unit tests would miss.
• Readable specifications -- JavaScript test files are readable by anyone familiar with Jest/Vitest conventions.
• Source of truth -- If a behavior isn't covered by a JavaScript test, it isn't guaranteed.

CLI behavior tests — the bun scripts under scripts/test-cli-*.ts, run by the workflows' cli job — form the second layer, verifying that the command-line tools produce correct output structure, reject malformed source with the right error, and that options like --coverage, --output=json, --source-map, and --timeout work end-to-end. These tests run in CI on every pull request.

Pascal unit tests (*.Test.pas) exist as a tertiary layer for behavior that cannot be reached through script code or other documented user-facing entry points. Even there, prefer stateless, repeatable input/output checks over tests that are tightly coupled to incidental implementation structure.

How the GocciaTestRunner Works

The GocciaTestRunner program:

1. Scans the provided path for .js, .jsx, .ts, .tsx, .mjs, and .mts files.
2. For each file, creates a fresh TGocciaEngine, applies source type from CLI/config or .mjs/.mts inference, attaches TGocciaRuntimeCore, applies the test-runner runtime profile, and installs the FFI runtime extension when --unsafe-ffi or the file's goccia.json enables it.
3. Loads the source and appends a runTests() call.
4. Executes the script — describe/test blocks register themselves during execution. Nested describe blocks are supported; suite names are composed with > separators (e.g., "Outer > Inner"). Skip state is inherited by nested describes.
5. runTests() executes all registered tests, reconciles snapshots through the installed host, and collects results.
6. When running multiple files, GC.Collect runs after each file to reclaim memory between script executions.
7. Aggregates pass/fail/skip counts across all files.
8. Prints a summary with total statistics.

CLI Behavior Tests (CI Integration)

The CLI behaviour tests are standalone bun scripts under scripts/test-cli-*.ts that the PR and main workflows run (via bun run) in the cli job on every pull request. They assert on command output, exit codes, and error envelopes rather than internal state, catching regressions in the user-facing interface. test-cli-parser.ts and test-cli-lexer.ts specifically own the parser/lexer rejection paths (malformed source must fail with the correct SyntaxError and envelope) that the JavaScript suite cannot express.

What the CLI tests cover:

This table is non-exhaustive — the scripts/test-cli-*.ts scripts that the cli job runs are the source of truth. The intent is to check all CLI options, all parser/lexer error paths, and all output-format correctness. To add a new check, add a case to the matching scripts/test-cli-*.ts (parser/lexer rejection → test-cli-parser.ts / test-cli-lexer.ts; CLI flags/output → test-cli.ts / test-cli-apps.ts); the cli job already invokes these scripts, so no workflow change is needed.

Pascal Unit Tests (Tertiary)

Pascal unit tests are the tertiary testing layer for low-level internals that are difficult to validate through JavaScript alone (e.g., value representation, function/scope internals, and test-framework state behavior).

These compile as standalone executables and run directly:

./build.pas tests
./build/Goccia.Values.Primitives.Test
./build/Goccia.Values.FunctionValue.Test
./build/Goccia.Values.ObjectValue.Test
./build/Goccia.Builtins.TestingLibrary.Test

Pascal Test Framework

Pascal tests use TestingPascalLibrary.pas which provides a TTestSuite base class and a generic Expect<T> fluent assertion API:

type
  TMyTests = class(TTestSuite)
    procedure TestSomething;
    procedure SetupTests; override;
  end;
​
procedure TMyTests.SetupTests;
begin
  Test('should do something', TestSomething);
end;
​
procedure TMyTests.TestSomething;
begin
  Expect<string>(Value.ToStringLiteral.Value).ToBe('hello');
  Expect<Double>(Value.ToNumberLiteral.Value).ToBe(42);
  Expect<Integer>(Length(Params)).ToBe(2);
  Expect<Boolean>(Value.ToBooleanLiteral.Value).ToBe(True);
  Expect<Boolean>(Value.ToNumberLiteral.IsNaN).ToBe(False);
end;

Generic Assertions

The primary assertion API uses a standalone generic Expect<T> function that returns a TExpect<T> record:

On failure, ToBe produces descriptive messages like Expected "hello" to be "world" or Expected 3.14 to be 2.71.

TTestSuite also provides a Fail(Message) method to force a test failure with a custom message.

FPC 3.2.2 AArch64 Compiler Bug

Expect<T> is intentionally a standalone generic function (not a method on TTestSuite) and TExpect<T> is a record (not a class). This design works around a known FPC 3.2.2 AArch64 compiler crash: when a class with a generic method is inherited across compilation units, the compiler raises an internal EAccessViolation. The standalone function + record pattern avoids this entirely while preserving the fluent Expect<T>(...).ToBe(...) syntax.

NaN Checks in Pascal Tests

When testing for NaN, use the IsNaN property on TGocciaNumberLiteralValue:

// Correct — uses the property accessor
Expect<Boolean>(Value.ToNumberLiteral.IsNaN).ToBe(True);
​
// Also correct — IsNaN delegates to Math.IsNaN(FValue) internally
Math.IsNaN(Value.ToNumberLiteral.Value)

TGocciaNumberLiteralValue stores a single Double in FValue using standard IEEE 754 bit patterns for NaN, Infinity, and -0. The IsNaN, IsInfinity, and IsNegativeZero property accessors delegate to Math.IsNaN, Math.IsInfinite, and an endian-neutral sign-bit check respectively. Prefer the property accessors for readability.

Testing the Test Framework Itself

Goccia.Builtins.TestingLibrary.Test.pas validates the JS-level assertion matchers from Pascal. Since the matchers call AssertionPassed/AssertionFailed (which update TGocciaTestAssertions internal state) rather than raising Pascal exceptions, the native tests use two helpers to verify both the return value and the recorded pass/fail outcome:

• ExpectPass(result) — Asserts the matcher returned UndefinedValue and CurrentTestHasFailures is False
• ExpectFail(result) — Asserts the matcher returned UndefinedValue and CurrentTestHasFailures is True

Each test starts with a BeforeEach override that calls FAssertions.ResetCurrentTestState to clear the failure flag. The CurrentTestHasFailures property and ResetCurrentTestState method are public on TGocciaTestAssertions for this purpose.

When to write Pascal unit tests vs JavaScript tests:

• JavaScript tests (preferred) — For any behavior observable from script code. This is almost everything.
• Pascal unit tests — Only for internal implementation details not reachable from script code (e.g., the test framework's own pass/fail recording logic).

CI Integration

GitHub Actions CI (.github/workflows/ci.yml) runs on push to main and tags, with a post-build job fan-out plus release packaging:

build → test             → artifacts
      → toml-compliance  →
      → json5-compliance →
      → test262          →
      → awfy             →
      → benchmark        →
      → cli              →

build — Installs FPC once per platform, compiles all binaries, uploads them as intermediate artifacts.

test (needs build, all platforms) — Downloads pre-built binaries, runs all JavaScript tests and Pascal unit tests. Outputs JSON files via --output=<file> for CI timing comparison.

toml-compliance (all platforms) — Downloads the prebuilt GocciaTOMLComplianceRunner, prepares the exact pinned toml-test checkout, and relies on the runner exit status. CI performs only lightweight validation of the report envelope before uploading it.

json5-compliance (all platforms) — Downloads GocciaTestRunner, verifies that the committed generated parser suite names the pinned JSON5 revision, and runs it together with the local stringify suite. CI relies on TestRunner's exit status and performs only lightweight validation of its JSON report before upload.

test262 (needs build, ubuntu-latest x64 only, non-blocking) — Runs the official conformance suite in bytecode mode from the shared pin in scripts/test262-suite-sha.txt, uploads the JSON report, and saves a main baseline cache for PR deltas. The run step is continue-on-error: true so known steady-state conformance failures do not block unrelated work; the downstream PR comment still gates regressions against the cached main baseline. See test262.md for the harness contract and Build System for the workflow wiring.

awfy (needs build, ubuntu-latest x64 only) — Runs the pinned AWFY JavaScript report set from perf/awfy/manifest.json under Goccia bytecode, QuickJS, and the latest Node Current release resolved at workflow time with five interleaved samples per engine. Uploads the normalized awfy-report artifact; on main, publishes the compressed report plus daily pointer to the awfy/ Vercel Blob namespace when BLOB_READ_WRITE_TOKEN is configured.

benchmark (needs build, all platforms) — Downloads pre-built binaries, runs all benchmarks.

cli (needs build, all platforms) — Downloads pre-built binaries and runs CLI behavior smoke tests via Bun: test-cli.ts (options across all apps), test-cli-lexer.ts (numeric-separator rejection), test-cli-parser.ts (error display), test-cli-config.ts (config-file loading and per-file inheritance), and test-cli-apps.ts (app-specific features, including GocciaScriptLoaderBare stdin/file checks, CLI-local print, and runtime-global absence). The x86-64 Linux leg also runs the pinned es-toolkit compatibility probes and uploads their classified JSON report. Windows runs additionally assert that the loader binary does not link against OpenSSL DLLs.

artifacts (needs test + toml-compliance + json5-compliance + awfy + benchmark + cli, main only) — Uploads release binaries after all checks pass. test262 is not a gating dependency — failing tests there cannot block a release.

release (needs test + toml-compliance + json5-compliance + awfy + benchmark + cli, tags only) — Packages and publishes release archives after the same gates pass.

The test, awfy, benchmark, cli, toml-compliance, json5-compliance, and test262 jobs run in parallel after build. The AWFY, TOML, JSON5, and test262 lanes all reuse the already-built binaries from the matrix build artifacts instead of installing FPC again, so platform-specific issues are caught on the same Linux, macOS, and Windows targets as the main test lanes (AWFY and test262 run on Linux only since those outcomes are platform-independent for this engine).

PR Workflow (.github/workflows/pr.yml)

Runs on pull requests targeting main, on ubuntu-latest x64 only:

build → test
      → benchmark → benchmark / timing comments
      → test262   → test262 comment
      → awfy      → AWFY Results comment
      → cli

The PR workflow also runs the pinned es-toolkit compatibility probes in the Linux cli job and retains the classified JSON report. It posts a Suite Timing comment with expandable test-runner and benchmark summaries. Each summary shows timing, top-level GocciaScript GC metrics, and selected FreePascal heap allocation metrics for interpreter mode and bytecode mode. GC memory rows aggregate the main thread plus all worker thread-local GCs. The test runner does not count worker shutdown reclamation as GC collections, while the benchmark runner explicitly collects between benchmark files, so collection counts are expected to differ. The comment hides negative FreePascal heap free-space deltas because they are valid allocator diagnostics but are noisy in a PR summary. See benchmarks.md for details on the benchmark comparison format.

The AWFY Results comment posts median timings and geomean ratios for the pinned AWFY report set under Goccia bytecode, QuickJS, and the latest Node Current release. The underlying awfy-report artifact keeps the five raw interleaved samples per engine, min/max/CV, environment metadata, and first-class failure outcomes.

The test262 conformance comment posts a non-blocking summary using the actions/github-script@v7 pattern, marker <!-- test262-results -->. The markdown is generated by bun scripts/run_test262_suite.ts --comment <results.json> <baseline.json|->. It contains:

• A per-category breakdown table (run / passed / Δ pass / failed / timeouts / Δ timeout / wrapper-infra / pass-rate / Δ rate) for each of built-ins, harness, intl402, language, staging, plus a totals row.
• A wrapper-infra-failure top-line warning when non-zero (gates the run as untrustworthy).
• An Areas closest to 100% table listing the three test262 directories (keyed by the first two path components, e.g. built-ins/Math, language/expressions) with the highest pass rate, filtered to areas with at least 25 attempted tests and below 100%.
• When a main baseline is cached, Δ vs main columns show absolute count delta and percentage-point delta. Deltas below ±0.05pp render as ±0pp to keep the table readable.
• A collapsible Per-test deltas section listing newly-passing, newly-failing, newly-timed-out, and no-longer-timed-out test IDs.

The website compatibility dashboard at /compatibility reads main-branch test262 reports from Vercel Blob at request time with CDN caching. Main CI publishes future dashboard points directly, and cd website && bun run backfill-test262 can seed historical days from retained artifacts or fresh historical reruns. Website builds do not download artifact ZIPs or bake test262 data into the deployment. This keeps the public pass-rate timeline and group coverage tables tied to generated CI data rather than hand-maintained percentages.

Weekly crons open (or update) a single PR every Monday with the latest upstream SHA so pins don't go stale:

Each workflow reuses a fixed branch (chore/<suite>-bump), so an unmerged PR is updated in place rather than replaced. See test262.md for details on the test262 harness contract.

Running test262 locally

./build.pas loaderbare  # build the binary the runner needs
bun scripts/run_test262_suite.ts --filter "built-ins/Array/*" --output=results.json

The runner clones test262 into a tempdir on first use, or accepts an existing checkout via --suite-dir. See bun scripts/run_test262_suite.ts --help for the full option set, including --categories, --max-tests, --mode={interpreted,bytecode}, --jobs, --timeout-ms, and --max-memory.

Coverage

The GocciaTestRunner and GocciaScriptLoader support JavaScript source-level coverage reporting via the --coverage flag. Coverage tracks which lines, branches, and functions of JavaScript source code are executed at runtime.

Coverage implies --mode=bytecode

Enabling coverage — via --coverage, --coverage-format, or --coverage-output — switches the engine to bytecode mode automatically, overriding any --mode from the command line or a config file, exactly as --profile does. The implication is silent: --mode=interpreted --coverage is not an error, it simply produces the bytecode report, and no flag combination yields an interpreter-mode one. The reason is that the interpreter's coverage path is structurally incomplete and unmaintained — it instruments only the entry file (imported modules run with coverage disabled, so their lines, branches, and functions never appear) and counts a statement hit per executed AST node rather than per executed source line, so its counts are not comparable to bytecode's. Coverage therefore always reports through the bytecode path, over the entry file and every module it imports. The switch is applied in each application's Validate (TTestRunnerApp.Validate, TScriptLoaderApp.Validate), after config-file application and before the tracker is initialized.

Usage

# Console summary (printed after execution)
./build/GocciaTestRunner tests --coverage
​
# lcov output (for Codecov, Coveralls, or genhtml)
./build/GocciaTestRunner tests --coverage --coverage-format=lcov --coverage-output=coverage.lcov
​
# JSON output (Istanbul-compatible)
./build/GocciaTestRunner tests --coverage --coverage-format=json --coverage-output=coverage.json
​
# GocciaScriptLoader also supports coverage
./build/GocciaScriptLoader example.js --coverage

What is Tracked

Line coverage: Which source lines were executed and how many times. Instrumented in the bytecode VM's main dispatch loop, with per-frame line-change deduplication so several instructions on one source line count as one hit.

Branch coverage: Which branch arms were taken at:

• if/else statements (branch 0 = consequent, branch 1 = alternate)
• Ternary expressions (? :)
• Short-circuit operators (&&, ||, ??)
• switch statement case clauses

Function coverage: Which user-defined functions were created and called. A function is registered when its definition is evaluated and is reported at its declaration line (what LCOV's FN: means), not at the first line of its body; created-but-uncalled functions are retained with zero hits, and repeated calls increment the function hit count. Definitions inside untaken control flow are never created and therefore do not appear.

Parallel runs: --jobs=N does not change any of these numbers. Each worker collects into its own tracker and the counts are added together at the end, so line, branch, and function hit counts for a given run are identical whatever --jobs is set to.

Output Formats

Report Path Keys

Every file gets exactly one record per run, under one canonical path, however it was reached. A file can enter a run by two routes at once — named on the command line, and imported by another file — and each route knows it by a different spelling: the command line supplies whatever the user typed, import resolution produces an absolute path. Keying by the incoming spelling used to split such a file into two records whose hits were never added together.

The canonical form, applied uniformly to the console summary, the lcov SF: records, and the JSON keys and "path" fields:

• repo-relative when the file is under a repository root — the nearest ancestor directory holding a .git entry (a directory in a normal clone, a file in a linked worktree or submodule);
• absolute when the file is outside any repository;
• / as the separator, on every platform, including Windows.

Repo-relative is what the consumers want: Codecov matches report paths against repository paths, so a build machine's absolute paths need a fixes mapping while repo-relative paths match directly, and genhtml resolves relative paths against its working directory. Forward slashes matter because imports resolve through FPC's ExpandFileName — on Windows the raw paths carry backslashes, which genhtml and Codecov commonly mishandle in an SF: record.

Canonicalization is textual (ExpandFileName plus the repo-root trim), so it does not resolve symlinks: two spellings that differ by a symlinked ancestor — macOS's /var → /private/var, say — still produce separate records. Reaching the same file by two genuinely different real paths is not something normal entry-plus-import runs do.

Paths with no on-disk backing are identities, not files — <stdin>, and multifile section names such as <stdin>[part1] — and are left verbatim. Report keys are never used to read source: the tracker remembers the native on-disk path behind each key, so reports still render source when the process runs from outside the repository.

JSX Source Map Integration

When coverage is used with .jsx/.tsx files, the JSX transformer produces a source map that maps transformed (post-JSX) coordinates back to the original source. The coverage system integrates this source map so that lcov and JSON reports reference original JSX source positions, not transformed positions. This ensures downstream tools (Codecov, genhtml, Istanbul) display accurate line and branch locations.

Architecture

Coverage uses a runtime boolean check (FCoverageEnabled on TGocciaVM). When --coverage is not passed, the boolean is False and branch prediction makes the check effectively free — no separate build is needed.

Data is collected by TGocciaCoverageTracker (Goccia.Coverage.pas), a per-thread tracker that follows the same Initialize/Shutdown pattern as TGarbageCollector and TGocciaCallStack. During parallel test runs each worker thread initializes its own thread-local instance; after all workers complete, TGocciaThreadPool.MergeCoverageInto merges their data into the main thread's tracker via TGocciaCoverageTracker.MergeFrom, which adds the source hit counts into the destination (via AddLineHits / AddBranchHits) rather than registering a single hit per covered entry. Output formatting is in Goccia.Coverage.Report.pas. A JSX source map is registered with TGocciaCoverageTracker during file registration and applied only at report generation time, so the recording hot path is unaffected: BuildTranslatedLineHits translates transformed line hits back to original coordinates, and branch positions are translated via TGocciaSourceMap.Translate during report emission.