Skip to content

Scenario Lifecycle

This document describes the lifecycle contract for Screeps Lab scenarios. The 0.3.1 implementation established bounded synthetic scenarios and terminal reports. The 0.3.2 implementation composes the same lifecycle with external subject resolution, deterministic world bootstrap, private-server deployment, controlled ticks, observation, generic assertions, artifacts, and cleanup. The 0.3.4 implementation adds subject-owned declarative assertion manifests while preserving Lab-owned lifecycle and reporting semantics.

Implemented Lifecycle

  1. queued
  2. preparing
  3. deploying
  4. executing
  5. observing
  6. evaluating
  7. reporting
  8. cleanup
  9. terminal

The terminal invariant is mandatory: every scenario run writes a final machine-readable report and a concise human summary. This applies to normal passes, failed assertions, captured subject errors, timeouts, preparation failures, and cleanup/reporting partial failures.

Result Separation

Scenario reports separate the lab engine from the experiment outcome:

engine.result    completed | aborted
scenario.result  pass | fail | error | aborted | cancelled

A subject assertion failure is not an engine crash. A captured subject runtime error is classified as scenario.result: "error" without treating the engine as failed when the engine captured, classified, and reported it. A timeout produces engine.result: "aborted" and an explicit scenario terminal result.

Reports also include terminalClassification:

pass
assertion-failure
subject-error
timeout
preparation-error
deployment-error
orchestration-error
cleanup-error

Operator Commands

npm run scenario:run -- 0.3.1-pass
npm run scenario:run -- 0.3.1-assertion-fail
npm run scenario:run -- 0.3.1-subject-error
npm run scenario:run -- 0.3.1-timeout
npm run scenario:run -- external-subject-smoke --subject-path ../screeps --ticks 100
npm run scenario:run -- external-subject-assertion-fail --subject-path ../screeps
npm run scenario:smoke

Reports are written under artifacts/scenarios/runs/<run-id>/. The latest single-run pointer is artifacts/scenarios/latest; the latest representative suite index is artifacts/scenarios/smoke-latest/.

Exit codes are:

0   scenario passed, or representative smoke suite matched expected outcomes
1   scenario completed but scenario.result was not pass
2   engine aborted
3   unexpected engine/internal error
64  CLI usage error

External Subject Contract

External subject scenarios are typed JavaScript objects created through createExternalSubjectScenario. The supported contract includes:

{
  id: "external-subject-smoke",
  world: {
    strategy: "bootstrap",
    reset: true,
    room: "E48N13",
    rcl: 5,
    spawn: "Spawn1"
  },
  subject: {
    type: "external-screeps-repository",
    path: "../screeps",
    branch: "scenario-external-subject",
    buildCommand: "npm run build",
    entryModule: "main",
    output: "dist",
    assertionManifest: "screeps-lab/assertions/mmo-regression.json"
  },
  execution: {
    ticks: 100,
    timeoutMs: 120000
  },
  assertions: [
    { type: "no-subject-errors" },
    { type: "ticks-completed", expected: 100 },
    { type: "subject-executed" }
  ]
}

The scenario engine validates the contract before starting the server. The external subject adapter validates and builds the external repository before the world is mutated when practical. World preparation is invoked through the server adapter's bootstrap service during deploying, before subject code installation. Snapshot-backed scenarios use the same world.strategy boundary.

When subject.assertionManifest is supplied, the path resolves relative to the prepared external subject checkout. Absolute paths and paths that escape the subject root are rejected. The manifest must be JSON with schemaVersion: 1, an id, and unique assertion IDs. Screeps Lab records the resolved path, subject-relative path, checksum, byte size, assertion count, subject revision, branch, dirty-tree state, tracked/modified/untracked manifest state when Git is available, and validation result.

Subject assertions are declarative and evaluated by Screeps Lab. The initial catalog reads only safe source: "memory" dot paths and supports path-exists, path-absent, equals, not-equals, numeric-comparison, allowed-values, collection-size, changed, unchanged, increased, and decreased. Numeric operators are limited to >, >=, <, <=, ==, and !=. Paths are relative to the Memory root, missing paths are distinct from present null values, and __proto__, prototype, and constructor segments are rejected. The manifest never executes subject-provided code or runtime expressions.

Artifacts

Every started run receives an isolated artifacts/scenarios/runs/<run-id>/ directory. External subject runs write scenario-definition.json, lifecycle-events.json, environment.json, world-preparation.json, subject-provenance.json, subject-build.log, subject-deployment.json, observations.jsonl, server.log, assertions.json, subject-assertion-manifest.json, subject-assertion-provenance.json, subject-assertion-results.json, scenario-report.json, cleanup.json, and summary.txt.

assertions.json contains Lab-owned assertion results. subject-assertion-results.json contains detailed subject assertion evidence, including before and after values for temporal checks. scenario-report.json keeps compact summaries to avoid repeatedly embedding large Memory branches. summary.txt reports Lab assertions and subject assertions separately.

Accelerated execution is opt-in. The server adapter starts paused, observes the authoritative starting tick, advances bounded chunks, pauses at observation or persistence boundaries, rejects undershoot and overshoot, and performs a final persistence wait. Controlled execution retains its existing persisted-per-tick contract.

When enabled, scenarios read the player's actual Memory.stats through the server adapter at the start, configured intervals, checkpoint-adjacent boundaries, and the final tick. Samples stream to telemetry.jsonl; numeric leaves are projected deterministically into telemetry.csv and summarized in telemetry-summary.json. scenario-report.json and suite child records contain only compact summaries and artifact references. Performance is separately recorded in simulation-performance.json and timeline.jsonl.

The environment artifact is intentionally a whitelist of run metadata. It does not serialize process.env, credentials, tokens, or subject source contents.

Notes

Scenario execution should preserve enough state to explain failures. Assertions should be deterministic where possible, and any random input should be captured with seeds or source metadata.

The scenario engine should not encode assumptions about one colony bot. It should operate through subject adapters and Screeps server interfaces.

Implemented building blocks include private-server lifecycle and controlled ticks, snapshot sandbox import, automated owned-room bootstrap, synthetic subject smoke, external subject deployment, the 0.3.1 scenario runner, the 0.3.2 external subject scenario runner, the 0.3.3 snapshot-backed external subject scenario strategy, and the 0.3.4 subject-owned assertion contract.

Snapshot-backed scenarios validate the snapshot artifact during preparing, then perform destructive reset and snapshot import during deploying before subject code installation. The strategy reuses the reset-only server importer and records snapshot-provenance.json, snapshot-import.json, world-preparation.json, and the normal combined terminal report.

Deferred work includes a rich scenario DSL, executable subject assertions, arbitrary Game expressions, broader telemetry-baseline comparisons, exact live-creep continuity assertions, scenario report indexing, screenshots, replays, distributed workers, fuzzing, and adaptive scenarios.