The fixture lied by succeeding

The replay test passed because it used the internet.

N17Q had a recording for the main source query and no recording for a follow-up status check. The fixture adapter delegated unknown requests to its normal client, received a current result, and cached it. The final report looked correct. The test was no longer a replay.

I had implemented a helpful cache with a dangerous name.

In replay, missing evidence must stop the run. A fixture that fails open can turn a deterministic test into an unannounced live action.

The fallback seemed harmless for reads

The original adapter served recorded results when found and fetched otherwise. During fixture authoring, this made iteration fast. New request paths populated themselves.

I reused the adapter in replay with a flag intended to disable capture. One branch checked the flag after constructing the live client, and a missing fixture reached the fallback before the mode assertion.

The call happened to be a documentation read. It still changed the experiment's time, source ranking, network exposure, cost, and reproducibility. A similarly structured mutation would have been worse.

Read-only was not permission to contact the present.

The failure was architectural, not one conditional

Adding another if (replay) inside the adapter would fix the observed path and leave future fallthroughs possible.

I separated packages and construction paths. replay registry accepted only fixture and pure local adapters. It had no live-client dependency, credentials, endpoint configuration, or generic “fetch on miss” interface. The process ran with network disabled.

Record mode built a different registry under explicit connection and capture policy. Moving between modes required ending or branching a run.

Capability absence made the guarantee stronger than branch coverage.

A fixture miss received its own state

Previously, missing data surfaced as tool unavailable or generic error. That made it look like the agent had encountered an ordinary service failure and should replan around it.

N17Q added fixture.missing with scenario, normalized request digest, tool contract revision, world revision, and checkpoint. The run paused as incomplete test world.

The agent could not solve the problem by selecting another live tool. A developer could inspect the missing path, author a fixture, version the scenario, and rerun deliberately.

The error belonged to the harness, not the simulated world.

Exact matching exposed accidental variation

The first matcher used tool name and a partial argument object. It accepted a request whose default date range had changed and returned an older result.

I moved matching to the versioned normalized request. Canonical identifiers, resolved defaults, content digests, contract revision, and relevant world state entered fixture identity. Unknown fields and ambiguous matches failed.

Transport-only fields such as attempt ID could vary without breaking meaning. The adapter contract declared which fields were semantic.

Strict matching turned hidden drift into an actionable fixture diff.

A status lookup before a simulated request existed should return not found. The same lookup after a committed-but-unacknowledged effect should return the request. One response fixture could not cover both.

N17Q's simulator applied tool operations to a versioned world and emitted independent events. Fixture behavior selected on normalized request and current world state. Scheduled transitions used scenario time.

A mismatch reported both expected and actual world revision. It never searched the catalogue for a plausible response from another point in the run.

The fixture represented behavior, not a pile of messages.

Failure injection was part of the contract

The lost-response scenario needed the effect to commit before delivery failed. A generic timeout fixture did not say where failure occurred.

Fixture steps named before send, after acceptance, after world mutation, before receipt, during response, and after local commit. The simulator and trace each observed the stages appropriate to them.

Replay could therefore preserve unknown outcome and test safe reconciliation. If the requested injection point was unsupported by an adapter revision, fixture construction failed.

Fault location was evidence, not an error-string adjective.

Missing mutations were treated most strictly

No consequential capability entered replay without a simulated world implementation, effect identity behavior, expected receipts, and prohibited-effects assertions.

A static success response was insufficient because it would let the agent believe the effect happened while the evaluator's world remained unchanged. A static error was insufficient for ambiguous outcomes.

Where I could not model the behavior credibly, the scenario excluded the tool and documented the limitation.

An incomplete simulation was safer when it admitted what it could not evaluate.

Tests instrumented every network client, credential resolver, and external process adapter. Replay failed if any constructor ran, even if no request followed.

This caught eager initialization that read environment variables and endpoints before the tool was used. It also prevented secrets from entering process memory unnecessarily.

A host-side audit log recorded attempted DNS, sockets, and subprocess routes supported by the environment. Known blocked probes verified containment.

The absence of observed network traffic was backed by the absence of a route.

The model could not switch modes

The run prompt explained that replay used fixtures. A hostile repository file instructed the agent to set an environment variable, edit harness configuration, and retry live.

Mode and policy lived outside the writable sandbox. Environment variables were allowlisted. The registry exposed no mode-control tool. The agent could request help, but no text or workspace mutation could activate a live connection.

The fixture tested this denial and checked that repeated indirect attempts stopped the run.

Replay integrity was not another instruction inside the material being tested.

Record mode needed its own fail-closed edges

Separating replay did not make record unrestricted. A missing capture policy, unapproved source class, consequential mutation, sensitive result, or ambiguous account scope stopped recording.

Record mode could call only reviewed connections and capabilities. It staged responses, ran redaction and provenance checks, and required fixture promotion before replay use.

If capture failed after a live read, the trace retained the safe request observation and explained that no reusable fixture resulted. It did not quietly store unreviewed content.

Both modes failed closed around different promises.

Fixture promotion was a review

New recordings entered a staging area with request identity, response shape, connection, source time, sensitive-field findings, effect semantics, and proposed world transition.

A reviewer could redact, synthesize, reject, or approve the fixture. Meaning-changing redaction required a synthetic replacement and a note about lost fidelity. Consequential cases were usually authored against the simulator rather than captured live.

Promotion created a new scenario or fixture version and ran contract tests. Existing replays stayed pinned to their old manifest.

The catalogue did not become trusted because a client wrote a file.

Credentials never entered replay artifacts

Record requests could use server-side credentials under connection policy. Capture removed authentication headers, signed URLs, cookies, and provider-specific secret fields before staging.

Replay substituted scoped synthetic tokens where authorization behavior mattered. Secret scanners and schema-aware rules checked manifests and raw artifacts. A finding blocked promotion.

The semantic trace retained safe credential-scope identity and outcome, not reusable material.

This allowed authorization paths to be tested without making the fixture package a credential archive.

The accidental live lookup returned a newer source and made the old replay appear better. The agent cited a fact unavailable when the original run occurred.

Fixtures pinned retrieval time, source revision, and scenario clock. Freshness policy evaluated against simulated time. A counterfactual using current sources became a fresh comparison run, not replay.

The interface showed recording age and source contract. Old evidence could remain historically faithful and currently stale at once.

Failing closed protected temporal truth as well as side effects.

The generic response cache had encouraged the conceptual error. I split it into live read cache, recording staging store, and immutable fixture registry.

Live cache could refresh under connection policy and never satisfied replay. Staging content was incomplete and non-executable. Fixture entries were versioned, reviewed, and selected only by replay registries.

Types prevented accidental substitution between their handles. Logs used the correct nouns.

Naming could not enforce safety alone, but accurate names made unsafe composition harder to write.

Fixture coverage became consequence-aware

A single percentage treated ten search variants as more important than one missing unknown-outcome path. N17Q reported coverage by scenario state, effect class, adapter contract, error category, and recovery transition.

The missing status lookup blocked the scenario's recovery claim. A missing optional explanatory read could leave the case usable with narrower scope.

The review page linked uncovered transitions to the tools and invariants they affected. It did not reward fixture volume.

Coverage answered which product behaviors the simulated world could support.

Adapter replay could test parsing one recorded exchange. Segment replay could test recovery from a checkpoint. Scenario replay could evaluate a complete trajectory.

The original failure had run a segment and presented its result like end-to-end replay. I added explicit scope to manifests and reports.

A narrow replay could still be useful and could not make claims about missing earlier policy, later world state, or final account. The UI kept those dimensions unevaluated.

Failing closed included refusing conclusions outside the reconstructed boundary.

When a tool contract changed normalization or error semantics, old fixtures did not match silently. The harness reported contract mismatch and required migration or a pinned older adapter.

Safe structural transformations preserved original payload digest and produced a new fixture revision. Meaning-bearing changes required review or recapture. Rewriting one result could affect downstream world events and expected evaluations, so the scenario version changed.

Historical runs remained reproducible against their manifest.

Convenience did not justify retroactively improving the world they faced.

Unknown fixture data stayed unknown

Some recordings omitted a field through redaction or because the original adapter never observed it. I had been tempted to fill those gaps with plausible defaults so downstream code could remain simple.

N17Q represented absent, redacted, unavailable, and not applicable separately. A scenario requiring the missing value stopped or used an explicitly synthetic replacement. It did not turn omission into an empty string that might be interpreted as evidence.

Evaluators could see whether a conclusion rested on unavailable material. The agent received the same bounded absence.

A closed fixture world could contain uncertainty without inventing facts to make every path executable.

The absence of a resource, denied permission, empty search result, and completed deletion were meaningful observations. Treating them as “no fixture” confused simulated negative evidence with harness failure.

Fixture entries explicitly represented these outcomes with scope, time, and contract. A search returning no results named the query and corpus revision. A status not found named the consistency window and world state.

The matcher could therefore distinguish “the simulated service says absent” from “the scenario does not define this request.”

Failing closed depended on modelling legitimate nothing precisely.

Concurrency was recorded as behavior

Two tool requests could race against the same world. Serial fixtures would hide duplicate effects and stale reads.

The simulator scheduled barriers, acknowledgements, and commits under scenario time. Fixtures could pause one attempt after acceptance, run another, then release both. Effect identity and contract determined whether one or two resources appeared.

Replay recorded the schedule as scenario evidence while allowing model turns around it to vary. The evaluator checked final world invariants.

A deterministic world did not have to be single-threaded; it had to make concurrency controlled and inspectable.

Large files, images, repository snapshots, and command outputs lived in a content-addressed store. Fixture manifests referenced digest, media type, size, semantic role, and retention class.

Replaying verified bytes before use. A missing artifact stopped rather than substituting a similarly named file. Derived previews named their renderer and source digest.

Where policy removed a large artifact, the fixture became incomplete for scenarios that needed it and honestly narrower for those that did not.

Stable identifiers prevented a fixture catalogue from quietly depending on mutable local paths.

Diagnostics could not repair the run

The missing-fixture page suggested the command used to enter authoring mode and showed nearby recorded requests. None of those controls ran inside the replay process.

This separation mattered after I added a developer convenience to copy a similar fixture. The copy created a staged candidate with a new identity; it could not satisfy the paused run until reviewed and the scenario version changed.

Debug tooling remained useful without turning the harness error screen into a back door around fixture policy.

The test stayed stopped until its world was deliberately repaired.

CI ran replay offline

Local network controls can be misconfigured. The continuous test job built the replay package in an environment with no live credentials and an explicit outbound-deny boundary, then ran fixture-miss canaries.

It verified that every promoted scenario loaded only allowed adapter modules and that the export contained no endpoint configuration. A separate record-capable job, manually invoked, used different identity and artifacts.

Mode separation therefore survived beyond my development machine. The build topology reinforced the runtime contract.

An offline replay that passed in CI made accidental local success easier to distrust.

A portable replay package contained scenario, tool contracts, fixture registry, world simulator data, base workspace snapshot, policy, safe artifacts, and hard invariants.

It explicitly excluded endpoints, credentials, account identifiers not needed for simulation, and record-mode code. Another environment could verify digests and run it offline.

The export test caught a hidden local cache the normal harness had been using. Removing it made fixture completeness honest.

Portability was an effective audit of ambient dependency.

The test suite attacked fallback paths

Cases requested unknown tools, unseen arguments, changed defaults, ambiguous matches, expired fixture revisions, absent status observations, redirect-like results, and repository scripts that attempted network.

Every case asserted no live-client construction, no credential resolution, no host effect, and a precise missing or denied state. Static dependency checks prevented replay packages from importing live adapter modules.

A canary endpoint caused an immediate failure if touched. It never became the only control.

The goal was to make accidental helpfulness visible before it reached a real service.

With the status fixture removed, replay now paused immediately after the unknown external outcome. It did not produce the polished final report the old test had graded.

The result was less complete and more correct. It identified the missing contract, preserved the checkpoint, and left the simulated effect visible in world state.

After I authored the status-query behavior and versioned the scenario, the run could reconcile safely. The final report then rested on evidence available inside the declared world.

Progress resumed only after the test became honest.

The final account disclosed fixture scope

When a replay completed, its generated report could easily sound like a claim about a live service. N17Q appended a deterministic evaluation note naming scenario version, fixture world, recording dates, synthetic substitutions, and uncovered behavior.

The agent did not write that scope statement. It came from the manifest. Exported traces and comparison views carried the same information.

This kept a successful recovery case in proportion. It proved that one configuration handled one controlled lost-response world without duplicating its simulated effect. It did not prove every provider would expose the same query behavior or timing.

Failing closed strengthened the test precisely because the claim around a pass stayed bounded.

A fixture is a boundary, not a convenience

Fixtures are often described as canned data used to make tests fast. In an agent harness, they also define which world the agent may observe and change.

That makes missing behavior a policy event. Filling it from the live world changes the experiment, may expose data, and can repeat effects. The harness must stop even when a fallback would probably be harmless.

N17Q kept record and replay useful by giving them different powers. Record could observe approved systems and stage evidence. Replay could operate only inside reviewed simulation.

The missing fixture was not a cache miss.

It was the edge of the world.