Replayed run, duplicate email

The first replay looked perfect until a second email arrived.

N17Q had loaded a saved run, restored its messages, and sent the same tool results back through the model. Near the end, the workflow reached the delivery step and invoked the live adapter. From the agent's perspective, this was faithful reproduction. From the recipient's perspective, it was a duplicate consequence with a misleading timestamp.

The bug was not an idempotency oversight. Replay had never been defined as a product mode. It was treated as another way to start a run.

A replay should reconstruct what the system knew and test how it behaved. It should not reenact what the original run did to the world.

The transcript was not the run

My earliest run export contained model messages, tool calls, tool results, and the final answer. Feeding that sequence back into a model could reproduce the shape of the conversation, but not the state that made each event legitimate.

The export omitted the offered tool catalogue, workspace revision, policy decision, approval receipt, effect identity, adapter version, fixture clock, and external observations. It also flattened several rejected proposals because they had never become assistant-visible results.

This made the transcript deceptively portable. It looked complete precisely because the missing pieces were product state rather than prose.

N17Q began treating a run as an event graph with references to immutable artifacts and versioned contracts. The conversational projection remained useful for reading. Replay consumed the graph.

The word replay hid several different jobs.

An exact reconstruction should show what happened without asking a model to decide again. A deterministic rerun should execute the original versions against captured fixtures. A counterfactual run should change one declared variable, such as model or policy, while holding the rest stable. A regression scenario should turn a historical failure into a maintained test.

These modes had different truth claims. Reconstruction could be exact if the evidence was intact. A rerun could be comparable, but model sampling and changed dependencies might still diverge. A counterfactual was an experiment, not an alternate history.

The interface required the operator to choose the question before it assembled the environment. That single choice prevented a great deal of accidental ambiguity.

Execution mode lived outside the prompt

The first proposed fix added “do not send email during replay” to the system message. It was easy to understand and incapable of carrying the safety boundary.

A model could miss the instruction, a summary could omit it, a provider could produce an old call, or an adapter could be reached by a non-model path. The product needed to make live delivery impossible.

N17Q assigned every run an execution environment at creation: live-disabled fixture, isolated sandbox, or an explicitly provisioned effect-capable environment. Replay creation always selected fixture or sandbox. That selection was stored outside model-controlled content and verified by the tool router.

No prompt could upgrade the environment. No historical approval receipt could change it. The replay did not possess live delivery credentials in the first place.

Connections became recorded dependencies

The original email adapter depended on account configuration, templates, destination normalization, provider behavior, and current credentials. Replaying against the live connection imported today's world into yesterday's run.

N17Q replaced each external connection with a versioned fixture contract. The fixture described accepted request shape, normalization rules, response timing, idempotency behavior, bounded failure modes, and observable world state. It did not need to impersonate every provider feature. It needed to reproduce the aspects the workflow relied on.

Run checkpoints named exact fixture versions. If the required fixture was absent, replay failed closed with a useful inventory of missing evidence.

That was preferable to silently falling through to a live adapter or inventing a generic success response.

The simulated world started from a snapshot

Tool outputs alone could not model stateful effects. If the original run created a draft, queried it, updated it, and then sent it, each step depended on what earlier steps had changed.

The replay environment restored a small simulated world from a scenario snapshot. Mutation fixtures changed that world through the same product-facing contracts as real adapters. Read fixtures observed it. The harness recorded state transitions independently of the agent trace.

This separation mattered when an adapter claimed success but the simulated world did not change. The workflow could not pass merely because the expected result string appeared.

For the email case, the world contained mailboxes, draft objects, send events, and provider receipts. A send produced a simulated delivery. It never crossed the fixture boundary.

Historical tool results were evidence, not instructions

Exact reconstruction displayed captured results. Deterministic rerun asked fixture adapters to produce new results from the restored world. Mixing the two created impossible sequences.

N17Q labeled every result with origin: historical observation, fixture output, derived safe projection, or current live observation. Replay context admitted only origins allowed by the selected mode.

A captured provider message could contain instruction-like text, stale links, or sensitive material. The context compiler treated it as bounded data with original provenance. It did not grant the old result authority over the new run.

This origin labeling also made comparisons honest. A fixture response matching the historical response was a useful equivalence. The historical response itself was not proof that the replayed adapter behaved correctly.

Approvals did not survive as authority

The historical trace included a valid approval for sending the original email. That receipt established why the original effect was allowed. It did not authorize another effect months later.

In replay, approval records became read-only evidence. Policy could evaluate how the original decision would be interpreted, and the interface could render the exact review surface. The execution gate rejected them for live consequences because their run, environment, world, time, and intent identities did not match.

Fixture mutations could use a simulated approval state when the scenario required it. This state proved a branch in the test harness, not human consent in the current world.

The distinction was central: reproducing an authorization decision is not reusing authorization.

Effect identities received a replay namespace

The original intent identity linked approval, transport attempts, provider keys, and delivery receipt. Reusing it in a fixture risked collisions in logs and, worse, accidental routing through an adapter that recognized the live key.

N17Q generated a replay world identity and mapped historical effects into it. The fixture could preserve semantic lineage while using non-live intent and idempotency namespaces. A counterfactual branch received its own child world rather than sharing mutations with its parent.

Reports displayed both references: derived replay intent and historical source intent. The relationship was traceable without implying they were the same consequence.

Any adapter receiving a replay-namespaced key while configured for a live connection rejected the request. Defence did not depend on a single router decision.

Time had to be controlled

The original workflow depended on expiry, quiet hours, retry delays, and a provider visibility window. Running it with the current wall clock changed policy and recovery behavior.

Each replay used a logical clock seeded from the historical checkpoint or scenario definition. Advancing time required an explicit harness event. Tool fixtures and policy received that clock rather than reading process time directly.

Wall time still measured test performance, but it did not decide product semantics. A ten-minute retry could be advanced instantly while preserving event order. An expiry boundary could be tested on either side without waiting.

The report always showed logical and execution time separately. “Completed in 240 milliseconds” no longer implied that the simulated workflow ignored its ten-minute hold.

Randomness became an input

Model sampling was only one source of variation. Generated identifiers, retry jitter, ordering of concurrent results, and randomized fixture faults could all change the trajectory.

N17Q recorded deterministic seeds and scheduling choices where the harness controlled them. IDs used namespaced derivation. Concurrent fixture completions followed a declared schedule unless the scenario was explicitly testing races.

I did not pretend this made remote model output bit-for-bit reproducible. Provider implementation and model versions could change. Instead, the trace named which sources were controlled and which were observational.

Regression assertions focused on product invariants and allowed trajectories: no live effects, at most one simulated delivery, approval before send, bounded attempts, and a final account matching world state.

Missing fixtures were first-class failures

The tempting convenience was to return a neutral placeholder for an unimplemented tool. That allowed the model to continue and produced a pleasantly complete-looking replay.

It also meant the trajectory no longer represented the original capability. A missing status query could push the model to send again. A fake empty search could erase evidence from the context.

N17Q stopped at the first unavailable fixture with capability, contract revision, expected input digest, and the reason it was required. The operator could add the fixture, exclude the scenario, or choose a reconstruction-only view.

There was no “best effort” switch that silently changed semantics. Incomplete evidence was more useful when it remained visible.

Secret handling became simpler

Historical traces sometimes contained redacted credential references and safe destination handles. Replay did not need the underlying secrets, but an early adapter initialization path still tried to resolve them.

Fixture environments used a separate registry containing only simulated connections. Historical secret references stayed opaque evidence. Context projections showed safe labels and digests exactly as the original interface had.

The process running regression tests received no live provider credentials. Network policy denied external access. Even a routing defect would meet an empty connection registry and a lower-level boundary.

This was one of the strongest benefits of treating replay as an environment. Secret absence became a designed property rather than careful test etiquette.

Network denial was part of the product contract

A unit test could mock the email client while another dependency fetched a template, resolved a redirect, or called telemetry. “The adapter is fake” did not mean the run was isolated.

The replay sandbox denied network by default and exposed fixture services through local, named endpoints. Attempts to reach an undeclared address generated trace evidence and failed the scenario. DNS resolution and subprocess inheritance followed the same rule.

Some research scenarios required captured web material. That content entered as immutable source fixtures with provenance and retrieval date. The replay did not refresh it automatically.

The system could therefore say something concrete: this run had no route to the recipient, not merely that the main send method was stubbed.

The UI stopped saying run again

The old control offered Run again, which collapsed reproduction, retry, and repetition into one reassuring phrase.

N17Q replaced it with Inspect recorded run, Replay in fixtures, and Create counterfactual. Each option summarized model version, policy revision, workspace snapshot, fixture completeness, logical clock, and effect isolation before starting.

Live repetition was a separate workflow beginning from a new user request. It prepared current artifacts, current approvals, and new effect identities. The interface did not place it beside diagnostic replay as though they differed only in speed.

Language shaped expectations. A person investigating a failure should not need to wonder whether a diagnostic button can contact yesterday's recipient.

Comparison happened at several layers

Looking only at final answers made two very different runs appear equivalent. Looking only at event equality made harmless wording changes look catastrophic.

N17Q compared context inputs, model proposals, policy decisions, tool transitions, simulated world changes, budget use, and final claims. Each layer had appropriate tolerances. Text could receive qualitative review while hard invariants remained exact.

A counterfactual model might choose a different read sequence and still reach the same safe outcome with lower cost. Another might produce an identical email while skipping a required approval branch. The first was useful divergence; the second failed despite textual similarity.

The comparison report linked every claim to trace and world evidence. It did not compress the result into one replay score.

The historical run had paused while a person reviewed a destination and rejected an overly broad audience. Replaying only automated events skipped the decision that shaped everything afterward.

N17Q represented human inputs as typed scenario events with actor role, visible evidence, selected action, and logical time. A deterministic rerun could use the recorded choice. A counterfactual could replace it with another declared choice. Neither path impersonated a current person or created a reusable approval.

Free-form feedback remained bounded content attached to the event. The harness did not infer permission from its tone. If a scenario needed authorization state, it supplied a fixture receipt whose scope existed only in the simulated world.

This let the replay test the pause and response path instead of editing humans out of the system.

Retention preserved the failure, not every byte forever

Replay evidence could include source documents, model outputs, tool observations, and workspace snapshots. Keeping everything indefinitely would turn a testing feature into an uncontrolled archive.

N17Q assigned retention and sensitivity classes at capture. A regression scenario kept the minimal fixture needed to reproduce an invariant. Large or sensitive artifacts could be replaced by approved synthetic equivalents while retaining the historical trace's digest and deletion record. Reconstruction views disclosed when source material had expired.

The system never substituted a synthetic fixture and called the replay exact. It named the transformation and the claims still supported.

Durable testing required selective preservation with honest gaps, not permanent duplication of every input.

Replay could not repair the historical record

When a rerun produced cleaner evidence, it was tempting to attach that evidence to the original run and make the old account look complete.

N17Q kept history append-only. A replay could add a linked analysis, reproduce a receipt in simulation, or demonstrate a fix. It could not backfill an event as though it had occurred earlier. Corrections named their author, basis, and time.

This protected the most important diagnostic signal: what the original system actually knew when it claimed success. A later test might prove the artifact was valid, but it could not prove the original run had validated it.

Replay improved the future without laundering the past.

Once effects were isolated, the email scenario revealed that the original run had allowed delivery immediately after an ambiguous draft update. The live adapter had happened to return quickly, hiding the unsafe branch.

I introduced a fixture delay between update acceptance and visible state. The agent queried once, saw stale content, and proposed another update. Because both updates shared an improperly broad identity, the second overwrote evidence from the first.

The regression failed without sending anything real. Fixing it required narrower effect identities and a postcondition check before delivery, not another prompt sentence.

Replay had become valuable precisely because it reproduced system pressure without reproducing consequence.

The duplicate email became impossible by construction

In the repaired fixture, N17Q restored the historical workspace and logical clock, mapped tool contracts into the simulated registry, and ran with no live credentials or network. The model proposed delivery. Current fixture policy allowed a simulated intent, which produced one mailbox event in the replay world.

The historical approval remained visible and unusable. The original provider key never left the trace. The report compared the new trajectory with the recorded run and identified a different recovery sequence.

No real mailbox could receive the message because no path to one existed.

That guarantee did not come from asking the model to remember that it was in a test. It came from making replay a world with its own identity, state, clock, tools, policy, and effects.

Replaying a run should let the system make the same decisions, encounter the same uncertainty, and reveal a better branch.

It should never ask the outside world to live through the same consequence twice.

That restraint made replay more useful, not less realistic: failures could be repeated freely, compared carefully, and converted into durable tests without turning diagnosis into another source of harm.

The world remained protected while the system learned.