Green benchmark, broken workflow

The N17Q coding scenario passed every benchmark I had written and opened the wrong simulated review request.

The model found the relevant files. Its tool arguments were valid. The patch fixed the synthetic issue. Focused tests passed. The final explanation described the diff accurately.

During the complete run, a broader validation command timed out after reporting a failure. Recovery created a second workspace from an earlier checkpoint. The final request used that second workspace's patch and the first workspace's green focused-test receipt. The failed broader test belonged to neither final claim in a way the policy understood.

Every component had done something locally reasonable. The workflow had assembled their evidence into a false permission to publish.

The scenario lived entirely in a fixture world

N17Q used a small synthetic repository, deterministic source library, local sandbox, fake review-request service, and no real external account. The task resembled ordinary maintenance: verify a dependency migration, change an adapter, update tests, and prepare one reviewable request.

The repository contained an unrelated local edit, a misleading old document, a broad test that failed only after several successful phases, and a request service capable of losing acknowledgement.

That controlled world let the harness know exactly what changed and inject failure at a precise point. The result was not evidence about a real employer or production agent.

It was evidence that my architecture could lose lineage under realistic pressure.

I had several test families:

• Question-and-answer cases for migration facts.
• Tool-selection cases with expected capability and arguments.
• Patch fixtures with expected changed files.
• Adapter contract tests for command and repository tools.
• Focused test assertions against the resulting tree.
• A model grader for final explanation quality.

They caught prompt regressions and malformed calls. They were quick enough to run often.

They also assumed that context, workspace, evidence, retry, policy, and external action would remain correctly connected between the pieces.

The assumption was exactly where the failure lived.

The task had more state than the prompt

One run involved task revision, base repository, initial local overlay, workspace instance, plan, model turns, files read, patch operations, command attempts, test receipts, policy decisions, approvals, external-effect intent, and final account.

The prompt described the goal. It could not serve as the authoritative store for all those changing relationships.

N17Q's first orchestrator kept several of them as convenient fields on a run object. Workspace and evidence records lacked stable lineage constraints.

When recovery branched, the fields still looked individually valid and no longer referred to one coherent world.

The broader test command produced setup success, several passing suites, one failing integration assertion, and then exceeded its time budget while another process remained active.

The command adapter returned timeout. The orchestration layer stored the bounded output as diagnostics and did not classify the failing chunk as admitted validation evidence. It also failed to prove the process tree had terminated cleanly.

Recovery treated timeout as if the command had never produced useful or dangerous state. It created a fresh workspace from the latest clean checkpoint.

The system had converted partial, uncertain execution into a blank square in its story.

A second workspace changed the lineage

The new workspace began from the same base commit and reapplied the accepted patch. Its content happened to match the first workspace at that moment.

It received a new workspace identity, overlay identity, command history, and tree digest. The earlier focused test had run in workspace one. The failed broad test had also run there. Workspace two had run no validation yet.

The policy screen displayed the patch digest and a list of green checks selected by task. It did not require check receipt and target tree to share lineage.

Byte equality at one snapshot hid a different evidence history.

The fake review-request adapter received one branch artifact from workspace two. Its schema passed. The artifact digest existed. The destination and title were correct. The service created exactly one request.

There was no duplicate and no malformed remote call. The wrongness was in the claim that this artifact had satisfied required validation.

This made the case distinct from N17Q's first duplicate-effect failure. Idempotency could not help. The external effect was singular and unauthorized by validly scoped evidence.

Reliability failed before the network boundary, in the composition of facts.

Component accuracy gave false reassurance

The model had selected the correct test command. The command tool had reported timeout accurately. The workspace service had restored the correct patch. The policy evaluator had received green focused-test evidence. The request adapter had delivered the approved artifact.

No one component owned the invariant that evidence used for release must belong to the exact artifact lineage being released and remain valid under required test scope.

The benchmark score was not wrong about its components. It was incomplete about their composition.

High local accuracy could not substitute for a missing system invariant.

Before changing code, I stated what had to be true:

An external review request may reference one immutable repository artifact only when every required validation receipt was produced from that artifact or an explicitly compatible ancestor, no later workspace event invalidated it, broader failures are resolved or deliberately waived, current policy allows the effect, and approval binds the exact evidence set.

That sentence was longer than “tests pass” because the product promise was longer.

I turned each relationship into data and executable checks rather than placing the sentence in a prompt.

Every workspace received stable identity, immutable base digest, initial-overlay digest, current overlay revision, tree digest, environment revision, and lifecycle state.

File operations and commands named the workspace and input tree. A command receipt named pre-run and post-run tree, environment, exit state, termination evidence, and bounded outputs.

Copying or reconstructing a workspace created a lineage event. Content equality could be recorded without collapsing identities.

The harness could now ask where evidence came from instead of inferring it from a nearby patch digest.

Evidence admission became a decision

Tool output did not automatically qualify as evidence for policy. N17Q created an evidence observation, then admitted it for a named purpose when its scope and lineage were valid.

A focused test receipt could support one validation requirement for the exact tree it ran against. Partial output from a timed-out broad test could support diagnosis and could not satisfy the completed broad-test gate.

Admission stored policy, artifact, workspace lineage, scope, freshness, and decision. Superseding a workspace changed eligibility without deleting historical observations.

Evidence could remain useful and not authorize the next action.

The interface previously displayed test names and pass badges. It now showed command, selected suites, repository tree, environment, completion, failures, skipped work, timeout stage, and receipt.

Required checks came from scenario and current policy. A focused pass and broad unresolved result could not be averaged into “mostly green.” A reviewer could explicitly waive a known failure only through a separate bounded decision with rationale.

The model's final answer used the same receipt vocabulary. “All tests pass” failed validation when only a subset had completed.

Green became evidence with a domain, not a color.

The command adapter distinguished not started, started, partial output, completed, canceled with confirmation, termination requested, termination unknown, and harness interrupted.

The workspace recorded possible side effects such as changed files and surviving processes. An uncertain environment could be quarantined and inspected, not reused automatically.

Partial failure entered the trace with exact source. Recovery policy decided whether a fresh workspace was required and which observations remained relevant.

Timeout no longer erased the activity that occurred before the clock expired.

Retry preserved task and changed attempt

The high-level maintenance task kept one identity. Recovery created a new attempt and workspace lineage. It did not copy validation admission wholesale.

Patch operations could be reapplied and produced a new artifact receipt. Sources and reasoning evidence could remain relevant if their revisions and scope still held. Command evidence required compatibility rules and usually rerun.

The checkpoint compiler explained what carried forward and what became stale.

Retry meant continue the intention under new state, not pretend nothing had changed.

The prepared review object included branch artifact digest, target base, workspace lineage, required validation receipts, unresolved failures, policy revision, adapter mapping, effect identity, destination, and expiry.

The UI rendered from that canonical object. A changed test, patch, workspace, or policy produced a new digest and invalidated approval.

The reviewer could open each check and see its tree. The model rationale appeared separately. Confirmation copy came from deterministic product state.

Human review could now inspect the relationship the benchmark had missed.

The final policy gate repeated the invariant

Immediately before invoking the fake request service, N17Q re-established requester authority, approval, artifact and workspace state, validation lineage, budgets, tool mapping, prior effects, and destination policy.

The gate ran inside the local transaction that activated one external-effect intent and wrote the outbox. It did not trust the queue's earlier allow.

A conditional remote request and idempotency identity handled the boundary afterward.

Planning policy reduced wasted work. Final policy protected the world that would actually receive it.

The original trace had model messages and tool JSON in time order. Finding the failure required reading hundreds of entries and manually connecting IDs.

N17Q added semantic events for attempt, workspace, artifact, evidence observation, evidence admission, policy, approval, effect intent, execution attempt, world observation, checkpoint, and run state.

Raw provider and command events remained linked. Causal edges answered which evidence authorized which action.

The incorrect cross-workspace relationship became a visible broken edge instead of an intuition after log reading.

The simulated world stayed independent

Repository content lived in content-addressed sandbox state. The fake request service kept its own event log and current resources. The agent saw only allowed tool observations.

Evaluation compared final tree and remote resource with scenario invariants. A generated summary could not redefine either.

In this case, the world contained one correctly formed request pointing to an artifact whose broad validation state was unresolved. That was enough to fail.

Independent ground truth found composition errors even when every trace message sounded plausible.

The report said the patch passed tests and the review request was ready. N17Q extracted operational claims and linked them to receipts.

Focused tests supported a narrower claim. The broad command contradicted completion. The external effect existed, but policy should not have allowed it.

The answer failed account accuracy independently of the world-state failure. A corrected harness-generated appendix described exact validation and consequence.

Good prose could not close an unresolved workflow.

The scenario definition grew richer

The revised benchmark specified initial repository and overlay, applicable instructions, tool registry, environment, policy, budgets, injected timeout point, required validation, allowed effects, prohibited evidence crossover, success invariants, and acceptable stop states.

It did not prescribe one exact model trajectory. The agent could diagnose first, patch differently, or ask for review. It had to preserve lineage and authority.

The scenario became an executable world around the task, not a prompt with an expected string.

This made the benchmark slower and substantially more meaningful.

Deterministic evaluators checked file scope, workspace and evidence lineage, required test receipts, policy, approval, effect count, world resources, budgets, and final status.

Model-based reviewers assessed code clarity, diagnostic reasoning, report usefulness, and whether the chosen path was unnecessarily wasteful. They cited events and artifacts.

A qualitative high score could not override a cross-workspace evidence violation. A hard pass could coexist with poor explanation.

Keeping dimensions separate stopped one appealing average from hiding the exact failure.

Replay reconstructed the fault without external action

The scenario package included base repository, initial overlay, command fixture, timed output sequence, sandbox profile, policy, and fake request service. Replay had no live network or credentials.

The timeout occurred after the same failing chunk. A new model could choose another path. The simulated request remained isolated to the branch world.

Missing fixtures failed closed. The run never opened a real pull request or contacted a package registry.

The workflow could be examined repeatedly without re-enacting consequences outside N17Q.

One mode fed the original recorded decisions and observations through the corrected projections to verify the UI and evaluator found the invalid lineage. Another started a new agent from the pre-timeout checkpoint.

The first reproduced the diagnosis. The second tested whether new policy and context led to rerunning validation or stopping.

I kept their results separate. Replaying recorded decisions was not evidence that a new agent would choose them. A better branch did not change original history.

Both modes answered useful and different questions.

The fix worked without a better model

Using the same model outputs from the original trace, the corrected final gate denied the request because no completed broad-validation receipt matched workspace two's artifact lineage.

The run returned to planning with a bounded reason. A new attempt ran the broad suite in workspace two and observed the known failure. The agent could fix it or stop with a reviewable patch and unresolved validation.

The world remained without a request until policy had valid evidence.

Containment did not depend on the model recognizing its earlier assumption.

I updated the context to state workspace identity, exact test scope, partial timeout evidence, and which checks required rerun. The model became less likely to claim completion and more likely to investigate the failure.

That improved trajectory quality and reduced denials. It did not remove the final gate or evidence schema.

Prompt and product state solved different parts of the problem. One helped the agent plan correctly. The other prevented an incorrect plan from becoming consequence.

The benchmark evaluated both.

Attempt cleanup became explicit

Workspace one had an uncertain command termination. The original recovery abandoned it without a verified teardown.

N17Q quarantined the overlay, inspected process state under sandbox guarantees, retained bounded diagnostics, revoked ephemeral resources, and produced a cleanup receipt. Unknown teardown prevented environment reuse.

The new workspace did not inherit caches or processes from the first unless a trusted immutable dependency layer allowed it.

Failure recovery included cleaning up the world, not only restarting thought.

Both workspaces began with the same deliberate unrelated edit. Reapplying the patch and running formatting risked absorbing it into the agent artifact.

The scenario asserted preservation through initial-overlay identity and final diff. The review grouped task changes separately and flagged overlap.

The corrected run kept the unrelated edit out of the simulated request. A benchmark focused only on the target file would have missed that responsibility.

The workspace contained state the agent inherited and did not own.

Budgets exposed recovery cost

Creating a second workspace, reapplying a patch, and rerunning tests consumed compute, turns, and artifacts. The old benchmark counted only model tokens for the successful path.

N17Q charged cumulative resources across attempts and retained a recovery reserve. The comparison showed that one policy configuration prevented publication early but required another validation run.

That cost was visible beside safety and task completion. It did not justify skipping evidence.

Dependability included understanding the price of uncertainty.

Human approval stopped being a ceremonial button

In the original view, a reviewer saw a good patch and green badges. In the corrected view, they saw one focused pass on workspace one, one incomplete broad run with a failure, reconstructed workspace two with no current validation, and exact requested effect.

The correct choice became obvious: do not approve yet.

Interface quality mattered because the state machine could preserve evidence and still present it misleadingly. Accessibility tests ensured the same scope and failure were available without relying on color or a trace canvas.

Meaningful intervention required meaningful evidence presentation.

Repository tests were inside the writable workspace. An agent could weaken or delete them. N17Q kept hard scenario invariants outside the sandbox and restricted changes to selected validation configuration unless the task required them.

The evaluator used its own world snapshots and receipts. It did not ask the agent whether it had respected evidence lineage.

This kept the system under evaluation from controlling the entire definition of success.

Independent assertions were the benchmark's own authority boundary.

False positives remained possible

Strict lineage could reject reusable evidence that was genuinely valid for a content-identical reconstructed workspace. Rerunning a long suite might waste time.

N17Q allowed explicit compatibility rules based on immutable tree, environment, command, inputs, and test isolation. Reuse created an admission decision with its rationale. It was never inferred from a matching green label.

For the timed-out broad suite, no completed receipt existed to reuse. For a pure focused test with identical inputs, policy could accept the earlier result.

Safety did not require ignoring valid equivalence; it required proving it at the right layer.

The run review offered synchronized views of task attempts, workspaces, evidence, policy and approval, remote effect, and final account. Selecting the request showed every admission used to authorize it.

The broken lineage appeared as evidence from workspace one attached to artifact from workspace two. Raw logs remained available behind each event.

A reviewer could explain the failure without reading private model reasoning or every token.

Semantic observability turned a distributed narrative into an inspectable product object.

The release gate became the scenario

The failing case joined N17Q's permanent fixture suite. Changes to trace schema, workspace service, command adapter, policy, approval, context compiler, or provider adapter had to preserve its invariants.

Variants changed timeout stage, content equality, required test scope, approval timing, and model plan. Counterfactual branches tested new configurations without live effects.

The scenario was more valuable than another hundred isolated tool-selection examples because it connected them under stress.

One real project failure became durable engineering memory.

The model cases remained useful for selecting files, choosing tests, interpreting failures, and writing explanations. Improvements there reduced wasted work and made safe completion more likely.

I stopped asking them to certify the orchestrator. A model could perform perfectly within the information provided and still be embedded in a broken workflow. A robust workflow could contain an unsafe proposal and still produce a safe stop.

The evaluation layers answered different questions.

Keeping the model benchmark became easier once its claim was narrower.

Adapter fixtures verified normalization, error mapping, timeout stages, output bounds, sandbox behavior, idempotency, query, and replay. They did not prove evidence from one adapter attempt would be joined correctly to another product object.

The scenario consumed those contracts and tested their composition. A failure could point back to an adapter or reveal missing orchestration state.

Unit and contract tests made end-to-end diagnosis tractable. End-to-end tests kept unit success in proportion.

The stack needed all three levels.

The fixed run was allowed to stop

One counterfactual reached the broad failure and lacked enough budget to diagnose and patch it safely. N17Q stopped with the exact diff, focused-test receipt, broad-test evidence, workspace state, and no external request.

The final account said validation incomplete and named the failing suite. A person could continue from the checkpoint.

This run failed the nominal completion target and passed authority, evidence, and reporting requirements.

A trustworthy system must distinguish unfinished from incorrectly finished.

The breakthrough was the unit of evaluation

N17Q began as a collection of model and tool tests. The workspace incident forced the project to treat the whole trajectory as the object under evaluation.

The unit now included task and configuration, model decisions, context, tools, policy, approvals, attempts, workspaces, evidence, effects, checkpoints, budgets, world state, and final account.

That did not make evaluation omniscient. Fixtures simplified reality, sandbox claims remained bounded, and model variation left untested paths.

It made the known system boundary executable.

A workflow is where correctness becomes consequence

An answer can be correct. A tool call can be valid. A patch can solve the issue. A test can pass. A reviewer can click approve. None of those facts alone says the external action is justified.

The product must preserve the relationships between them through retries, timeouts, branch changes, provider sessions, and human decisions. It must know which evidence belongs to which artifact and which authority still applies.

N17Q caught its failure because the simulated world and semantic trace disagreed with the confident final story. The fix lived in identities, admission, state, policy, and review—not one better completion.

The benchmark had passed what it measured.

The workflow revealed what mattered.