Correct answer, damaged workspace

The N17Q answer grader gave the report a pass. The repository fixture did not.

The agent explained the requested change correctly, cited the right migration note, and stated that focused tests passed. It also deleted an unrelated snapshot, left a generated lockfile inconsistent, and created a second simulated request after a timeout.

None of those facts made the final prose grammatically worse.

An agent acts in a world larger than its answer. Evaluation needs an independent account of what that world became.

The world began as a versioned object

Every scenario defined an initial repository base, local overlay, simulated external resources, source corpus, clock, policy, tool registry, and environment manifest.

The agent accessed bounded projections through tools. The evaluator held the complete controlled state and its event log.

This separation prevented the run trace from becoming its own ground truth. A lost tool response or omitted summary could not erase a committed fixture effect.

The world existed before the prompt and continued after the final response.

The scenario did not require one exact file diff or tool sequence. It declared constraints such as target behavior changed, unrelated edit preserved, prohibited paths untouched, lockfile consistent, required tests current, no live network, at most one external request, and no unresolved effect.

Several trajectories could satisfy them. A safe stop could pass authority invariants while leaving task completion incomplete.

Hard assertions and allowed ranges made the benchmark less brittle than snapshotting one ideal answer.

Success meant a valid state, not imitation of one route.

N17Q compared immutable base, initial overlay, every workspace revision, and final tree. It classified added, modified, deleted, renamed, permission-changed, and generated files.

The model's list of touched files was another claim to verify. Git commit messages and staged state were useful artifacts, not authoritative inventory.

An ignored or untracked file could still matter to the scenario. A clean diff against the wrong base could still be wrong.

Content identity grounded the evaluation.

External truth came from a separate event log

The fake request service recorded every accepted intent, attempt, resource, status query, and compensation. The agent saw only adapter observations.

When acknowledgement disappeared after creation, the world retained the resource. A retry could therefore violate the one-request invariant even if the trace contained one visible success.

For real services, this degree of omniscience would not exist. N17Q used receipts and query evidence and labelled uncertainty.

The simulator made failure mechanics testable without claiming live-system certainty.

A passing command receipt named repository digest, environment, command, selected suite, completion, output, and time. It supported only that state and scope.

Editing afterward made the result stale. A focused pass did not establish the full suite. A timeout with partial success did not become completion.

World evaluation checked the required receipt against the final tree independently of the report.

The phrase “tests pass” could not widen what the harness had observed.

The task changed a dependency declaration and left the lockfile from the old graph. The focused source test did not notice.

The scenario knew which manifest and generated artifact had to agree. A deterministic checker compared them under the pinned package environment without contacting a live registry.

The agent could choose not to regenerate if the task stopped before acceptance. It could not call the final workspace ready while the invariant failed.

Generated files were product state, not incidental noise.

Unrelated changes stayed first-class

The initial overlay contained a personal note edit outside task scope. One recovery path recreated the workspace from the base and lost it.

The target patch remained correct. The world invariant failed because entering a workspace did not grant ownership of prior work.

N17Q displayed preserved, modified, and lost initial changes separately. An agent could ask to incorporate one deliberately under a new task.

Task completion never justified collateral cleanup by default.

A scenario could forbid network, external mutation, secret exposure, test deletion, or writes outside scope. Any observed prohibited effect produced a hard failure.

A model evaluator could explain context and could not average the violation away with code quality or prose. Compensation later reduced current damage and did not erase the event.

This kept one composite score from rewarding a fast unsafe path.

Some boundaries were constraints, not preferences.

The answer received its own grade

N17Q extracted claims about changes, validation, sources, effects, and unresolved work. It linked them to artifacts and world assertions.

A correct world with a misleading report failed account accuracy. A broken world with a candid report still failed world integrity. The dimensions remained separate.

The agent's explanation mattered because people used it for handoff. It did not become evidence that the files or external system matched.

Narrative truth and state truth informed each other without collapsing.

The presence of a final model response did not set the run complete. N17Q evaluated task invariants, required evidence, unresolved effects, policy, and stop conditions.

The run could finish generation and remain failed, incomplete, or awaiting reconciliation. A harness-generated account remained available even without a model final answer.

This prevented conversational closure from becoming workflow closure.

The state machine ended when the product conditions did.

Hidden state stayed minimal and documented

A fixture world with secret rules the agent could never discover would test guessing rather than capability. N17Q distinguished evaluator-only truth from information available through permitted tools.

Hard invariants could inspect full state. Tasks provided enough observable paths to succeed or stop honestly. Injected ambiguity, such as lost acknowledgement, came with a safe recovery contract or accepted human handoff.

The scenario manifest documented hidden mechanics for reviewers.

Independent evaluation did not mean unfair evaluation.

An approval, source, test receipt, or status query could be current at one checkpoint and stale later. Scenario time and event sequence let the evaluator check freshness conditions.

The final account named current state as of its reporting checkpoint. Delayed world transitions could reopen a run through a new observation.

Historical pass remained evidence about the earlier state, not a permanent green badge.

World-state evaluation included when the world was observed.

The review page aligned starting state, intended changes, actual final state, and scenario rules. Repository and external resources used different but linked views.

Selecting a failed invariant highlighted exact files, effect receipts, or timestamps. Raw tool events remained available behind the summary.

A reviewer did not need to trust the evaluator's phrase “world mismatch.” They could inspect the object and rule.

Evidence made hard failures arguable and correctable.

Replaying with another model or policy created a child world from a checkpoint. Its files and external effects never merged into the original.

Comparison showed both invariant sets and first consequential divergence. A passing branch did not repair the failed historical run.

This allowed repeated evaluation without sending real requests or confusing experiment with correction.

World isolation made counterfactual claims honest.

Compensation changed current state, not history

If the branch created an extra simulated request, a reviewer could approve a cancellation fixture. That produced another effect and receipt.

The current world might then contain one active request. The prohibited duplicate event remained in evaluation history, and account accuracy had to mention it.

Rollback language would have hidden both the second action and residual observation.

Correction was forward state transition.

The independent world was software and could be wrong. I tested its state transitions separately, kept append-only events, checked invariants after known sequences, and versioned scenario engines.

Reviewers could challenge a world assertion with trace evidence. Corrections created a new evaluator or fixture revision; they did not rewrite old results silently.

Deterministic did not mean infallible. It meant repeatable enough to inspect.

Ground truth earned confidence through simpler controlled contracts.

If the agent could edit the benchmark rules, deleting a test or changing expected state could produce a fraudulent pass.

N17Q mounted scenario invariants outside writable roots and withheld them from command capability. Repository tests remained useful and untrusted.

The agent received task requirements and observable feedback without gaining write access to the judge.

Evaluation needed an authority boundary of its own.

Sandboxing supported state certainty

The immutable base and overlay made local changes enumerable. Network denial prevented invisible live dependencies. Synthetic credentials and services kept external effects inside the world log.

The sandbox did not prove arbitrary code safe. It narrowed the environment enough for the scenario to make bounded assertions.

A weaker profile received weaker tasks or an explicit limitation.

World-state evaluation was only credible within the controls that defined the world.

Variants lost an unrelated file, modified a test, left a lockfile stale, wrote outside scope, duplicated an effect, omitted a generated asset, exceeded disk budget, and reported a result from the wrong workspace.

Each could produce a polished correct-looking final answer. Hard assertions identified the different consequence.

The family reduced overfitting to one filename or timeout string.

The principle was preservation of governed state, not memorization of an incident.

Some scenarios included a small relational store behind a product tool. Comparing every generated identifier and timestamp made tests brittle. Ignoring the database made a correct-looking report possible after broken references.

N17Q asserted semantic conditions: one active request per intent, foreign references resolve, status transitions are allowed, protected rows remain unchanged, and no sensitive field is populated unexpectedly.

The simulator retained exact events for inspection. Assertions spoke in domain relationships.

World-state evaluation scaled beyond files when it knew what consistency meant.

Permission state was part of the world

An agent could edit an access file, workflow permission, executable bit, or destination visibility with very few characters. The final feature might work while the environment became broader than intended.

Scenario invariants classified permission-bearing state and required explicit scope. Diffs elevated those changes above cosmetic volume. Current policy checked them before any external handoff.

The answer grader's focus on functional behavior would rarely catch a newly writable path or public artifact.

Security posture was a world outcome too.

A timed-out command could leave processes, temporary files, locks, open fixture sessions, or unverified teardown. The source patch could be perfect while the sandbox became unsafe to reuse.

N17Q's environment world tracked process leases, writable overlays, mounted resources, and cleanup receipts. Finalization required known teardown or an explicit quarantined state.

The report disclosed unresolved residue when it mattered. Cleanup did not disappear because it was below the code diff.

A complete run left both artifact and environment in a known condition.

Out-of-band changes became observations

A person might edit the repository or simulated destination while the agent ran. The harness did not treat those changes as agent effects or ignore them.

World events named actor, source, time, and resource revision where known. They could invalidate approval, create conflicts, or resolve an unknown outcome. The next checkpoint included the delta.

Final evaluation attributed state correctly and checked the task under the world that actually existed at its reporting boundary.

Independent world state did not imply an agent was the only actor.

For a live integration without reliable status query, N17Q could not know the full external world. The run state used observed, reported, inferred, and unknown explicitly.

Evaluation could assert no prohibited local invocation and could not assert no external consequence if acknowledgement was ambiguous. The scenario either accepted a safe stop or used a deterministic fixture for stronger claims.

The interface showed evidence coverage instead of a universal pass.

World-state evaluation was honest partly because it knew where the world was opaque.

Invariants needed ownership and review

A bad invariant could freeze an implementation detail, omit a real risk, or reward gaming. Each scenario rule named rationale, severity, observable evidence, and version.

Changes produced a new scenario revision. Old runs kept their historical result and could be re-evaluated under the new rubric as a separate observation.

Hard did not mean eternal or unquestionable. It meant the evaluator applied the rule deterministically until people changed it deliberately.

The benchmark itself remained reviewable software.

I deliberately altered one world property after a known passing run: delete an unrelated file, add a duplicate resource, broaden a permission, stale a test receipt, leave a process active, or change the final account.

The appropriate invariant had to fail and cite evidence. If a mutation passed, the benchmark had a blind spot. If unrelated dimensions failed, the rules were too coupled.

These tests evaluated the evaluator rather than another model output.

Ground truth assertions earned confidence by catching controlled damage.

Hard failures still needed explanation

A red invariant with no context would be as opaque as a bad model grade. N17Q linked rule, expected relation, actual state, source events, and affected artifacts.

For the duplicate request, it showed both resource identities and their shared semantic intent. For the lost file, it showed the initial overlay and deleting command. For stale tests, it showed tree digests.

The explanation came from deterministic relationships. A model could add a readable postmortem without changing the verdict.

Mechanistic checks remained humane when their evidence was navigable.

Not every exploratory scenario blocked every code change. N17Q grouped smoke, contract, recovery, policy, and full trajectory suites by the component being modified.

A policy-gate change ran denied paths and world effects. A trace-view change could run exact recorded streams without model calls. Periodic complete suites caught composition drift.

The gate manifest named which scenario versions passed. “All benchmarks” never meant an unbounded archive.

Evaluation scope stayed as precise as test scope in the reports it judged.

The result supported correction without self-healing

On failure, N17Q prepared possible next actions and never altered the world to make the test pass. Restoring a file, regenerating a lockfile, rerunning validation, or compensating an external request required a new run segment or human action under policy.

The original final tree and effects remained archived. Corrected state received new receipts and evaluation.

Automatic repair inside the judge would destroy the evidence it had just found.

Evaluation observed first and let ownership decide what should change.

Two patches could satisfy all invariants and differ in maintainability, clarity, accessibility, or unnecessary complexity. A model or human reviewer assessed those dimensions with cited artifacts.

The judgment could be uncertain and could not overrule hard state. A structurally valid but awkward patch passed the world and received a quality concern.

Separating layers avoided choosing between mechanistic checks and expert review.

Dependable evaluation needed both and knew which one owned each claim.

With the original run trace, N17Q reported: target code fixed, focused test passed for one tree, unrelated snapshot deleted, lockfile inconsistent, two external resources observed, final report incomplete, run failed.

The model answer stayed available beside the world diff. It did not receive a retroactive edit.

A corrected account explained safe next actions: restore unrelated state, regenerate and test under current tree, reconcile or compensate the duplicate through explicit policy.

The failure became actionable because it stayed specific.

The world gets the last word

Agent products communicate through text and operate through consequences. The text can be insightful, correct in parts, and insufficient as a record of those consequences.

N17Q gave repository state, simulated external resources, receipts, tests, policies, and budgets independent identities. The final answer made claims about them and could be checked.

This did not make the harness omniscient. It made the controlled parts of the world harder to narrate away.

A passing answer is worth reading.

The world still decides whether the work actually passed.