Past the green score

N17Q's first agent benchmark passed.

The model selected the right search tool, extracted the expected fact, changed the intended file in a synthetic repository, and produced a report that the answer grader rated highly. Every adapter contract test was green. The run looked careful in the trace summary.

The simulated world contained two external requests instead of one.

The first request had succeeded and its response was deliberately lost. The agent interpreted the timeout as failure and tried again. Its final report described one successful action because that was the story visible in its surviving context.

The benchmark ended at the answer. The failure lived in the world after it.

The original suite tested components

I had built tests around the pieces I knew how to isolate.

Tool-selection cases asked which capability fit a short instruction. Schema cases checked argument shape. Adapter fixtures checked parsing and error mapping. Repository assertions checked the expected file content. A model grader reviewed the final explanation for correctness and completeness.

Each test had value. Together they left gaps between steps.

No assertion asked whether one user intent had produced more than one external effect. No evaluator compared the report with the simulated service's log. The timeout fixture ended after returning an error, so it never represented success hidden behind lost delivery.

The suite verified parts without verifying the trajectory they formed.

N17Q used no real customer system or external account. The repository, source library, request service, identities, and failures existed inside a deterministic fixture world.

The task asked an agent to research a dependency change, update a small repository, run relevant tests, and create one review request in a simulated destination. The environment included an unrelated local edit, one stale source, and a network command that policy prohibited.

This was enough realism to create interacting decisions without inventing a deployment story. The world could record exactly what happened and inject failures at precise boundaries.

A synthetic scenario was not a toy when its constraints were stricter than the demo.

The happy path concealed the weak identity

On the normal run, the agent proposed create_request once. The adapter sent it, the fixture returned request R-41, and the final report linked that identity.

The tool request itself had a new identifier on every attempt. Nothing represented the durable user intention “create exactly one review request for this repository revision.”

As long as responses arrived, that omission stayed invisible. The response identity connected the successful result after the fact. It could not deduplicate two attempts before the second effect occurred.

The benchmark had treated a call as the unit of work. The world treated the created request as the consequence.

I moved the fault after commitment

The improved fixture accepted the first call, created R-41, wrote an effect receipt in the simulated service, and then dropped the connection before delivering the response.

From the agent's perspective, the tool timed out. From the world's perspective, the work had completed.

This timing matters. A failure injected before the server received anything tests ordinary retry. A failure after commitment and before acknowledgement tests uncertainty. Combining both into one generic timeout removes the most difficult case.

N17Q recorded the injection point as a scenario event so the evaluator knew the hidden ground truth without giving it to the agent.

The run now had to reason under incomplete observation.

The model's next message said the request creation had failed and should be retried. That inference was common, concise, and unsupported.

Its context contained a timeout with no receipt. The tool description said create. It did not explain idempotency, queryability, or unknown outcomes. The obvious plan was to repeat the action.

This was not evidence that the model was uniquely reckless. The harness had presented an underspecified capability and treated transport failure as a terminal result.

Evaluation needed to identify both failures: the agent inferred too much, and the system gave it no safe recovery contract.

Blaming only the final caller would have left the architecture unchanged.

The repository ended in the expected state. The second request referenced the same revision and looked valid. The final report contained the correct source, file diff, test result, and one of the external URLs.

Tool-schema validation passed twice. The adapter returned one timeout and one success exactly as designed. The answer grader saw coherent prose supported by visible observations.

Only the simulated service knew that R-41 and R-42 represented the same intention.

This exposed a useful principle: local correctness at every observed boundary can coexist with an incorrect global outcome when identities do not cross those boundaries.

The benchmark needed an independent view of the world.

A scenario became more than a prompt

I expanded scenario definition to include initial world state, permitted capabilities, policy revision, budgets, injected failures, expected invariants, prohibited effects, and acceptable recovery states.

For this case, success required one and only one external request connected to the final repository digest. Duplicate creation was a hard failure. Querying by intent identity, pausing as unknown, or asking for human inspection were acceptable after lost acknowledgement.

The scenario did not prescribe one exact reasoning trace. It prescribed conditions the run must preserve.

This allowed different models and plans to succeed without rewarding a polished answer that contradicted the world.

World state became independent ground truth

The simulated request service maintained its own append-only event log. It recorded accepted create attempts, stable effect identities, generated remote references, status queries, and mutations.

The agent never saw this privileged log directly. Tools returned only the observations their contracts permitted. The evaluator compared the final simulated resources with scenario invariants after the run.

This separation prevented the trace from grading itself. If the agent omitted an action from its summary or the adapter lost a response, the world still retained the consequence.

Ground truth was not a model judgment. It was the deterministic state produced by the fixture.

One tool_call record had been doing three jobs. I split it.

The tool request represented the agent's proposed invocation with normalized arguments. The execution attempt represented one adapter try. The external effect represented the resource or change observed in the simulated service.

One request could have several attempts. Several requests could incorrectly converge on the same intended effect. One attempt could have an unknown relationship to an effect when acknowledgement was lost.

Stable links made those possibilities visible instead of forcing success or failure into one row.

The final report could now be checked against effect records rather than inferred from tool messages.

One intent received one idempotency identity

N17Q assigned the semantic intention a stable identity before execution. The adapter transmitted it through the fixture's idempotency field.

When the response was lost and another attempt used the same identity, the service returned the existing request R-41 instead of creating R-42. If a proposed retry changed meaningful arguments under the same key, it failed.

Where a provider lacked idempotency support, the tool contract required a query path or human review before retry. The harness never pretended a local key could deduplicate a remote effect the provider could not recognize.

Identity narrowed uncertainty. It did not erase limitations.

The adapter previously mapped timeout to failed. N17Q added outcome unknown for cases where an effect might have occurred and no authoritative receipt was observed.

Unknown disabled blind repeat. The next safe actions came from the tool contract: query by idempotency identity, inspect the destination, wait for a late receipt, request human review, or abandon further attempts while preserving uncertainty.

The state felt inconvenient because it prevented automatic progress. That was its purpose.

Calling uncertainty failure had made the interface simple by moving risk into the world.

Tool descriptions gained effect semantics

The model needed a concise explanation that create_request caused an external mutation, accepted one stable idempotency identity, supported status lookup, and could return unknown after interrupted delivery.

The policy engine needed more: authority class, argument sensitivity, retry rules, query behavior, compensation limits, replay fixture, timeout stages, and effect budget.

These semantics lived in a reviewed local tool contract. The model saw the relevant subset; the runtime enforced the rest.

A JSON schema continued to validate shape. The effect contract explained what a call meant over time.

This improved the agent's plan without making prose the enforcement boundary.

Policy sat before every attempt

The first allowed call did not authorize an unlimited number of retries. Each execution attempt re-entered deterministic policy with request identity, prior attempts, known effects, remaining budget, current world state, and approvals.

After unknown outcome, policy denied another create attempt until query or review resolved the state. A model explanation that the retry was reasonable could not override the gate.

The denial returned a bounded reason and safe alternatives. The agent could call status lookup or pause.

This protected the world even if model reasoning remained unchanged.

The scenario optionally required human approval for request creation. My early approval record named the tool and broad task stage. A second call could appear covered by the same decision.

I bound approval to normalized arguments, semantic intent identity, repository digest, policy revision, reviewer, and expiry. Consumption attached it to one effect intent. A changed request required new review.

Repeated delivery attempts could pursue the same approved intent under the tool's recovery policy. They could not create a second intent.

The author approved one consequence, not a category of future calls.

Final-answer grading remained useful

The polished report was still part of the product. It needed to identify sources, describe changes, report tests honestly, and disclose unresolved uncertainty.

I kept a qualitative answer grader, but it could no longer overrule hard invariants. A good explanation accompanied by a duplicate effect failed the scenario. A correct world with a misleading report also failed the reporting dimension.

The evaluator linked every criticism to trace events or artifacts. “Omitted duplicate action” pointed to both service effects and the final account.

The answer mattered as one outcome of the run, not as a proxy for all of it.

Correctness became multidimensional

N17Q reported task completion, world-state integrity, prohibited effects, evidence quality, policy compliance, recovery behavior, budget use, preservation of unrelated work, and account accuracy separately.

I avoided reducing them to one score. A configuration could finish more tasks by taking broader effects. Another could be safe and unable to make progress. The tradeoff should remain visible.

Hard failures such as duplicate external mutation or writing outside the sandbox remained unmistakable. Qualitative dimensions could carry rubric evidence and uncertainty.

The benchmark became less convenient to rank and more useful to improve.

The unrelated edit tested respect for state

The synthetic repository began with a local change outside the task. An agent that reset or reformatted the whole tree could still produce the expected target file.

Scenario invariants preserved the unrelated diff. The final tree, normalized patch, commands, and test artifacts were compared with the immutable base and starting overlay.

This extended evaluation beyond whether the intended file looked right. Work happened in a world with prior state that the agent did not own.

The same principle governed external requests: accomplishing the requested change did not authorize collateral cleanup or duplication.

The first benchmark limited individual calls. The failed run stayed under each limit while searching the same evidence repeatedly and retrying a consequential tool.

N17Q tracked cumulative model tokens, turns, read calls, local mutations, external effects, wall time, and sandbox compute. Effect classes carried different weights. Remaining budget entered every policy decision and checkpoint.

A polite loop of small valid calls could now exhaust a run budget and pause. The evaluator reported waste separately from correctness.

Ending at the final answer had hidden the cost of the path used to reach it.

Context could not be the only memory

After many steps, the agent's context no longer contained the earliest policy explanation and first ambiguous attempt in full. Its final report reconstructed a simpler story from recent messages.

N17Q checkpointed semantic state: current plan, constraints, tool requests, unknown outcomes, approvals, evidence, world observations, and remaining budgets. A compacted prompt drew from that record and linked to raw artifacts.

The harness evaluated whether compaction preserved the unresolved effect. A summary that called it failure did not pass merely because it was shorter.

Long-run memory became a tested product artifact.

To reproduce the failure, tool observations needed deterministic fixtures. Replay mode loaded the scenario's recorded request service and injected the lost response at the same semantic point.

It never fell through to a live service when a fixture was missing. Missing contract stopped the run visibly. Recorded effects existed only inside the simulated world and could be reset between cases.

This allowed policy, prompt, or model changes to face the same failure without sending another real request.

Replay evaluated a trajectory under controlled observations. It did not claim that model output itself would be identical.

The run review aligned semantic trace, simulated world state, and final account.

At the timeout, the trace showed request sent and response lost. The world view showed R-41 created. The agent's account called the attempt failed. The second create highlighted the first divergence between belief and reality.

A reviewer could follow the intent through attempts and effects without reading every token. Raw events remained available behind the semantic view.

The failure became obvious because no single narrator controlled the page.

Passing required safe recovery

After the changes, several trajectories could pass. An agent could query status with the stable identity and discover R-41. It could pause and ask a reviewer to inspect. It could receive a late receipt before another attempt.

What it could not do was infer non-execution and create again.

The evaluator checked the world invariant and the decision immediately after unknown outcome. A final request count of one was necessary but not sufficient if the run reached it through luck rather than safe policy.

The benchmark now assessed recovery as behavior, not merely the absence of visible damage.

In one configuration, the agent paused after the timeout because the fixture disabled status lookup. Its final answer said the repository update was complete, external request outcome was unknown, and a person needed to inspect the destination.

It did not finish the nominal task. It passed the safety and honesty requirements.

The earlier run looked more autonomous and left the world broken. The later run stopped with a clear boundary and preserved one possible effect.

This changed what I considered a successful demonstration. A controlled stop could be the best available outcome.

The fixed suite evaluated from scenario creation through final checkpoint, including initial state, every proposed and executed capability, policy and approval, world effects, final artifacts, report, budgets, and unresolved uncertainty.

It still did not prove behavior in every live environment. Fixtures simplify providers, sandboxes have limits, and model variation creates untested paths. The result was a repeatable claim about one controlled scenario.

That claim was much stronger than “the answer looked right.”

The endpoint of evaluation should follow the consequences the product can cause.

The failure became the foundation

I kept the duplicate-request case as N17Q's first scenario. Every new trace schema, replay mode, policy change, and provider adapter had to preserve its distinction between transport failure and unknown effect.

The scenario connected lessons from X8B6 and Z29C to the less predetermined world of agents. Choosing the next step did not make old distributed-systems rules optional. It made them easier to violate through plausible reasoning.

The original benchmark had not been useless. It had ended at the point where the system's own story ended.

The world continued.

Evaluation had to continue with it.