The invariant outranked the grader

The evaluator gave the run a 9.2 out of 10. The run had sent an unapproved message.

Its explanation was polished: the assistant understood the task, selected relevant tools, recovered from a transient failure, and produced a concise final answer. The rubric penalized the approval miss, but strong scores elsewhere outweighed it.

That arithmetic made sense only if every property was a preference.

Some properties were boundaries. N17Q needed evaluation to distinguish quality that could trade off from invariants that could not.

A score invited compensation

Weighted rubrics were convenient for ranking. They also implied that enough helpfulness could offset a forbidden effect, enough fluency could offset fabricated evidence, or enough speed could offset corrupting the workspace.

The problem was not the choice of weights. Any additive score allowed compensation unless a special rule sat outside it.

N17Q retained scores for dimensions such as clarity, efficiency, planning, and explanation. It moved product invariants into a separate verdict layer evaluated first.

A failed invariant could not be repaired by praise in another column.

Invariants described permitted worlds

The clearest checks inspected system and simulated-world state, not prose.

No external mutation without current authority. At most one effect for one intent. No network outside declared connections. No writes beyond the workspace. Final completion claims require verified postconditions. Replay creates no live effects. A denied capability is not invoked through an alias.

Each invariant had a stable identifier, scope, evidence query, and failure payload. It returned pass, fail, or indeterminate. Indeterminate did not quietly become pass.

These rules defined which run outcomes the product was willing to call valid.

Hard did not mean universal

An invariant belonged to a scenario, capability class, or product mode. “No external writes” was correct for a diagnostic replay and wrong for an explicitly approved delivery workflow.

N17Q compiled the applicable invariant set from scenario contract, execution environment, policy revision, and declared success conditions. The trace stored that exact set before execution.

Tests rejected a run that changed its evaluator contract after seeing the result. New rules could analyze old traces as a separate retrospective evaluation, but did not rewrite the original verdict.

Hardness came from scope and enforcement, not from pretending every task had the same boundaries.

The model grader remained useful

Deterministic checks could prove that one resource existed. They could not fully judge whether a research synthesis was insightful, a plan was proportionate, or an explanation was understandable.

N17Q used model graders for bounded qualitative questions with evidence packages. The grader received the relevant artifact, rubric, source references, and trace projection. It returned dimension scores, cited evidence, uncertainty, and critique.

Those judgments informed comparison and diagnosis. They never changed hard verdicts. A model grader could argue that an invariant was misconfigured; it could not mark the violating run valid.

Keeping that role modest made the grader more credible.

The inverse mistake was trying to turn every desirable behavior into a deterministic assertion. Checks for exact phrases rewarded templates. Fixed step counts punished useful adaptation. Brittle call sequences mistook one implementation for correctness.

N17Q limited hard checks to facts the product could define and observe reliably. Qualitative rubrics handled coherence, economy, relevance, and good judgment. Statistical measures handled tendencies across a suite.

The separation prevented “hard” from becoming a synonym for easy to code.

An invariant deserved that status only when violating it invalidated the run within the declared contract.

Evidence came from independent state

The earliest checker parsed the assistant's final answer for claims such as “tests passed” and “email sent.” A careful liar could satisfy it. An honest but terse assistant could fail.

N17Q queried trace events, policy decisions, approval receipts, effect ledger, sandbox snapshots, test artifacts, and simulated world. The final answer was one evidence source and often the subject of a consistency check.

The system verified completion from postconditions, then checked whether the assistant's account matched. Neither text nor grader confidence could create a missing receipt.

Evaluation became an observer of the run rather than another reader persuaded by it.

The invariant ran at the earliest useful boundary

Some rules could prevent harm before it occurred. The tool router blocked unoffered capabilities. The final gate denied stale approval. The sandbox denied network. Other checks could only inspect a result, such as whether a generated report omitted an important limitation.

N17Q placed preventive enforcement in the execution path and repeated corresponding assertions after the run. Detection-only invariants stayed clearly labeled.

This distinction mattered in reports. “No unauthorized send occurred” was stronger when the gate and world state agreed than when a retrospective evaluator simply found no receipt.

Evaluation complemented controls; it did not replace them.

Fail closed required an indeterminate state

If the effect ledger was unavailable, the checker could not prove that no duplicate delivery occurred. Treating a missing query as zero effects rewarded observability failure.

Every invariant declared required evidence. Missing, malformed, stale, or contradictory inputs produced indeterminate with a reason. For a release gate, indeterminate blocked the pass. For exploratory analysis, it remained visible without being counted as success.

The harness distinguished system-under-test failure from evaluator infrastructure failure where possible. Both prevented a trusted verdict, but they led to different repair work.

Uncertainty remained a result, not an exception swallowed by the score pipeline.

Once one hard invariant failed, the runner could stop evaluation immediately. That saved compute and hid interacting defects.

N17Q grouped checks by evidence cost and side-effect freedom. It ran all safe deterministic checks against the sealed trace and world, then reported a primary failure and related failures. Expensive qualitative grading could be skipped when the result was already invalid, unless the task was diagnostic.

No failure gained more authority because it appeared first. Ordering optimized useful feedback.

The report could say that the unapproved send invalidated the run and that its final answer also omitted the consequence.

Severity was not hardness

A hard invariant could catch a small but absolute contract breach, while a severe qualitative defect could make a technically valid answer unusable.

N17Q represented verdict class and impact separately. Hard failure answered whether the run belonged in the permitted set. Severity answered how much consequence or repair the finding carried. Qualitative dimensions answered how well the permitted run served its goal.

This avoided labeling every hard rule “critical” and every soft judgment “minor.”

A valid but incoherent report could score poorly. An invalid run could have a low-impact fixture-only violation. Neither category erased the other.

Scenario success was conjunctive

For the delivery fixture, success required a correct artifact, an approval bound to that artifact, one authorized effect, a matching receipt, and a final account consistent with world state. These were not weighted suggestions.

N17Q represented the scenario result as required predicates plus optional quality dimensions. All required predicates had to pass. The quality profile ranked valid runs and explained tradeoffs.

This made leaderboards less dramatic and more meaningful. A fast run that skipped review was not “nearly as good” as a slower valid run.

The harness optimized only inside the space of acceptable outcomes.

Statistical reliability sat above individual verdicts

One passing trajectory did not prove a model or policy reliable. N17Q ran scenarios across seeds, scheduling variations, model versions, and fault injections.

Each run first received its hard verdict. Suite reports then measured valid-run rate, invariant-specific failure frequency, cost distribution, recovery quality, and qualitative scores among valid runs.

Averages never converted failed runs into partial passes. Confidence intervals and sample counts accompanied rates. Rare catastrophic failures remained individually inspectable.

The suite could compare systems without burying the tail inside a mean.

A check that sometimes failed against identical sealed evidence was not a nuanced judge. It was an unreliable program.

Deterministic invariants used versioned queries over immutable trace and world snapshots. Their fixtures included positive, negative, missing-evidence, and adversarial cases. Changes to a rule reran its corpus before entering the registry.

If external services were needed, the check was not hard-gating until their evidence was captured and deterministic. Model output never sat inside a hard predicate.

The confidence to block a run had to come from testable evaluator behavior.

Two model graders could differ on whether the final explanation was sufficiently clear. N17Q retained their independent judgments, evidence citations, model identities, and rubric versions.

Aggregation rules could produce a review queue or a bounded score. They did not pretend consensus where none existed. A human resolution became another labeled observation.

If a grader claimed an unauthorized action was acceptable, the report showed the critique beside the hard failure and treated the claim as rubric or grader evidence—not an appeal decision.

Disagreement improved the evaluation when it remained inspectable.

The evaluator had its own attack surface

Tool output and generated artifacts could contain text aimed at the grader: declarations of success, fake rubric instructions, or fabricated citations. Passing raw traces into a model invited prompt injection at evaluation time.

N17Q built typed, bounded evidence packages. Instruction fields and observed content were separated. The grader's task contract was fixed outside the artifact. Sensitive data was projected according to policy.

Deterministic checks parsed schemas and identities, never prose from untrusted results. Model graders cited evidence handles that the system could verify.

Evaluation was another trust boundary, not a privileged afterthought.

Policy and evaluation used different questions

Policy asked whether the system may take an action given current state. Evaluation asked whether a completed or attempted trajectory satisfied a declared contract.

Using evaluator scores in the live gate created circular behavior: a model predicted that its plan would probably score well and treated that as authority. Using policy decisions as the only evaluation missed poor outcomes after allowed actions.

N17Q linked the two without merging them. A post-run check verified that every invocation had an eligible decision. Outcome checks inspected what followed.

Permission did not guarantee quality, and quality could not grant permission.

Operators sometimes wanted to continue after a hard failure in a synthetic scenario to collect diagnostic evidence. Calling this an override would imply that the invalid run became valid.

N17Q allowed continue for observation in non-live environments. The original failure remained, downstream events carried a tainted lineage, and the final verdict stayed invalid. The action required a separate diagnostic capability and could not cross into external effects.

If a product requirement genuinely changed, maintainers versioned the invariant contract and reran evaluation under the new rule. They did not attach a one-off note saying “ignore.”

Evidence collection could continue without laundering the breach.

The UI led with validity

The old report displayed a large 9.2, a green trend arrow, and a small warning lower on the page. Visual hierarchy told readers the run was excellent.

N17Q placed the hard verdict first: Invalid — external effect lacked eligible approval. Under it, the report showed world impact, evidence, causal path, containment, and repair. Qualitative scores appeared in a section explicitly labeled “What the run did well despite invalidity.”

Valid runs showed their quality profile without a celebratory universal score. Indeterminate runs named missing evidence before any ranking.

The interface stopped asking a decimal to carry moral and technical meaning it did not possess.

Counterfactual grading could not change history

After fixing the approval gate, I replayed the old trace against the new invariant set. The retrospective report showed that the historical run would fail today's contract too. A fixture rerun demonstrated the corrected system blocking the send.

Neither artifact changed the original evaluation record. The first preserved what the old evaluator had concluded; the retrospective analysis documented why that conclusion was inadequate.

This chronology mattered. Improving a benchmark should reveal past blind spots, not silently make the archive look wiser.

Evaluator versions were part of the evidence lineage.

Hard rules still needed human ownership

Hard checks could shape product behavior as decisively as policy. Leaving them as anonymous test assertions made changes difficult to notice and failures difficult to contest constructively.

N17Q gave each invariant an owner, rationale, threat or correctness model, applicable scopes, required evidence, and change history. Modifying one required review of both the rule and its fixture corpus. Deprecation stayed visible in historical reports.

An appeal therefore targeted something concrete: perhaps the check used the wrong resource identity or the scenario contract had omitted a legitimate repeated effect. The answer could be a rule fix without turning the violating run into an exception.

Governance made hardness accountable rather than arbitrary.

Minimal counterexamples accelerated repair

A failed scenario could contain hundreds of events. Dumping the entire trace on the maintainer obscured the state transition that violated the rule.

Each invariant query returned a bounded witness: relevant identities, predecessor events, world entries, expected predicate, and observed contradiction. The report linked outward to full context. For duplicate effects, the witness showed the one intent and two resource creations. For stale approval, it showed the approved and executed digests.

The harness also attempted safe trace slicing against dependency edges, never deleting evidence from the canonical run. A small reproducible fixture could then join the evaluator's regression corpus.

Hard failures became engineering inputs, not merely red badges.

Success conditions included required non-events

Some outcomes depended on what did not happen: no external send, no write outside scope, no second create after ambiguity. Absence was harder to prove than presence.

N17Q bounded these claims with sealed environments, complete effect ledgers, sandbox enforcement, and scenario completion points. “No live effect” meant no eligible adapter path, no network route, and no recorded world mutation within the declared run boundary. It did not claim omniscience over systems it could not observe.

Checks named that boundary in their evidence. If telemetry coverage was incomplete, the result was indeterminate.

A negative invariant was credible only when the system could explain where it had looked.

Optimization respected the validity frontier

Automated prompt and policy experiments wanted one number to maximize. Feeding them the average quality score recreated the original problem at a larger scale.

N17Q filtered invalid configurations first, then compared valid candidates on task quality, cost, latency, and review burden. Where no candidate met the invariant floor, the experiment returned no winner. It did not select the least frequent violator and call it best.

Multi-objective reports showed tradeoffs instead of hiding them in arbitrary weights. A faster configuration that increased indeterminate evidence could be rejected even before a hard failure appeared.

Optimization became search within the permitted region, never bargaining over its boundary.

This also improved experiments that ended without a valid candidate. Instead of hiding the result, N17Q reported which invariant formed the frontier, which evidence was missing, and what configuration changes had already failed. An unsuccessful search became a map of constraints. That was more useful than a leaderboard whose winner still required a disclaimer before anyone could trust it.

The 9.2 became an invalid run with useful qualities

N17Q reran the scenario. The model prepared the right message and attempted delivery without a matching approval receipt. The final gate blocked the adapter, the simulated mailbox remained unchanged, and the assistant stopped with a precise request for review.

All hard predicates passed in the repaired trajectory. Qualitative graders still found the planning efficient and the explanation strong. Those judgments now described quality inside a permitted outcome.

The earlier run remained invalid even though its prose was better and its task appeared complete.

A hard invariant is not a very important rubric item. It defines the boundary within which rubric tradeoffs make sense.

The evaluator can criticize that boundary, reveal that its evidence is inadequate, or help maintainers design a better one.

It cannot award enough points to make the forbidden world acceptable.

That limit is what lets every softer judgment remain genuinely useful, debatable, and safe to improve over time.

Together.