One benchmark score hid five failures

K81R's benchmark improved from 71 to 78 after a model change.

Exact identifier retrieval had regressed. Refusals became less useful. The model wrote smoother prose and cited more passages, including several that did not support their claims.

The average rose because the final-answer grader liked the new wording.

One number had turned five different systems into one impression.

I rebuilt the evaluation around failure location and consequence rather than a single answer score.

The pipeline had five accountable stages

I grouped K81R's work into:

1. Corpus: ingest the correct authorized source revisions with faithful structure and authority metadata.
2. Retrieval: find relevant lexical and semantic candidates inside the permitted scope.
3. Evidence: select sufficient, current, coherent units and decide answerability.
4. Generation: produce bounded claims that preserve qualifiers and task intent.
5. Verification: validate citations, claim support, contradictions, rendering, and review state.

The interface and durable answer record sat across them. A failure could originate in one stage and appear only at the end.

The benchmark reported every stage before any end-to-end summary.

Retrieval cannot find a heading the parser discarded or an exception separated from its rule.

Corpus fixtures checked:

• Expected source IDs and immutable revisions.
• Current, superseded, draft, and historical relationships.
• Heading lineage and source offsets.
• Complete lists, code blocks, and table headers.
• MDX component adapters and unsupported-node inventory.
• Permission attributes and deletion state.
• Coverage gaps by project and node type.

A visual fidelity set compared representative source rendering with parsed evidence views.

If ingestion failed, later stages received no credit for guessing the missing meaning.

One recall metric hid literal failures under many broad conceptual wins.

Tasks were grouped into:

• Exact identifiers and contract names.
• Quoted phrases.
• Filenames, paths, and errors.
• Natural-language paraphrases.
• Cross-project concepts.
• Mixed literal and conceptual queries.
• Current versus historical authority.
• No-support questions.

Metrics included relevant-source recall within a scan-sized depth, first relevant rank, misleading-distractor rank, source diversity where appropriate, and retrieval reason accuracy.

Lexical, semantic, and fused lists were reported independently. A fusion improvement could not hide one broken lane.

Permission tests were blocking, not averaged

A restricted passage appearing in an unauthorized candidate set was not balanced by ninety-nine relevant public results.

Leakage fixtures checked results, facets, suggestions, snippets, cache reuse, evidence proposals, prompt context, refusals, and diagnostics. Any unauthorized inclusion failed the generation regardless of average relevance.

Other blocking invariants included invented source IDs, unsafe rendered markup, and duplicate consequential commands.

Some failures should not be converted into a small score penalty.

A retriever could return relevant sources and an automatic selector could choose the wrong five.

Evidence fixtures defined task clauses and required relationships. Evaluation asked:

• Is every necessary entity and condition covered?
• Are sources current or correctly historical for the question?
• Are contradictions present and classified?
• Did structural expansion preserve qualifying context?
• Is the token budget dropping a required unit?
• Does the set contain inaccessible or redundant evidence?
• Is the answerability decision correct?

False answer and false refusal were separate errors. A system refusing everything could not score as safe.

To isolate the model and prompt, I ran them against immutable evidence-set revisions. Retrieval and selection did not change between candidates.

The grader examined:

• Required claim coverage.
• Unsupported additions.
• Entity, quantity, modality, and time qualifiers.
• Correct task scope.
• Appropriate explicit inference.
• Useful refusal when the fixed set was insufficient.
• Structured-output validity and stability across repeated runs.

Several complete outputs were compared, not only the best run. A model that answered correctly once and invented on four repetitions remained unstable.

Verification tests used adversarial claim pairs

Citation presence was easy to satisfy. Support required harder cases:

• Same topic, different condition.
• Same amount, different unit.
• Current claim, superseded source.
• Sentence under “Rejected alternatives.”
• Code exists but is not registered.
• Two sources support facts but not the causal link.
• Restricted source supplies a fact cited to a public neighbor.

Deterministic integrity checks, automated support classification, and human labels were reported separately. A verifier model did not become ground truth because it agreed with another model.

False support received greater weight than conservative review-required labels.

End-to-end remained necessary

Perfect stage fixtures could still fail when assembled. The full scenario tested query, scope, index generation, selected evidence, answerability, generation attempts, claim streaming, source links, refusal page, and immutable answer record.

It included temporal actions: permission changes during a job, source revision after selection, one retrieval lane timing out, reconnection during claim streaming, and a model returning malformed structure.

The end-to-end result was a narrative report with stage outcomes. I retained a compact overall status for regression triage and refused to make it the headline number.

Integration evaluation showed how failures propagated; stage evaluation showed where to repair them.

Consequence shaped prioritization

Not every mistake deserved the same urgency.

A related source ranked sixth instead of fourth affected scanning. A missing negation could reverse a claim. A permission leak exposed restricted data. A fabricated retry policy could influence an external effect if copied into another workflow.

Cases had consequence classes and allowed product actions. A low-consequence brainstorming task could tolerate explicit unsupported proposals. A reference answer about current policy required source support and review. An external action remained outside K81R regardless of score.

The benchmark informed boundaries; it did not authorize them automatically.

Averages hid sparse severe failures

I reported distributions and failure counts by class. A 99 percent pass rate could contain one repeated leakage case. Median latency could hide a claim buffer that never closed on long tables.

The dashboard showed:

• Blocking invariant failures.
• Regression cases newly failing.
• False answer and false refusal counts.
• Support precision by qualifier type.
• Worst and tail latency per stage.
• Model variability across repetitions.
• Coverage gaps and untested categories.

Aggregate values remained available for trend and capacity. They never replaced the case list.

Using a model to judge prose was useful for style and some semantic comparisons. It introduced sensitivity to prompt, model, ordering, and verbosity.

I included hand-labelled calibration examples, obvious counterexamples, and duplicated cases with superficial wording changes. Grader outputs recorded model and prompt revision, rationale, and cited text. A grader update ran against the same frozen outputs before activation.

Deterministic checks overrode a flattering model grade for invalid IDs, wrong units, forbidden markup, or inaccessible evidence.

The grader was one instrument, not the benchmark's authority.

Evaluation data had versions and lineage

Each case stored:

• Case and revision ID.
• Corpus and index generation.
• Authorized scope.
• Query and task classification.
• Expected relevant, misleading, and prohibited sources.
• Required and forbidden claims.
• Useful refusal requirements.
• Consequence and blocking rules.
• Label authorship and disputed status.

Changing a source or judgment created a new case revision. Reports named the exact evaluation set.

“78” without the set and system versions was not a result anyone could inspect.

Benchmarks built only from answerable questions rewarded eagerness. K81R needed cases where the best result was a source-first refusal, contradiction page, or closed evidence gap.

No-answer fixtures distinguished:

• Complete corpus with no supporting source.
• Incomplete index.
• Missing relationship.
• Active contradiction.
• Inaccessible necessary evidence.
• Question outside corpus authority.

Evaluation checked safe disclosed facts, hidden details, gap specificity, and useful next actions.

Refusal quality became a product capability rather than an error count.

Historical questions prevented recency bias

Current authority preference improved current questions and could bury the exact document needed to reconstruct 2018 or 2021.

Cases asked what the project believed at a date, which decision changed, and how a superseded approach differed. Relevant sources and ranking policy shifted with temporal intent.

The generation had to preserve period language and avoid using a later API or decision as though it already existed.

The benchmark protected the archive's journey, not only current documentation.

Time to first token encouraged raw streaming. K81R measured:

• Time to first inspectable source.
• Time to complete evidence proposal.
• Time to answerability result.
• Time to first structurally valid claim.
• Time to first evidence-linked visible claim.
• Time to complete draft and verification.

Stage resource use and provider attempts accompanied latency. A faster model that required repeated repair could be slower and more expensive to a usable result.

Performance evaluation used the artifact the reader could trust.

Changes carried hypotheses

Every parser, ranking, selection, prompt, model, or verifier change named the stage and expected case classes. Reports highlighted results outside that hypothesis.

If a chunking change improved table support and exact identifiers regressed, the regression could not hide in an overall increase. If a model improved qualifier preservation but false refusals rose, the trade remained visible.

One dimension changed at a time where practical. Larger coordinated changes kept ablations or comparison generations.

Evaluation became a learning process instead of a leaderboard.

I kept a small end-to-end summary for quick comparison. It linked directly to stage distributions, blocking failures, and notable cases. A release could never be accepted from the summary alone.

The model update that moved 71 to 78 no longer looked unambiguously better. It improved prose coverage, regressed exact retrieval through a changed query rewrite, increased citation presence, and worsened support precision. The next iteration kept the model improvement, restored literal routing, and tightened claim validation.

That repair was possible because the benchmark stopped hiding where quality came from.

One score had hidden five failures. The useful result was not five new scores. It was a map from observed harm to the stage capable of fixing it.

The release report kept examples near numbers

Every metric row linked to representative passed, failed, improved, and regressed cases. The report showed query, expected evidence, actual candidates, evidence set, claims, and interface state. A reviewer could move from “support precision fell” to the exact qualifier or citation relationship responsible.

This discouraged arguments over abstract thresholds divorced from product consequence. A small average change caused by many harmless rank swaps differed from one current policy source disappearing below the scan depth.

I kept the report compact enough to read and detailed enough to challenge. Evaluation became evidence about the system rather than a ceremony producing a release number.

Benchmark success did not close the investigation

A passing generation entered a bounded observation period under synthetic probes and ordinary personal use. New failures could add cases, revise assumptions, or reveal an untested stage interaction. Passing meant the known gates held for a named system and corpus.

It did not mean K81R was generally correct, secure under every threat, or finished. The final handoff would need to state those limits beside the green checks.

Some properties could be tested without writing the only correct prose. I transformed fixtures and checked invariants:

• Reordering irrelevant distractors should not change the selected authority.
• Replacing an entity ID should update the answer and citations consistently.
• Removing a required capability source should change an answer into a gap.
• Marking a source superseded should alter current-state authority but preserve historical retrieval.
• Narrowing permissions should never add evidence or claims.
• Paraphrasing the query should preserve relevant sources within a bounded rank range.

These tests caught brittle prompt and ranking behavior that one golden answer would miss. A failure still required inspection; the invariant made the expected relationship explicit.

Counterfactual evidence tested dependence

To check whether a claim actually depended on its citation, I removed or replaced one evidence unit and reran answerability and generation.

If the same specific claim survived after its only supporting source disappeared, the model was likely drawing from general patterns or another unnoticed passage. If changing a numerical source did not change the quantity, the answer was not grounded in the selected evidence.

Counterfactual runs were expensive, so I used them on high-consequence regression cases rather than every query. They provided stronger evidence about system dependence than citation presence alone.

Labels had disagreement and review states

Even in a personal project, I reviewed hard cases at different times and disagreed with my earlier labels. Instead of overwriting them, the evaluation record stored each judgment, rationale, and adjudicated result.

Cases with unresolved support or relevance disagreement remained in a contested set and did not drive a simplistic pass threshold. They were useful for qualitative comparison and verifier calibration.

The benchmark became more credible after uncertainty entered the labels rather than being forced into model output only.

A correct answer could be hard to use if citations opened the wrong revision, refusal hid available evidence, source inspection reset search state, or claim streaming moved focus.

End-to-end fixtures evaluated:

• Claim and evidence adjacency.
• Keyboard and screen-reader navigation.
• Source revision and currentness visibility.
• Layout stability during progressive claims.
• Disclosure-safe error states.
• Preservation of evidence selection across navigation.
• Clear distinction among draft, verified, reviewed, and accepted.

Screenshots, accessibility snapshots, and temporal state assertions complemented semantic graders.

Product quality included the path by which a person could challenge the answer.

Cost was reported beside quality

The model update that improved prose also used more input context and triggered more repair attempts. I recorded embedding, retrieval, reranking, generation, verification, and repair cost or resource use per case where measurable.

The dashboard compared cost to first inspectable source, valid claim, complete draft, and accepted result. A cheap lexical answer could beat a full generative pipeline for an identifier query. An expensive counterfactual check could be justified for one critical policy fixture.

Optimization followed task consequence instead of one average cost per answer.

Passing tested cases said little about untested document types, languages, projects, permissions, and interaction paths.

I maintained a coverage map across year, project, source type, structural node, query class, authority relation, permission class, answerability state, claim qualifier, and interface route. Sparse cells were visible. Adding five variants of the retry example did not create broad confidence.

The map guided new fixtures from observed failures and archive growth. It also prevented the handoff from claiming exhaustive correctness.

Release gates differed by change type

A parser change had to pass structural fidelity and citation spans. A ranking change emphasized retrieval classes. A model update emphasized frozen-evidence claims, refusals, and variability. A renderer change emphasized safety and accessibility. Every change still ran blocking permission and source-ID invariants.

This made verification proportional without allowing a component owner to ignore downstream consequence. A parser could pass its unit fixtures and fail end-to-end because claim citations no longer opened.

The release report named the affected stages and complete gate set.

Search clicks, copied answers, and accepted drafts can be useful signals and are confounded by current ranking, interface position, convenience, and reviewer habits.

K81R's personal use offered too little data for statistical claims. I used explicit challenges and reviewed corrections as candidate cases, minimized behavioral logging, and never treated a click as proof of relevance or acceptance as proof of factual support.

Q2F8's purpose rule applied to evaluation telemetry. More data was not automatically better judgment.

Drift monitoring used stable probes

External model or infrastructure behavior could change without an application release. A small set of synthetic, non-sensitive probes ran periodically against exact identifiers, refusal, structured claims, citation integrity, and injection-like output.

The probes recorded provider and system versions, latency, usage, and stage outcomes. A change triggered investigation and could disable generation while source search remained available. Probe success did not certify the full corpus.

This separated ongoing availability and behavioral drift from the larger offline evaluation used for releases.