The index generation is part of provenance

I reran the same K81R question and got different sources.

The documents had not changed. I had improved the parser, changed passage boundaries, and rebuilt embeddings. The new results were better, but the old answer still claimed it had been generated “from the archive” as though the archive alone explained what the model saw.

Source citations named the end of the retrieval chain. They did not name the machine that selected those sources.

I made the index generation part of provenance.

An index is a derived product

K81R's search representations depended on:

• Source corpus and immutable revisions.
• Parser and supported node types.
• Passage-construction and context rules.
• Text normalization and lexical analyzer.
• Embedding model and vector dimensions.
• Metadata and permission schema.
• Candidate depths, rank fusion, and tie-breaking.
• Build failures and excluded documents.

Changing any of them could change whether a passage appeared, its source span, or its rank.

“Searched the same documents” was therefore incomplete provenance.

A generation was immutable after activation

Each build received a generation ID and manifest. Files and records under that ID did not change in place.

The manifest contained:

generation_id
source_corpus_revision
parser_version
passage_policy_version
lexical_schema_and_analyzer
embedding_model_and_dimensions
ranking_policy
permission_schema
build_started_and_completed
source_count
failed_source_inventory
artifact_digests

An atomic active-generation pointer selected which complete build served new searches. Fixing one source or parameter created another generation.

Immutability made old results explainable and rollback possible.

Source documents changed at different times. “Indexed at 14:08” did not establish which revisions were included.

The corpus builder produced a manifest of source ID, immutable revision, status, relationships, access attributes, and content digest. Its own revision identified that exact set. The index generation referenced it.

If one document failed parsing, the generation either failed complete coverage or declared the specific omission. It did not claim to represent corpus revision 44 while quietly lacking a source.

The search page could say “Index generation 18 covers corpus revision 44 except two reviewed unsupported diagrams.” That was less tidy and more truthful.

Parser version determined heading lineage, source offsets, node types, and whether a table or MDX component survived faithfully.

An answer record linked selected evidence to source revision and parser representation. Opening the citation used the source; inspecting why that passage existed used the parser manifest. A new parser could split the same text differently without invalidating the old source reference.

I stored parser fixtures and fidelity results with the release. A version string alone would not explain behavior after code disappeared.

Provenance needed enough artifact to reproduce or at least inspect the transformation.

Passage policy was separate from parsing

The same parsed nodes could be grouped into different retrieval units. One generation might attach list introductions automatically; another might expand after ranking.

The passage policy recorded type-specific rules, token bounds, required context, overlap behavior, oversize handling, and representation templates. Each passage had an ID distinct from its source-node identity.

Evidence records stored the selected representation and exact source spans. Rebuilding passage policy did not rewrite old evidence sets.

This separation helped evaluation locate regressions. A parser could remain correct while passage grouping worsened recall.

The embedding model was not a transparent dependency

Changing the model altered vector dimensions, similarity geometry, supported input limits, tokenization, latency, and sometimes task behavior.

The manifest recorded provider or local model identity, dated version where available, dimensions, input normalization, truncation policy, and embedding artifact digest. I did not store only a friendly model name that could later point to different behavior.

Source text and embeddings followed different retention. The vector was a derived representation, not a substitute archive.

A new model required a full vector generation and retrieval comparison. I never mixed vectors from incompatible spaces in one index.

Lexical analyzers changed exact evidence

Stemming, stop words, case handling, punctuation, field weights, synonyms, and code-token rules affected literal retrieval.

The lexical manifest included schema mappings and analyzer configuration. Diagnostic queries stored analyzed terms for development. A generation change could explain why workflow_call now ranked in code rather than being split into generic words.

Curated synonym updates were corpus policy changes with authorship and review, not anonymous runtime dictionaries.

Exact search felt deterministic only within a known analyzer and corpus.

Ranking policy completed the chain

Candidate lists could remain identical while fusion constant, exact tiers, status preference, or tie-breaking changed final order.

The generation manifest included candidate depths and ranking policy revision. A query record added its actual effective filters, lane status, and scope. Result cards in diagnostic mode showed lane ranks and reasons.

I separated generation-wide ranking configuration from per-query intent such as “historical only.” Provenance recorded both.

The final order was reproducible enough to inspect without claiming every external service response was deterministic forever.

Reindexing in place could serve half old and half new passages. I built a complete candidate generation under a new ID, verified counts and invariants, ran offline evaluation, and only then moved the active pointer.

The build checked:

• Every expected source had a disposition.
• Passage and source identities were unique.
• Lexical and vector records referenced the same allowed source inventory.
• Permission attributes matched the corpus manifest.
• Embedding dimensions and digests were consistent.
• Retrieval fixtures linked to valid source spans.

A failed build never became partially active.

This borrowed M31V's immutable release lesson for search data.

Every candidate generation ran the same versioned task set. Reports broke results down by exact identifier, conceptual paraphrase, structure, current-versus-historical authority, permission, and no-answer behavior.

The report listed improvements and regressions with source examples. Aggregate recall or reciprocal rank alone could not authorize activation. A severe permission or exact-identifier regression blocked the generation even if average relevance improved.

Evaluation fixtures named the corpus revision they expected. A source edit did not silently change the benchmark under the same ID.

The activated generation linked to its report and accepted limitations.

Rollback changed new searches, not old records

If generation 19 regressed after activation, the pointer could return new queries to generation 18. Searches and evidence sets created under 19 remained labelled 19. Their provenance did not pretend rollback had erased them.

An answer could be superseded after a rerun under 18 or a fixed 20, but its original evidence and output stayed immutable.

Old generations remained available for a bounded inspection window. Source revisions and compact manifests outlived large vector artifacts where retention allowed.

Rollback was operational control, not history rewriting.

The generation explained the index. A query record explained how it was used:

• Exact question and normalized form.
• Filters and temporal intent.
• Authorized search-scope identity and policy revision.
• Index generation.
• Retrieval-lane availability and latency.
• Candidate ranks and fusion result.
• Selected evidence-set revision.

This let me distinguish a model or generation regression from one vector-lane timeout that produced partial results.

Detailed candidate traces used short retention outside saved evaluations and evidence, respecting the privacy cost of queries.

Reproducible did not mean identical forever

External model endpoints could change or become unavailable. Database engines and libraries could rank ties differently after upgrades. Hardware and concurrency affected timing.

I used “inspectable replay” rather than promising byte-for-byte reproduction. The system preserved source, manifests, configuration, artifacts where practical, outputs, and stage results. It could explain which conditions were no longer reproducible.

For critical regression fixtures, local deterministic representations and pinned dependencies offered stronger replay. For general generation, provenance bounded uncertainty rather than eliminating it.

Honest reproducibility included limits.

The internal comparison view showed one query across two generations:

• Added, removed, and moved source passages.
• Parser or passage-boundary changes.
• Lexical and semantic lane differences.
• Authority and permission status.
• Evidence-set impact.
• Whether the expected task improved.

Clicking a result opened both source spans. The view did not reduce change to one relevance-score delta.

This made index upgrades reviewable by the kind of evidence they changed.

Old answer freshness became specific

An answer could be old for several reasons:

• The source corpus gained revisions.
• The active index improved retrieval over the same corpus.
• Authority or permissions changed.
• A model or verification policy changed.

K81R showed each separately. “New sources available” differed from “A newer index would choose different evidence.” A reviewer could rerun the relevant stages without assuming the old answer was false.

Freshness became a relationship among source, retrieval, and review systems.

Retaining every generation forever duplicated source text and embeddings. The manifest classified artifacts:

• Source corpus manifests and minimal provenance.
• Parsed nodes and passage records.
• Lexical index.
• Vector index.
• Evaluation reports.
• Query traces.

Large indexes expired after the provenance and rollback window unless an accepted audit need justified retention. Sensitive source revocation could make a generation immediately ineligible and accelerate deletion. Answer records retained exact permitted excerpts or source references according to their own policy.

Provenance was not an excuse for unlimited copies.

Progress bars based on percentage of files hid whether one failed source was important. The build view listed every source as indexed, unsupported with accepted limitation, failed, or excluded by policy.

It showed node and passage counts, embedding failures, analyzer anomalies, permission-schema mismatches, and the oldest source revision not covered. Retrying a failed source created a new build attempt under the candidate generation; activation still waited for a coherent final manifest.

Operational visibility answered “What corpus does this generation actually represent?”

The answer page exposed only useful provenance first

Ordinary readers saw source revision, currentness, evidence set, and whether a newer corpus or index existed. A details panel showed parser, embedding model, lexical schema, and ranking generation for technical inspection.

I did not put a wall of infrastructure IDs beside every sentence. Provenance had layers. The primary interface supported source judgment; diagnostics supported system evaluation.

All layers referred to the same immutable answer record.

Before generation IDs, a changed answer invited speculation: model randomness, new document, parser fix, or ranking tweak. Afterward, I could locate the difference.

The first wrong retry answer used corpus 27, parser 1, fixed windows, embedding generation 3, and semantic-only rank policy. The corrected system used the same source history with structure-aware parser 4, hybrid generation 18, and explicit authority metadata.

The explanation did not prove the new answer universally correct. It made the path from source to evidence inspectable enough to evaluate.

The index generation is part of provenance because retrieval is an authored transformation of the corpus. Citations name where evidence came from. The generation explains why that evidence was the part the system saw.

The final comparison replayed the original retry question across generations 3 and 18. Generation 3 lost the exception at a token boundary, ranked a superseded retry note semantically, and offered no lexical exact lane. Generation 18 preserved the capability table, ranked the current state definition through both lanes, and exposed the older note as superseded. The sources had been present in both corpora. The retrieval machinery determined which relationships became visible.

That result made the manifest more than operational metadata. It was necessary context for evaluating the answer's evidence path. Without it, a citation could show the destination and conceal the route that had excluded a better source.

Two builds with the same named parser policy could behave differently after a tokenizer, Markdown parser, database extension, or numerical library changed.

The generation recorded a resolved dependency manifest, runtime versions, database and index-engine versions, and platform details relevant to deterministic parsing and ranking. Container or environment artifact digests linked the manifest to runnable code where retention allowed.

I did not preserve an entire environment forever for every experiment. Accepted generations kept stronger artifacts; discarded candidates kept compact metadata and reports. The retention level was itself documented.

Configuration provenance without implementation provenance was incomplete.

Randomness was seeded where appropriate

Most parsing and lexical indexing were deterministic. Some approximate vector-index builds and evaluation sampling could vary with insertion order, concurrency, or seed.

The build manifest recorded source ordering, seed, parallelism, and engine settings where controllable. Repeated-build checks measured rank stability on critical fixtures. If an engine could not guarantee exact ordering, ties and expected tolerance were explicit.

I avoided claiming a generation could be recreated byte-for-byte solely from a model name. The original artifacts and observed outputs remained the strongest evidence.

Curated terminology aliases, supersession links, and document relationships influenced retrieval without changing source prose.

The corpus manifest referenced a metadata revision containing those annotations, their authorship, and review. A generation could therefore explain why “uncertain delivery” expanded toward outcome-unknown or why a historical note received current replacement context.

Changing an alias or authority relationship built a new generation and evaluation report. Curatorial work became visible rather than hidden inside ranking configuration.

Partial rebuilds needed a coherent generation boundary

Re-embedding one changed document was cheaper than rebuilding everything. In-place updates risked making a generation ID describe several build moments and configurations.

I allowed incremental construction into a new generation. Unchanged artifacts could be content-addressed and reused if their source, parser, passage, model, permission, and analyzer inputs matched. The new manifest still listed the complete corpus and produced a new immutable generation.

Reuse reduced cost without making provenance incremental and ambiguous.

A generation missing two unsupported diagrams might still answer a code-identifier query and should not answer a question whose evidence could live in those diagrams.

The manifest exposed gaps by project, document, node type, and reason. Answerability checked whether the task's scope intersected a known gap. The refusal page could say that textual sources were complete while diagram content was not indexed.

Coverage was more useful than a single “build succeeded” flag.

Index activation had an authority

The build worker could create candidates and could not activate them unilaterally. A release decision compared integrity checks, permissions, evaluation, coverage gaps, latency, and accepted limitations, then advanced the active pointer with a stable change ID.

The action was idempotent and recorded previous and new generation. Rollback used another decision. Query servers reported which pointer revision they had observed, so mixed rollout was visible.

This was deliberately formal for a personal project. K81R existed to make evidence infrastructure inspectable, including the evidence used to replace that infrastructure.

The browser cached search pages and could show generation 18 results after generation 19 activated. That was acceptable for navigation only when labelled and unsafe for a current answer without revalidation.

Query responses and cache keys included generation. The UI showed when a newer generation was available and preserved selected source revisions. Refreshing search did not mutate an evidence set automatically.

A stale browser result remained understandable because its retrieval generation travelled with it.

If search latency spiked or results disappeared, W93H-like incident records captured active generation, index engine state, failed partitions, release identity, and representative query traces under synthetic fixtures.

Rollback could restore service while the incident retained exactly which index had failed. A postmortem linked observed symptoms to manifest and activation decision rather than referring vaguely to “the search index.”

Search data was deployed software in operational terms. It deserved the same incident identity.

Provenance had a compact human summary

Raw manifests were precise and difficult to read. K81R generated a deterministic summary:

Built from corpus revision 44: 218 source revisions. Parsed with structure policy 4. Embedded with model E3. Ranked with hybrid policy 7. Two diagram nodes were excluded under accepted limitation L12.

Every phrase linked to manifest detail. The summary contained no model-generated interpretation and could be included in answer diagnostics or release notes.

Readable provenance made the deeper record more likely to be used.