The chunk boundary broke the citation

K81R cited a sentence saying retries were allowed.

The next paragraph began with “Except when the adapter cannot establish whether the first request was accepted.” That paragraph lived in the next chunk.

The citation was exact. It was also misleading because the indexing boundary had cut the rule away from its exception.

Fixed token windows made every passage fit the embedding model. They did not make every passage independently meaningful.

Chunking had become an invisible editor

The first ingester extracted text, counted tokens, emitted a window, overlapped the next window, and repeated. The approach was predictable and easy to batch.

It also made editorial decisions without recording them:

• A heading could be separated from its section.
• A list introduction could live apart from the list.
• A code block could be split mid-function.
• A table row could lose its headers.
• A warning could be detached from the instruction it qualified.
• A supersession note could fall outside the retrieved window.

Overlap reduced some cuts and produced duplicate partial evidence. The system still could not explain why a boundary was safe.

I treated parsing and passage construction as product architecture rather than preprocessing.

The source document remained authoritative

An index passage was not a new document. It was a retrieval representation of an immutable source revision.

Every parsed node carried:

• Source document ID and revision.
• Original byte or character offsets.
• Node type.
• Heading ancestry.
• Sibling and parent relationships.
• Authored order.
• Parser version.
• Visibility and status inherited from source.

The renderer could always open the exact source revision and highlight the span. Reindexing changed retrieval units without changing source history.

This prevented a common provenance shortcut: citing the text copied into a vector store as though the copy were the authority.

I parsed authored nodes before measuring tokens

Markdown and MDX sources already expressed structure. The parser emitted nodes for headings, paragraphs, lists, code fences, tables, blockquotes, and callouts. Other document formats would need equivalent adapters and fidelity checks.

Token count became a constraint applied after structure existed. A short section could remain one passage. A long section split at paragraph or list-item boundaries. A code block stayed intact unless it exceeded the hard model limit, in which case the ingester marked it oversize and used a language-aware fallback with explicit continuation.

The parser never silently discarded unsupported nodes. It produced a failure inventory or a bounded plain-text representation with a fidelity warning.

Ingestion quality began with knowing what the source contained.

Heading lineage supplied latent context

A paragraph saying “This state cannot be retried automatically” was ambiguous alone. Under:

Z29C
  External delivery
    Adapters without idempotency or status lookup

it became specific.

I stored the complete heading path and included a compact form in lexical and semantic representations. The displayed evidence showed the path separately from source text so the citation did not imply headings were part of the paragraph.

Heading text influenced retrieval but retained provenance as authored structure. Generated synthetic summaries did not replace it.

This improved both search and review. A result could be judged before opening the whole document.

The retry policy used a list of adapter capability combinations. Fixed windows split the list, and one item looked like a general rule.

The parser represented the list container, introductory paragraph, and items. Passage construction could retrieve one matched item while automatically attaching the introduction and enough sibling context to establish the classification scheme.

For display, the exact matched item was highlighted and the supporting list remained visible. Claim citations could reference the item's source span plus its required context envelope.

I capped automatic expansion. A list with fifty items did not become one massive passage. The evidence set could expand further on demand, and answerability recorded when a claim required items not selected.

Structure informed context without forcing every parent to travel forever.

Tables required headers and scope

K81R initially converted tables into lines of cell text. A result for “no status lookup” returned No | No | Manual review, with no indication which columns those values belonged to.

The table representation kept caption or preceding explanation, headers, row identity, and source spans. Lexical indexing included header-value pairs:

idempotency: no
status lookup: no
automatic retry: no
recovery: evidence review

The UI rendered the relevant row in the original table with headers. The embedding text could use a normalized representation, but citations opened the authored table.

If merged cells or complex formatting could not be represented faithfully, ingestion marked the table for review instead of inventing a simple grid.

A fixed window split resolveRetryPolicy between a guard and its return value. Another result retrieved a function without the comment naming its input contract version.

For supported languages, a lightweight syntax parser identified declarations and blocks. Small functions remained intact. Large files created passages around declarations with imports and comments selectively attached. The preceding prose paragraph and heading remained linked, not always embedded into the code passage.

I kept exact code in a lexical field and used a normalized semantic representation for conceptual retrieval. Line numbers and source revision anchored display.

When syntax parsing failed, the ingester fell back to fenced-block identity rather than claiming AST precision.

Code chunking became language-aware only where the implementation could be tested.

Oversize sections required honest degradation

One incident timeline contained a long table beyond the embedding input limit. Truncation would have hidden late recovery evidence while preserving a normal-looking passage.

I created an oversize policy:

• Preserve the full source node and identity.
• Split retrieval representations at safe structural boundaries.
• Record continuation relationships and lost context risk.
• Embed a bounded representation for discovery.
• Require source expansion for claims spanning segments.

The interface labelled partial representations. A citation could point to the exact row, and a reviewer could open the full timeline.

Model limits shaped the index; they did not authorize pretending a partial node was complete.

Context expansion happened after retrieval

Embedding every passage with all ancestors and siblings created duplication and blurred specificity. Embedding too little lost meaning.

I used a two-stage approach. Candidate representations contained the node, heading path, and small required context. After ranking, a structural expander assembled an evidence unit according to node type and task.

For a paragraph, it might include the previous definition. For a list item, the introduction and selected siblings. For code, the declaration and preceding explanation. For a table row, headers and caption.

Expansion rules were versioned and recorded in the evidence unit. The source spans remained individually addressable.

Retrieval found a location. Evidence construction decided what needed to travel with it.

Passage identity could not depend on chunk number

The first IDs looked like document-42/chunk-17. A parser change moved the same paragraph to chunk 19 and broke saved evidence links.

I derived node identity from immutable source revision, structural path, and source span. A passage representation had a separate ID including parser and construction versions. Evidence records referred to source nodes and the representation generation that selected them.

If an author inserted a paragraph, a new source revision naturally produced new node identities. Old answers still rendered from the old revision. The system could attempt lineage between revisions for comparison but did not claim identity automatically.

Stable provenance required distinguishing source continuity from index convenience.

With overlapping windows, one relevant paragraph appeared in three chunks. Recall looked high because several “relevant” results occupied the top ranks. The reader saw duplicates and less evidence diversity.

Evaluation grouped candidates by source span and semantic unit. A task received credit for finding the relevant evidence once. Duplicate overlap consumed the scan budget and counted against diversity.

The interface merged representations that pointed to the same source node, preserving multiple retrieval reasons. Adjacent distinct nodes remained separate until an evidence set expanded them deliberately.

Removing naive overlap lowered raw chunk recall in one report and improved the product result list.

Different material carried meaning at different scales. A definition could be one paragraph. An architecture decision needed context, considered alternatives, and outcome. A code function needed its declaration. A table needed headers.

I set bounds rather than one target:

• Minimum useful semantic unit.
• Preferred range for embedding and scanning.
• Hard model limit.
• Type-specific attachment rules.
• Maximum automatic evidence expansion.

The index-generation manifest recorded these values. Evaluation compared generations on the same source corpus.

Tuning remained empirical, but the units being tuned had semantic names.

Citation spans were smaller than evidence units

A model might need a full section to understand a claim while only one sentence directly supported it.

K81R stored both:

• The evidence unit supplied during generation.
• The exact source span attached to the claim.

The citation renderer opened the narrow support in surrounding context. Verification could check that the cited span belonged to the supplied evidence and source revision.

This prevented a whole-page citation from disguising weak support and avoided pretending the model reasoned from only the highlighted sentence.

Context and support were related, not identical.

Structure-aware chunks made current passages clearer but did not decide which revision governed.

Document status, effective date, and supersession relationships remained metadata outside the chunk text. Ranking used them according to query intent. Evidence sets displayed contradictions and temporal relationships.

The old retry note and current policy could both be structurally perfect. A current-state answer needed the authority relationship to choose. A historical question might need both.

Chunk quality cannot substitute for document governance.

Permissions were inherited before passage construction

A parser could create child nodes from a restricted document. None should enter a global candidate pool and be filtered only at display.

Every node inherited source access scope and any narrower structural restriction. Authorized candidate filters applied before lexical and vector retrieval. Structural expansion could attach only nodes available under the same effective request scope.

I tested a public paragraph adjacent to a restricted callout. Expansion returned the public source without the callout and marked the context envelope incomplete. It did not hint at hidden text through a citation gap or token count.

Evidence construction respected authorization as a structural boundary.

Unit tests checked node types and offsets. They did not reveal a table rendered without a caption or a list whose nested level collapsed.

For representative documents, I rendered source and parsed evidence views side by side at wide and narrow widths. Fixtures covered headings, nested lists, code, tables, callouts, long paragraphs, unusual punctuation, and unsupported MDX elements.

The test compared visible content, ordering, links, line references, and accessibility structure. Parser changes produced visual artifacts for review.

Retrieval ingestion had become part of the reading interface, so visual fidelity belonged in verification.

Changing the parser in place would make saved query results and answer records impossible to interpret.

I built a complete new index generation from one pinned corpus revision. It recorded parser, passage, lexical, embedding, and expansion versions. Offline evaluation compared it with the active generation. Only after it passed did an atomic pointer make it current.

Old generations remained long enough to render and inspect existing answer provenance. Their source documents remained authoritative even after the vector index expired.

Failed documents prevented a generation from claiming complete coverage unless the gap was explicitly accepted and shown.

The migration treated indexing as a data product, not a cache warmed opportunistically.

Better chunks changed the refusal behavior

The retry question now retrieved the exception with its capability table and heading lineage. When the adapter contract was absent, answerability could name the missing relationship. With fixed slices, the system often saw enough related words to answer and not enough structure to know what it lacked.

Improved parsing did more than raise relevance. It made insufficiency more legible.

A result could say, “This passage defines the rule for adapters without a status query, but the selected evidence does not establish which adapter version the question names.” That refusal depended on evidence units retaining their scope.

Users did not need to learn token windows or embedding limits. They did need citations that preserved qualifiers, code that remained intact, tables with headers, and links that opened the exact source.

The system therefore hid chunk mechanics and exposed source structure. Diagnostic views kept parser and representation details available for evaluation.

The original citation failed because the index had edited the document through an arbitrary boundary and then presented the fragment as evidence.

K81R improved when retrieval units stopped pretending to be authored objects. A passage was a versioned representation with lineage back to the document.

The model still consumed bounded context. The product stopped letting that bound silently decide what a source meant.

Footnotes and references carried delayed meaning

One architecture note placed a qualification in a footnote. Plain-text extraction moved footnotes to the end of the document, far from the sentence they modified.

The parser represented reference markers and definitions explicitly. A retrieved sentence could attach its referenced footnote inside the evidence unit while the citation still opened the original marker and definition locations. Broken or reused references became ingestion errors.

External links kept displayed label, destination, source span, and retrieval date only when the archive had actually captured the referenced material. A link was not treated as evidence for the linked claim merely because it appeared in a source.

Authored indirection required lineage, not flattening.

Some project notes used components for warnings, decision status, diagrams, or expandable detail. Rendering their raw tag names into the index produced noise; dropping them removed meaning.

I created an allowlisted adapter registry. A decision-status component emitted structured status metadata and visible text. A warning preserved its severity and body. Presentational layout components exposed their children without pretending the layout name was content. Unknown components caused a fidelity warning and retained a safe textual fallback.

The adapter version became part of the parser manifest. Components capable of execution never ran during ingestion.

This kept MDX expressiveness from turning the retrieval pipeline into either a code-execution surface or a lossy text scraper.

Images and diagrams were not silently ignored

K81R's initial corpus was text-heavy, but several architecture notes depended on diagrams. The parser recorded image source, authored alternative text, caption, surrounding section, and asset digest.

If the alternative text and caption carried enough meaning for retrieval, they became a labelled non-visual representation. The system did not claim to understand unlabeled diagram content. Such nodes appeared in the ingestion gap report and could make a question unanswerable from text evidence.

A citation to an image opened the original asset and source context. Generated descriptions were never inserted as authored alternative text without review.

Coverage reports became more honest after non-text material remained visible as a limitation.

Sentence and word boundary rules tuned for English can split abbreviations, code-adjacent prose, or other languages poorly. The archive was primarily English, so I documented that scope and avoided a claim of universal parsing.

The parser used source language metadata when available and conservative paragraph boundaries otherwise. Token measurement followed the target embedding tokenizer, while source offsets remained in the original text. Normalization never changed citation content.

Fixtures included technical abbreviations, inline code, Unicode punctuation, and mixed code/prose lines. A new language would require its own fidelity and retrieval evaluation before the generation claimed support.

Deduplication preserved independent provenance

The same paragraph sometimes appeared in a release note and later retrospective. Text-hash deduplication could save index space and erase which source made which claim at what time.

I deduplicated embedding computation opportunistically but kept separate source nodes, status, dates, and relationships. Search could group identical text while showing every provenance path. An evidence set selected the relevant source revision, not a floating canonical paragraph.

If two sources copied an error, identical text did not become stronger evidence through frequency. Authority and document relationships still decided how it could support a claim.