Retrieval is not a chat feature

K81R's first interface had one text box and one promise: ask a question about the project archive.

That promise made retrieval look like plumbing. The visible product was the answer; search merely supplied context to a model.

Then I asked for the exact Z29C adapter contract that prohibited automatic retry. The system retrieved the correct document, passed a nearby paragraph to the generator, and returned a fluent summary that omitted the qualifying sentence.

The useful result had existed before generation. The interface hid it behind a less reliable transformation.

I changed the product model. Retrieval would be a complete user outcome. Generation would become an optional operation over a visible, bounded evidence set.

The archive already had questions search could answer

K81R indexed personal project notes, architecture decisions, release manifests, incident reconstructions, case-study drafts, and technical references accumulated through P6X4. The corpus was bounded, versioned, and synthetic where operational detail was needed.

Many questions were navigational:

• Where is the write-fence procedure?
• Which document defines outcome-unknown?
• Show decisions changed during P6X4's consolidation.
• Find the release that introduced render intent version three.
• Which notes mention both idempotency and status lookup?

Returning matching sources, headings, passages, dates, and revisions answered these questions directly. A generated paragraph added latency and another failure mode.

I separated finding, inspecting, selecting, and synthesizing instead of forcing each through the last step.

Search had its own product contract

I wrote the contract before choosing algorithms:

Given an authorized corpus revision and a query, return inspectable source passages with stable identity, location, revision, retrieval reason, and enough surrounding structure to judge relevance.

The contract excluded answer generation. It required:

• Direct navigation to the source.
• Visible document title, type, date, and revision.
• Heading lineage and passage boundaries.
• Match explanation appropriate to the retrieval method.
• Filters for project, date, document type, and status.
• A way to expand context without losing the matched passage.
• Disclosure when the index was behind the source corpus.

Search could now succeed or fail on its own terms.

The question box had overloaded several intents

“What did Z29C do after a timeout?” looked like one conversational question and contained multiple possible tasks:

• Locate Z29C documents mentioning timeout.
• Identify the authoritative adapter contract.
• Compare several revisions.
• Explain the current rule.
• Recommend an action in a current workflow.

The last task required live state and policy that the archive did not contain. A chat interface encouraged one response to slide across all five.

I added explicit modes through progressive controls rather than a mandatory form. A query began as Search. A person could select passages and choose Compare or Draft answer. Consequential workflow action was outside K81R entirely.

This did not prevent natural-language queries. It prevented natural language from erasing task boundaries.

The archive contained an early Z29C note recommending three retries and a later decision rejecting that rule for adapters without idempotency or status lookup. A generated answer tended to choose one or blend them.

Search displayed both with dates, revision status, and heading context. The later decision record was marked current; the older note was visibly superseded but still inspectable.

That disagreement was part of the story. It explained why the rule existed.

I added relationship fields to ingestion:

• Supersedes and superseded-by.
• Amends.
• Implements.
• References.
• Derived-from.
• Contradicts, when manually curated.

The retrieval page used these relationships to show history without asking generation to infer temporal authority from prose.

A result needed more than a snippet

Traditional search snippets optimized for query-term density. For technical evidence, a dense excerpt could omit the definition, precondition, or exception one sentence away.

K81R stored authored structure during parsing: document title, section hierarchy, paragraphs, lists, code blocks, tables, callouts, and source offsets. A result returned the matched unit plus a bounded context envelope chosen by structure.

A paragraph under “Adapters without query support” included its heading chain and adjacent list when the list defined the exception. A code result kept the complete fenced block and its preceding explanation. A table match included headers and the full relevant row.

The interface could expand to the enclosing section or open the exact source revision. The initial result remained small enough to scan.

Retrieval quality was partly a document-rendering problem.

Result identity survived reindexing

The first index assigned passage IDs from array position. Changing the parser split one paragraph differently and made saved links point elsewhere.

I defined source identity from document ID, source revision, structural path, and source offsets. The index generation had its own identity, but a passage referenced the immutable source location it represented.

When a new parser changed chunk boundaries, old answer and evidence records could still render from their stored source revision and offsets. New searches used the new generation. The system did not pretend the passages were identical merely because their text overlapped.

This became important later for provenance. Retrieval is reproducible only when source, parser, and index generation are distinguishable.

Lexical search remained first-class

Embeddings made semantic retrieval compelling, but exact identifiers, filenames, API names, error codes, and rare phrases were common in the archive.

A query for effect_9f2 should not be transformed into “concepts related to effects.” A query for outcome-unknown should find the literal state definition even if a semantically similar note ranked higher.

K81R supported lexical search with token, phrase, prefix, and field-aware matching. Results showed highlighted terms and fields. Semantic retrieval ran alongside it for conceptual questions, not in place of it.

The interface let direct identifier matches appear with high clarity rather than disguising every rank as model intelligence.

Date, project, document type, current status, and source collection changed what counted as a useful result.

The first prototype retrieved broadly and filtered after ranking. That allowed inaccessible or explicitly excluded passages to influence candidate limits and later generation even if they were hidden from display.

I applied authorization and hard scope before retrieval. User-selected filters also constrained candidate generation where possible. Soft preferences such as “prefer current decisions” influenced ranking while preserving visible superseded evidence when it directly matched.

The query summary showed the effective scope: “Searching 184 current and historical notes in Z29C and P6X4 through corpus revision 27.” Empty results therefore had an interpretable boundary.

Search was not one global similarity function over whatever data happened to be indexed.

Search results became selectable evidence

Each result had a Select action. Selection created an evidence-set draft containing source identity, revision, exact passage, heading context, retrieval query, and index generation.

The set was visible in a side region and editable. A person could remove a weak passage, add a contradictory one, expand context, or reorder material before requesting a comparison or draft.

Generation never received an invisible top-k result list. It received the selected evidence artifact plus the task. The evidence set persisted even if the generation failed.

This made retrieval output useful across several actions:

• Share a reading list.
• Record support for an architecture decision.
• Compare old and current policy.
• Export references.
• Draft a bounded answer.

The search product did not disappear when an answer feature was unavailable.

I avoided a single “confidence” percentage. Lexical and semantic systems produced scores with different meanings and distributions. A high vector similarity was not probability that a passage answered the question.

Results used concrete explanations:

• Exact phrase in heading.
• Identifier match in code.
• Current decision for selected project.
• Semantically related passage.
• Referenced by an already selected document.

An expandable diagnostic view showed rank contribution and index generation for evaluation. The ordinary interface described why a result might be worth inspecting without claiming epistemic certainty.

Trust came from source inspection, not a decorative score.

Empty results were a useful result

The chat prototype treated no retrieval as a prompt problem and still generated from general model knowledge. K81R's archive question then received an answer that sounded relevant but was not about the archive.

Search returned a bounded empty state:

• The effective corpus and filters.
• Query interpretation, including exact phrases.
• Suggestions to remove a filter or try a known identifier.
• Related source collections that were not currently included.
• A clear statement that no evidence had been selected.

Generation stayed unavailable until evidence existed or the task explicitly allowed general brainstorming outside archive claims.

Not finding support was information. The interface stopped converting that information into plausible prose.

Search latency had a different budget

Direct retrieval needed to feel exploratory. A person might reformulate several times, scan, and select. Waiting for generation on every query would have made that loop slow and expensive.

I streamed or progressively rendered result groups only when each group was stable and inspectable. Exact lexical matches appeared quickly. Hybrid ranking settled before presenting semantic candidates as ordered results. Facet counts could update later without rearranging selected passages.

Queries were cancellable because they had no durable external effect. Evidence-set changes were durable local state. Generation was a separate accepted job with its own record.

Separating the stages produced clearer latency and cancellation semantics.

Retrieval evaluation used findability tasks

An answer benchmark would not reveal whether search itself was good. I built tasks with known relevant source passages and distractors:

• Find the exact definition of a workflow state.
• Locate the latest authority decision and its superseded predecessor.
• Retrieve an identifier inside code.
• Find a concept described without the query term.
• Preserve a table row and headers.
• Exclude a similarly worded inaccessible fixture.
• Show that no source supports a fabricated project claim.

I measured whether relevant evidence appeared in a scan-sized result set, its rank, whether context was sufficient to judge it, and whether the route linked to the correct source revision.

This was not the same as measuring whether a model later wrote an acceptable answer.

Query logs can reveal interests and sensitive identifiers. For the personal laboratory, I used synthetic tasks and local evaluation fixtures. The product design still specified purpose and retention.

Evaluation runs stored query IDs, corpus generation, expected passages, rankings, and failure classification. Free-form exploratory queries were not retained indefinitely by default. Saved evidence sets preserved the user's chosen sources without requiring a permanent raw-query history.

Q2F8's lesson applied: search usefulness did not justify collecting every expression of curiosity forever.

The interface supported source-first reading

Opening a result placed the matched passage in its full document, highlighted without replacing the original text. The table of contents showed its heading lineage. Revision status and related decisions remained visible.

Back navigation preserved query, filters, scroll position, and evidence selection. Search did not punish inspection by resetting the result list. Keyboard commands supported moving through results and selecting evidence without trapping focus.

On narrow screens, the evidence set followed results in reading order and could open as a labelled region. Selection state remained clear without relying on color.

These details made source inspection a primary flow rather than a detour from chat.

Generation earned an explicit verb

The action was not “Ask.” It was “Draft an answer from 4 selected passages.”

The wording declared input scope and provisional status. The generated result listed the evidence set, claim links, model and prompt revision, and review state. Removing a passage created a new evidence-set revision; it did not silently mutate an existing answer record.

If the selected passages contradicted one another or did not support the task, answerability checks could refuse or suggest a narrower question.

The interface made generation a transformation that consumed retrieval output. It was no longer the invisible default destination of every query.

Search could be the end

The most useful K81R session in January ended with three selected passages and no generated answer. One passage defined Z29C's unknown state. Another documented an adapter with idempotency support. A third showed the recovery console policy.

Reading them answered my question and preserved nuance I would have had to reconstruct from a summary.

I saved the evidence set beside a P6X4 decision record. It was concise, attributable, and usable without a model.

That outcome changed how I judged the product. A session did not fail because it ended at sources. It succeeded when the person found enough evidence to continue responsibly.

Retrieval is not a chat feature. It is an information product with its own identity, policies, interface, evaluation, and stopping point.

Generation can build on it. It should not make retrieval disappear.

Sources remained valuable even when synthesis was unnecessary, unavailable, or refused.

A saved search preserved a question, filters, corpus scope, and ranking preferences. Running it later against a newer index could produce different passages. An evidence set froze exact source revisions and spans selected at one moment.

I kept both identities. A reviewer could rerun the search to discover newer material without silently changing the evidence behind an existing decision. The interface showed “3 new results since this evidence set was created” and offered a comparison.

This distinction prevented a dynamic query from masquerading as stable provenance. Search is a recipe; selected evidence is an artifact.

Corpus freshness was visible before querying

K81R's ingestion ran asynchronously. A newly committed P6X4 note could exist in the source collection before lexical and vector indexes included it.

The search page showed source corpus revision, active index generation, ingestion completion time, and any failed documents. A query response inherited those values. If the index lagged, direct source navigation remained available and the UI did not say “no documents exist” when it only knew “no indexed passage matched.”

For evaluation, I could pin a corpus and index generation. For ordinary exploration, the page offered refresh only when a newer complete generation was ready. It did not mix results from half-built indexes.

Freshness became part of the search result's truth conditions.

Facets could leak or mislead

Project and document-type counts looked like harmless navigation. Computing them across the global index before authorization could reveal that a restricted project or rare document existed.

Facet aggregation used the same pre-retrieval scope as candidates. Counts described the authorized corpus and active filters. Very small sensitive categories could be omitted according to policy rather than exposing one hidden record through a number.

I also distinguished total source documents from indexed passages. A project with fewer passages did not necessarily contain fewer documents; chunking strategy affected counts. The interface used facets for navigation, not as corpus analytics pretending to be exact knowledge volume.

Ranking changes needed release notes

Improving an analyzer or fusion strategy could move familiar results. I treated index-generation changes like product changes.

An offline comparison reported tasks improved, regressed, or unchanged. A small set of saved diagnostic queries rendered old and new ranks side by side with source reasons. Only a complete generation became active, and existing evidence links remained on their original source revisions.

The release note described behavior in plain language: exact code identifiers now rank before semantic neighbors; superseded decisions remain visible but receive a lower default preference. It did not announce an unexplained relevance-score increase.

Retrieval deserved change management because people learn where evidence appears.

Search could support correction without editing sources

A misleading result sometimes reflected missing relationship metadata rather than poor text. The interface allowed a curator role within the personal lab to mark a document superseded, add an alias, or relate a decision to its implementation. The correction produced a new corpus revision and audit record.

It did not rewrite historical source content. It changed current retrieval metadata with provenance. Evaluation then showed whether the intended query improved and whether unrelated tasks regressed.

This made search quality an interaction between authored documents, curated relationships, and algorithms—not a parameter-tuning contest hidden from the archive.