Exact identifiers still belong to lexical search

The embedding index understood that “safe retry after a timeout” was related to idempotency.

It could not reliably find adapter_no_query_v2.

K81R's archive contained identifiers, filenames, contract names, database fields, version labels, error strings, project names, and deliberately unusual phrases. Those tokens often carried more meaning than a fluent paraphrase. Semantic retrieval was strong when a question and source used different wording. It was weak when one exact string was the question.

I had treated lexical search as a legacy baseline. It became half of the product.

Embeddings compress meaning and discard some identity

An embedding maps text into a dense vector useful for similarity. The representation does not promise that a rare token remains independently recoverable.

Two passages about adapter retries could be near each other even if only one named outcome-unknown. A code and a prose explanation could embed similarly while differing in the exact version or identifier needed for action.

This was not a defect in the idea. It was a mismatch between representation and task.

I classified queries by cues:

• Exact identifier or code-like token.
• Quoted phrase.
• Filename or path.
• Error fragment.
• Natural-language concept.
• Mixed query with an identifier and conceptual qualifier.

The system did not force every query through one semantic representation.

The first missed result looked absurd

I searched for unknown-outcome and received passages about uncertainty, retries, and failure classification. The authoritative state definition ranked below them because the source used the exact token once in a heading and then explained it in language shared by many other notes.

A lexical index placed the heading first immediately.

The failure felt obvious because humans recognize the query as a literal symbol. An embedding pipeline sees text unless the surrounding system preserves lexical intent.

I added an exact-token route that normalized case according to field, preserved punctuation meaningful to identifiers, and searched headings, code, paths, and body with different weights.

Semantic candidates still helped after the exact definition. They no longer displaced it.

Tokenization was domain design

Default text analysis split some identifiers poorly. DocumentRendered, document_rendered, document-rendered, and document.rendered needed both exact and component-level matching.

I indexed several forms:

• Original keyword form for exact match.
• Lowercased keyword where identifiers were case-insensitive.
• Word components split on case and separators.
• Path segments for filenames.
• Language-aware tokens for prose.

The query parser recognized backticks and quotes as strong literal intent. It did not strip punctuation from error codes or semantic versions blindly.

I kept the original text and source offsets so highlights reflected the source rather than normalized tokens. Analyzer changes created a new index generation and evaluation run.

Search configuration was part of provenance, not an invisible tuning knob.

An identifier in a heading, contract declaration, or code block carried different weight from the same string in a historical aside.

The lexical index stored fields for:

• Document title.
• Heading lineage.
• Code and inline code.
• Path and filename.
• Body prose.
• Tags and project.
• Current or superseded status.

Exact title and code matches ranked strongly. Current authoritative decisions received a preference, not an absolute filter. A superseded document could still be the intended target of a historical query.

Field-aware results also supported better explanations: “Exact contract name in code under ‘Delivery adapters’” was more useful than a generic relevance score.

Developers often remember the beginning of an identifier. Prefix matching helped outcome-unk find outcome-unknown and DocumentRend find DocumentRendered.

Unbounded prefix expansion could produce enormous candidate sets and surprising matches. I enabled it only for code-like terms above a minimum length and capped expansions. Short natural-language fragments used ordinary analysis and suggestions.

Fuzzy matching was even more constrained. It could recover a likely typo in a long identifier, but the interface labelled the correction and kept the original query visible. It never silently rewrote a version number, digest, or effect ID.

Precision mattered more than pretending every input had a result.

Error messages needed phrase structure

An exact runtime error often includes volatile values among stable words. Searching the full pasted line could miss the source because identifiers and timestamps differed.

I parsed known variable shapes into placeholders for retrieval while preserving the original query:

Lease expired for job render_84 at 14:08:31

could produce a stable phrase candidate:

lease expired for job <id> at <time>

The lexical search tried the original phrase, stable subsequences, and normalized pattern. Results explained which form matched.

I kept the pattern library small and inspectable. A generic model-generated rewrite would have made it hard to know why a result appeared.

Lexical search failed when the query said “work accepted outside the process” and the source called it an “external effect.” Embeddings retrieved the relevant Z29C discussion despite no shared rare term.

This made semantic retrieval valuable for:

• Conceptual paraphrases.
• Natural-language questions.
• Older notes using different terminology.
• Related design patterns across projects.
• Discovery when the reader did not know the exact name.

I resisted framing one system as accurate and the other as approximate. Each represented relevance differently and failed differently.

Hybrid retrieval needed to preserve both kinds of evidence without converting their scores into a false common probability.

Raw scores could not be added honestly

The lexical engine produced scores based on term statistics and field weights. The vector index produced similarity values whose ranges shifted with model, corpus, and query.

Adding normalized scores required arbitrary calibration. Min-max normalization changed when one extreme candidate entered the set. Hand-tuned weights looked scientific and were brittle across query classes.

I combined rank positions using reciprocal rank fusion in a later iteration. At this stage, I first displayed separate lexical and semantic groups with duplicates merged by source passage identity.

This was less elegant and more diagnosable. I could see when exact search supplied the authoritative identifier and semantic search supplied useful context.

A single relevance number was not a requirement.

I wrote deterministic cues for strong literal intent:

• Backticks or quotes.
• Paths and file extensions.
• Snake, kebab, or camel case.
• Version-like or error-code shapes.
• Long mixed alphanumeric identifiers.

Strong cues guaranteed a lexical retrieval lane. They did not disable semantic retrieval for mixed queries. DocumentRendered authorization benefited from exact contract matches and conceptual policy passages.

Natural-language queries used both lanes by default. The interface allowed Exact only and Meaning only as advanced controls, but the ordinary experience did not ask people to understand retrieval algorithms.

Routing determined candidate effort, not which results the person was allowed to inspect.

Duplicate passages needed stable merging

The same passage could rank in both systems. I merged by immutable source revision and structural offsets, not text equality.

Two revisions containing identical text remained distinct evidence because status and context could differ. Two index chunks overlapping the same paragraph became one source result with multiple retrieval reasons. A code block and its prose explanation remained separate when their source spans differed.

The merged result listed “Exact identifier” and “Semantically related to retry safety” as separate signals. It did not sum them into confidence.

This preserved provenance while reducing visual duplication.

I generated a report of code-like tokens, document frequencies, analyzer output, and whether each could be retrieved exactly. The report caught identifiers lost through punctuation, lowercasing, or minimum-token rules.

Fixtures included:

• outcome-unknown
• workflow_call
• useDeferredValue
• adapter_no_query_v2
• 23505
• render-worker.ts
• Content digests and effect IDs

Some values, such as full random IDs and secrets, should not be indexed broadly. The ingestion policy classified fields before indexing. Synthetic fixtures exercised retrieval without introducing sensitive real data.

Index coverage was a data-governance question as well as a search-quality question.

Lexical failure could reveal documentation weakness

Several concepts had three near-synonyms across the archive. Search results looked fragmented because the writing had never established a canonical term.

I did not force the index to hide that inconsistency. A terminology page mapped preferred terms and historical aliases. Authors could update current documents while preserving old revisions. Query expansion used the curated aliases and showed the expansion.

For example, “uncertain delivery,” “ambiguous outcome,” and outcome-unknown became related without pretending they were always identical.

Retrieval exposed vocabulary drift that architecture documentation needed to address.

A deprecated decision could contain the exact query in its title and outrank a current document. Literal precision did not make it current or correct.

Results displayed source type, date, revision, and relationship status. Ranking preferred current authority for unqualified queries but preserved a visible historical result when its literal match was strong.

If a query included a date or explicitly selected historical documents, that preference changed. The system never discarded older evidence merely because a newer file existed.

Exact retrieval answered “where is this token?” Authority metadata answered “what may I conclude from this source?”

Permissions applied before both lanes

The vector and lexical indexes had to enforce the same access scope. Filtering only after retrieval could let inaccessible candidates consume the top-k set or enter a model context invisibly.

K81R built authorized candidate filters from corpus, project, document status, and subject scope before querying. Index generations recorded which access-policy revision and document visibility metadata they contained. Consequential reads revalidated current source authority before displaying content.

I tested a rare identifier existing only in an inaccessible fixture. Neither lexical suggestions nor semantic neighbors revealed its presence to an unauthorized query.

Hybrid retrieval doubled the places policy could fail. It did not double the policy definitions.

A single recall score hid the failure that started the work. I created categories:

• Exact identifier.
• Quoted phrase.
• Error fragment.
• Filename and path.
• Natural-language paraphrase.
• Cross-project concept.
• Mixed exact and conceptual query.
• No supporting source.

Each task named relevant source passages, unacceptable distractors, and the scan depth at which a result remained useful. Identifier tasks emphasized first-rank precision. Conceptual tasks allowed several relevant passages and emphasized coverage.

I reported both lanes and the combined result. A hybrid improvement could not excuse a regression in literal retrieval.

The interface exposed mode through language

When a query contained a strong exact cue, the result summary said “Exact and related matches for adapter_no_query_v2.” Quoted phrases remained visibly quoted. Suggested spelling changes required acceptance.

Result cards labelled Exact, Phrase, Code, Title, or Related. These labels helped a person decide where to look first without requiring knowledge of BM25 or embedding distance.

An advanced diagnostic view showed ranks, analyzer tokens, and index generation for development. The ordinary interface stayed oriented toward sources.

The system explained retrieval behavior at the level needed to inspect evidence.

Exact keyword lookup was inexpensive and returned quickly. Semantic search required embedding the query and consulting the vector index. For strong identifier-only queries, I could show exact results immediately and load related results later without rearranging selected evidence.

For natural-language queries, I waited for both candidate lanes before fusing order. A result list that jumped as semantic candidates arrived was harder to scan.

Caching used normalized query and authorized corpus generation. Sensitive scope remained part of the key. A cached broad result could not be filtered into a restricted answer after the fact.

Different retrieval paths supported different latency strategies because the product contract remained source-first.

Lexical search preserved the archive's texture

The identifiers in the archive were not noise around meaning. They were part of the meaning: exact workflow states, contract versions, filenames, API names, and errors that connected prose to systems.

Embeddings helped when I did not know the archive's vocabulary. Lexical search helped when the vocabulary was the evidence.

K81R became stronger after I stopped describing literal retrieval as a fallback. It was a first-class representation with its own analyzer, fields, evaluation, explanations, and failure modes.

The eventual hybrid system did not ask one method to imitate the other. It let exact identity and semantic relationship meet at the source passage.

Rare technical phrases still belonged to lexical search because losing them would mean losing the part of the archive that could be verified exactly.

The final regression fixture searched for a newly added contract name that appeared once in code and nowhere in prose. It had to rank first, preserve punctuation, link to the exact source revision, and remain absent outside its authorized scope. Semantic neighbors could follow, but none could replace it. That small test captured the larger rule: when identity is the evidence, similarity is context rather than an answer.

General-purpose analyzers often remove common words. In code and API names, a short word can distinguish meaning: use, as, in, or not may be part of an exact phrase or identifier.

I kept a language analyzer for prose and a literal subfield for every source. Quoted queries and code-like cues searched the literal field without stop-word removal. Natural-language queries could still benefit from stemming and common-word handling.

The evaluation included phrases such as “is not a receipt” because negation carried the lesson. An analyzer that reduced it to receipt would rank text saying the opposite.

K81R did not need one tokenizer to serve prose, code, paths, and decisions equally.

Version numbers needed exact comparison

A query for workflow@2 should not return workflow@20 merely because prefix or token analysis looked close. Semantic versions, contract revisions, dates, and numeric error codes received typed query parsing where their syntax was known.

The index stored the original keyword and, for supported forms, structured components. Filters could ask for a specific contract version or range without pretending text relevance decided numeric order.

Unknown number-like strings stayed literal. The system did not infer that a document ID was a version.

This modest parsing prevented a class of results that looked superficially exact while naming the wrong artifact.

Copied text could contain typographic quotes, different dash characters, composed or decomposed accents, and invisible spacing. Overaggressive normalization could also change code or identifiers.

For prose phrase matching, I normalized compatible quotation and Unicode forms while retaining source offsets. For code and identifiers, normalization was conservative and case behavior came from the field contract. The diagnostic view showed normalized query tokens when a match was surprising.

Fixtures included curly apostrophes in titles, em dashes, accented words, and visually similar but distinct identifier characters. Security-sensitive confusables were not merged silently.

Literal search still required interpretation. The difference was that its transformations were bounded and inspectable.

The first result highlighter reconstructed snippets from normalized tokens and occasionally emphasized text across a sentence boundary that had never appeared as one phrase.

I changed matching to return exact source spans. Phrase highlighting reflected positions from the indexed source unit, and semantic results highlighted the whole supporting passage rather than inventing term-level causality. Expanded context preserved the original document.

For normalized error patterns, the result explained which stable subsequence matched and showed the unmodified source line.

A highlight is an evidence claim: this part of the source contributed to retrieval. It deserved the same provenance discipline as a citation.

Query suggestions came from authorized vocabulary

Autocomplete initially used a global list of frequent terms and titles. That could reveal a restricted project name before search authorization applied.

Suggestions were generated within the authorized corpus scope and separated into exact identifiers, document titles, and curated aliases. Sensitive random IDs were never offered through popularity. The system limited logging of abandoned suggestion queries.

Accepting a suggestion preserved the visible term and its type, so the result summary could say “Contract identifier” rather than pretending a natural-language query had been inferred.

Literal retrieval begins before submission. The suggestion surface had to respect the same access and evidence rules.