Reciprocal rank fusion beat score alchemy

K81R's hybrid search began with a formula I could tune until any demonstration looked good.

combined = 0.55 × lexical + 0.45 × semantic

The arithmetic implied that a lexical score of 8.2 and a cosine similarity of 0.76 were quantities on a shared scale. They were not. One depended on corpus term statistics, field weights, document length, and query shape. The other depended on an embedding model and vector distribution.

I normalized both, added weights, and moved the problem into the normalization.

Reciprocal rank fusion gave K81R a simpler compromise: combine ordering rather than inventing common units.

Raw scores answered different questions

The lexical score increased when rare query terms appeared in important fields and when documents matched according to the analyzer and ranking function. It could vary dramatically between a one-token identifier and a long natural-language question.

The vector score described proximity in one embedding space. Its useful range depended on model, corpus, passage construction, and query. A score that looked strong for one topic could be ordinary for another.

Neither was a probability that a passage supported the answer.

Adding them produced a number with no stable interpretation. More decimal places did not improve it.

Min-max normalization moved with the candidate set

I tried normalizing each lane to zero through one within its returned candidates. The top candidate in every lane became one, even when the lane had weak evidence. Adding one extreme result changed every other normalized value.

A semantic search with several nearly tied generic passages looked as confident as an exact title match. A lexical lane with no shared term still gave its least bad candidate a normalized position.

Z-score normalization depended on distribution shape and enough candidates. Hand-set thresholds drifted after index and model changes.

The normalized numbers were convenient for a formula and misleading in the interface.

Rank was limited and useful

Reciprocal rank fusion used each result's position:

RRF(document) = Σ 1 / (k + rank)

for the lexical and semantic result lists. The constant k controlled how quickly contribution fell with rank. A passage appearing high in both lists accumulated support. A top exact match from one lane still remained competitive even if absent from the other.

RRF discarded score magnitude. That was a limitation and the point. It used the piece of information the lanes could share more honestly: ordering among their own candidates.

I recorded the chosen constant in the index-generation manifest and evaluated changes rather than presenting it as universal.

A document not returned by a lane contributed nothing. If lexical search retrieved 100 candidates and semantic search only 10, the lexical lane had more chances to contribute tail results.

I set lane depths based on recall evaluation and latency budgets, then tested sensitivity. Exact identifier queries needed only a small lexical set to find the intended source. Broad conceptual questions benefited from a deeper semantic set. Mixed queries used enough of both to allow intersection.

The query router could vary retrieval effort, but fusion always recorded actual lane, rank, and depth. A result absent because of a candidate limit was distinguishable from one that scored poorly.

Hybrid retrieval quality begins before the fusion formula.

Exact matches received a rule, not a giant weight

RRF alone could place a semantically popular passage above the exact code identifier a person typed. I added a small number of interpretable priority rules before ordinary fusion:

• Exact stable identifier in its native field.
• Exact quoted phrase in a title or heading.
• Exact source ID or path.

These matches entered a clearly labelled exact tier. RRF ordered remaining hybrid candidates and could order ties within the tier using field and status rules.

I avoided multiplying lexical scores by an enormous constant. The product rule was discrete: when the query clearly names an identity, show that identity first if authorized and present.

The interface could explain the decision without exposing score math.

Duplicate passages merged by source identity

The same source node often appeared in both lists. I merged by immutable source revision and span, retaining both lane ranks and reasons.

Overlapping index representations of one paragraph collapsed into one result. Identical text in two document revisions did not, because provenance differed. A current decision and its superseded predecessor remained separately inspectable.

The fused score belonged to the result representation for that generation. It was not stored as a timeless property of the source.

Merging reduced duplicate cards and made agreement between retrieval methods visible.

A superseded note could match a query exactly. A current decision might use different language and rank lower semantically.

I kept retrieval relevance and document authority as separate features. For an unqualified current-state query, a ranking policy preferred current authority within closely related candidates. Historical filters or explicit dates changed that policy. Superseded results remained visible with status.

I did not force every current document above every historical one. An unrelated current note should not beat the exact historical source a person named.

The result explanation could say “Exact phrase in a superseded decision; current replacement linked below.” Fusion found the text. Governance framed what it meant.

Filters applied before ranks existed

Post-filtering a global top ten could leave too few authorized or scoped results and let hidden candidates influence rank positions.

Lexical and semantic lanes both received the effective corpus, project, document-type, date, status, and access scope before retrieval. Facets and caches shared that scope. RRF saw only candidates the requester could inspect.

If vector infrastructure could not apply a required policy filter safely, the system did not use that lane for the query. It never retrieved broadly and trusted the final renderer to hide sensitive evidence.

Security changed candidate generation, not just display.

One overall ranking metric rewarded common natural-language tasks and hid exact failures. I grouped evaluation:

• Identifiers and code.
• Quoted phrases.
• Filenames and errors.
• Natural-language paraphrases.
• Cross-project concepts.
• Mixed literal and conceptual queries.
• Current-versus-historical decisions.
• No-evidence questions.

For each case, I recorded relevant passages, misleading distractors, and useful scan depth. Exact tasks cared strongly about rank one. Conceptual tasks allowed several relevant sources. Historical tasks judged authority labelling as well as rank.

RRF was kept because it improved coverage across classes without erasing the literal lane.

Offline relevance labels had limitations

I authored judgments from my own archive knowledge. That made the set precise and subject to my expectations.

I recorded graded relevance: directly answers, supplies necessary context, related but insufficient, misleading, or irrelevant. Counterexamples deliberately used similar vocabulary. Later review could challenge a label with source evidence.

The evaluation was a regression tool for K81R, not a universal benchmark. I did not report its score as general search quality.

The best value was diagnostic: which query class and which source relationship regressed after a change?

A fused score stayed out of the UI

RRF produced a number useful for ordering. Showing “87% relevant” would have misrepresented it.

Result cards displayed source identity, status, heading lineage, passage, and retrieval reasons:

• Exact term in heading.
• Related by meaning.
• Ranked by both lexical and semantic search.
• Current decision for selected project.

An advanced diagnostic panel showed lane ranks, fused value, candidate depths, and index generation. The main interface helped a person inspect evidence.

Ranking scores operated the list. They did not certify claims.

Ties revealed policy choices

Several passages received the same RRF score because their rank combinations matched. I needed deterministic tie-breaking for reproducible evaluation and stable scanning.

The order used, in sequence:

1. Exact-tier rule.
2. Query-intent authority preference.
3. Best individual lane rank.
4. Structural field priority.
5. Stable source identity.

The policy was documented and versioned. I did not use random or ingestion order.

Stable ties made saved diagnostic queries comparable across builds and prevented results from dancing when nothing meaningful changed.

Changing the embedding model altered similarity space and candidate ordering. RRF made score calibration less fragile but did not make a model swap free.

I built a new index generation, reran every retrieval task, compared lane and fused ranks, inspected regressions, and activated the generation atomically. Existing answer records kept their original model and index provenance.

The fusion constant and candidate depth were reevaluated because a new lane distribution could affect recall even though raw scores were absent from fusion.

Rank fusion reduced one form of coupling. Evaluation still governed the system change.

Freshness could not be a hidden boost

I experimented with a recency multiplier and watched historical evidence disappear beneath newer summaries.

Freshness mattered only relative to question intent and document governance. Current status was a discrete relationship, not “newer is always better.” A 2019 incident note remained the authority for what happened in that incident. A 2023 retrospective could contextualize it without replacing it.

I removed generic recency boosting. Date filters, current-status preference, and explicit supersession relationships handled temporal intent more honestly.

The search system stopped treating chronology as popularity.

Fusion could amplify bad chunks

A poorly constructed passage containing many query terms and broad semantic context ranked well in both lanes. Agreement between methods did not make the evidence good.

Parser fidelity, structural boundaries, authority metadata, and source rendering remained upstream requirements. The evaluation included misleading fragments created by bad chunking. RRF could not rescue them and sometimes promoted them.

This reinforced the stage model: ingestion, retrieval, selection, and answerability needed separate checks. A better ranking algorithm cannot restore an exception already cut away.

Diversity belonged after relevance

The top fused results often contained several passages from one document. That could be appropriate for a narrow identifier query and limiting for a broad comparison.

I added an optional diversification pass based on query task. It preserved the highest-ranked result, penalized redundant source spans, and allowed related documents to enter within a bounded margin. The interface never diversified exact-identity results away from the named source.

Evidence selection displayed source coverage and let a reviewer include multiple sections deliberately.

Diversity was a product preference, not proof that different documents were independently authoritative.

Latency remained predictable

Lexical retrieval was fast. Query embedding and vector search took longer. For strong exact queries, the interface could reveal a stable exact tier early and append a separately labelled Related section without reordering the selected result.

For hybrid natural-language queries, I waited for both lanes before showing fused order. A list that reshuffled while someone scanned was not worth a few earlier generic cards.

Timeout of one lane produced a partial-results state naming which representation was unavailable. It did not silently present lexical-only order as the normal hybrid result.

Operational behavior stayed visible without teaching score formulas.

The most useful diagnostic screen showed queries where lexical and semantic ranks diverged sharply.

Some revealed exact identifiers semantic search missed. Others revealed vocabulary gaps lexical search could not bridge. Several revealed poor parsing or a broad document dominating vectors. The disagreements suggested improvements to ingestion, terminology, or evaluation—not only ranking weights.

With weighted scores, I had tuned until disagreement disappeared inside one number. With lane ranks preserved, disagreement became evidence about the corpus and task.

Reciprocal rank fusion was not inherently optimal. It discarded magnitude, depended on candidate depth, and required a chosen constant. More sophisticated learned ranking could outperform it with sufficient representative judgments and monitoring.

K81R did not have that data. RRF was appropriate because it was robust, inspectable, easy to reproduce, and difficult to overinterpret if the UI kept the score private.

The breakthrough was not a new ranking invention. It was refusing to manufacture calibration I had not earned.

Lexical and semantic retrieval could agree through rank while retaining their different meanings. The fused list became better after the formula claimed less.

The final regression query paired a rare identifier with a conceptual condition. The exact contract ranked first, the current explanatory decision followed, and a superseded near-match stayed visible with its status. Neither lane could produce that sequence alone. The fusion worked because each result still carried the reason it had entered, and the policy could resolve identity without pretending relevance was certainty.

Corpus size changed lexical behavior

K81R's archive grew unevenly across projects. Adding many 2023 AI notes changed document-frequency statistics and could lower lexical scores for terms that became common, even when old source relevance had not changed.

RRF insulated fusion from the raw magnitude shift but lexical rank could still move. I pinned corpus revisions in evaluation and compared rankings when adding a new project. Regressions were classified as healthy new competition, analyzer drift, or lost authoritative evidence.

For project-scoped queries, term statistics came from the effective searchable corpus where the engine supported it. Global and scoped ranking could legitimately differ, and the query record preserved scope.

The archive was part of the ranking algorithm.

Several retrospectives paraphrased the same Z29C lesson. The vector lane returned them as a block and pushed a distinct adapter contract below its candidate limit.

I added source-span deduplication and light pre-fusion diversification inside the semantic lane. It kept the best passage from a near-duplicate group and allowed other concepts into the candidate set. Exact current sources were never merged with historical paraphrases merely because vectors were close.

The grouping threshold was evaluated and recorded. It reduced crowding without becoming an authority decision.

Candidate diversity before fusion mattered because RRF could only rank what each lane exposed.

Missing lanes affected result semantics

If the embedding service timed out, an early implementation returned lexical results through the same page with no warning. A user could assume hybrid search had found no conceptual matches.

The query result recorded lane status: complete, timed out, unavailable, or policy-disabled. Partial results were labelled and were not cached as complete hybrid responses. Evidence selected from a partial query carried that state into the set.

For exact identifiers, lexical-only results could still answer the task. For broad conceptual research, the interface suggested retrying before concluding the corpus lacked evidence.

Graceful degradation required telling the truth about which representations participated.

Fusion parameters had change budgets

The constant k, lane depths, exact-tier rules, and tie-breakers formed a ranking policy. Changing several together could improve aggregate results and make the cause unknowable.

I varied one dimension at a time against frozen generations where possible, then ran end-to-end candidate latency and scan tests. A policy change needed a written hypothesis, class-level results, notable regressions, and a rollback generation.

Small improvements that moved many familiar exact results were rejected unless they solved a consequential failure. Ranking stability had product value.

This discipline kept tuning from becoming an endless search for a prettier benchmark.

I sketched features a learned reranker might use: lane ranks, exact fields, heading depth, document status, relationship to the query project, structural type, and source diversity. The archive lacked enough independent, representative judgments to train or validate one responsibly.

Synthetic clicks would have encoded the current ranking back into the model. Sparse personal usage could not support a broad claim. I kept the feature inventory and did not build the learner.

RRF provided a strong transparent baseline against which a future learned system would need to demonstrate benefit, stability, and explainability.

Retrieval reasons were tested too

A result explanation could say “Exact heading match” even after a parser change moved the term into body metadata. That would undermine source inspection despite correct rank.

Evaluation asserted the reason fields beside relevance: exact span exists, field is accurate, semantic-only results do not show literal highlights, and both-lane results retain both paths. The UI derived labels from recorded retrieval evidence rather than guessing from the final passage.

The explanation was part of the search contract, not decorative copy generated after ranking.