Search permissions belong before retrieval

K81R's first authorization test passed and the system still leaked a restricted fact.

The final result list removed inaccessible sources. Citation links checked permissions again. The generated answer cited only visible documents.

The vector retriever had already placed a restricted passage in the top candidates. The prompt builder included it, and the model paraphrased the fact without citing that passage.

Filtering the output had protected the links. It had not protected retrieval.

I moved authorization to the point where the candidate corpus was constructed and followed it through every derived surface.

Retrieval changes what the system knows for a request

Search is not a neutral scan followed by display filtering. Candidate membership affects:

• Which passages occupy the top-k limit.
• Ranking positions and fused results.
• Facet counts and query suggestions.
• Snippets and highlights.
• Evidence-set proposals.
• Model context and generated claims.
• Caches and diagnostic logs.

If inaccessible material enters any of these stages, hiding the final card is too late.

The effective authorized corpus became an input to retrieval, not a predicate applied to its output.

One policy produced several index filters

K81R documents carried project, collection, document status, visibility class, subject relationships, and occasionally narrower structural restrictions. The authoritative policy service evaluated subject, action, resource, and current policy revision.

The search layer could not call that service independently for millions of candidate passages after ranking. I created a search-scope token from a bounded policy decision:

subject_scope_id
allowed_collection_ids
allowed_project_ids
visibility_classes
policy_revision
expires_at

Lexical and vector adapters translated the same scope into their supported prefilters. Unsupported policy shapes failed closed or used a smaller authorized index partition. They did not retrieve broadly and promise to clean up later.

The token was signed and internal. It was not a client-authored filter.

Document permissions propagated to passages

During ingestion, every source node and retrieval representation inherited document identity, revision, access class, and policy-relevant attributes. A public paragraph inside an otherwise restricted document did not become public accidentally; visibility needed an explicit structural rule.

If a source changed visibility, a new corpus revision and index generation updated its passages. Until the new generation became active, consequential access used current source validation, and queries could exclude a stale generation whose scope metadata was unsafe.

Passage metadata was a search projection of authority. It remained rebuildable and versioned.

The vector itself contained no permission. Its candidate record did.

Filtering before top-k protected result quality too

Imagine the five nearest semantic passages are restricted and the next five are public. Retrieving five globally and filtering produces an empty public result despite relevant accessible sources.

Pre-filtering lets the vector search find nearest candidates inside the allowed corpus. Lexical term statistics and ranks also reflect the effective collection rather than positions occupied by hidden documents.

Security and relevance aligned here. The authorized user received better results because inaccessible material never consumed the candidate budget.

I tested exact and conceptual queries where a restricted near-duplicate would otherwise dominate.

A project filter showing “1 document” can disclose a restricted project's existence. A title suggestion can reveal a code name. A result-count change can show that hidden evidence matches a rare query.

Facet aggregation and suggestions used the authorized corpus before counting. Sensitive small categories could be suppressed according to policy. Error and empty states avoided distinguishing “hidden matches exist” from “no matches exist” when that distinction was not allowed.

The interface showed the effective visible scope: “Searching 184 sources available in Z29C and P6X4.” It did not show a global total beside an accessible subtotal.

Navigation metadata was part of the security surface.

Query rewriting could leak vocabulary

Curated aliases and spelling suggestions came from indexed terms. A global vocabulary could suggest a restricted contract identifier to someone who typed its prefix.

Suggestions were generated within authorized scope or from a public curated dictionary. Query expansion recorded which visible alias produced a term. Model-assisted rewriting, when evaluated, received no broader source context than the query scope and could not introduce hidden entities from a global memory cache.

The original query remained visible, and expansions could be inspected.

Retrieval begins before the search button; autocomplete and rewrite belonged inside the same boundary.

The first semantic-query cache used normalized question and index generation. A broad-scope result could be returned to a narrow-scope request and filtered later, reintroducing hidden candidate influence.

Cache keys included search-scope identity, policy revision, corpus/index generation, query, filters, and retrieval policy. Highly individualized scopes reduced hit rate. I accepted that rather than sharing unsafe candidate sets.

Where many users shared an identical public corpus, the scope ID allowed safe reuse. Cached evidence sets and answers retained their own source access records and revalidated on open.

Performance optimization did not get a looser authorization model.

The prompt builder accepted an immutable evidence-set revision. Every unit had already passed pre-retrieval scope and current source validation. Before serialization, the generation job revalidated the requester and intended output scope.

No background “helpful context” search ran under a service-wide identity. The model saw exactly the units visible in the set manifest plus permitted task instructions.

If a required source became inaccessible between selection and generation, the job stopped with an answerability state. It did not omit the source and continue producing a conclusion from a now-incomplete set.

Authorization changes could alter whether an answer was possible.

Service identities had separate scopes

Indexing required reading source material and writing search representations. Querying required only the current request's authorized candidates. Generation required selected evidence, not database-wide read access.

I used distinct runtime identities:

• Ingestor: read approved source collections, write a new index generation.
• Search API: query indexes with enforced scope tokens, no raw corpus sweep.
• Evidence loader: open exact authorized source revisions.
• Generation worker: read one evidence manifest and write one answer record.
• Evaluation worker: access synthetic and explicitly approved fixtures.

The model adapter had no direct search or source credentials.

This limited the consequence of a prompt or adapter defect.

Search debugging pages listed top candidates, even those filtered later. Logs recorded raw query strings and source IDs. Evaluation snapshots captured passages.

I applied role-based access to diagnostics, redacted or hashed sensitive identifiers where full values were unnecessary, and set short retention for exploratory queries. Ordinary request logs recorded scope ID, result counts, generation, latency, and classified failure without copying passage text.

Synthetic fixtures carried detailed traces for development. Real personal notes remained minimized.

Security boundaries include the tools built to prove security.

I created public and restricted documents with nearly identical structure. The restricted one contained one distinctive fact and rare identifier.

Unauthorized tests asserted that the fact or identifier never appeared in:

• Exact or semantic results.
• Snippets and highlights.
• Suggestions and facets.
• Result counts distinguishable from a no-match corpus.
• Evidence proposals.
• Generated answers and refusals.
• Caches later opened under another scope.
• User-visible diagnostics.

The public neighbor could not become a citation costume for a fact learned from the restricted source.

This scenario caught the original leak and several subtler variants.

Embedding indexes needed deletion and revocation paths

A source removed from the corpus could leave its vector and lexical terms in an old index generation. Deleting the primary document did not erase retrieval representations automatically.

Index generations had source inventories, retention, and retirement. A sensitive revocation could mark a document denied immediately at query prefilter while a clean generation rebuilt. Old generations lost query eligibility and were deleted under policy once no provenance record required their metadata.

Answer records retained source references according to their own lifecycle and revalidated access. They did not make old vector stores permanent archives.

Q2F8's copy classification applied to embeddings too: they were derived representations with a source and deletion contract.

Permission changes could invalidate selection

An evidence set created under policy revision 12 might include a source unavailable under revision 13.

Opening the set revalidated units. The interface could show a redacted placeholder and explain that the set was no longer complete without revealing the hidden passage. Generation and sharing used current scope.

The historical record preserved that a source had been selected, under restricted audit access, but ordinary viewers could not inspect or derive from it.

I did not silently remove the unit and present the old answer as still fully supported.

A reviewer with broad access could generate an answer and then share only public citations. The prose might still contain facts inferred from restricted evidence.

K81R computed an output-scope ceiling from every evidence unit used by each claim. Publishing or sharing to a broader scope required regenerating or reviewing from an evidence set restricted to that scope. Removing citation links was not enough.

Claim records made this enforceable: every claim carried its evidence dependencies. A response with mixed scopes could be partially shared only when independent public claims remained supported without hidden context.

Information flow followed evidence, not the reviewer's ability to copy text.

Timing and counts were considered side channels

Perfectly equal timing was not realistic, but obvious differences could reveal that hidden candidates existed or expensive restricted sources were processed.

Pre-filtering reduced this risk because inaccessible candidates were never loaded or reranked. Empty and denied queries followed similar response shapes. Public errors did not report the number of filtered sources. Rate limits and logging avoided per-hidden-document variation.

I documented residual timing risk and kept the personal-lab threat model bounded. The system did not claim resistance to every statistical side channel.

Security work included stating what it had not solved.

Changes to access attributes, filter translation, cache scope, or index visibility created a new policy revision and test matrix. An index manifest declared the visibility schema it supported. A query refused to use a generation incompatible with current policy requirements.

This prevented a new authorization rule from being applied only in the application while an old vector index lacked the attribute needed to enforce it before candidate selection.

Migration built and verified a compatible generation before activating the rule where possible. Urgent revocation used a deny overlay that applied before both lanes until rebuild completed.

The answerability system could say that required evidence was unavailable under current scope. It did not decide whether the requester should gain access. That process lived outside K81R's generative path.

The refusal page avoided listing hidden requirements. It could offer an access request link only when a disclosed resource and established workflow allowed it. A model never drafted authorization policy into existence.

Search, generation, and refusal consumed policy decisions. They did not negotiate them through prompts.

The server remained the boundary

The client sent filters and displayed results. It never received a broad candidate list to hide locally. Inspecting network traffic revealed only authorized passages and counts.

Source links resolved through server authorization. Evidence selection commands validated passage identity and scope rather than trusting that an ID had appeared in the UI. The client could not construct a restricted evidence set by guessing stable IDs.

The design followed an old lesson from Q2F8 and P6X4: a well-behaved interface is not enforcement.

The resulting architecture had more explicit scope in cache keys, index metadata, evidence manifests, and tests. It did not make ordinary search cumbersome.

Authorized candidates ranked normally. Facets reflected visible material. Evidence selection behaved the same. Broad-scope reviewers could inspect broad sources; narrow-scope users received relevant results within their corpus.

The key difference was invisible and crucial: restricted material never became an uncredited ingredient in the answer.

Search permissions belong before retrieval because candidate selection is already information use. Once a passage enters ranking or context, hiding its citation cannot make the influence disappear.

The boundary must hold before similarity, ranking, synthesis, and caching ever get the chance to use it.

Physically separate indexes for visibility classes simplified some threat boundaries and multiplied build, ranking, and migration work. One metadata-filtered index simplified operations and depended heavily on correct prefilter support.

I used a public index for openly shareable archive content and a restricted index family for scoped personal material in the lab. Queries constructed a plan across only the partitions authorized for the request, then fused results while retaining partition provenance. No service identity could query every partition by default.

The choice was not presented as universally safer. It matched the small number of stable visibility classes and made an accidental unfiltered public query incapable of reaching restricted vectors.

Partition strategy became part of the threat model and index manifest.

Result counts were scoped facts

Search pages often say “42 results.” In a hybrid system, that number could mean all lexical matches, semantic candidates above a threshold, fused candidates retrieved, or an estimated total.

K81R displayed counts only when they had a clear authorized meaning. Lexical result totals came from the scoped index. Semantic search showed “Top related passages” rather than inventing a corpus-wide total. Fused pages avoided adding incompatible counts.

This reduced a side channel and improved accuracy. A number can leak hidden corpus structure and still misdescribe the visible search.

I experimented with generating document summaries and keywords during ingestion. Those derived fields could reveal restricted content if copied into a broader metadata index.

Every enrichment inherited source visibility, revision, model provenance, retention, and deletion. Public suggestion indexes accepted only public authored or reviewed metadata. Generated keywords never crossed scope because they seemed innocuous.

The ingestor's broader read capability did not authorize it to publish derivatives broadly. Output policy was explicit per field.

Revocation interrupted in-flight generation

A generation job could start with valid evidence and continue after one source was revoked. Revalidating only at job creation left a window before answer publication.

The worker checked evidence authorization before model invocation and again before committing the answer record. Long-running jobs listened for scope-generation changes and could abort early. The committed answer carried its policy revision and output scope.

This did not retract text already seen by an authorized requester. It prevented a newly unauthorized result from becoming a shareable current artifact after policy changed.

A model with a generic source-search tool could issue a broader query than the evidence workflow. I kept tool execution outside the model for K81R's 2023 path. The application constructed authorized retrieval and supplied selected evidence.

In a future tool-using design, every call would receive the same bounded subject scope from the orchestrator, not parameters supplied solely by generated text. Tool output would return through source validation before entering context.

Prompt instructions such as “search only this project” were not authorization.

A broad-scope reviewer might compare a public decision with a restricted draft. The evidence set correctly inherited restricted output scope. If the useful conclusion could be supported entirely by public evidence, the reviewer created a public-only set and regenerated or rewrote the claim.

The system showed which claims depended on which sources, making partial downgrading possible without assuming it. Reviewer access alone did not confer the right to publish derived restricted facts.

Information classification travelled through synthesis.

Backup and analytics copies stayed in scope

Search indexes were not the only derivative. Query caches, evaluation snapshots, failed prompt payloads, observability traces, and backups could contain passages or identifiers.

The data inventory linked each copy to source class, purpose, encryption, retention, and deletion. Synthetic fixtures replaced source content in most diagnostic tests. Restore drills rebuilt indexes under current policy instead of reviving obsolete visibility metadata blindly.

Removing a source from active search was incomplete while a broad debug snapshot could still expose it.