The schema closed around the data

The event name was registered. The sensitive data arrived through metadata.

Q2F8's first catalogue defined each event's purpose and core properties, then allowed an optional object for integration-specific context:

interface EventEnvelope {
  name: RegisteredEventName
  version: number
  properties: RegisteredProperties
  metadata?: Record<string, string | number | boolean>
}

The escape hatch seemed practical. Applications could add debugging detail without waiting for a catalogue update. The collection service still knew which event had been sent, and the value types looked harmless.

I connected three synthetic applications and inspected the first day's rejected and accepted fixtures. metadata contained an email-shaped test identifier, a free-form validation message, a URL with a document title, and an error object flattened into a string. All were plausible integration choices. None had been reviewed as part of the event's purpose.

The schema was closed around a small center and open around everything consequential.

I removed the metadata object, rejected unknown properties at the server boundary, and made every schema extension a new versioned decision. The change reduced integration convenience and turned the catalogue into an actual limit.

A typed container can still be unbounded

The metadata definition allowed only strings, numbers, and booleans. That appeared safer than arbitrary JSON.

Primitive type says little about sensitivity.

A string can contain a name, email address, search query, access token, medical detail, complete stack trace, or serialized object. A number can be an age, account balance, precise location coordinate, or database identifier. A boolean can describe a consequential state.

The type system constrained representation, not meaning.

Even the property names were open. One integration used userEmail; another used contact; a third placed the same value inside label. Any downstream scanner would have to infer semantics from an unbounded key space and incomplete naming.

The event's registered properties had descriptions, purpose links, retention, and tests. Metadata had only a generic value type. It was a parallel schema owned by every caller and reviewed by nobody.

This distinction changed my definition of a closed contract. A schema is not closed because its outer object is typed. It is closed when every admitted field and meaning is explicitly enumerated at the authoritative boundary.

Convenience concentrated risk at the worst moment

The escape hatch was justified as a temporary aid during integration. That is exactly when developers know least about which context will become permanent.

A handler catches an unexpected validation failure and includes the server message “for debugging.” A route has a useful user label nearby and attaches it rather than a stable category. An error wrapper serializes everything enumerable. The event begins flowing before anyone inspects real shapes.

Once stored, removing the field does not remove historical values. Queries may depend on it. Exports may include it. Retention and deletion now need to account for an undocumented property. A future schema cannot confidently state what old records contain.

The metadata object made the cheapest local action—attach context—the most expensive lifecycle action—accept unknown meaning into retained storage.

Q2F8 moved friction earlier. A new property required a proposal before collection. The author had to name its decision, allowed values, sensitivity, retention effect, and why existing evidence was insufficient.

This slowed the first event and reduced years of uncertainty the project was explicitly trying to avoid.

Synthetic data found a real architectural defect

Q2F8 used only generated accounts and events. It would be easy to dismiss the email-shaped value because it did not identify a real person.

The fixture was valuable precisely because it exercised the shape of sensitive data without creating the harm. A realistic integrator had found a path the policy model did not govern.

I expanded the synthetic corpus with:

• Email-like and phone-like strings.
• Free-form notes mentioning another test identity.
• URLs containing query parameters.
• Long stack traces and serialized objects.
• High-precision timestamps.
• Small-group category combinations.
• Values near schema length and numeric boundaries.

The goal was not to build a universal sensitive-data detector. It was to challenge whether a proposed field could admit meanings outside its contract.

Detectors ran as secondary warnings. A value not matching a pattern could still be sensitive; a string matching an email pattern might be harmless synthetic content. The authoritative protection was the closed schema and minimization review.

Generated hostile fixtures gave the architecture evidence without becoming an excuse to test on real personal data.

Unknown properties failed before storage

The revised collection gate validated the exact property set for an event name and version.

definition: report_filter_applied v1
allowed:
  selectedRangeClass: enum(current, older, custom)
  initialResultState: enum(nonempty, empty)
  resultAfterChange: enum(nonempty, empty, not_checked)
additional properties: false

An envelope containing query, email, or debug failed admission. The service did not store the payload in a quarantine table for later review. Quarantine would recreate the open collection under a different name.

The response returned a stable error code and safe field names where policy allowed:

schema_extra_property
event=report_filter_applied
version=1
property=query

Values never appeared in the response or ordinary logs. The integration dashboard linked to the definition and suggested the synthetic fixture workflow for reproduction.

The server, not generated TypeScript alone, enforced the property set. A JavaScript caller, stale bundle, or handcrafted request could not bypass it.

Closed meant fail before retention, not accept now and clean later.

Property names became public policy

Adding a field to an event may look smaller than adding an event. In Q2F8, it received the same kind of review because one field can change the privacy character of the whole record.

For each property, the catalogue stored:

name and description
allowed type or finite values
decision contribution
sensitivity notes
source and transformation
cardinality expectation
validation and truncation rules
retention consequence

Cardinality mattered operationally and privately. An open string with millions of unique values can behave like an identifier even if it is labeled “category.” A high-precision timestamp combined with a route and rare outcome can narrow a population substantially.

Transformations were explicit. If the client converted a raw query into a category, the allowed category list and fallback behavior belonged to the contract. The server did not trust a property called queryCategory to be safely categorized merely because of its name.

Property definitions made review possible at the level where data actually entered. The event's purpose could no longer hide a revealing field inside a friendly envelope.

Enumeration beat sanitized free text

One proposed fix kept open strings but applied redaction patterns: remove email addresses, long numbers, and URL query parameters before storage.

Redaction is useful at some boundaries. It was the wrong primary model here.

Pattern-based sanitization cannot know every personal or confidential value. It can corrupt legitimate categories, miss Unicode or contextual forms, and encourage callers to send raw data because the server promises to clean it.

For the studied decisions, finite enumerations were usually enough:

validation outcome: missing_required | malformed | rejected_by_service
range class: current | older | custom
result state: empty | nonempty
help source: inline | summary | explicit_request

The client mapped local detail to a registered category before constructing the event. The server verified membership. Raw messages and values stayed in the transaction or controlled diagnostic path where their own policy applied.

When a real decision required a numeric measure, the definition bounded range and precision. Latency could be bucketed rather than stored at microsecond precision if the decision only distinguished responsive, slow, and timed out.

Sanitization remained defense in depth at logs. Schema design minimized what needed sanitizing.

Versioning preserved old meaning

The first impulse after closing schemas was to edit an existing definition whenever a property became necessary. That would make event version 1 mean different things across time.

Q2F8 treated used definitions as immutable. Adding, removing, renaming, reinterpreting, or changing the allowed values of a consequential property created a new version.

report_filter_applied v1
  selectedRangeClass, initialResultState, resultAfterChange
 
report_filter_applied v2
  adds triggerSource under a new evaluation question

The event name could remain when the core behavior and purpose stayed compatible. A materially different purpose often deserved a new event identity as well as version.

Stored records carried the definition version. Queries declared which versions they understood and normalized explicitly. The catalogue did not project all historical records into the newest shape through null fields that erased why a property was absent.

Old clients could continue sending v1 until its admission window ended. The server never interpreted their missing v2 property as an integration bug.

Versioning turned schema evolution into a visible policy transition instead of mutable JSON folklore.

Generated clients improved ergonomics without gaining authority

Closed schemas create integration friction if every caller hand-builds envelopes. Q2F8 generated TypeScript helpers from catalogue definitions:

events.reportFilterAppliedV1({
  selectedRangeClass: 'older',
  initialResultState: 'empty',
  resultAfterChange: 'nonempty'
})

The helper offered editor completion, prevented extra object-literal properties in typical typed code, attached name and version, and documented allowed values.

JavaScript consumers received runtime development validation from the same generated schema. Both fixtures installed the packed client artifact rather than importing catalogue internals.

Generated code was convenience, not policy. Type assertions, JavaScript calls, stale helpers, or forged requests could still reach the endpoint. Server validation remained mandatory.

The code generator also produced negative fixtures: extra properties, wrong enum values, out-of-range numbers, and version mismatches. A catalogue change could not update client types while leaving server validation stale because both artifacts were tested against the same immutable definition digest.

Ergonomics made the safe path easier. Authority made the unsafe path fail.

Logging was another open schema

After removing event metadata, I found the same pattern in operational logs:

logger.error('event rejected', { event, error, request })

The logger accepted arbitrary objects. Error serializers and request middleware decided which values became retained text. A closed analytics schema could therefore leak rejected data into a longer-lived log store.

Q2F8 created structured safe log events with enumerated fields:

collection_rejected
  requestReceipt
  eventName
  eventVersion
  integrationId
  reasonCode
  safePropertyName?
  payloadDigest

Request bodies, headers with credentials, and error objects did not flow automatically. An unexpected exception logged a protected diagnostic identity and stack from trusted code, not the unparsed payload.

Log schemas had their own retention and access. Debug mode in the synthetic environment could expose generated payloads locally without teaching deployed paths to retain samples.

The metadata incident widened the lesson. Every generic context bag at a persistence boundary deserves suspicion, including logs, traces, queues, and error reports.

Queues preserved schema identity

The collector briefly queued accepted envelopes before purpose storage. If the queue contained raw input and workers validated later, an unknown property would still be retained during the delay.

Validation and admission moved before enqueue. The queued message contained only the normalized registered fields plus the admission receipt:

event definition digest
normalized allowed properties
admission receipt
retention deadline
destination policy class

Workers rechecked the definition digest they supported before writing. A deployment mismatch paused processing rather than falling back to storing the original envelope.

The raw request existed only in bounded request memory and was not written to a dead-letter queue. Failed normalized messages could enter a protected dead-letter path because their schema was already known. Even there, retention was short and payload access limited.

This prevented asynchronous architecture from moving the policy boundary downstream. A queue is storage, even if messages usually live for seconds.

The schema needed to close before the first durable handoff.

A single JSON payload column would preserve event flexibility but make enforcement depend entirely on application validation. For the experiment's small catalogue, I used purpose-specific tables or normalized columns for maintained properties, with definition and admission identity beside them.

This was intentionally less flexible. A worker could not add an unregistered key without a schema and migration change. Query code saw typed columns and finite categories. Indexes reflected actual decisions rather than arbitrary JSON paths.

Not every event required its own table. Related definitions with a shared purpose and stable property family could use one storage schema. The grouping decision was explicit in the catalogue.

Database constraints reinforced enums, ranges, nullability, and definition versions. They did not replace the admission gate, which understood current consent and policy. They protected the store if a worker or migration was wrong.

The storage shape made undocumented drift inconvenient, which was a feature. Flexible JSON remained appropriate for some systems; here, flexibility had already proven to be the mechanism by which policy disappeared.

The table design reflected the promise that only known data would exist.

During rollout, an old client still sent metadata. Silently dropping it would allow the main event through while hiding an integration defect. Rejecting the entire event lost otherwise valid measurement.

I chose rejection during the synthetic migration because no product function depended on analytics availability. The error made the stale caller visible and prevented any ambiguity about what had been admitted.

For a future essential operational path, the tradeoff might differ. A versioned compatibility adapter could accept one exact known old shape, discard a specifically documented field before durable handling, and produce a migration warning. It would need an owner and removal deadline.

What remained forbidden was generic extra-property tolerance. additionalProperties: true could not become a compatibility strategy.

Compatibility preserves known historical contracts. It should not preserve the ability to invent new historical contracts at runtime.

The old metadata path was removed from generated clients, server schemas, fixtures, logs, and documentation. Source search and rejection counters confirmed no remaining synthetic integrations used it before the compatibility error message retired.

Schema diffs became privacy diffs

Catalogue review initially displayed JSON before and after. A one-line property addition could disappear inside formatting.

Q2F8 generated a semantic diff:

added property: triggerSource
type: enum(user, automatic)
purpose contribution: distinguish deliberate filter changes
cardinality: 2
sensitivity: behavioral source
retention: unchanged, 30 days
client versions affected: report-filter helper >=2
policy acknowledgement: not required under current test decision

Changes to type breadth, maximum length, precision, enum set, required status, purpose, category, retention, or deletion policy appeared separately. A change from enum to arbitrary string was labeled as a substantial broadening even if both were represented as strings in TypeScript.

The diff could not decide whether the addition was ethical or legally appropriate. It made the consequence difficult to hide behind syntax.

Activation created an immutable definition version and generated new client and server artifacts. Review state itself did not make the event admissible.

Schema evolution became one of the system's main product surfaces because it controlled what the architecture was capable of knowing.

Closing collection would be incomplete if query tools exposed a broad raw envelope or joined purpose classes freely.

Each analysis request named a registered purpose and definition versions. A purpose-scoped view exposed only the fields reviewed for that use. The ordinary product surface never offered an unrestricted raw export.

Derived tables carried source lineage and their own closed schema. A new aggregate property required review just as an input property did. Aggregation did not erase the meaning or risk of rare combinations automatically.

Query logs recorded schema identity and result size, not row payload. Access credentials could read only the purpose store needed for the current analysis.

This reduced the temptation to use available data for an unrelated question. The system could not make misuse impossible, but architecture stopped presenting unrestricted exploration as the easiest path.

A closed schema should constrain the full lifecycle: admission, queue, storage, query, derivation, and deletion.

Removing metadata forced several outcomes.

Some context became a reviewed property with finite values. Some moved to operational logs under a separate narrow policy. Some remained local because it was useful only for immediate debugging. Some disappeared because nobody could connect it to the event's decision.

The resulting events were less adaptable to unanticipated questions. They were easier to interpret, test, retain, and delete. A reader could know the maximum shape of a record from its immutable definition rather than sampling the data and hoping no caller had been creative.

The architecture also became easier to secure. Validators had a finite surface. Storage credentials could target purpose tables. Deletion jobs could select known schemas. Synthetic fixtures could cover property boundaries. Rejected inputs did not become an alternate dataset.

Constraint reduced machinery that open flexibility would require later.

The escape hatch had been the real API

The incident changed how I review extensible envelopes.

Whenever an API contains metadata, context, extra, attributes, or arbitrary key-value tags, that field often becomes its most durable and consequential surface. The named properties receive design attention. The escape hatch receives real data.

There are legitimate uses for extensible maps, especially when the system's purpose is transporting user-defined data. Q2F8's purpose was the opposite: admit only reviewed evidence under declared decisions.

The correct schema therefore closed around the data at every durable boundary.

A registered event name was not enough. A TypeScript interface was not enough. Redaction was not enough. The server had to reject unknown meaning before storage, and every later representation had to preserve that constraint.

The synthetic email address did not expose a person. It exposed an architecture that would have accepted one.

Closing the schema removed that capability before the experiment ever handled real data. That was a better outcome than building a more sophisticated detector for things the system had no reason to collect.