Retention without a job is documentation

Q2F8 said events were retained for thirty days. The database contained events four months old.

The policy catalogue was not wrong. Every definition had a retentionDays field. Documentation generated a table showing the period. The settings detail could explain it. No code used the number after admission.

I had mistaken a declared intention for system behavior.

The missing piece was not a nightly DELETE statement alone. Retention had to survive event-version changes, partial job failure, queue delay, backups, derived aggregates, and records whose policy identity was malformed. The system needed to show that expired data became inaccessible within a bounded window without deleting too broadly when policy was uncertain.

Q2F8 moved retention into each admitted record as a computed deadline and built deletion as an observable workflow. The catalogue supplied the rule. Admission fixed the deadline under the immutable definition. A worker removed expired records in bounded batches and produced receipts, lag measures, and canary evidence.

Only then did “thirty days” become more than prose.

A global age query erased purpose

The first cleanup script was obvious:

DELETE FROM events
WHERE created_at < NOW() - INTERVAL '30 days';

It assumed every event shared one period and that record creation time equaled retention start. Q2F8's catalogue already contradicted both assumptions.

Operational latency events lived for seven days. A finite product evaluation used thirty. Some immutable preference evidence followed a separate period and store. A queued event could be observed on the client one day and admitted later. A corrected event definition could shorten future retention without rewriting the past.

The delete query had no event version or purpose identity. Updating its constant would make a policy change broad and invisible.

Each admitted record instead carried:

admittedAt
eventDefinitionId and digest
retentionPolicyId
expiresAt
admissionReceiptId
storageClass

expiresAt was computed from server admission time and the immutable policy active at admission. The job did not rerun today's catalogue against historical records and change their meaning silently.

The deadline made lifecycle a property of the record, not a guess embedded in maintenance code.

Retention began at an authoritative event

Which timestamp should start the period?

Client occurrence time could be wrong or deliberately old. Queue insertion time could precede policy admission. Storage write time could lag after the server had accepted responsibility. For Q2F8's event stores, retention began at server admission.

expiresAt = admittedAt + definition.retentionDuration

This slightly extended the life of delayed events relative to their claimed occurrence. It gave the system one trusted and auditable start.

Other data might require a different rule. A temporary upload could expire from creation, a legal hold from an explicit release, or a draft from last activity. The important part was naming the lifecycle event in the policy instead of defaulting every table to created_at.

The catalogue stored both duration and start rule. The admission service materialized the resulting deadline. A policy change could select a new rule only through a new definition version.

Retention is a relationship between a purpose and time. Duration without a defined start is incomplete.

Deadlines were immutable but could be shortened

Once admitted, a record's deadline represented the promise under which it entered. Extending it later because an analysis became interesting would broaden retention without a new collection decision.

Q2F8 prohibited automatic extension. A policy could shorten deadlines for existing records through an explicit migration that wrote a new earlier deadline and receipt. Extending existing data required separate review of whether the original purpose and policy allowed it; the experiment normally rejected the proposal and used new collection if justified.

The storage constraint was:

newExpiresAt <= currentExpiresAt

Ordinary application credentials could not update the deadline at all. A narrow lifecycle migration identity could only move it earlier and had to reference a policy decision.

This prevented a generic backfill or replay from refreshing updated_at and accidentally resetting retention. The deletion job selected the explicit deadline, never last modification time.

Immutability made downstream behavior predictable. Shortening remained possible because reducing retention reduced capability rather than expanding it.

Time-to-live metadata became a security boundary, not a convenient column applications could edit.

Indexing was part of deletion correctness

A worker selecting expired rows without an appropriate index can scan the whole event table, interfere with normal writes, and eventually be disabled because cleanup is too expensive.

Purpose stores indexed expiresAt with stable record identity. The worker selected a bounded ordered batch:

SELECT id, expires_at, retention_policy_id
FROM measurement_events
WHERE expires_at <= :cutoff
ORDER BY expires_at, id
LIMIT :batch_size;

The cutoff came from the server clock captured once per batch. Stable ordering ensured retries revisited a predictable frontier. The delete statement matched both ID and expected deadline so a concurrently shortened or corrected record did not receive unexplained treatment.

Large stores could partition by coarse expiry window, but the small personal experiment stayed with indexed batches. The design documented when partition removal would become worthwhile and what verification it would need.

Retention performance was not separate from privacy. A policy that becomes prohibitively expensive at scale will fail operationally. The storage shape needed to make the promised deletion routine, bounded, and boring.

Deleting every expired record in one transaction produced long locks and an all-or-nothing failure. Q2F8 used small batches under a lease.

For each batch, the worker:

1. Captured a cutoff and worker identity.
2. Selected the oldest eligible records.
3. Validated their policy identities.
4. Deleted them transactionally from the purpose store.
5. Wrote a compact deletion receipt outside the payload store.
6. Updated the lifecycle frontier.

If a worker crashed before commit, the rows remained. If it crashed after commit but before publishing metrics, reconciliation inspected the receipt and store frontier rather than repeating a destructive guess.

Batch identity derived from store, cutoff window, and sequence. Receipts contained counts and policy classes, not deleted payload values.

The worker yielded between batches and enforced a time budget. Normal application traffic retained priority. Backlog clearing could increase bounded concurrency after monitoring confirmed store health.

Small batches made failure local and evidence manageable. They also allowed the job to stop when it encountered an unknown policy without blocking deletion of earlier well-understood records indefinitely.

Unknown policy failed safe in both directions

One synthetic record had an event definition ID missing from the catalogue due to a fixture migration. Deleting it immediately might satisfy minimization and erase evidence needed to understand corruption. Keeping it indefinitely would violate the declared lifecycle.

The worker quarantined only its lifecycle metadata—not the payload into another store—and paused that policy partition. An alert named record identity, unknown definition, admission receipt, and current age. Access to the payload remained restricted and normal analytics queries excluded it.

Resolution could recover the immutable definition, assign the original or a shorter deadline, then resume deletion. It could also delete under an emergency minimization procedure with a recorded decision if reconstruction was impossible.

The system avoided guessing longer retention. It also avoided a broad delete query capable of crossing stores or policy classes because metadata was malformed.

Keeping records past their deadline and sweeping away eligible records too early were different failures of the same policy. Privacy does not excuse destroying data outside declared authority, especially when some records may support necessary operations or rights evidence.

Fail-safe behavior meant restriction, visibility, and bounded resolution rather than silent permanence.

Job success was not rows deleted

A cleanup worker can report thousands of deletions while the oldest expired record remains stuck forever in another partition.

Q2F8 monitored the lifecycle frontier:

oldest expired record age
expired records remaining
deadline-to-deletion lag by policy class
batch success and failure
unknown-policy records
stores with no recent completed scan
canary lifecycle status

The primary service level was the lag between expiresAt and verified absence. A bounded grace window allowed scheduling and retries. Crossing it meant the promise was behind, even if the job was busy.

Counts were useful for capacity. They were not proof of completeness. A scan completion receipt recorded the cutoff through which every eligible record had been considered under a catalogue generation.

Dashboards avoided payload or subject identifiers. Diagnostics linked protected record identities only when investigation required them.

The monitoring question was not “Did the cron run?” It was “How old is the oldest data that should no longer exist, and can the system explain why?”

Canaries exercised months in minutes

A thirty-day policy is difficult to test by waiting thirty days. The retention engine accepted a controllable clock and synthetic policy with a short duration in the isolated environment.

The canary sequence was:

admit known synthetic event
verify purpose query returns it
advance controlled time beyond expiresAt
run lifecycle worker
verify query and direct storage path no longer return it
verify deletion receipt contains no payload
verify a neighboring unexpired record remains

The deployed test environment created a real canary with a short approved test retention and observed wall-clock execution. It used unmistakably synthetic content and its own partition.

Canaries covered selection, deletion, query visibility, and receipts. They did not prove every old record was gone, which is why frontier scans and counts remained necessary.

One canary deliberately used a malformed policy ID to verify the job stopped and alerted without broad deletion.

A retention promise needed a continuously exercised path. Otherwise the least frequent workflow could carry the strongest claim and fail only after months of backlog accumulated.

Derived data had its own deadline

Deleting raw events did not remove daily aggregates derived from them. The first design called those aggregates anonymous and kept them indefinitely.

Aggregation is a transformation, not an automatic release from lifecycle. A small cell or combination of dimensions can still single out a person or group. The aggregate also remains useful only for a declared purpose.

Each derived table gained:

source definition versions
transformation version
purpose
dimensions and suppression rules
retention deadline or review rule
deletion behavior

The aggregation job emitted lineage and removed raw records according to their own deadlines. Aggregate retention could be longer only when its risk and purpose were reviewed separately. Small cells were suppressed or combined before availability.

When the evaluation decision ended, the derived table expired even if its storage cost was negligible. “Already aggregated” did not become “useful forever.”

Lifecycle propagated through transformations rather than stopping at the first delete statement.

Backups could resurrect expired data

The live database deletion worked. Restoring yesterday's backup could bring the record back.

Immediately rewriting every encrypted backup after each event expiry would be complex and risk the integrity of recovery media. Q2F8 used finite, documented backup retention and a restore gate.

Backups expired under a short schedule appropriate to the synthetic experiment. Deletion and erasure workflows wrote compact tombstones containing subject or record identity, policy, and effective deletion—never the deleted payload. Before a restored database could serve reads, a reconciliation step replayed tombstones and current retention cutoffs against it.

The audit view reported the maximum persistence window in backups separately from live deletion. It did not say “gone everywhere instantly.” Backup access remained narrow, encrypted, and unavailable to normal analytics.

This did not solve deletion from every hypothetical medium. It prevented recovery from silently reversing a completed lifecycle and made residual windows visible.

Backup design is part of retention because a dormant copy is still data the system controls.

Replicas and caches followed verified absence

Deleting from the primary store could leave a read replica lagging or a query cache serving old results. The worker's commit was necessary and not sufficient for user-facing absence.

Purpose query services included record deadlines in their filters so an expired row was not returned even before physical deletion. Cache entries had lifetimes shorter than the retention grace and were invalidated by lifecycle events. Replicas exposed replay position, allowing verification that the deletion commit had propagated.

The completion receipt distinguished:

logically unavailable at deadline
deleted from primary
propagated to active replicas
cache invalidated or naturally expired
backup maximum persistence documented

The interface used precise language. A retention job could report complete for active query paths while separately showing backup expiry policy.

Layered evidence prevented one database row count from standing in for the whole system.

This was more machinery than the tiny event volume required for performance. It was the point of the experiment: understand which paths a data-lifecycle promise actually crosses.

Retention changes were migrations

Changing retentionDays from thirty to fourteen was not a documentation edit.

A new event definition version applied fourteen days to future admissions. Existing records could keep their original deadline or be shortened through an explicit migration. Q2F8 chose shortening where the purpose no longer justified the longer period.

The migration preview reported:

records affected by policy and definition
current and proposed expiry ranges
records becoming immediately expired
estimated batch work
derived stores affected
backup and cache considerations
rollback limitations

Shortening is not fully reversible after deletion. Approval needed to recognize that consequence. The job rate was bounded so an accidental policy mapping could be stopped before crossing every partition.

Extending future retention required a new purpose review and policy version. Existing deadlines did not extend automatically.

Treating retention changes as data migrations connected product policy with operational consequence. A number in a catalogue could move a large deletion frontier.

Erasure and retention reused mechanics but not meaning

Account erasure could delete records before their ordinary deadlines. Retention expiry removed records because their purpose time had ended. Both used store adapters and receipts, but they remained separate workflows.

The worker accepted a deletion authority:

retention policy + cutoff
or
erasure request + subject scope

Receipts named which authority applied. An erasure exception under the test policy did not alter general retention. A retention batch did not claim to complete a subject-wide request across other stores.

Shared deletion mechanics reduced duplicate code. Separate identities preserved why the effect occurred and what completeness meant.

This distinction also helped audit. A record absent before its deadline could be explained by erasure rather than appearing as unexplained loss.

Lifecycle operations should share reliable tools without collapsing different policies into one generic purge.

The first job was a cron expression. If the scheduler stopped, nothing inside the retention system noticed.

Q2F8 stored expected scan cadence and the last completed frontier per store. A monitor alerted when no scan advanced within the grace period. Starting a worker after downtime resumed from the oldest frontier; it did not process only “today's” expiry window and skip the gap.

Workers used leases with expiry and fencing identity so two schedulers could not both treat themselves as exclusive after a pause. Deletion remained idempotent at record identity, but receipts and counts still needed one active sequence.

Manual runs used the same code path and authority, with a bounded cutoff. There was no special administrator command that deleted arbitrary dates without a policy preview.

Schedule, ownership, and recovery turned a script into an operation the system could depend on.

A policy promise cannot assume the scheduler is immortal.

The audit screen showed debt before success

The original catalogue page showed retention duration as a reassuring column. The revised view paired it with execution evidence:

declared duration
oldest active record
oldest expired record
current deletion lag
last complete scan receipt
canary result
derived stores
backup maximum window
open failures

A “30 days” label beside an expired-record lag of 47 days could no longer look compliant through typography.

The interface emphasized exceptions and missing evidence. Green status required the canary, frontier, and scheduler to agree. Unknown states appeared as unknown, not assumed healthy because the database query timed out.

This made retention a product surface for maintainers. The number and the job were finally visible together.

The audit view did not expose event payloads to prove deletion. It used identities, counts, policy versions, and synthetic canaries.

Once retention worked, several secondary benefits appeared.

Queries scanned less irrelevant history. Access reviews covered a smaller store. Erasure discovery traversed fewer records. Backups shrank. Old schema versions left active storage sooner, reducing compatibility code. An accidental access path had less data available to expose.

These were not the purpose of deletion, but they showed how privacy limits can simplify systems.

The opposite had been true while records accumulated. Every undefined old event increased query ambiguity, migration cost, and breach consequence even if storage remained cheap.

Deleting on purpose is maintenance, security, and architecture work at once.

A policy became behavior only through evidence

The retention system ended with a simple invariant and a substantial implementation:

After a record's explicit deadline plus bounded grace,
active query and storage paths no longer expose it,
and the system can provide a receipt without recreating its payload.

Per-record deadlines connected policy to data. Indexed batches made deletion operationally safe. Frontiers and lag showed completeness. Canaries exercised the path. Derived stores, caches, replicas, and backups received explicit treatment. Unknown policy paused instead of guessing.

The catalogue's retentionDays field remained important. It stopped pretending to do the job by itself.

Documentation can state an intention and help people understand it. It cannot make a row leave a database. A scheduler invocation can delete rows and still miss partitions, replicas, or months of backlog.

Retention becomes real when the lifecycle is encoded at admission, executable under bounded authority, and observable through verified absence.

Without that, the period beside an event is only a sentence about what the system hopes will happen later.