A write fence is kinder than permanent dual writes / Bob's Site

The safest migration plan seemed to be writing every change to both the old document service and P6X4.

If either system failed, the other would still have the data. Reads could switch gradually. Rollback would remain available. No dramatic cutover would be necessary.

The first failure proved the opposite. The old write committed, the new write timed out, and the client did not know whether P6X4 had accepted it. A retry could produce a second revision on one side. The migration now needed to reconcile two authorities while preserving one user's intent.

Permanent dual writes did not remove a cutover. They stretched it across every request.

Two successful calls were not one decision

The prototype placed two database calls beside each other:

await oldDocuments.update(command)
await harborDocuments.update(command)

Reversing the order changed which side could be ahead. Running the calls concurrently made the result faster and more ambiguous. Retrying failures required idempotency on both systems and a coordinator that knew each outcome.

I could add a distributed transaction protocol, but neither database nor application had been designed for it. More importantly, the product decision already had a natural authority. A document revision should be accepted once against one current version.

I replaced dual writes with a write fence: a durable routing rule that named exactly one authoritative writer for a scoped set of document identities and epochs.

Readers could observe both systems during migration. Writers crossed the fence deliberately.

The fence was data, not deployment folklore

The earliest routing flag lived in gateway configuration: WRITE_TO_HARBOR=true. It could differ across replicas during rollout and offered no history for one request.

The fence became a database record:

scope: documents/bucket-07
epoch: 3
writer: harbor
effective_revision: 1842
changed_at: 2022-08-09T15:00:00Z
change_id: cutover_07_3

Every write request resolved the current fence, included its epoch in the command, and reached the selected writer. The writer rejected commands from an older epoch. A stale gateway therefore could not continue writing to the former authority after the fence moved.

The scope began with deterministic hash buckets of document identity. This allowed a small cohort without selecting only easy records. System documents and migration fixtures had separate scopes so operational tests did not distort user-like traffic.

The fence history made it possible to explain which authority accepted any revision.

Shadow reads came before authoritative writes

Before moving a fence, P6X4 imported the scoped records and maintained a read model from the old authority's committed changes. Requests continued writing only to the old system.

For selected reads, the gateway fetched both versions and compared normalized domain responses. The shadow result never reached the interface and could not trigger effects.

Normalization removed representation differences that carried no meaning: field order, generated transport identifiers, and equivalent default values. It preserved revisions, policy outcomes, and ordered history where order mattered.

Differences were classified:

Expected representation change.
Known unsupported legacy case.
Projection lag within policy.
Migration defect.
Unknown and blocking.

A comparison percentage alone would have hidden concentrated defects. I required every mismatch class to have an explanation and scenario. The cohort did not move until blocking differences were resolved.

Shadowing was useful because reads were non-authoritative. I did not shadow external writes by sending them twice.

The cutover had a quiet point

Moving a bucket required a bounded pause for new writes.

The procedure was:

Mark the scope closing under the old epoch.
Reject or briefly queue new writes with a retryable response carrying the stable intent ID.
Allow accepted old writes and outbox publication to drain to a known sequence.
Verify P6X4 had imported through that sequence.
Record content and invariant checks.
Atomically advance the fence epoch and select P6X4.
Open writes against the new authority.

The pause was measured in the lab and represented honestly in the interface. For an editable document, a local draft could remain while the server said “This document is moving; saving will resume shortly.” The client did not claim success before the authority accepted the revision.

I preferred a short, explicit write interruption to months of invisible conflict risk.

Stable intent survived the pause

A browser could submit just as the fence closed, lose its response, and retry after the writer changed.

Each mutation carried a stable intent ID, document identity, expected revision, and payload digest. The old and new authorities both retained importable intent receipts. During final synchronization, P6X4 received accepted old intents with their results.

After cutover, retrying an intent accepted by the old authority returned the imported result. Reusing the same ID with a different digest was rejected. A command that the old authority had definitely refused could be evaluated by P6X4 under the new epoch.

If the old outcome was unknown, the scope did not cross until reconciliation resolved it. The migration did not turn an ambiguous write into two attempts against different authorities.

Z29C's distinction between intent and attempt became the key to a safe database cutover.

Rollback was a window with conditions

“We can always switch back” was not a plan. Once P6X4 accepted writes, the old system became stale. Switching the routing flag would lose new revisions unless they were copied back, which recreated dual authority.

I defined a bounded rollback window for each bucket. During it:

P6X4 was the only writer.
Committed P6X4 revisions were exported to a compatibility log the old reader could ingest.
No P6X4-only schema feature was enabled for the cohort.
External effects referenced stable artifact and intent identities understood by both paths.
The old system remained read-capable and its restore procedure was rehearsed.

A rollback followed another fenced transition: close P6X4 writes, drain to a sequence, import the compatibility log, verify, advance the epoch, and reopen the old writer.

After the window, I deliberately removed backward-write compatibility and declared rollback to mean restoring P6X4 or applying a forward fix. The old service could not remain a permanent emotional safety object.

P6X4's target schema introduced explicit document revisions and immutable render requests. The old model stored current values with a change log.

I separated structural migration from semantic activation.

First, additive tables and nullable fields were deployed. Importers populated revision history and recorded source sequence. Both old and new readers could operate. Then the fence moved one cohort while commands stayed within the shared semantic subset. Only after the rollback window closed could P6X4 accept features the old system could not represent.

Destructive cleanup occurred in later releases after no retained commands, compatibility readers, or restore points required the old shape.

This expansion-and-contraction sequence was slower than changing the schema at cutover. It kept each risky decision observable: can the data move, can authority move, can new semantics activate, and can old structure disappear?

The migration had invariants, not just row counts

Equal row counts can preserve equal numbers of wrong records.

For every bucket I checked:

Stable document identities were unique.
Revision numbers formed the expected sequence or carried an explicit gap reason.
Each current document referenced its highest committed revision.
Render artifacts referenced existing source and template revisions.
Accepted intents mapped to one result digest.
Policy relationships referred to valid subjects and resources.
Outbox sequence had no unaccounted hole through the cutover point.

Content digests compared canonical representations. A sample was not enough for structural invariants, so those checks ran across the cohort. Larger payload comparisons could be batched, but every failure blocked movement.

The verification report was stored beside the fence change. It became evidence for why the system believed P6X4 was ready to write.

Reads announced their source during the window

Mixed authority created diagnostic confusion. A read response therefore included the authority epoch and source revision in internal metadata. Logs and comparison reports carried them. The public interface exposed source detail only when it affected action, such as an edit conflict or temporary migration pause.

Cache keys included the epoch. Otherwise, a cached old-authority response could survive cutover and appear to reverse a successful P6X4 write. Projection consumers also stored the authority epoch and rejected events from an older writer after the final sequence.

This prevented late transport from reopening history. An old event arriving after cutover remained evidence from epoch two, not a new command to overwrite epoch three.

I interrupted the migration at each step:

Gateway cached the old fence after closure.
One old write was still leased by a worker.
The final outbox event was delayed.
P6X4 import committed but verification process crashed.
Fence advanced but the response was lost.
New writer became unavailable before the first accepted command.
Rollback export lagged behind P6X4 authority.

The system always resumed from durable fence and sequence records. Procedures were idempotent by change ID. A repeated “advance fence” call returned the current epoch rather than advancing twice.

The drill in which the new writer failed immediately was especially useful. The scope remained on P6X4, accepted writes were durable, and the client received a temporary availability error for new intents. I did not automatically flap authority back to the old system. Authority movement required the same evidence and quiet point in both directions.

Availability pressure was not permission to create two writers.

Permanent replication still had a place

The rejection of dual writes did not mean data stopped moving.

P6X4 published authoritative document facts through a transactional outbox. Search, reporting, and the temporary compatibility reader consumed them. These were projections with source sequences, idempotent application, lag measures, and rebuild paths.

The distinction was direction and authority. One writer made the decision. Other representations followed and could be recreated. If a projection failed, the authority did not ask it which title was true.

For the rollback window, the old system's compatibility view was deliberately read-only. Its familiar schema did not make it a peer authority.

Replication was sustainable when disagreement had a resolution rule. Dual authority did not.

After every bucket had passed its rollback window, I removed old write routes, compatibility exports, old database credentials, fence branches in the gateway, and migration-only dashboards. The fence history remained as audit evidence, but normal requests no longer paid a routing tax.

I added a scenario that tried to start the retired writer and expected failure because its write credential no longer existed. Another sent an old-epoch command to P6X4 and expected an explicit stale-epoch refusal.

A migration feature that remains forever becomes architecture. Deleting it was part of correctness.

The kinder plan had a real moment of commitment

Permanent dual writes had felt kind because they postponed an irreversible choice. In practice, they made every partial failure ask the system to choose between conflicting records with incomplete evidence.

The write fence concentrated risk into a rehearsed, observable transition. Before it, the old authority decided. After it, P6X4 decided. Stable intents survived the boundary. A limited rollback path existed while both representations remained compatible. Then the obsolete writer lost its credentials and its claim.

The plan admitted that moving authority has a moment of commitment. Naming that moment made it possible to protect.

A short pause was visible. Two sources of truth would have been a permanent uncertainty.

An edit request could receive 409 authority-epoch-changed even when its expected document revision was still current. I did not collapse that response into a generic conflict.

The response included the current epoch, authoritative endpoint, accepted-intent lookup path, and whether the submitted intent had any known receipt. The client first queried the stable intent. If no authority had accepted it, the same intent and digest could be submitted against the new epoch. If a receipt existed, the client adopted that result. If the outcome remained unknown, the UI preserved the draft and requested reconciliation instead of silently sending again.

This protocol distinguished three events that looked alike from the browser:

The document changed.
Authority moved.
The network lost an acknowledgement.

Only the first required merging content. The second required routing the same decision. The third required finding evidence before another attempt.

I kept local draft state keyed by document and intent, not by server host. Moving authority therefore did not discard the person's work or create a new action identity.

Foreign keys complicated a scoped move

Hash buckets divided documents cleanly until relationships crossed buckets. A collection could reference documents whose authority moved at different times. A delivery policy could apply to a group rather than one document.

I classified relationships before selecting migration scope. Invariants that had to be checked atomically moved together or remained behind an authoritative query. Read-only references used stable global IDs and tolerated the other side being remote during the window. Projections could follow asynchronously.

I rejected a database-level foreign key in P6X4 where the referenced authority still lived in the old store. Instead, the accepting command validated the reference through a versioned authority lookup and stored the observed revision. After both scopes moved into the same database, a later migration could introduce a local constraint if it represented the domain rule.

This made some temporary integrity checks more explicit and less convenient. Pretending cross-database references were locally enforced would have been worse. The migration plan documented which invariants were strong, which were validated at acceptance time, and which were projections.

Scope design became part of architecture. A bucket was useful only when moving it did not bisect an invariant I could not safely weaken.

The quiet point could be delayed indefinitely if new work continued entering just before closure. Closing a bucket therefore changed admission before draining began.

Interactive writes received a bounded retry response and retained their intent IDs. Background commands for the scope stopped being scheduled. Existing leases had deadlines; workers either committed under the old epoch or released without side effects. The outbox relay prioritized the closing scope through its final sequence without starving other work.

I set a maximum closing duration. Exceeding it aborted the cutover, reopened the old epoch, and recorded which lease or publication prevented progress. The procedure did not wait forever while the interface implied an ordinary brief pause.

The limit exposed one render job that held a database transaction while performing CPU work. Fixing that design—commit durable intent, perform work outside the transaction, then record a receipt—improved the system independently of migration.

Cutover pressure became a diagnostic for operations that did not have clean commitment boundaries.

Monitoring followed cohorts and epochs

Aggregate error rates could hide one damaged bucket. Every migration view grouped commands, read mismatches, projection lag, and recovery records by cohort and authority epoch.

I watched:

Accepted intents without terminal results.
Rejected stale-epoch commands.
Old-authority events arriving after the final sequence.
Revision conflicts compared with the pre-cutover baseline.
Projection lag from the new source.
Differences between authoritative and compatibility reads.
Requests still routed to retired endpoints.

Thresholds produced a hold, not an automatic rollback. Some problems—especially uncertain external effects—could be made worse by moving authority again. The runbook required classifying whether the fault involved availability, data integrity, routing, compatibility, or external consequence.

The cohort stayed small until it had exercised ordinary edits, stale clients, rendering, delivery, deletion, and restore. Passing a few synthetic health requests did not widen the migration.

Before the first fence move, I restored the old authority and P6X4 importer to a separate environment and replayed the migration. During the rollback window, I restored P6X4 at a sequence after cutover and verified that its fence and accepted intents were consistent.

A backup from before the authority epoch could not simply replace a live post-cutover database. Recovery needed the fence history and all authoritative changes through a known point, or the scope needed to close while a verified restore took over under a new epoch.

The restore drill checked that projections and caches were rebuilt rather than trusted from a mismatched point. External effect receipts received special attention: restoring an intent without its accepted receipt could allow a duplicate delivery.

This removed “we have backups” from the cutover checklist and replaced it with named recoverable states.

The old writer became incapable, not merely unused

Traffic graphs reaching zero were not enough. A stale script, forgotten job, or manual endpoint could still write.

At the end of each rollback window, the old database role lost write grants for the migrated scope. Old command routes returned a permanent moved response with the current authority epoch. Scheduled jobs were disabled and their credentials revoked. A canary test attempted a write through every retired path and expected refusal.

Only after that did I call P6X4 the sole authority.

This was the final kindness of the fence: it made the old path incapable of creating disagreement. Documentation and intention had governed the transition; enforcement closed it.