A transactional outbox is not eventual consistency everywhere

P6X4's transactional outbox solved one narrow problem: record an authoritative database change and the durable intention to publish related work in the same local transaction.

I nearly turned it into a philosophy for the whole application.

If every module emitted events and every other module caught up eventually, boundaries would remain decoupled. The architecture would avoid cross-module transactions. The outbox would make publication reliable.

The resulting prototype allowed a document revision to exist without its access policy, an accepted render request without a stable template revision, and an audit history that might arrive later. It was consistently asynchronous and locally incoherent.

An outbox protects a boundary after a decision. It does not tell me where the decision ends.

The atomic promise is local and precise

In P6X4, one transaction could commit:

• A new document revision.
• The authority and policy context under which it was accepted.
• An audit fact about the decision.
• A render intent with stable identity.
• An outbox record that would wake asynchronous execution.

Either all became durable or none did.

The outbox record did not mean the render was complete, search was current, or notification had been delivered. It meant the local authority had accepted a request and preserved enough information for subsequent work to resume.

I wrote the promise in the API response:

Revision 7 was accepted. Rendering is pending under intent render_84.

That response was stronger than “message published” and weaker than “document ready.” It matched what the transaction knew.

Local invariants should not wait for consumers

The early service architecture placed document metadata and access policy in different stores. Creating a restricted document emitted an event that a policy consumer eventually applied. During lag, the document existed with a default or missing policy.

No interface wanted that state. I had introduced it to preserve service autonomy.

Consolidation moved policy relationships required for document validity into the same authoritative transaction. Other policy projections could lag, but the system never accepted a document that lacked the decision needed to govern it.

I used a simple test:

If a consumer is unavailable for an hour, may the source decision remain valid and useful?

Search indexing could. Notification could. Rendering could remain pending. A required access decision could not.

An outbox is valuable only after the local transaction contains a coherent state worth publishing.

The outbox row was not the domain event

My first table stored a generic JSON payload and destination. Application code serialized a broker message directly into it.

That coupled the transaction to transport topology and made retries reproduce old routing decisions. It also treated every outbox item as a published fact, even when some were commands or wake-ups.

I split the concepts:

• A domain fact recorded authoritative meaning and contract version.
• A work request recorded a stable command or job identity.
• An outbox delivery recorded that a durable item needed publication to a transport destination.

The outbox could reference a fact or request by ID and carry a serialized contract envelope appropriate to the current external boundary. Internal database workers could lease the durable request without a broker message at all.

This made transport replaceable without rewriting the source transaction. It also kept authority in domain records rather than a queue-shaped table.

Publication was at least once

The relay selected unsent rows, published them, and marked them sent. A crash could occur after the broker accepted a message but before the database recorded completion. The relay would publish again.

I did not call the path exactly once. Messages had stable identities, and consumers handled duplicates according to their effect:

• A projection applied a source revision idempotently.
• An internal job recorded a consumption receipt in the same transaction as its state change.
• An external effect used stable effect identity and adapter-specific idempotency where available.
• An observational metric either tolerated duplicate transport or deduplicated at its own semantic identity.

The outbox closed the “database committed but message was never durably scheduled” gap. It did not close every duplicate-effect gap downstream.

Naming the remaining failure window prevented the pattern from becoming magical thinking.

Ordering was scoped to authority

A single global outbox sequence would have serialized unrelated work and still not guaranteed broker-wide observation order.

Document facts carried the document revision. Projection consumers applied only a higher complete revision. Delivery workflow records used their own intent and attempt sequences. The outbox table had a database sequence for relay progress and diagnosis, but domain ordering came from the authority that could define it.

I tested duplicate, late, skipped, and reversed deliveries. A search projector receiving revision eight before seven kept eight. If revision eight required a complete snapshot, seven could be ignored. If a delta contract required continuity, the consumer detected the gap and loaded or rebuilt from authority rather than applying nine to unknown state.

Wall-clock timestamps were evidence, not order. Broker partitions and retries could deliver a later publication first.

One “queue depth” metric was insufficient. Work could accumulate as:

• Committed outbox records not yet published.
• Broker messages not yet received.
• Accepted jobs not yet leased.
• Leased jobs not making progress.
• Projection source revisions not yet applied.
• External effects waiting for an adapter window.
• Unknown outcomes awaiting reconciliation.

Each stage needed a measure tied to consequence. The oldest unsent outbox age indicated publication health. Projection revision lag indicated search freshness. Oldest accepted render intent indicated user wait. Adapter next-eligible time explained delayed delivery.

The interface did not expose all transport queues. It described stable product states such as rendering pending or search updating. Operational views connected those states to pipeline evidence.

An outbox can move a bottleneck. Good instrumentation shows where it moved.

A relay could repeatedly fail to serialize or publish one outbox row. If it processed strictly in sequence, later unrelated work stalled. If it skipped the row silently, the source fact might never reach a required consumer.

The relay classified failures:

• Temporary transport failure: retry with bounded backoff.
• Unsupported or invalid contract: quarantine and block required destinations.
• Retired optional destination: record disposition under retirement policy.
• Sensitive payload policy violation: stop publication and raise a recovery record.
• Missing source record: invariant breach requiring investigation.

Rows carried independent destination status when one fact had multiple external consumers. A failed optional analytics publication did not prevent a required search projection signal, though both failures remained visible.

The recovery console began with source meaning and required consequence, not a raw “republish” action.

Cleanup could not outrun consumers

Deleting sent outbox rows immediately removed useful delivery evidence. Retaining them forever duplicated domain payloads and expanded privacy exposure.

I used a bounded retention policy. The system kept delivery state and message identity long enough for consumer deduplication, investigation, and configured replay windows. Durable domain facts and workflow receipts lived in their owning stores. The outbox was not the permanent event archive.

Cleanup required every required destination to reach a terminal publication state and the row to exceed its retention. Sensitive fields were minimized in the serialized envelope; consumers loaded authority when safe instead of receiving broad snapshots.

Restore tests checked the relationship. Restoring a source transaction without its unsent outbox record could strand work. Restoring an old outbox after the effect receipt could create a duplicate publication that consumers still needed to handle.

Backup consistency belonged to the pattern.

Search could lag because it supported discovery, not authoritative mutation. Its contract named the source revision it had applied and revalidated access before returning sensitive results.

When the outbox relay stopped, document writes continued. Projection lag rose. The interface could say that recent changes might not appear in search and offer direct navigation by stable identity. Once publication resumed, the indexer converged or rebuilt from authoritative revisions.

This was deliberate eventual consistency:

• The temporary disagreement was named.
• The maximum tolerated lag had an operational objective.
• Unsafe decisions revalidated authority.
• The projection had a source order and rebuild path.
• The UI had a truthful degraded state.

The outbox supported that contract. It did not create the contract by existing.

Notification lag had different consequences

Notification followed completed facts and could lag without invalidating the document or delivery. A delayed message affected awareness, not the underlying outcome.

The notification intent used a stable effect identity derived from fact, recipient scope, and template version. Duplicate publication therefore found the same intent. An adapter timeout could still produce an unknown outcome, handled through Z29C's evidence model.

I did not let notification completion gate the source transaction. I also did not represent notification as a casual consumer whose failures vanished into logs. Its state was independently recoverable because the person might otherwise miss an important outcome.

Different asynchronous consumers deserved different guarantees even when both received through the same outbox mechanism.

Rendering crossed a resource boundary, not an authority boundary

The document transaction accepted a render intent and immutable source references. A worker process performed CPU-heavy work later. Its result committed an artifact manifest and terminal job state.

The web process and worker came from one P6X4 release, but interruption remained possible. A lease was not a receipt. Temporary files were named by job attempt, and publication of the final artifact used an atomic move or content-addressed key. Retrying a known-failed attempt could not create two authoritative artifacts for one render intent.

The outbox woke the worker or published to an external execution path. The durable job table remained the source of work. Losing a wake-up did not lose the accepted intent because a sweeper could discover due jobs.

This avoided turning the broker into the only record that work existed.

A synchronous call was sometimes correct

The event-first design made every module relationship asynchronous. After consolidation, many queries and validations became ordinary in-process calls.

When the document module needed current template compatibility before accepting a revision, a synchronous query to the local template policy was honest. The caller required the answer to decide. Publishing an event and allowing temporary invalidity would not improve decoupling.

The call still used a public package interface and could fail with a typed result. It did not need serialization, retry, and a pending product state.

I used asynchrony when the caller could make a coherent decision without the result. I used synchronous composition when the result was part of that decision.

This made P6X4 less uniformly event-driven and more consistent with its actual invariants.

A central messaging module initially let any package insert arbitrary outbox payloads. It became a side door around domain APIs.

Each authoritative package created its facts or requests through domain operations. The platform outbox accepted a typed reference and envelope only within the caller's database transaction. It handled leasing, publication receipts, and transport adapters without deciding payload meaning.

Contract ownership remained with the publisher or command handler. Destination policy was configured explicitly. The platform could report that publication failed; it could not decide that a document fact was optional.

The separation kept reusable delivery mechanics from becoming a central domain authority.

Failure drills established the boundary

I tested crashes at six points:

1. Before the source transaction committed.
2. After commit but before relay discovery.
3. After broker acceptance but before sent receipt.
4. After consumer receipt but before projection commit.
5. After projection commit but before consumption receipt.
6. During outbox cleanup and restore.

The system converged without losing the source decision or applying an unsafe duplicate. Some paths produced repeated messages. None pretended repeated delivery could not happen.

I also disabled consumers for an hour of synthetic work. The source modules had to remain coherent, and every delayed consequence had to be visible. If disabling a consumer made the source invalid, the boundary belonged inside the transaction instead.

That test was more useful than debating eventual consistency abstractly.

The pattern remained small

P6X4's outbox implementation used an ordinary relational table, indexes for due unsent work, short leases, bounded batches, stable message identity, and destination receipts. A relay process could scale modestly without becoming a general workflow engine.

Long-running step state lived in Z29C-like workflow tables. External effect evidence lived in the delivery ledger. Domain history lived in domain facts and revisions. Search checkpoints lived with the projection.

Keeping those responsibilities separate prevented the outbox from accumulating scheduling, orchestration, compensation, analytics, and permanent event storage merely because it sat between a database and a broker.

Consistency followed meaning

The final P6X4 design was strongly consistent about a document revision, its authority context, accepted render intent, audit fact, and publication obligation. It was eventually consistent about search, notification, and other projections. It represented rendering and delivery as durable workflows with explicit intermediate and unknown states.

Those choices were not contradictory. They described different consequences.

A transactional outbox made the transition from local truth to later work reliable enough to resume. It did not make every boundary asynchronous, and it did not absolve consumers from idempotency, ordering, recovery, or data lifecycle.

The pattern became useful again after I made its claim smaller.

Commit what must be true together. Publish what another capability may process later. Make the temporary disagreement explicit. Do not call the entire application eventually consistent because one row waits in an outbox.

Multi-tenant fairness needed a relay policy

One synthetic account generated thousands of projection events and delayed a small document's notification behind them. A first-in global scan was technically ordered and operationally unfair.

I partitioned relay claims by destination and bounded source scope, then used round-robin batches with per-destination concurrency. Required domain ordering remained per document revision; unrelated documents did not need to block one another. The oldest unsent age was measured per class so aggregate throughput could not hide starvation.

Rate-limited destinations stored their next eligible time rather than failing and re-entering generic retry. The relay could continue publishing other work while preserving the delayed item.

This policy belonged to outbox delivery, not domain truth. A document revision remained accepted regardless of when an optional notification received capacity.

Disaster recovery included relay position

Restoring the application database restored authoritative records and their outbox obligations together. If the broker retained messages from a later point, consumers might observe duplicates or facts beyond the restored authority.

The restore procedure established a new topology generation, rebuilt projections from the restored source boundary, and treated broker deliveries as untrusted until their source identity and revision existed. Stable consumer receipts absorbed valid duplicates. Orphaned later messages were quarantined rather than applied.

External effect receipts were restored with their intents. Losing a receipt while keeping an outbox command could cause a duplicate action; losing the command while keeping an accepted intent could strand it. Backup tests asserted these pairs.

The outbox made source and publication obligation atomic during normal operation. Recovery still had to restore their relationship deliberately.

An outbox row could wait across a deployment. A new publisher therefore could not assume every relay or consumer understood its latest envelope immediately.

Contracts carried explicit versions. The release manifest declared which versions each runtime role could read and write. Deployment brought compatible consumers and relays online before publishers emitted a new version. Old handlers remained until retained rows and broker messages had crossed their retention boundary.

I preferred additive evolution and complete snapshots for projection facts where payload size allowed. Upcasters filled only values implied by older semantics. Missing information remained an explicit legacy case.

One application release reduced accidental version combinations, but queue time meant old data still met new code. The outbox pattern remained temporal architecture.

Reconciliation looked beyond sent_at

A row marked sent proved that the relay had recorded broker acknowledgement. It did not prove every required consumer had reached the intended state.

For critical projections and effects, reconciliation compared authoritative source position with consumer checkpoints or workflow receipts. It found gaps caused by retention mistakes, consumer defects, or incorrectly acknowledged messages. Recovery chose replay, rebuild, or evidence review according to the consumer's semantics.

I did not make every publisher wait for end-to-end acknowledgement; that would recreate synchronous coupling. I made convergence measurable after the transaction returned.

This distinction kept the outbox promise honest: durable publication responsibility, followed by independently observable completion.