The side effect crossed the transaction boundary

Z29C could commit its own truth atomically. It could not include the synthetic delivery service in the same PostgreSQL transaction.

That boundary created a small impossible-looking sequence. The workflow needed to record that a delivery was required, perform the external action, and record that it had completed. A process could stop between any two steps.

If Z29C committed “done” before the call, a crash could skip the delivery. If it called first, a crash could leave the delivery accepted and local state unfinished. If it held a database transaction open during the network call, the external service still would not roll back when PostgreSQL did.

The side effect had crossed the transaction boundary. The solution was not to hide the boundary. It was to turn the gap into a protocol that could resume.

The three obvious orderings all had a hole

I implemented or sketched each naive ordering.

Mark complete, then call

Z29C commits the step as complete and calls the adapter afterward. A crash between them creates a durable lie: dependants can advance while the required effect never began.

Call, then mark complete

Z29C calls the adapter and commits completion after receiving success. A crash or lost response between them creates uncertainty. Repeating may duplicate the effect.

Call inside an open database transaction

Z29C begins a transaction, marks the attempt, calls the adapter, then commits. The network call holds locks and a database connection. If the local commit fails after external acceptance, the effect remains. The destination does not participate in PostgreSQL rollback.

The third version looked atomic in one function and remained distributed in reality.

No statement ordering made two independent systems one transaction. I needed durable identity and recovery on both sides of the call.

The local transaction recorded the obligation

Z29C began a workflow by committing intent, definition version, initial step state, and an outbox event together.

For a side-effecting step, a worker acquired a lease and committed an attempt plus a stable external request identity before making the call:

workflow w-17
step deliver-v1
attempt a-3
input digest sha256:...
external request key w-17/deliver-v1
state invoking

The transaction did not say delivery had happened. It said Z29C had durable reason to perform or verify one request under a named identity.

If the worker crashed before calling, recovery could inspect the checkpoint. The adapter contract determined whether the prepared identity was safe to send. If the worker called and disappeared, the same identity gave query or idempotent retry a handle.

Local atomicity protected the obligation and its evidence. It did not counterfeit external completion.

The strongest synthetic adapter accepted an idempotency key. On first acceptance it stored key, normalized request digest, result, and external reference atomically. Repeating the same key and request returned the original result. Reusing the key with different input was rejected.

The protocol became:

1. Z29C commits stable request key and input digest.
2. Z29C sends the request.
3. Destination atomically accepts or returns the prior result under that key.
4. Z29C commits the external receipt.

A lost response allowed step three to repeat safely. The two systems still did not share one transaction, but repeated communication converged on one external effect.

The key represented one logical side effect, not one network attempt. Attempts could change while the key stayed stable.

I tested the receiver contract with concurrent duplicate requests, response loss, and key reuse with changed input. An idempotency header without durable receiver behavior would be decoration.

The destination's implementation completed the protocol. Z29C could not impose idempotency unilaterally.

Queryable outcome was the next best evidence

Another adapter did not support idempotent creation but accepted a stable client reference and exposed status lookup.

After an ambiguous call, Z29C queried before repeating. The query could return:

• Found with external reference and accepted input digest.
• Not found after the adapter's documented visibility window.
• Still processing.
• Found with conflicting identity or digest.
• Temporarily unavailable.

“Not found” was not immediately safe if the destination's query index lagged behind acceptance. The adapter declared a visibility bound or left the state uncertain. Z29C waited and queried again within its workflow deadline.

If absence became authoritative and the creation semantics allowed a new attempt, Z29C could retry with the same client reference. If not, human review remained.

Queryability repaired lost acknowledgement through evidence. It did not make creation itself idempotent and required a trustworthy relationship between lookup and effect.

The recovery experience depended on what the external system was willing to expose.

The hardest adapter could neither deduplicate by client key nor query by reference. After a request may have left Z29C, no automatic operation could prove whether delivery existed.

The step entered outcome-unknown. Dependants that required the delivery receipt stayed blocked. The recovery console showed request identity, input digest, time, transport observation, and any manual correlation clues.

A person could attach verified evidence, decide to accept bounded uncertainty, or authorize another attempt with the duplicate risk visible. The resolution created a typed receipt or waiver; it did not edit the attempt to look successful in retrospect.

This was less satisfying than automatic recovery and more truthful than blind retry.

The adapter's nominal feature remained “deliver document.” Its operational contract was materially weaker. Z29C surfaced that weakness as product behavior instead of hiding it in exception handling.

Some external boundaries cannot be made safe from one side. The correct automation may be to stop.

The transactional outbox solved a different gap

Z29C also needed to publish internal workflow events after database state changed. Writing a step receipt and then sending a queue event had another two-system gap.

Here a transactional outbox helped. The same PostgreSQL transaction committed the workflow transition and an outbox row. A publisher delivered outbox rows at least once. Consumers deduplicated by event ID.

If the publisher crashed after sending but before marking the row delivered, it sent again. The event identity made repetition harmless. If it crashed before sending, the row remained.

This solved reliable publication from Z29C's database to its internal event transport because Z29C controlled the durable source and consumers accepted duplicate delivery.

It did not make the external document delivery transactional. Placing an adapter request in an outbox ensured the request would be attempted; it did not ensure the destination would perform it once or expose the outcome.

I kept the distinction explicit: transactional outbox provides durable handoff, not universal exactly-once effect.

An inbox made internal consumers idempotent

Projection builders and notification candidates consumed workflow events. The outbox publisher could deliver duplicates.

Each consumer stored processed event identity in the same transaction as its local state change. Repeating an event returned the prior result. Events also carried workflow and projection revisions, so a gap or out-of-order delivery triggered repair rather than silent advancement.

This inbox pattern worked because Z29C controlled both event identity and consumer storage. It did not require pretending the broker delivered exactly once.

For derived projections, replaying from durable workflow state could repair corruption. For notification delivery, the candidate had its own stable identity and re-projected current state before sending.

The architecture accepted at-least-once transport and created effectively-once local transitions where semantics allowed.

Again, the property came from the receiver's transaction. The sender alone could not guarantee it.

Receipts were typed conclusions

The first step receipt stored { success: true }. It proved little.

A delivery receipt needed:

workflow and stable step ID
input digest and definition version
external request identity
adapter and contract version
external reference
observed completion state
method: direct response | idempotent replay | status query | manual evidence
observation time
bounded result summary

Dependent steps declared which fields they required. A completion notification could not proceed from a boolean if it needed the external reference and document digest.

A manual reconciliation receipt named author, evidence, and reason. A waiver named residual uncertainty. A compensation receipt linked the original effect rather than replacing it.

The receipt was durable enough to answer why the workflow advanced. It avoided storing credentials or full sensitive payloads.

Typed evidence made “complete” a reviewable claim instead of a status flag set after a function returned.

If recovery reused one idempotency key with a request whose body changed, the destination could reject it or, worse, treat it ambiguously.

Z29C normalized side-effect input before persisting the request digest. Adapter code generated the outbound shape deterministically from that immutable normalized input and its version.

Time-dependent values were explicit inputs. The adapter did not insert the current time into a field that participated in identity on each retry. Random identifiers were allocated once and stored. Default values came from a versioned policy at preparation, not whatever code happened to run later.

The request log retained a redacted canonical summary and digest. Recovery compared the regenerated request before sending.

If a bug fix required materially different payload, Z29C created a new external request identity through a migration or replacement step. It did not quietly change bytes under the old key.

Determinism made retry semantics testable across process and deployment boundaries.

Current authority was checked just before the boundary

A workflow could be created under one policy and reach delivery later. The original authorization established that Z29C could record the intent. It did not grant permanent authority for every future side effect.

Before committing the invoking checkpoint, the worker requested a short-lived capability scoped to workflow, step, adapter, action, and synthetic subject. The policy decision was recorded without storing the credential.

If policy changed, the step entered review or terminal rejection before calling externally. The workflow history distinguished definition from current permission.

The request identity could exist before authority was granted; existence did not imply execution. An expired capability during a long call did not revoke an already accepted external effect. Recovery still used receipts.

This placed authorization at the real consequence boundary while preserving durable intent and explanation.

A transactional record of permission was evidence of a decision, not a remote undo mechanism.

Removing network calls from PostgreSQL transactions improved both correctness clarity and capacity.

The worker used short transactions to acquire a lease, persist the invoking checkpoint, and later commit observation or receipt. The external call occurred between them without holding row locks or an idle transaction.

Another recovery worker could not execute routinely because the lease and state prevented it. If the lease expired, recovery followed request identity rather than relying on a transaction still being open.

This avoided lock timeouts tied to adapter latency and reduced connection occupancy. It also made the failure window explicit in the schema.

I did not use “eventual consistency” as a blanket excuse. The workflow had defined intermediate states and invariants. Dependants could not advance without the required receipt. The user-facing projection could say waiting, unknown, or review required.

Short local transactions plus a resumable protocol were stronger than one long transaction that only looked global from inside the code.

Distributed transactions were not available by wish

I considered whether a two-phase commit protocol could include the external adapter. The synthetic services did not expose a prepare/commit interface or participate in Z29C's transaction coordinator. Many ordinary HTTP APIs do not.

Building a coordinator around systems that cannot prepare durably would merely rename the same uncertainty. Holding a reservation at the destination could also introduce expiry, heuristic outcome, and recovery complexity.

Z29C used the guarantees actually available: local ACID transactions, stable identities, idempotent receiver behavior where supported, status queries where available, and human review otherwise.

The design did not claim exactly-once execution across every boundary. It aimed for one durable intent and one explainable accepted outcome.

Stronger coordination could be appropriate for a closed system designed to participate. It was not something the caller could add through architecture diagrams alone.

Guarantees must be shared by every boundary that names them.

Compensation crossed the boundary again

If a later step failed after delivery, Z29C could sometimes request withdrawal. That was another external side effect with the same transaction problem.

Compensation had its own step ID, request key, adapter capability, attempt history, and receipt. A timeout after withdrawal could produce unknown compensation outcome. The original delivery receipt remained.

The workflow definition stated what compensation could and could not restore. Withdrawing a document might prevent future access and could not erase a notification already seen. A resource release might fail after another system had built work on the reservation.

Calling compensation rollback would imply the original transaction had never committed. It had.

The recovery console showed original effect and counteraction as separate facts. A final projection such as reversed after partial completion preserved the history.

Every side effect that crosses a boundary deserves its own protocol, including the one intended to undo another.

Fault injection targeted every commit gap

The scenario harness terminated workers at the critical boundaries:

• After local obligation commit, before adapter call.
• After destination acceptance, before response.
• After response, before local observation commit.
• After local receipt, before queue acknowledgement.
• During outbox publication.
• During status query.
• During compensation acceptance.

For each adapter capability, tests asserted the permitted recovery. An idempotent destination received the same key. A queryable destination was checked. An opaque destination stopped at unknown. A local receipt made queue redelivery a no-op.

The harness also changed code versions between interruption and recovery. Request normalization and stable step identity had to survive deployment.

These tests did not eliminate distributed failure. They proved Z29C would represent and recover each known gap without inventing atomicity.

The hardest bugs lived between commits, so the test clock stopped there deliberately.

Before the redesign, a timeout became Something failed—retry. Afterward, the page could say:

• Request recorded; delivery has not started.
• Waiting for the delivery service.
• Delivery may have completed; checking its status.
• Delivery verified from external reference x-82.
• Delivery status cannot be verified automatically; review required.

The person did not need to understand transactions. They benefited from a system that distinguished waiting, absence, and uncertainty.

Cancellation language also improved. Before external invocation, Z29C could prevent delivery. After a receipt, it could request withdrawal with stated limits. During unknown outcome, it could stop future work while preserving investigation.

Storage and protocol decisions became emotional affordances: safe retry, understandable waiting, and fewer duplicate consequences.

The boundary remained visible by design

PostgreSQL gave Z29C a strong local source of truth. The external destination had its own truth. The interval between them could not be compressed into a boolean without losing information.

Z29C stored an obligation before the call, a stable request identity at the boundary, observations after it, and a typed receipt before advancing. An outbox reliably published local changes. Adapter contracts determined whether unknown outcome could be resolved automatically.

The result was more state than a simple job table and less false confidence than a global exactly-once claim.

Transactions remained essential. Their value came partly from knowing where they ended.

That knowledge also prevented one common repair from making matters worse. A database restore could return Z29C to a state before an external receipt was recorded while the delivery still existed. Recovery procedures therefore treated external references and adapter reconciliation as part of restoration. Restoring local rows was not enough to restore global knowledge. The consistency checker identified invoking steps without receipts and queried or paused them before workers resumed. Backups protected Z29C's evidence; they could not rewind the destination.

The side effect crossed that ending. Z29C became reliable when the architecture crossed it with evidence instead of pretending the line was not there.

The gap never disappeared. It became named state, bounded authority, and a recovery path that another process could continue safely.

That was a stronger, more useful guarantee than an imaginary global commit.