The queue did exactly what I asked

Z29C's first duplicate delivery was not caused by a broken queue.

The queue stored the job durably. One worker leased it. The worker called a synthetic delivery adapter. The adapter accepted the document, assigned a delivery reference, and delayed its response beyond the worker timeout. The worker raised an exception. The lease expired. The queue made the job available again. A second worker delivered the document again and received a prompt response.

Every infrastructure component followed its contract.

The duplicate came from the workflow contract I had failed to write. An exception meant “this attempt did not return successfully.” I had treated it as “the external action did not happen.”

The queue did exactly what I asked. I had asked the wrong system to decide whether repetition was safe.

The happy path hid two commits

The first Z29C prototype looked like a conventional background job:

async function deliverDocument(job: Job) {
  const result = await adapter.deliver(job.document)
  await jobs.complete(job.id, result)
}

The code suggested one operation: deliver and complete. It crossed two independent durable boundaries.

The external adapter committed a delivery in its own system. Z29C committed job completion in PostgreSQL. No transaction included both. Between those commits, the worker could time out, crash, lose a response, or fail to write local state.

The code had at least these outcomes:

• External delivery did not begin; local completion absent.
• External delivery rejected; local failure recorded.
• External delivery accepted; local completion recorded.
• External delivery accepted; acknowledgement lost; local outcome unknown.
• External delivery accepted; local database write failed; local outcome unknown.

The last two looked identical to the queue: the job was unfinished. They looked very different to the intended recipient of the synthetic document.

The workflow model needed to represent the gap instead of compressing both commits into one function call.

Z29C used synthetic adapters deliberately. I was not operating a real fulfilment or notification service. The test environment existed to create failure timing that ordinary mocks tend to omit.

The delivery stub had a controllable sequence:

1. Validate request.
2. Persist accepted delivery with a stable external reference.
3. Optionally delay, drop, or return the response.
4. Expose status lookup only when the adapter contract allowed it.

The duplicate scenario committed step two and dropped step three.

My original mock returned success or threw before doing anything. Under that model, retrying every exception was safe. The mock had encoded the assumption I was trying to test.

I changed adapter fixtures to control failure at semantic boundaries: before acceptance, after acceptance, during status lookup, and after compensation. Every side-effecting adapter needed at least one “effect happened, response did not” scenario.

The more realistic stub made a well-behaved queue reveal a poorly behaved workflow.

Retry was a policy decision disguised as queue configuration

The prototype configured three attempts with exponential backoff. The settings looked operational: balance recovery against load.

They also made a domain decision. Every retry asserted that repeating the job after any captured error was safe and still useful.

That assertion differed by step:

• A pure document transformation could run again safely.
• A local database state change could use a transaction and stable operation key.
• An external adapter accepting an idempotency key could return the existing result.
• An external adapter queryable by a client reference could be checked before repetition.
• An external side effect with neither property could not be retried automatically after an ambiguous timeout.

One queue-level attempt count could not express those contracts.

I moved retry policy into step and adapter definitions. Queue redelivery became an opportunity for recovery logic to inspect evidence, not automatic permission to execute the step body again.

The infrastructure still handled leases and scheduling. The workflow decided what another attempt meant.

“Failed” was too confident

After the timeout, Z29C marked the attempt failed. That label carried two implications: the intended effect had not happened, and trying again was a route toward success.

The only observation was that the worker did not receive a usable response before its deadline.

I split outcomes:

rejected
  The destination returned a durable domain rejection.
 
failed-before-effect
  Z29C has evidence the side-effect boundary was not crossed.
 
outcome-unknown
  The request may have been accepted; available evidence is insufficient.
 
succeeded
  A durable receipt or verified external reference exists.

Transport errors mapped according to when they occurred and what the adapter could prove. A connection refusal before a request left the process was different from a timeout after the request body had been sent. Even then, network APIs could not always reveal a perfect boundary.

Unknown was not indecision. It was a precise statement that prevented the scheduler from manufacturing certainty through repetition.

The user-facing page said Delivery status needs review, not Delivery failed, when that was all Z29C knew.

The queue created a new job record for each retry. Logs and metrics treated them as separate failures followed by one success. The intended delivery appeared to succeed eventually while the duplicate consequence stayed hidden.

I separated stable identities:

• Workflow ID: one durable user intent.
• Step ID: one logical part of the versioned workflow.
• Attempt ID: one worker execution of that step.
• Request ID: one external request identity, reused or replaced according to adapter semantics.
• External reference: the destination's durable result when available.

Retries created attempts under the same step. The run timeline could show two delivery attempts and two external references for one intended effect—a clear invariant violation.

Metrics moved from “jobs eventually successful” to outcomes per logical step, duplicates detected, unknown-outcome age, and recovery path. A later successful attempt did not erase earlier evidence.

One intent could survive many attempts without becoming many intents.

The queue lease prevented two ordinary workers from processing the same job concurrently. When it expired, the queue could safely let another worker examine the record.

It could not prove where the previous worker had stopped.

The external delivery existed outside PostgreSQL and outside the lease. A worker could send the request, lose contact, and remain alive. Another could begin after expiry. The first could later receive a response. Perfect lease renewal would reduce overlap and still not make the external effect transactional.

I retained leases for scheduling and added checkpoints:

1. Record attempt and lease.
2. Validate current policy and workflow state.
3. Persist external request identity.
4. Invoke adapter.
5. Persist adapter observation.
6. Commit step receipt and release dependants.

If a worker disappeared after checkpoint three, recovery had a key to query or retry safely where supported. If it disappeared before three, no external request should have begun under the adapter contract.

The lease answered who may act now. The receipt answered what had happened.

The adapter contract became executable metadata

The first adapter interface exposed one method: deliver(payload). Recovery code had no way to discover what the destination supported.

I added declared capabilities:

type SideEffectContract = {
  acceptsIdempotencyKey: boolean
  queryByClientReference: boolean
  compensation: 'none' | 'supported-with-limits'
  unknownOutcomePolicy: 'query' | 'review'
  retryableRejections: string[]
}

The scheduler did not trust the declaration blindly. Contract scenarios verified behavior against the synthetic adapter. An adapter claiming idempotency had to return the same external reference for repeated keys and reject a key reused with different input.

Adapters with query support had to distinguish not found, found, temporarily unavailable, and ambiguous response. A missing query result inside an eventual-consistency window did not immediately authorize another creation.

The nominal action—deliver a document—could have different recovery behavior across adapters. The contract made that difference visible to the workflow and console.

Capability determined product behavior.

The external request needed a stable reference before sending

The prototype generated an external client reference inside the adapter call. If the worker crashed, Z29C might not retain it.

Recovery needed the reference before the side-effect boundary. I derived or allocated it from stable workflow and step identity, persisted it with the attempt, then invoked the adapter.

For an idempotent destination, the reference or idempotency key identified the logical request. Repeating the call returned the original external result. For a queryable destination, status lookup used the same reference. For a destination with neither feature, persisting the request identity still helped correlate manual evidence even if it could not make retry safe.

The input digest was stored beside the reference. Reusing a key with different document bytes became an explicit error.

This order narrowed the crash window:

commit request identity → call external boundary → commit receipt

It could not eliminate uncertainty between the last two arrows. It ensured the uncertainty had a durable name.

The first console listed failed jobs with a prominent Retry button. In the duplicate scenario, it guided me toward the harmful action.

The revised view led with evidence:

• Intended document digest and logical step.
• Attempt and request identities.
• Time the adapter call began.
• Last transport observation.
• Presence or absence of an external receipt.
• Adapter capabilities and known uncertainty window.

Available actions followed the contract. If status lookup existed, Check destination came first. If the destination reported the delivery, Z29C attached the verified reference and completed the step. If it reported no match after the documented visibility window and idempotent retry remained safe, Retry became available with the same key.

For an opaque destination, the console preserved outcome unknown and offered a documented manual investigation path. It did not provide a universal force-success control.

Recovery UI should not make the most dangerous action the easiest merely because it is familiar.

The duplicate needed its own receipt

Once the test had produced two external deliveries, marking the step successful with the second reference would hide the first.

Z29C recorded an integrity incident on the step. Both external references remained linked. The workflow entered review even though the intended document was definitely delivered.

The resolution named the accepted canonical result and the disposition of the duplicate. In the synthetic adapter, a duplicate could be withdrawn through a separate compensating request. That action had its own ID, result, and limitations. Withdrawal could not erase a notification already observed.

The final state was not “succeeded normally.” It was completed with duplicate effect; duplicate counteracted or completed with unresolved duplicate, depending on evidence.

This prevented eventual success from laundering an earlier correctness failure.

The scenario fixture asserted that a workflow with more than one external reference for a one-effect step could never project a clean completion.

Alerting on exceptions would have missed the lesson

The first worker emitted an exception and the second succeeded. A conventional error-rate alert might show a brief retry and recovery.

The important signal was semantic: one logical step produced two accepted external references. Z29C checked invariants across attempts and receipts.

Other semantic signals included:

• A succeeded step with no receipt.
• A dependant beginning before required receipt fields existed.
• An outcome-unknown step retried under an opaque adapter.
• An idempotency key associated with two input digests.
• A lease expired after an external request began and no recovery review occurred.

Worker errors remained useful diagnostics. They did not define workflow health.

W93H had taught me to alert on user-facing capability rather than resource threshold alone. Z29C extended the idea: operational telemetry should understand the logical intent whose consequence it is measuring.

A queue can be healthy while the workflow duplicates work perfectly.

The fix was not “increase the timeout”

A longer timeout would have made this particular delayed response arrive. It would also make a worker occupy capacity longer and fail under a slightly longer delay.

Timeouts bound waiting. They do not decide whether an external action occurred.

I kept adapter-specific deadlines based on useful response windows and recovery cost. The fix was durable identity, explicit outcome states, adapter capabilities, query-before-retry, and receipts.

Timeout tuning still mattered. A deadline shorter than normal destination latency would create unnecessary unknown outcomes. A deadline longer than the workflow's useful window would delay recovery. Those were scheduling decisions, not correctness guarantees.

The scenario varied the lost response timing around every deadline. Correctness could not depend on the response arriving just before the timer.

This distinction stopped performance configuration from carrying a semantic promise it could never make.

The synthetic destination supported withdrawing a delivered document. It was tempting to call withdrawing the duplicate a rollback.

The original delivery had happened. It may have been observed. It created an external reference and an audit entry. The withdrawal was another side effect with its own possible timeout.

Z29C recorded compensation as a new step linked to the duplicate. If withdrawal succeeded, the external state changed again. If its response was lost, the compensation itself entered outcome unknown and required the same evidence discipline.

The interface said duplicate delivery withdrawn rather than duplicate never happened.

This mattered for future design. A workflow cannot promise atomic rollback across systems that have already exposed effects. It can define counteractions and their limits.

The duplicate scenario became the first place Z29C abandoned rollback language.

The postmortem changed the model, not the queue

The corrective actions were not “use a more reliable queue” or “retry less.” The queue had retained and redelivered unfinished work correctly.

I changed Z29C:

1. One stable workflow and step identity survived all attempts.
2. External request identity was persisted before invocation.
3. Attempt outcome included outcome-unknown.
4. Each adapter declared and tested idempotency, query, and compensation capabilities.
5. Recovery inspected receipts before scheduling another side effect.
6. The console organized evidence and safe actions instead of failed jobs.
7. Semantic invariants detected multiple effects for one step.
8. The exact response-loss scenario became a maintained regression fixture.

The queue kept doing durable scheduling and lease recovery. It stopped owning the decision to repeat.

That division made both systems easier to explain.

The ordering of acknowledgement changed too. A worker did not acknowledge the queue message merely because the adapter returned. It first committed the step observation or receipt in PostgreSQL. Only then could the queue message disappear.

If that final acknowledgement was lost, the queue might redeliver the message. The next worker loaded the durable step receipt and completed without calling the adapter again. Duplicate message delivery became harmless at the workflow boundary.

If the local receipt commit failed after external acceptance, the message returned and recovery found the persisted request identity but no outcome. It queried or paused according to the adapter contract. That remained the irreducible uncertainty window.

This ordering did not create one transaction across PostgreSQL and the queue. It made repeat delivery safe through idempotent message handling and durable workflow state. A transactional outbox carried subsequent workflow events so projection consumers could also tolerate duplicates.

The pattern accepted at-least-once delivery where it was useful and refused to let at-least-once side effects follow automatically. Infrastructure duplication and product duplication were different failures.

Reliable execution begins after durable delivery

Queues solve important problems. They separate request latency from work, buffer load, retain jobs, and let workers recover after interruption. Z29C needed all of those properties.

The duplicate showed the boundary of the abstraction. A queue knows whether its message has been acknowledged under its protocol. It does not know whether an arbitrary external consequence occurred before acknowledgement was lost.

I had modeled the job as the unit of truth because the queue made jobs visible. The real unit was one user intent realized through steps, attempts, external requests, and receipts.

After the correction, redelivery meant “another worker may inspect and recover this logical step.” It no longer meant “run the function again.”

The queue had never lied. The workflow had asked it a question it could not answer.

Correctly.