Z29C

Z29C was a 2021 personal project for running multi-step work safely across interruption. D4U7 had an outbox in which one durable contribution eventually produced one server result. Z29C raised the difficulty: a single user intent could validate data, reserve a resource, generate a document, deliver it through an external adapter, and record completion. Any step could time out after the external system acted but before Z29C received the answer.

A queue was not a workflow

I built the project around synthetic fulfilment and publishing scenarios with stubbed external services. It was not a commercial workflow platform or evidence of operating a real transaction business. The environment existed to make retries, unknown outcomes, compensation, and human recovery concrete.

The first prototype placed jobs on a queue and retried exceptions. It performed well while failures were clean. Then the delivery stub accepted a document and delayed its response beyond the worker timeout. The queue retried, delivered the same document again, and marked the second attempt successful.

The queue had done exactly what I asked. The workflow model had failed to distinguish “the step did not happen” from “the step may have happened but I do not know.”

Z29C was built around that distinction.

Every workflow began with a durable intent record created in the same transaction as its initial state. The intent named its type, initiating subject, validated input digest, policy version, and stable workflow ID.

Submitting the same user action with the same idempotency key returned the existing workflow. It did not create another run merely because the browser had missed the response.

The interface acknowledged creation only after the intent record existed. Before that boundary, the user could retry the submission. After it, the user could reopen the workflow by ID and inspect progress.

The workflow input was immutable. A correction created a new revision or a new workflow according to the domain. Mutating the request under an existing ID would have made receipts impossible to interpret.

I separated intent from execution. The intent said what outcome had been requested. Execution records showed the attempts made to realize it. A workflow could retain one intent across several leased workers and step retries.

This reused X8B6’s stable operation identity at a larger scale. The user experience of safe retry depended on an end-to-end identifier, not a spinner or disabled button.

A durable step model

Each workflow definition was a versioned graph of named steps. I kept the first definitions mostly sequential; explicit branching entered only where the domain required it.

A step moved through states that distinguished knowledge rather than worker activity alone:

pending
ready
running
succeeded
failed-retryable
failed-terminal
outcome-unknown
waiting-for-review
compensating
compensated

State transitions and their receipts were stored in PostgreSQL. A worker acquired a short lease on a ready step. The lease prevented concurrent execution and did not imply success. If the worker disappeared, the step returned for recovery according to its recorded attempt state.

The workflow projection could be rebuilt from intent, definition version, step transitions, and receipts. The projection served the interface and scheduler; the durable records remained the explanation source.

Steps declared their retry semantics. A pure calculation could retry automatically. A database update inside Z29C could use a transaction and stable operation key. An external side effect required an adapter contract that described idempotency, queryability, and compensation.

The scheduler never used “retry everything three times” as a universal policy.

Step receipts

A successful step produced a durable receipt before dependent steps became ready. The receipt contained the step ID, attempt ID, input digest, adapter version, result summary, external reference where available, and completion time.

The receipt was not a log line. It was structured evidence that the workflow used to decide what could happen next.

When an adapter supported an idempotency key, Z29C derived one from workflow and step identity. Repeating the request returned the external system’s prior result. The adapter receipt preserved that external reference.

When an adapter supported status lookup but not idempotent creation, Z29C queried by a stable client reference before attempting again. When it supported neither, a timeout moved the step to outcome-unknown and required review.

This was the project’s central breakthrough. A timeout was a transport observation, not proof that the side effect failed. Recording unknown outcome prevented automation from converting uncertainty into duplication.

Dependent steps could require particular receipt fields. A notification step, for example, needed the generated document digest and delivery reference rather than the general fact that generation had succeeded.

The unknown-outcome console

The first operator view was a table of failed jobs with a retry button. It encouraged the most dangerous action exactly where knowledge was weakest.

Z29C’s recovery console organized a step around four questions:

• What did Z29C intend to request?
• What evidence exists that the external system received or completed it?
• Which adapter operations are safe now: query, retry, compensate, or escalate?
• What would each action change in the workflow?

For the delayed delivery scenario, the console showed the stable client reference, request time, connection result, absence of a receipt, and a query action. If the stub reported that delivery had occurred, the reviewer could attach the verified external reference and complete the step. If it reported no match and the idempotency window remained valid, retry became available.

The console did not expose a generic force-success control. Manual resolution created a typed receipt with author, evidence, and reason. The workflow then proceeded from an accountable state transition.

The interface used ordinary language for the initiating user: “Delivery status needs review,” not “worker failed.” Technical details remained available in the evidence panel.

Leases and worker recovery

Workers polled ready steps in bounded batches and acquired leases using a transactional update. A lease carried owner, expiry, and attempt ID. Long-running adapters renewed it while making progress.

If a lease expired, a recovery task inspected the attempt before scheduling anything else. A step that had not crossed its side-effect boundary could become ready. A step with a sent request and no receipt followed its adapter’s unknown-outcome policy.

I inserted checkpoints around each step:

1. Record attempt and lease.
2. Validate current workflow and policy.
3. Record side-effect request identity.
4. Invoke the adapter.
5. Store the adapter result or transport uncertainty.
6. Commit the step receipt and release dependent steps.

The checkpoints made crash tests precise. I could terminate a worker after any one and assert the allowed recovery.

Heartbeats alone were not enough. A worker could remain alive while an adapter call stalled. Each step carried an execution deadline and adapter-specific timeout. Lease renewal stopped before the deadline so recovery could begin.

Transactions stopped at their real boundary

Z29C used database transactions for its own state. It did not pretend that one PostgreSQL transaction could include an external delivery service.

For internal changes, intent, step transition, outbox event, and local receipt could commit atomically. For external work, the database recorded enough identity and state to resume a protocol across the boundary.

A transactional outbox published workflow events after commit. Consumers deduplicated by event ID. Publishing twice was acceptable; losing the only event was not.

The first implementation marked a step running, called the adapter, then marked it complete. A process crash after the adapter call left only the running state. Adding the request identity before invocation narrowed the uncertainty and gave recovery a query key.

I avoided distributed transactions because the adapters did not participate and the operational complexity would have disguised the same uncertainty under a stronger name.

Z29C’s design accepted that exactly-once end-to-end execution is often unavailable. It aimed for one durable intent, idempotent handling where supported, visible ambiguity where not, and one explainable final outcome.

Retry taxonomy

Errors entered categories with different policies:

• Transient transport failure before request delivery was known.
• Rate limit with a declared retry time.
• Temporary remote unavailability.
• Validation rejection that required input change.
• Authorization failure that required policy or credential review.
• Unknown outcome after a possibly accepted request.
• Permanent domain rejection.

Backoff used jitter and honored adapter-provided retry times. The scheduler enforced a total attempt budget and workflow deadline. Automatic retries stopped when the chance of recovery no longer justified delay or load.

The error record retained the adapter’s stable code and a redacted bounded explanation. Generic exception strings did not become workflow policy.

A retry created a new attempt under the same step identity. Attempt history showed what changed: time, adapter version, request identity, response, and decision. The final receipt never erased failed attempts.

This taxonomy turned retry from an instinct into a domain decision. “Try again” was appropriate only when the system could state why another attempt was safe and useful.

Compensation was not rollback

Some completed steps could be counteracted. A reserved synthetic resource could be released; a published test document could be withdrawn. Compensation was a new side effect with its own identity, receipt, failure modes, and possible unknown outcome.

Z29C never labeled compensation a rollback. The original action remained part of history. The compensating action might not restore every external consequence, such as a notification already read.

Workflow definitions declared which prior steps required compensation when a later terminal failure occurred. The engine scheduled them in a safe order and paused where a person needed to choose.

The first saga-like workflow compensated automatically on every later failure. A scenario showed that releasing a scarce resource after a recoverable delivery delay made the user’s intended outcome harder to complete. Compensation policy began considering failure category and workflow deadline rather than firing from one generic branch.

The console explained both the original and compensating effects. A “clean” final status did not hide that the workflow had performed and later counteracted work.

Versioned workflow definitions

A running workflow kept the definition version with which it began. Deploying a new graph did not reinterpret existing step records.

Compatible code fixes could support old definitions through versioned handlers. A structural migration required an explicit plan that mapped old pending steps to new ones while preserving completed receipts.

I created fixtures for workflows paused in every state and loaded them against new releases. Tests verified that the interface, scheduler, and recovery actions still understood their definition and adapter receipts.

The first migration attempted to rename a step and update running records mechanically. External idempotency keys derived from the old name no longer matched. Stable step IDs became distinct from display labels, and running workflows retained them forever.

Definitions included the policy version that authorized external actions. A later policy restriction could stop an old pending step before execution even though its workflow graph remained valid. The console showed that distinction: definition says what comes next; current policy says whether it is still allowed.

The internal states were too detailed for the initiating user. Z29C projected them into meaningful progress:

• Request recorded.
• Preparing.
• Waiting on an external service.
• Needs your review.
• Completed.
• Could not complete.
• Reversed after partial completion.

The page showed completed milestones and the next consequential state. It never displayed a smooth percentage for work with discontinuous external steps.

Refreshing or opening the workflow on another device read the durable projection. The browser was not responsible for keeping the run alive.

Notifications fired on meaningful transitions: review required, deadline approaching, completed, or terminal failure. Repeated worker errors did not produce repeated user messages.

A cancellation request entered the workflow as intent. If no side effect had begun, cancellation could complete quickly. Otherwise it followed step-specific stop or compensation rules. The interface did not promise that closing a page or pressing cancel could erase already accepted external work.

Scenario replay

Z29C contained a deterministic adapter harness. Each stub could accept, reject, delay, rate-limit, lose a response after acceptance, return conflicting status, or change authorization.

Scenarios terminated workers at each checkpoint and advanced a controllable clock through leases, deadlines, and retry windows. Assertions covered both state and allowable actions.

One core scenario required the following sequence:

1. Delivery accepts a document.
2. The response is lost.
3. The worker lease expires.
4. Recovery queries by stable reference.
5. The adapter reports the accepted delivery.
6. Z29C records a verified receipt and does not deliver again.

Another adapter could not query. The same sequence had to end in outcome-unknown with no automatic retry.

These scenarios made adapter capability part of product behavior. Two services offering the same nominal action could produce different recovery experiences because their contracts exposed different evidence.

The harness was the ancestor of N17Q’s later tool fixtures. Z29C tested deterministic workflow code around synthetic external behavior; N17Q would use recorded tool contracts to evaluate probabilistic agent decisions.

Observability organized around runs

Z29C logged semantic events: intent recorded, step ready, lease acquired, request identity stored, adapter outcome observed, receipt committed, retry scheduled, review resolved, and compensation completed.

The run timeline separated workflow state from worker process logs. A step could have several attempts and one final receipt. Raw logs linked to the attempt that produced them.

Metrics tracked age of ready work, lease expiries, unknown outcomes, retry categories, and review wait time. I avoided a simple success rate because a workflow incorrectly duplicated and then “succeeded” would look healthy under it.

A consistency checker rebuilt projections, verified that succeeded steps had receipts, ensured no dependent step preceded its requirements, and found expired leases. Repair operations were idempotent and retained their own records.

W93H’s incident model influenced this design directly. The run timeline distinguished observations, actions, and conclusions. It made the workflow explainable without treating every log line as an equal event.

Workflow definitions named the authority each step required. A worker did not receive one general credential capable of every adapter operation. It requested a short-lived capability scoped to adapter, action, test subject, and workflow step.

The policy service checked the current workflow, definition version, initiating identity, approval state, and execution environment before issuing that capability. The receipt recorded the policy decision without retaining the credential itself.

This mattered during long waits. A workflow could be created while an action was allowed and reach the step after the policy changed. Z29C revalidated authority immediately before the side-effect boundary. The run history showed that the definition remained the same while current policy blocked execution.

Each workflow also carried budgets: maximum attempts, total runtime, external-call count, and optional synthetic cost units. A rate-limited adapter could not keep a run alive indefinitely. Crossing a budget paused or terminated according to the workflow contract and produced a visible reason.

I separated a budget from a timeout. A step might complete within its individual deadline and still consume the workflow’s remaining attempt allowance. The projection showed both the current wait and the remaining recovery room.

Secrets were never serialized into workflow input, events, or receipts. Adapter workers resolved scoped credentials at execution time. Error handling redacted request headers and limited response excerpts. A durable workflow record must survive longer than many credentials and therefore should not become their archive.

The synthetic environment included a deliberately over-permissioned adapter test. A workflow asking only to query delivery status received a credential that could also create deliveries. The policy assertion failed even though the workflow used it safely. Least authority was treated as a property to verify, not a promise about developer intention.

These controls anticipated N17Q. Once software chooses and invokes tools more autonomously, scoped authority and cumulative budgets become even more important. Z29C established them while every step was still predetermined by a versioned graph.

The scheduler grouped ready work by workflow and subject rather than letting one workflow flood every worker. Per-adapter concurrency limits protected fragile external stubs. Rate-limit receipts changed the next eligible time without occupying a worker.

A large fan-out scenario revealed that marking thousands of steps ready in one transaction made ordinary workflows wait. Z29C activated bounded windows and released the next window only as prior work settled. The user-facing projection still represented the whole intent.

Queue age was measured by step class. Unknown-outcome review and ordinary transient retry did not disappear inside one backlog number. A personal deployment had small volume, but modeling pressure at low scale made the scheduling behavior explainable before capacity became an emergency.

I tested shutdown by stopping workers while they held leases, then starting a new version. No graceful-shutdown promise was required for correctness. Grace shortened recovery; durable attempts and leases provided it.

Recovery paths I designed too late

The queue-and-retry prototype treated every exception as evidence that work had not happened. A lost response produced duplicate delivery. Outcome-unknown became a first-class state.

The failed-jobs table offered a generic retry button where the correct action depended on adapter evidence. The recovery console exposed query, retry, compensate, or review according to contract.

I initially called compensation rollback and implied history had been restored. Treating it as another side effect made limitations and receipts visible.

Step display names entered idempotency keys. A harmless copy edit could have changed external identity. Stable internal step IDs fixed it.

The first workflow progress bar advanced by step count. A quick validation and a slow consequential delivery appeared equal. Milestones and named waiting states replaced false precision.

When automation became recovery work

Z29C was the first project in the sequence to coordinate several durable side effects under one user intent. It combined transactional state, leased execution, adapter contracts, step receipts, explicit uncertainty, retry taxonomy, compensation, version migration, and human review.

Its core achievement was not automation. It was recovery. The system could stop at an arbitrary boundary and state what it knew, which actions remained safe, and what evidence a person needed to continue.

X8B6’s idempotent operation receipt became a per-step execution protocol. D4U7’s attention model shaped the recovery console. W93H’s semantic timeline shaped the run record. Q2F8’s policy versions constrained what old workflows could still do.

The next project, P6X4, would challenge a different assumption. Z29C’s clean adapter boundaries made a multi-service architecture attractive. I would deliberately split a larger synthetic system too far, then learn how to consolidate it without losing the boundaries and receipts that actually mattered.

Guarantees the adapters could not supply

Z29C used synthetic workflows and adapters. It did not prove behavior across every payment, messaging, or fulfilment provider. External systems can violate their documented idempotency or query contracts. Manual reviewers can make unsafe choices.

The engine reduced ambiguity where adapter evidence allowed and exposed it where it did not. It could not create exactly-once guarantees beyond the real boundaries of the participating systems.

Technology: PostgreSQL, durable intent and step records, leased workers, transactional outbox events, versioned workflow definitions, idempotency keys, adapter receipts, controllable scenario time, and a server-rendered recovery console.