Workflow definitions outlive deployments

I renamed a Z29C step from send to deliverDocument, deployed the change, and stranded a running workflow.

The old workflow had completed preparation and stored its next step as send. The new scheduler loaded the latest definition, found no such step, and treated the record as corrupt. My attempted fix renamed the database row. That changed the value used to derive the external idempotency key.

The display-label improvement had become an execution migration with side-effect identity consequences.

A workflow can remain alive longer than a process, release, or deployment. Z29C needed to preserve the definition under which work began and distinguish stable identities from current presentation.

Latest code was not the definition of old intent

The first engine stored workflow type and current step name. On every poll it loaded the latest graph for that type.

Changing the graph could reinterpret running state:

• A new step appeared before work already completed.
• A removed step made an old receipt unreachable.
• A renamed step changed idempotency identity.
• A branch condition evaluated under new logic.
• A timeout or compensation policy changed mid-wait.
• New input defaults altered a retry payload.

That behavior made deployment an implicit bulk migration of every unfinished workflow.

Z29C began storing a definition version with intent creation. The version identified the graph, stable step IDs, dependency and receipt contracts, timeout policy, compensation graph, and handler interface expected by that run.

New workflows used the new definition. Existing workflows stayed interpretable under the old one until an explicit migration said otherwise.

Deploying code no longer rewrote the meaning of work by proximity.

Stable step ID was not display copy

The string send had served as UI label, database key, log field, and external idempotency component. Those concerns evolved at different rates.

I separated:

stepId: deliver-document-v1
displayLabel: Deliver document
handlerKind: document-delivery

The stable step ID never changed for a running definition. Display copy could improve. Handler kind could route several versioned definitions to compatible implementation code.

External request identity derived from workflow ID and stable step ID, plus an explicit effect revision when the domain required a new side effect. Renaming a label no longer changed it.

Receipts and dependencies referenced stable IDs. The interface projected current labels appropriate to the definition and could use newer explanatory copy without changing execution identity.

This looked like extra naming until the first migration. Identity that participates in retry or evidence should never be accidental user-facing text.

Handler code and definition version were related, not identical

Keeping every old application binary running forever would be expensive. Z29C supported multiple definition versions through versioned handler contracts in current deployments.

A handler declared which definition and payload versions it could execute. Compatible bug fixes could improve old workflows without changing their graph or stable identities. The attempt receipt recorded handler build and adapter version actually used.

If a new handler could not preserve old behavior safely, the release retained the older handler module or stopped and required migration. Deleting support while workflows still referenced it failed a deployment compatibility check.

The definition described intended sequence and evidence contracts. The handler build described implementation. A receipt connected them.

This allowed security or correctness fixes to reach old runs while keeping their logical plan stable. It also prevented one opaque version field from pretending graph and code were the same artifact.

Compatibility became declared and tested.

Running-state fixtures became release inputs

Unit tests usually started a new workflow under the current definition and ran it to completion. They did not load yesterday's partial state into today's code.

I captured fixtures at every durable boundary:

• Intent recorded before first scheduling event.
• Step ready but not leased.
• External request identity persisted before invocation.
• Outcome unknown during status query window.
• Rate limited until a future time.
• Waiting for manual review.
• Forward work failed with compensation pending.
• Compensation outcome unknown.
• Workflow completed with historical receipts.

Each fixture named its definition, payload, adapter-contract, projection, and local schema versions. The new build had to load, render, schedule, recover, cancel, and archive each supported state correctly.

The test did not necessarily advance every old fixture automatically. Some states correctly required migration or review. The release failed if the application merely crashed or silently reinterpreted them.

A deployment was compatible only with the work already in the database.

Structural migration was a workflow of its own

Some changes justified moving a running workflow to a new definition. I stopped updating rows with ad hoc database statements.

A migration plan named:

• Source and target definition versions.
• Eligible source states.
• Mapping of stable pending steps.
• Treatment of completed receipts.
• External request identities that must remain unchanged.
• New steps introduced and why they were safe.
• Compensation obligations.
• Roll-forward and stop conditions.
• Author and evidence.

The migration had a stable operation ID and receipt. Repeating it returned the same transition. If the server lost a response, the console could query outcome.

Completed effects were never migrated into having different identities. A new step that represented a new side effect received a new stable ID. Pending pure work could sometimes map directly.

The workflow timeline showed definition migrated from 3 to 4 and preserved the old graph for historical interpretation.

Migration became explicit product state rather than a deploy-time side effect.

Pending was easier to migrate than unknown

A step that had never begun could often move to an equivalent new step after validation. A step with outcome unknown carried a request whose external effect might exist.

Changing its stable ID, adapter key, or input normalization could destroy the recovery handle. Z29C generally kept unknown-outcome steps on their original definition until resolved. A migration could update the recovery handler while preserving every external identity and evidence contract, but it could not pretend the request was new.

Similarly, a completed receipt remained tied to its original step. The target definition could declare that receipt satisfies a new dependency only through an explicit mapping whose semantics were checked.

I wrote migration eligibility by epistemic state:

pending: often movable
ready: movable before new attempt identity
running: wait for checkpoint or stop safely
outcome-unknown: preserve identity; migrate only recovery implementation
succeeded: immutable historical receipt
compensating: preserve counteraction identity

Execution state determined migration freedom more than row shape.

Database schema evolved separately

Definition versions did not solve application schema changes. A new release could change tables or receipt fields needed by old handlers.

I used additive database migrations first: add new nullable or defaulted fields, deploy code that reads old and new shapes, backfill where appropriate, then remove obsolete columns only after no supported definition or fixture needed them.

Receipt payloads carried explicit versions and readers tolerated bounded older shapes. A projection could rebuild old events into a current read model without mutating the historical payload.

Destructive schema cleanup checked active and retained definition versions. The presence of zero currently running workflows was not enough if archived records still needed to render or audit correctly.

Database migration, workflow-definition migration, and event-payload migration became separate plans with relationships.

This was slower than changing all layers at once and far easier to recover when one assumption was wrong.

Pinning a definition did not grant permanent authority. A 2021 workflow created under policy version 3 might reach an external step after policy version 4 prohibited the action.

Before the side-effect boundary, Z29C evaluated current policy. The run history could show:

Definition 3 schedules document delivery. Current policy 4 blocks delivery for this synthetic subject. Review required.

The graph said what came next. Policy said whether it remained allowed now.

This avoided two extremes: reinterpreting the entire old workflow under new code, or allowing old intent to bypass current restrictions forever.

Policy decisions had their own receipts. An approved bounded exception did not modify definition 3. A denied step could trigger definition-declared compensation or stop according to current recovery policy.

Stable execution semantics and current authority were compatible because they answered different questions.

Adapter contracts also changed over time

An external adapter could gain idempotency support, change its query consistency window, or deprecate an operation. A running step needed to know which contract governed its existing request.

The request checkpoint stored adapter and contract version. Recovery used that version's promises for key validity, status query, and compensation.

A newer adapter implementation could safely query an older request if compatibility was declared. It could not assume a new idempotency window applied retroactively to a key issued under the old contract.

Before invoking a pending step, Z29C could select a current compatible adapter version under the definition's requirement. Once invocation began, external request identity anchored recovery to the accepted contract.

The release fixture included response loss across an adapter upgrade. The new process had to find the old request rather than create one under the new identity scheme.

Integration contracts have time just like workflow graphs.

Scheduler state survived code changes

Rate limits, backoff, workflow deadlines, and next eligible times lived durably. Deploying a scheduler did not reset them.

Each schedule decision named policy version and source observation. A new algorithm could apply to future decisions while preserving existing nextEligibleAt, or migrate schedules explicitly if the old calculation was unsafe.

Leases remained short and expired across deployment without correctness loss. A graceful shutdown reduced recovery latency; fixtures assumed processes could disappear without one.

A worker acquired a step only if its build supported the workflow definition and handler contract. Unsupported work stayed visible rather than being repeatedly leased and failed.

The queue message remained a wake-up hint. Durable workflow and schedule state determined action after any release.

This kept “deploy completed” from becoming “all timers and retries restarted.”

The system's patience outlived its processes.

UI copy could evolve without rewriting history

Old workflows had technical labels that became clearer over time. I wanted to improve them without making a 2021 run appear to have used concepts invented later.

Z29C separated current explanatory copy from historical authored records. A current interface could call a state Delivery status needs review while the event retained its original stable code outcome-unknown/v1.

Definition-specific milestone order remained accurate. A new step introduced in version 4 did not appear as skipped in version 3. Archived timelines used event and definition identities, then rendered them through compatible current components.

When a term's meaning changed materially, the interface displayed the historical label or an annotation rather than silently translating it.

This was the content version of schema compatibility. Better language could improve recovery while history stayed grounded in the model that had produced it.

Display is a projection too.

Rollout verified mixed versions

During deployment, old and new workers could coexist. Both needed to respect leases, fencing, definition support, and adapter identity.

The release scenario ran:

1. Old worker persists external request identity.
2. It loses its lease before the response.
3. New worker build starts.
4. New recovery handler queries under the old adapter contract.
5. Late old worker reports success.
6. Current handler reconciles one receipt.

Other scenarios left a rate wait, manual review, and pending compensation across the rollout.

Build identity appeared in attempt records. A late observation from an old worker could contribute evidence without committing current state under a stale fence.

I avoided a deployment scheme that required draining every long workflow. The durable model existed so work could span releases. A release that demanded total quiescence would discard that advantage.

Mixed-version behavior became a normal test, not a rollout surprise.

A completed workflow could remain useful long after handler support ended. Its timeline, receipts, and final projection needed interpretation.

Z29C retained a compact immutable definition artifact: stable step IDs, dependency and compensation graph, receipt contracts, display metadata, and version digest. It did not require executing old code to render history.

Archived receipt payloads remained versioned. Current readers could display bounded old shapes or state clearly that a deprecated detail required raw export. The archive did not load arbitrary historical application binaries.

Retention policy eventually removed full input or logs where their purpose ended, while keeping the minimal lineage required for the chosen workflow history period.

This prevented implementation cleanup from making old outcomes inscrutable.

A definition was part of the receipt for what the engine had done.

Observability grouped by definition version

An outcome trend across a definition change could hide the reason behavior shifted. Z29C tagged workflow and step metrics with bounded definition and adapter versions.

I compared:

• Completion and terminal outcome by definition.
• Unknown-outcome recovery path by adapter contract.
• Migration count and state.
• Active workflows on definitions approaching support end.
• Handler compatibility failures, which should remain zero before deployment.
• Running workflows blocked by current policy rather than definition error.

The personal project used deterministic scenarios rather than invented scale. The dimensions still made regression attribution possible.

I avoided unbounded labels such as workflow ID in metrics. Detailed timelines carried specific identities; metrics used stable version classes.

Deployment health included the old work the new release had inherited.

Retirement was a planned transition

A definition could not remain executable forever. Retiring it required knowing every active state and available path.

Z29C produced a retirement report:

• Active workflows by step and epistemic state.
• Pending work eligible for migration.
• Unknown or compensating work that must preserve identity.
• Handler and adapter support required until resolution.
• Archived artifacts retained after execution support ends.

New intent stopped entering the old definition first. Eligible pending workflows migrated through receipts. Remaining runs completed or reached explicit review. Only then could executable handler support disappear.

An emergency security issue could accelerate retirement and block current policy immediately. The workflow history still remained; urgency did not justify mutating receipts.

Retirement was another workflow with conditions, not a deletion after the deployment graph turned green.

If a new release introduced a handler defect, reverting the application artifact could restore the prior code. Workflows created while the new release was active still named their accepted definition versions. The old application might not understand them.

Release planning therefore required two safe directions: the new build could continue old definitions, and the prior build could either read new durable shapes safely or deployment policy prevented new definitions from entering before the release was proven.

For consequential definition changes, I separated availability from activation. The code supporting definition 4 could deploy while new intent continued using definition 3. Scenario tests exercised mixed handlers and stored state. Only then did a versioned configuration switch new workflows to 4. Reverting that switch affected future intent, not workflows already created under 4.

If definition 4 needed to be abandoned, its active workflows followed an explicit migration or completion path. I did not relabel them as version 3 because the application binary rolled back.

This resembled database expand-and-contract deployment. Capability arrived before data began depending on it, and old capability remained until dependence ended.

The distinction made application rollback useful without allowing it to falsify workflow history. Code can move backward quickly. Durable intent and external receipts move only through explicit transitions.

The rename became harmless in the right layer

After separating identity, I could change Send to Deliver document in display metadata without touching the stable step deliver-document-v1. Running workflows rendered the clearer label. External keys and receipts remained intact.

A later definition could introduce deliver-document-v2 with a materially different evidence contract. Migration would be explicit. New workflows could adopt it while old ones resolved under v1.

The original bug had come from using one convenient string to carry presentation, control flow, storage, and external identity. Versioning alone would not fix that conflation. Stable semantics and layered identities did.

Deployments are brief. Workflows can wait through rate limits, human review, outages, and policy changes. The system becomes durable when the plan and evidence outlive whichever release happens to be running today.

Z29C stopped asking the newest code to reinterpret the past. It asked new code to prove that it could continue it.