A lease is not a receipt / Bob's Site

At 10:03, Z29C's worker lease expired. At 10:04, another worker was allowed to recover the step. Neither timestamp said whether the synthetic delivery had happened.

The first worker had acquired the step correctly, persisted an attempt, and called the adapter. Its heartbeat stopped while the adapter response was delayed. The lease expired according to PostgreSQL time. Recovery saw no local completion receipt.

The early implementation concluded that the worker had failed before completing the action. It repeated the step.

The lease had answered who was allowed to work. I had read it as evidence of what work existed outside the lease.

Ownership and outcome needed separate records.

The lease solved a real scheduling problem

Z29C had several workers polling ready steps. Without coordination, two workers could select the same row and begin routine execution together.

A worker acquired a time-bounded lease through a transactional update. The record named:

workflowId
stepId
attemptId
leaseOwner
leaseVersion
leasedAt
expiresAt

Only one non-expired lease could govern a ready step. The worker renewed it while making expected progress. Completion or a durable transition released it. If the worker disappeared, expiry allowed recovery without a permanent lock.

This was valuable. It bounded abandoned ownership and prevented normal duplicate execution.

The mistake was not using a lease. The mistake was extending its claim beyond the resource that enforced it. PostgreSQL could decide which worker held the Z29C step. The external delivery stub did not participate in that lease and did not revoke a request already accepted.

A correct local lock was not a distributed receipt.

Expiry meant “ownership may be reassigned”

I rewrote the lease contract as one sentence:

After expiresAt, another recovery worker may inspect and transition the step according to its durable attempt evidence.

It did not say:

The prior worker is dead.
The prior request did not reach the destination.
The prior process cannot still write.
The step is safe to execute again.
The external side effect is absent.

Networks and scheduling make those implications unsafe. A worker can pause long enough to miss renewal and resume later. A response can arrive after its local deadline. A process can remain alive while its database connection fails. The destination can commit independently.

Expiry enabled a new decision. It was not the decision itself.

The recovery worker loaded the attempt state, request identity, adapter contract, and receipts before choosing query, retry, review, or terminal failure.

That additional read turned lease expiry from a trigger for repetition into a trigger for reasoning.

Heartbeats measured one path of liveness

The worker renewed its lease every few seconds. A healthy heartbeat meant the worker could reach PostgreSQL and its renewal loop had run recently.

It did not prove the adapter call was progressing. The call could be stuck while a separate heartbeat task continued. Conversely, a brief database outage could stop renewal while the adapter completed normally.

I associated progress observations with step checkpoints rather than relying only on process heartbeat:

Attempt created.
External request identity persisted.
Adapter invocation began.
Adapter response observed.
Step receipt committed.

The lease renewal included the latest checkpoint version. A worker could not renew an old lease version after recovery had superseded it.

Long-running pure computations could expose bounded progress and cancellation points. Opaque external calls could not pretend to. They had an execution deadline and a clear unknown-outcome path.

Heartbeats remained useful for detecting abandoned workers. They stopped serving as proof of external progress.

A fencing token protected participating resources

Every lease acquisition incremented a version. Z29C used that version as a fencing token for its own writes. A stale worker returning after expiry could not commit a step receipt under an older lease version.

The database transition checked:

step lease version == attempt lease version
and lease owner == attempting worker
and workflow state still permits completion

If recovery had acquired version 8, a late write from version 7 was rejected. The observation did not disappear; it entered a late-attempt record for reconciliation rather than mutating current state.

Fencing works only where the resource checks the token. Z29C's PostgreSQL state did. The synthetic delivery adapter initially did not. Sending lease version 7 in a header would accomplish nothing unless the destination stored and compared it under a matching domain identity.

For adapters that supported a stable idempotency key, that key served a more relevant purpose: repeated calls for one logical effect returned one result. I did not describe every external service as fenced simply because Z29C generated a number.

Tokens need enforcement, not ceremony.

A step receipt represented Z29C's accepted conclusion. An adapter receipt represented the external system's result. They could be separated by a failure window.

The sequence became:

Persist attempt and request identity.
Call adapter with stable key where supported.
Observe external result and durable external reference.
Commit Z29C step receipt under current lease version.
Release dependant steps.

If the worker lost its lease after step three but before four, a late completion write could be rejected. Recovery still needed to retain the external reference as evidence. The adapter's stable key or query path allowed Z29C to verify it and create a current receipt.

The receipt was not merely “worker returned success.” It named workflow, step, input digest, adapter version, request identity, external reference, observed result, and method of verification.

Outcome came from evidence across the boundary. Lease ownership governed who could commit the conclusion now.

Renewal stopped before the execution deadline

A worker that renewed forever around a stalled call could hold the step indefinitely. A lease needs a bounded relationship to useful work.

Each attempt had an execution deadline determined by step and adapter policy. Lease renewal stopped before that deadline, leaving enough time to record a timeout observation and transition safely. The total workflow had a wider deadline and attempt budget.

The deadlines did different jobs:

Adapter timeout bounded one call's waiting.
Attempt deadline bounded one worker's execution authority.
Lease expiry allowed recovery after lost ownership.
Workflow deadline bounded the useful lifetime of the intended outcome.

Increasing one did not repair the others. A longer lease could reduce premature recovery and delay detection of a dead worker. A longer adapter timeout could reduce unknown outcomes and occupy scarce concurrency.

I tested boundary timing around each deadline. Correctness could not depend on the response arriving on one favorable side of a timer.

Time controlled when to reconsider. It did not reveal whether an effect existed.

The recovery scan did not enqueue every expired row

My first recovery task selected expired leases and marked their steps ready. That transition erased the reason they had stopped.

The revised task classified durable checkpoint state:

No request identity persisted: safe to make the step ready if policy still allows it.
Request prepared but invocation not begun under a checkpoint guarantee: safe to retry.
Invocation began and adapter is idempotent: retry with the same key or query first.
Invocation began and destination is queryable: schedule a status query.
Invocation began with opaque outcome: enter human review.
External result exists but local receipt missing: verify and reconcile.
Local receipt exists: complete queue acknowledgement without re-execution.

The scan created a recovery decision event and, where appropriate, a new attempt. It never changed an ambiguous step directly to ready.

This made expired-lease volume less interesting than recovery-category volume. Ten safe pure-computation retries and one opaque external unknown were not the same operational condition.

Recovery began with classification.

The late worker was still part of the story

In one scenario, worker A lost its lease, worker B queried the destination, and then A received the original success response.

Worker A could not commit current state under its stale lease. Discarding the response would throw away valuable evidence. It posted a late observation through a separate append-only path naming its attempt and external reference.

Worker B's recovery could consume that observation if it matched request identity and input digest. If the destination query had already found the same result, the evidence converged. If it reported a different reference, Z29C raised an integrity conflict.

The stale worker never regained authority over the step. It remained a source of evidence about its own attempt.

This separated command and observation. Fencing prevented late mutation; the incident-style event path preserved what the process had learned.

Killing stale output entirely would make safety look like ignorance. Accepting it as current state would defeat fencing. Typed late evidence allowed both concerns to hold.

Graceful shutdown improved latency, not correctness

I added shutdown handling so a worker stopped acquiring leases, completed safe local checkpoints, and released work it had not begun. It reduced recovery delay during deployments.

I did not make correctness depend on it. Processes can be killed, hosts can fail, and the network can disappear before a shutdown hook runs.

At every checkpoint, abrupt termination had a defined recovery:

Before attempt commit, no durable ownership existed.
After attempt commit but before request identity, recovery could restart preparation.
After request identity but before invocation checkpoint, adapter contract determined safety.
After invocation, unknown-outcome rules applied.
After external observation but before local receipt, reconciliation applied.
After local receipt, queue redelivery became a no-op.

The scenario harness terminated worker processes at each boundary. A clean shutdown was one friendly path through the same model.

Reliability comes from durable state that survives impolite endings, not from confidence that every process will say goodbye.

Lease duration followed work class

One fixed lease duration produced opposite failures. Short pure calculations recovered quickly after crashes. Long external calls lost leases during normal latency. Making every lease long delayed recovery for ordinary steps.

Step definitions supplied a lease policy within bounded platform limits:

Initial duration based on expected checkpoint interval.
Renewal cadence.
Maximum attempt deadline.
Whether progress could be observed safely.
Grace available to persist a final observation.

Rate-limited waits did not hold worker leases. The scheduler stored nextEligibleAt and released capacity. Human review did not hold a lease. A lease represented active execution, not every kind of waiting.

The scheduler also avoided renewing a lease when the current workflow was cancelled or policy changed. The worker transitioned at its next safe checkpoint.

This reduced the number of expired leases without treating expiry as failure by itself.

Duration was scheduling data grounded in step behavior.

Worker clocks could drift. If each worker decided lease expiry from its own wall clock, two processes could disagree about current ownership.

PostgreSQL time governed acquisition, renewal, and expiry. The worker treated returned expiry as informational and renewed before a conservative local deadline. Server-side comparisons decided whether the lease remained valid.

Network delay could consume part of the renewal window. The worker did not assume a renewal sent before local time was accepted. It waited for the database result and stopped external work at a safe boundary if renewal failed.

Monotonic local time measured durations within the process where appropriate. It did not override the database's lease authority.

W93H's clock work had taught me to name clock domains. Z29C used one authoritative domain for its own lease state and retained source times for adapter observations separately.

Even a local coordination primitive needs a clear clock owner.

Leases did not flow into user progress

The first Z29C page displayed Worker active and Worker lost. Those states exposed infrastructure and made a lease expiry look like the user's request had failed.

The product projection used meaningful states:

Preparing.
Waiting on an external service.
Recovering execution.
Delivery status needs review.
Completed.

A new worker could acquire the step without creating a new user-visible item. A lease expiry during a pure retry might produce no notification. An unknown external outcome produced a clear action state because consequence, not worker liveness, had changed.

Technical details showed lease history for diagnosis. The main narrative remained one intent and its evidence.

This made infrastructure recovery less alarming and semantic uncertainty more visible.

The person did not need to understand worker ownership to know whether trying again was safe.

Metrics distinguished churn from uncertainty

Z29C tracked lease acquisitions, renewal failures, expiry age, and recoveries. Those helped tune worker behavior and find capacity problems.

It separately tracked:

Steps entering outcome unknown.
Time until external receipt or manual resolution.
Stale-worker late observations.
Fencing rejections.
Duplicate external references.
Recovery decisions by adapter capability.

A rising lease-expiry rate could indicate worker churn without external correctness impact. One opaque unknown outcome could deserve more attention than many safe computation retries.

I avoided a single “job failure” graph that combined them. The metric names kept their level: lease behavior, attempt behavior, step knowledge, and workflow outcome.

In the synthetic environment, deterministic fault scenarios supplied the counts. There was no invented production scale.

Operational clarity began by not allowing liveness metrics to masquerade as result metrics.

Adapter concurrency was another lease with another scope

Z29C also limited concurrent calls to each synthetic adapter. I first reused the step lease count as the limit. A worker holding a step during local preparation consumed adapter capacity before making a call, while a late in-flight call whose step lease had expired stopped counting even though the destination still saw it.

I created a separate adapter permit acquired immediately before invocation. It named adapter, operation class, attempt, expiry, and request identity. The permit bounded how many calls Z29C intentionally started. It still could not cancel a request already accepted or prove that an expired call had ended remotely.

Recovery treated permit expiry with the same restraint. A new call could start only if step semantics allowed another request, not merely because a concurrency slot appeared free. For an unknown opaque outcome, the logical capacity remained represented in a conservative adapter budget until review or a bounded safety window passed.

This exposed an unavoidable trade. Holding capacity conservatively could reduce throughput after network ambiguity. Releasing it optimistically could exceed the destination's real concurrency if old calls remained active. The adapter contract chose the policy and made the risk visible.

Separating step ownership from adapter permits clarified both. A worker could own a step while waiting to acquire destination capacity without holding the permit. A rate-limit response released the worker and scheduled a future eligibility time. A status query used a different operation class from creation.

Leases coordinate access to a named resource. Z29C had several resources, and none of their leases certified the outcome of another.

The invariant became narrower and stronger

After the redesign, Z29C could defend these claims:

At most one current lease governs routine step execution in PostgreSQL.
Once its lease is stale, a worker can no longer commit the step it once owned.
Lease expiry always enters evidence-based recovery.
External request identity exists before a side-effecting invocation where the adapter contract permits.
A completed step has a durable receipt.
Absence of a receipt after invocation is unknown, not proof of absence.

It could not claim that only one external request was ever in flight unless the adapter itself enforced a stable idempotency or fencing boundary. It could not infer process death from heartbeat loss. It could not promise exactly-once effect through lease ownership.

The narrower claims were more useful because tests could verify them.

Ownership ends; evidence remains

Leases are excellent tools for coordinating fallible workers. They make abandoned work recoverable and normal concurrency controllable. Z29C kept them at the center of scheduling.

The mistake had been treating a lease as a certificate about the world beyond Z29C's database.

When a lease expired, ownership ended. The attempt, request identity, external reference, and observations remained. Recovery combined those facts with adapter capabilities before taking another action.

A receipt answered whether the step could advance. A lease answered who was currently allowed to try to produce or verify that receipt.

The two records often changed near each other. They never meant the same thing.