Rate limits are scheduling data

Z29C's first response to a rate limit was to hold the worker, sleep, and try again.

The synthetic adapter had returned 429 with a retry time. The worker still owned its lease, occupied one process slot, and knew nothing about other workflows calling the same destination. Ten limited steps produced ten sleeping workers. When their timers ended together, all ten retried together and received another limit.

I had treated rate limiting as an exception inside one attempt. It was scheduling information about shared capacity over time.

Z29C began storing the destination's signal, releasing workers, and deciding when each logical step became eligible again.

A 429 response was an accepted observation

The first error mapper converted every non-success HTTP response into AdapterError. Retry middleware chose exponential backoff.

A rate-limit response carried more structure:

• The destination had received enough of the request to classify it.
• The request was not accepted as the intended side effect under that response.
• A retry might become useful after a stated interval.
• The limit might apply to an account, endpoint, operation class, or broader destination.
• Repeating immediately would consume capacity and likely extend harm.

Z29C stored a typed observation with status, stable adapter code, scope where known, response time, and Retry-After value. It did not retain sensitive headers or full bodies.

The step transitioned from running to scheduled waiting, not failed-retryable in a generic queue. Its current attempt ended. No worker remained leased.

The destination had supplied a scheduling constraint. Preserving it made cooperation possible.

Retry-After needed parsing and a clock owner

HTTP Retry-After can represent a delay in seconds or an HTTP date. The two forms have different clock assumptions.

A delay could be added to Z29C's receipt time using the scheduler's authoritative database clock. An absolute date came from the destination's clock and could be skewed relative to Z29C.

The adapter normalized the value into:

observedAt
rawRetryAfter
earliestRetryAt
interpretation: delay | http-date | invalid
clockUncertainty

For an invalid or absent header, adapter policy supplied bounded backoff. For an absolute date in the past, Z29C did not immediately unleash every waiter; it applied a small minimum delay and jitter while recording the anomaly.

The original header remained available as bounded evidence. Changing parsing logic later did not rewrite what the destination had sent.

W93H's clock lesson returned in a scheduler. A timestamp is useful only when its clock domain and interpretation are clear.

Sleeping was state, not work

A sleeping process consumed memory, a worker slot, a database connection if implemented carelessly, and a lease that could expire while nothing productive happened.

Z29C committed nextEligibleAt on the step and released the lease. The scheduler selected only ready steps whose eligibility time had arrived.

The state distinguished reasons:

• Waiting for adapter retry window.
• Waiting for workflow-defined delay.
• Waiting for a dependency receipt.
• Waiting for human review.
• Waiting for current authority.

All were non-running states. Their wake conditions differed.

No in-memory timer was required for correctness. A process restart or deployment did not lose the wait. An index over eligible time let workers find due steps. A periodic recovery scan caught missed scheduling events.

This converted time from control flow inside one worker into durable workflow data.

The system could now show and reason about the wait even when no process remained alive.

The limit scope determined who should wait

One 429 response might apply only to a single subject, one adapter account, one endpoint, or the entire destination. Applying it to the wrong scope either wastes capacity or continues overload.

Adapter contracts named known limit dimensions:

adapter
credential or account class
operation class
synthetic subject
global destination

The response could update a scheduler gate for the relevant key. New steps under that gate received an eligibility time before acquiring a worker. Unrelated operation classes continued if the contract said their capacity was separate.

When the destination did not reveal scope, Z29C chose a conservative adapter-level gate and recorded the limitation. It did not infer a precise per-subject quota from one response.

The synthetic harness modeled several scopes and asserted that a rate limit on document creation did not block status queries needed to resolve unknown outcomes, unless the adapter declared one shared pool.

Recovery traffic can be more important than new work. Scope made that priority expressible.

Jitter prevented a synchronized retry wave

If every step received Retry-After: 60, scheduling all of them at exactly one minute recreated the burst.

Z29C treated the destination time as the earliest allowed retry and spread eligible work with bounded jitter and concurrency control. Jitter never moved a request earlier than the declared window.

The scheduler also admitted a limited number of calls as the gate reopened. If the first calls succeeded, capacity could increase cautiously. If they returned another limit, the gate extended without waking the whole backlog.

I kept the policy simple and versioned. The personal synthetic environment did not justify an elaborate adaptive controller. The important properties were:

• Respect the earliest retry time.
• Avoid simultaneous release.
• Limit in-flight calls by adapter and operation class.
• Preserve fairness across workflows.
• Stop at workflow and attempt budgets.

Randomness had a deterministic seed in scenario tests so schedules remained reproducible.

Jitter was not noise added after scheduling. It was part of cooperative load shaping.

Concurrency and rate were different constraints

An adapter could allow five simultaneous calls and one hundred calls per minute. Meeting one constraint did not imply meeting the other.

Z29C used short-lived adapter permits to limit in-flight calls. A separate rate gate tracked earliest times and bounded admission over a window according to known contract data.

A long request occupied concurrency but counted once toward rate. A quick burst could exhaust rate with low concurrency. Status queries and creation requests could have different pools.

The scheduler acquired an adapter permit immediately before invocation and released it after the call observation. Waiting for a permit did not begin the external attempt. A process could release its step lease and reschedule if capacity wait would exceed a bounded interval.

I did not claim Z29C could perfectly reproduce the destination's hidden algorithm. It used documented limits and response evidence to behave conservatively.

The distinction prevented one “max retries” or “max parallel” setting from standing in for all capacity behavior.

Workflow deadlines constrained useful waiting

Honoring a six-hour retry window could be correct for the adapter and useless for a workflow whose intended result was needed in one hour.

Before scheduling another attempt, Z29C compared:

• Earliest adapter eligibility.
• Step execution deadline.
• Overall workflow deadline.
• Idempotency-key validity window.
• Remaining attempt and external-call budget.
• Time required for dependant steps.

If the earliest safe retry could not complete within the useful workflow window, the step entered review or terminal failure according to definition. The interface said the destination's next window fell beyond the request deadline.

Z29C did not violate the rate limit to chase the deadline. It also did not wait silently for an outcome that could no longer be useful.

This made time a product constraint. Adapter patience and user patience were related schedules with different owners.

The workflow could offer a deliberate alternative adapter only if the definition had modeled its side effects and identity safely.

Some destinations retain idempotency keys for a bounded period. A long rate-limit wait could approach or cross that window.

Retrying after expiry might create a duplicate if the original outcome was unknown. Z29C stored the destination key's valid-until time where the contract exposed one and compared it with nextEligibleAt.

Possible outcomes:

• Safe window remains: retry same key after rate limit.
• Status query remains available after key expiry: query first.
• Safe window expires before eligibility: enter review before automatic creation.
• Original response was an explicit rejection before effect: a new attempt may be allowed under policy.

The scheduler did not reset the key merely because time passed. A new key represented a new possible effect and required an explicit transition.

This joined correctness and scheduling. A retry can be allowed by capacity and unsafe by identity.

The interface surfaced the approaching evidence deadline rather than presenting one generic waiting spinner.

Fairness began at workflow identity

One synthetic workflow could fan out hundreds of delivery steps. A simple ready-time queue allowed it to occupy every adapter slot when a gate reopened.

Z29C grouped eligible work by workflow and synthetic subject, then admitted bounded rounds. Smaller workflows could progress without waiting behind one fan-out.

Unknown-outcome status queries received a reserved portion of adapter capacity because they reduced uncertainty and could unblock or prevent duplicate effects. New low-priority creation could wait.

Fairness did not mean equal throughput for every step. Workflow deadline, operation consequence, age, and explicit priority influenced selection within documented bounds. I avoided a hidden business-value score.

The scheduler recorded why a step was selected: eligible time, gate, priority class, and fairness cursor. replay runs with the same scenario seed produced explainable order.

Even at personal-project scale, this model exposed starvation bugs that a single FIFO queue hid.

Rate limiting was a shared scheduling problem, not a private sleep in each worker.

Backoff followed error category

Not every retryable condition provided Retry-After. Z29C used adapter-specific backoff with jitter for transient unavailability and transport failures known to be safe to retry.

The categories remained separate:

• Rate limited: obey destination earliest time and scope gate.
• Remote unavailable: exponential backoff within workflow budget.
• Local capacity unavailable: scheduler delay without counting an external attempt.
• Authorization blocked: no automatic retry until policy or credentials change.
• Validation rejected: requires input change.
• Outcome unknown: query or review before repeating side effect.

Generic exceptions did not select a backoff path. The adapter produced a typed observation.

This prevented a 401 from being retried for hours and an unknown outcome from becoming another creation. It also prevented local worker scarcity from consuming adapter retry budget.

Retry policy became an interpretation of evidence, not a decorator around every function.

User-facing progress named the wait

The first UI displayed Retry 2 of 3. It made the system look active and said nothing about when or why.

Z29C projected:

• The delivery service asked Z29C to wait until 14:30.
• Your request remains within its completion window.
• No action is needed.

If the wait crossed the workflow deadline:

• The next safe delivery attempt is later than this request allows. Choose whether to extend the deadline or stop remaining work.

The page did not show an exact second-by-second countdown that implied perfect clocks. It displayed the chosen timezone and refreshed from durable state.

Repeated 429 responses did not generate repeated notifications. A meaningful transition—deadline at risk, review required, or completion—did.

The person saw scheduling truth instead of worker attempt theater.

A workflow could be cancelled while waiting for a rate gate. No worker held a lease, so sending a cancellation signal to processes would accomplish nothing.

The cancellation transition marked pending steps no longer eligible and removed or invalidated their scheduler entries. The shared adapter gate remained because it applied to other workflows.

If an attempt was already in flight, step-specific cancellation and outcome rules applied. The scheduler could stop new calls and could not unsend the active one.

Reopening or revising a workflow created new eligibility under current policy and did not resurrect old timers blindly. Every scheduled row named the workflow and step revision that had created it.

This was another benefit of durable scheduling state. Control acted on the logical workflow even when no active process existed.

The interface could confirm exactly which future actions cancellation prevented.

Deployments did not reset patience

An in-memory timer would disappear during a worker release. A generic retry count might restart under new code. Z29C stored observation, policy version, attempt count, and next eligible time durably.

A new scheduler release respected existing gates and deadlines. If policy changed, it applied through an explicit migration or future observations rather than silently recomputing old waits.

The system retained the raw Retry-After interpretation so a bug fix could audit affected schedules. Recalculating an absolute date required acknowledging clock assumptions.

Running workflows remained on their definition versions. Current adapter policy could impose stricter capacity without rewriting history.

The scenario harness stopped every scheduler process during a rate wait, advanced time, deployed a new handler version, and verified that no early burst occurred.

Patience belonged to the workflow state, not the lifespan of one worker.

Metrics measured pressure and cooperation

Z29C tracked:

• Rate-limit observations by adapter and operation class.
• Declared versus observed retry windows.
• Steps waiting behind each scope gate.
• Time spent rate-limited relative to workflow deadline.
• Retries attempted before eligibility, which should remain zero.
• Burst size as gates reopened.
• Starvation and fairness by workflow class.
• Idempotency windows endangered by delay.

I did not invent provider scale. Deterministic synthetic scenarios supplied known capacity and asserted scheduler behavior.

A low error rate could coexist with poor cooperation if Z29C hammered immediately and the stub eventually accepted one call. The relevant behavior was respecting the signal while keeping useful work moving.

The metrics described scheduling pressure, not a generic reliability percentage.

Quota exhaustion was not always a retry window

One adapter used the same 429 status for a short burst limit and for a synthetic monthly quota. Retrying the second condition with exponential backoff would keep the workflow alive without a plausible path to success.

The stable adapter code and bounded response metadata distinguished:

• Short window with an earliest retry time.
• Daily or monthly quota whose reset time exceeded many workflow deadlines.
• Account disabled pending a configuration change.
• Per-subject limit that required a different workflow decision.

Z29C scheduled only when the response supplied a credible future eligibility. A quota beyond the workflow deadline entered review immediately with the current usage period and reset evidence. The person could stop, extend the request if still useful, or choose an explicitly modeled alternative adapter. The scheduler did not invent capacity.

This also changed alerting. A burst limit was normal cooperative flow at low frequency. Repeated quota exhaustion indicated planning or configuration. An account-disabled response was an authority or account-state problem. One red “rate limit” counter would collapse different repairs.

I retained a conservative fallback for unknown 429 shapes, but the fallback had an attempt and time budget. If the adapter could not establish when another request might be accepted, Z29C stopped automatic retries before spending the whole workflow window.

The distinction kept patience from becoming avoidance. Waiting is useful only when the system can name the condition expected to change.

When a step moved from rate-limited to eligible, Z29C recorded which gate, policy version, destination observation, jitter choice, and fairness cursor supported the decision. The receipt was compact and made later analysis reproducible.

If a bug scheduled work before Retry-After, the timeline could compare raw response, normalized earliest time, and selected attempt. If a workflow starved, the fairness receipt showed which higher-priority classes consumed the gate.

This was not an audit log of every scheduler loop. It recorded consequential eligibility transitions. A new scheduler implementation could be evaluated against captured scenarios without pretending its ordering was invisible infrastructure.

Scheduling decides whose work reaches an external boundary and when. In Z29C that was important enough to explain.

The adapter response became part of the plan

The first 429 had looked like a failure interrupting execution. After the redesign, it changed the plan durably.

Z29C recorded when the destination would next consider work, which calls shared the constraint, how much workflow time remained, and which safe operations deserved scarce capacity. Workers became disposable executors of eligible steps rather than sleeping owners of future promises.

This reduced resource waste and made the user-facing wait explainable. It also prevented synchronized retries from turning one limit into a sustained overload.

Rate limits are often described as something a client “handles.” The important change was to stop handling them privately inside a request loop.

The destination had supplied scheduling data. Z29C became a better participant when it let the scheduler hear it.