The recovery console should not start with Retry

Z29C replaced a failed-jobs table and one dangerous button with an evidence-first interface that exposed what happened, what remained unknown, and which actions were safe.

Z29C's first recovery console was a table of failed jobs. The final column contained a blue Retry button.

It was visually clear, consistent, and most dangerous on the rows where it seemed most useful.

A timed-out synthetic delivery could already exist. A step might have failed validation and require changed input. A worker lease could expire after an external request began. A rate-limited adapter might have named a future retry time. A compensation attempt could be the side effect now in doubt.

The table flattened those conditions into one word and placed repetition at the end of every row.

I redesigned the console around evidence and safe next actions. Retry became one possible operation, not the organizing principle.

The row was the wrong unit

The table row represented a worker job or attempt. Recovery concerned a logical workflow step under one user intent.

One delivery step could have:

  • Two worker attempts.
  • One stable external request key.
  • A timeout after invocation.
  • A late success response from the stale worker.
  • A status query from the current recovery worker.
  • One final verified receipt.

Displaying each attempt as an independent failed job made repetition look like progress and scattered the evidence needed for one decision.

The redesigned page began with workflow intent and stable step. Attempts appeared as an expandable history. External references, request identity, adapter contract, and current receipt state stayed together.

The console could now answer: what outcome was requested, what has Z29C tried, and what does it know now?

Changing the unit corrected the interface before any visual design work.

Current knowledge came before error text

Exception strings dominated the original table. They were useful diagnostics and poor recovery summaries.

The step header stated one of Z29C's evidence states:

  • Rejected: destination provided a durable reason the request was not accepted.
  • Retryable before effect: evidence supports another safe attempt.
  • Outcome unknown: the request may have been accepted and no sufficient receipt exists.
  • Rate limited until: destination supplied a scheduling boundary.
  • Policy blocked: current authority does not permit invocation.
  • Compensation incomplete: counteraction has not produced its required receipt.

Below it, the console showed the observation that justified the state. Raw logs and stack traces lived in attempt details and linked source diagnostics.

A transport timeout did not become the title when its semantic consequence was unknown outcome. A database constraint error did not become a generic server failure when it represented an input conflict.

Recovery began with the strongest supported claim.

The evidence panel separated facts from conclusions

W93H's incident interface had distinguished observations, actions, and hypotheses. Z29C needed an execution version.

The evidence panel grouped:

  • Intent: immutable command, input digest, definition and policy versions.
  • Requests: attempt and external request identities, invocation checkpoints.
  • Observations: adapter responses, timeouts, queries, late worker messages.
  • Receipts: local and external conclusions accepted by the workflow.
  • Gaps: the exact evidence required but absent.

For an unknown delivery, the gap read:

No verified external reference or authoritative absence exists for request key w-17/deliver-v1.

The wording made the next operation apparent. Query could fill the gap. Blind Retry could create another effect.

The panel did not ask a reviewer to infer state from chronological logs. It preserved chronology as supporting detail and projected the current evidence question.

I first hard-coded buttons by status. That moved the generic retry policy from the worker to the UI.

Z29C derived actions from step state, adapter capabilities, workflow definition, current authority, deadlines, and evidence:

  • Check destination when status query existed.
  • Retry with the same key when idempotent creation remained valid.
  • Wait until 14:30 when the adapter supplied a rate window.
  • Revise input after validation rejection where the workflow allowed correction.
  • Attach verified receipt for manual evidence.
  • Request withdrawal after a completed effect with supported compensation.
  • Authorize duplicate-risk attempt only in review with consequences explicit.
  • Stop remaining work when the workflow could end without counteraction.

Every action response included why it was available and what state it could produce.

The UI remained a projection of server policy. It did not decide safety in the browser.

Retry was the most common recovery in pure computations. It did not deserve primary placement on every page.

For outcome unknown under a queryable adapter, Check destination was primary. For rate limiting, no immediate action was primary; the next eligible time and cancellation option mattered. For validation rejection, Review input led. For a terminal failure after partial completion, Review recovery plan led.

If no action could improve evidence safely, the page said Manual evidence required and explained the boundary. It did not add a disabled Retry as decoration.

I used button hierarchy to represent consequence. A duplicate-risk attempt required an explicit review flow and confirmation of the one-effect invariant it might violate. It never sat beside a harmless Refresh button.

Visual prominence became a claim about what the system recommended. The recommendation had to come from the contract.

Retry included its identity and scope

When Retry was safe, the confirmation explained what would repeat:

Z29C will make another delivery attempt for the same logical step using external idempotency key w-17/deliver-v1. The adapter guarantees repeated requests under this key return the original result until 18:00.

For a pure calculation:

Z29C will recompute the document from immutable input digest d-44. No external effect has begun.

The person did not need to memorize the identifiers. Their presence in expandable detail made the claim auditable.

The action had its own idempotency key. Losing the console response did not schedule two retries. The page refreshed from workflow revision and showed the accepted recovery attempt.

I avoided “Retry workflow” when only one step was safe to repeat. Dependants and completed effects retained their state.

Specificity reduced both fear and accidental scope.

The original console offered Mark as complete. It could turn any row green without evidence.

Manual completion required a typed receipt appropriate to the step:

  • External reference or deterministic result identity.
  • Input digest comparison.
  • Evidence source and observation time.
  • Author and reason.
  • Adapter and contract version where known.
  • Residual uncertainty or limitation.

The form changed by evidence contract. A generated artifact needed its content digest. A delivery needed its destination reference. A compensation needed the target and resulting state.

If evidence could not satisfy the receipt, the reviewer could create a waiver or accepted-risk resolution. The final projection displayed that distinction.

There was no generic force-success control. Development-only fixture manipulation remained outside the product console and could not operate on ordinary runs.

Manual recovery earned the same durable explanation as automation.

Even in a one-person project, two tabs could open the same recovery page. The invited scenario could also involve more than one study participant reviewing low-consequence test state.

Every recovery command named the workflow projection revision it had inspected. If another action resolved the step first, the server returned current state instead of applying a stale Retry or waiver.

The page showed:

This step changed while you were reviewing it. Delivery was verified from external reference x-82. Your action was not applied.

Drafted manual evidence remained locally copyable. A fresh action required the current revision.

Short-lived UI edit indicators could warn about likely collision but did not lock recovery. Server preconditions remained the authority.

This prevented a second reviewer from creating a new side effect after the first had filled the evidence gap.

The recovery console was itself a concurrent command surface and followed Z29C's identity rules.

The timeline explained how the state formed

The evidence summary answered what was true now. The timeline answered how Z29C reached it:

14:02 attempt a-1 acquired lease 7
14:02 request identity persisted
14:03 adapter response deadline passed
14:03 outcome marked unknown
14:04 lease expired
14:05 status query scheduled
14:06 stale worker reported late success x-82
14:06 current recovery verified x-82

Events labeled source and epistemic role: attempt, observation, scheduler decision, receipt, or manual action. A late event showed both source time and discovery time.

The timeline did not visually imply that a Retry had caused recovery when a query or late response supplied the evidence. It linked every transition to its input.

This made postmortem review possible without requiring raw worker logs. Detailed logs remained attached to attempts for debugging.

Recovery UI and observability used one semantic event model.

Z29C's technical console was for recovering the personal synthetic system. The initiating user-facing view needed a smaller account.

It showed:

  • Requested outcome and durable ID.
  • Completed milestones.
  • Current meaningful state.
  • Whether any action was required from the initiator.
  • What already happened and could not be cancelled.

An unknown delivery appeared as Z29C is checking whether delivery completed or Delivery status needs your review, depending on available automation. Worker leases and adapter exception codes stayed behind technical details.

If manual input was necessary, the link opened the focused recovery question rather than the whole console dashboard.

The two views shared one workflow projection. The simpler view did not lie by mapping unknown to failed, and the technical view did not force infrastructure vocabulary into the main journey.

Good recovery design reveals detail according to the decision a person must make.

Accessibility required stable state, not live-log noise

The original table updated rows as attempts retried. Screen-reader announcements became a stream of worker activity. Focus could move when a failed row disappeared after success.

The redesigned page used a stable heading for workflow and step. Meaningful state changes produced a concise update notice. Raw log additions did not enter the live region.

Action availability was communicated in text, not only button enablement or color. Disabled controls referenced the missing evidence or earliest time. Confirmation dialogs named the effect and recovery guarantee rather than asking “Are you sure?” generically.

The evidence sections used headings and definition lists. The attempt timeline was a list with links to details. After an action, focus returned to the updated state heading and announced the accepted receipt or conflict.

Manual receipt errors preserved entered evidence and moved focus to a summary. A lost network response reused the command identity and did not clear the form.

The console became calmer by announcing domain transitions instead of operational churn.

Some recoveries could not be made fully safe. An opaque adapter might leave outcome unknown until the workflow deadline. A person could authorize another delivery with duplicate risk.

The preview showed:

  • What effect might already exist.
  • Why automatic verification was unavailable.
  • Which external reference or time window had been checked.
  • What a duplicate could change.
  • Whether compensation was possible and its limits.
  • Which future steps would remain blocked or proceed.

The action required an authored reason and created an accepted-risk receipt. It was not hidden behind a generic advanced menu, and it was not the default.

Conversely, harmless operations did not receive dramatic confirmation. Checking status and refreshing evidence could proceed directly.

Friction matched consequence rather than implementation complexity.

The interface could not remove risk. It could stop disguising risk as routine repetition.

Filters organized by required recovery

The dashboard originally filtered by job status: failed, retrying, running. I changed the categories:

  • Evidence query available.
  • Safe automatic retry scheduled.
  • Input or authority review required.
  • Outcome unknown with no automatic evidence path.
  • Compensation incomplete.
  • Approaching workflow or idempotency deadline.
  • Resolved with waiver or accepted risk.

This grouped steps that needed similar attention even if their worker errors differed.

Sort order considered consequence and deadline, not number of failures. One opaque unknown delivery could appear above fifty safe scheduled retries.

No fabricated production queue existed; the personal scenario harness generated runs. The dashboard still benefited from modeling prioritization honestly.

The worklist answered “what decision is needed?” rather than “which process threw?”

Actions themselves produced recoverable workflows

A status query, manual receipt, or compensation request could fail or lose its response. The console did not fire them as untracked commands.

Every consequential action received a stable operation ID and returned a receipt. Query attempts joined the step timeline. Compensation became a linked step. Manual resolution committed under a projection precondition.

If the browser lost a response, reopening the page showed whether the command had been accepted. Pressing the button again reused identity where appropriate.

This recursive property mattered. A recovery console that helps resolve unknown outcome should not create a new unknown command merely because its own request timed out.

The UI used the same protocol it exposed. It did not sit outside the system as an all-powerful admin panel.

Recovery remained explainable at every layer.

The console had no green ending shortcut

Resolved steps could have different endings:

  • Verified success.
  • Verified absence and safe terminal failure.
  • Completed after idempotent retry.
  • Completed through manual receipt.
  • Counteracted after partial completion.
  • Closed with accepted uncertainty.
  • Duplicate effect detected and repaired.

The summary preserved the outcome kind. A green check alone would make them equivalent.

The workflow could close only when required recovery conditions had receipts or explicit waivers. Temporary credentials, held resources, and pending compensation remained visible after forward progress stopped.

This borrowed W93H's multi-clock recovery model. Returning the main path to healthy did not erase cleanup and learning.

The console's job was not to make every row disappear. It was to make every remaining state defensible.

The console was difficult to review when every interesting state required waiting for a real synthetic failure. I connected it to Z29C's deterministic adapter harness and created a scenario mode containing no external authority.

Fixtures opened directly into lost response, rate limit, stale policy, duplicate reference, compensation unknown, and manual-evidence states. The page used the same projection and action derivation as ordinary runs, while every external result came from the controllable stub.

This let me test whether a person could predict each action before using it. After reading the evidence, they described what Check, Retry, Stop, or Withdraw would attempt and which uncertainty would remain. If the explanation differed from the contract, I changed hierarchy or copy before assuming the participant needed more training.

Scenario mode also supported destructive-action drills. A duplicate-risk retry could be followed through to two external references and a compensation decision without creating real consequences. Keyboard and screen-reader flows used the same captured states repeatedly.

The mode was visibly synthetic and unavailable under ordinary workflow credentials. It could not mutate live records or attach fixture receipts to them. Its purpose was product verification, not an admin backdoor.

Recovery interfaces are often seen only during stressful, rare conditions. Designing the state as a repeatable scenario made the safest action familiar before it mattered and kept dangerous affordances from surviving merely because they were hard to exercise.

Retry returned to its proper place

After the redesign, Retry still appeared often. Pure computations failed transiently. Idempotent adapters tolerated the same key. Authoritative absence made another creation safe. Rate windows eventually reopened.

The difference was that the button followed the evidence. It named the logical step, preserved identity, respected current authority and deadlines, and produced a durable attempt.

On the dangerous rows, another action led—or no automatic action existed.

The first console had optimized for speed after failure. Z29C needed to optimize for safe recovery from incomplete knowledge.

A recovery interface should begin with the question the system can answer: what happened, what do we still not know, and which action can improve that state without creating a worse one?

Retry is one possible answer. It should never be the question.

Safety stays visible.