The shell script did not know where it failed

R7K1's first preview environment was one shell script:

build image
start database
run migrations
start application
write proxy route
send success message

With set -e, the script stopped when a command returned nonzero. I treated that as failure handling.

One run lost its terminal connection after the proxy update. The script never sent success. Retrying created a second database, found the application container already running, and rewrote the route. Nobody knew whether the first run had failed before or after activation.

The script knew which command it was currently executing. R7K1 needed to know which durable state transitions had completed.

An imperative sequence hid the state machine

The script looked like six steps and contained many states:

requested
building
artifact ready
provisioning data
migrating
starting application
checking readiness
activating route
ready
removing
removed
failed at any boundary

Those states existed whether or not they had names. After interruption, containers, databases, images, and routes remained in some combination.

I moved lifecycle state into a durable environment record. The controller read the current state, selected one eligible transition, recorded an attempt identity, performed the external effect, verified it, and committed the next state.

The shell commands remained useful adapters. They stopped owning workflow memory.

A script can be an excellent implementation of one transition. It becomes a fragile workflow engine when its process lifetime is the only record of progress.

Exit status described one process

set -e helped stop on some command failures. It did not prove that a successful command produced the intended external state or that a timed-out command produced none.

Starting a container could return success before the application was ready. A proxy reload could return after accepting configuration while another process still served the old route. A database command could commit and lose the client response.

Each transition needed an observation:

build -> artifact exists and manifest matches
database create -> named database can be queried with expected owner
migration -> schema version and migration receipts match
application start -> instance exists and reports candidate identity
route activate -> proxy resolves route to expected environment generation

Exit status remained diagnostic evidence. The transition completed only when R7K1 could query the effect it cared about.

This changed automation from “command returned zero” to “desired state is now observable.”

Every effect received stable identity

The retry problem came from commands that generated new names each time. create database on retry created another resource because the workflow lacked a stable requested identity.

The environment request declared IDs before external work:

environment ID: env-271
database ID: db-env-271
application instance ID: app-env-271
route generation: route-feature-search-8

Adapters created or found resources under those IDs. If the resource already existed, they verified ownership and configuration before returning it. A same-name resource with a different environment receipt caused a conflict, not reuse.

Stable identity made create transitions idempotent enough to retry. The operation might execute several times; the world contained one intended resource.

Generated provider IDs were recorded after creation, but R7K1's requested identity remained the lookup key it controlled.

The environment record became the receipt connecting internal intent to external resources.

Transition intent was recorded before the effect

A controller crash between external success and state update could leave the record behind reality. Recording only after the command did not eliminate outcome unknown.

R7K1 wrote a transition attempt first:

attempt ID
environment ID
from state
intended transition
expected external identity
started time
lease owner and expiry

After a crash, another controller found the incomplete attempt and reconciled. It queried the external system for the declared identity. If the effect existed and matched, it completed the receipt. If it did not, it retried. If evidence conflicted, it entered review rather than guessing.

This resembled X8B6's local operation and server receipt, but the side effect spanned infrastructure APIs instead of one database transaction.

The intent record did not make the external call atomic. It made uncertain outcomes recoverable by naming what to inspect.

Two workers could observe the same environment as artifact-ready and both begin provisioning. R7K1 used a short lease on the environment transition.

The lease had owner, generation, and expiry. A worker renewed it during long work. Another worker could take over only after expiry and still had to reconcile the previous attempt before starting new effects.

The lease was not the ultimate correctness guarantee. A paused worker could resume after losing its lease. External operations still used stable identities and transition commits checked the current lease generation.

This fencing prevented a stale worker from marking a newer attempt complete. Provider adapters rejected or ignored operations carrying obsolete generation where supported.

Leases reduced concurrent noise. Idempotent effects and state checks protected the workflow when leases overlapped.

The design did not pretend a timestamp lock could stop a process already in motion.

Transitions were small and independently retryable

The original script reran from the beginning after failure. R7K1 split it:

requested -> artifact-ready
artifact-ready -> data-ready
data-ready -> app-starting
app-starting -> app-ready
app-ready -> route-active
route-active -> ready

Each transition named prerequisites, external effect, verification, receipt, timeout, and compensation or cleanup behavior.

A failed readiness check did not rebuild the artifact or recreate the database. A route activation failure did not rerun migrations. Reconciliation resumed from the durable last completed state.

I resisted making transitions microscopic. “Create database” and “grant user” belonged together only if the adapter could verify and recover them as one data-ready contract. The right boundary followed a meaningful capability and recovery action.

Small meant independently accountable, not one API call per state.

Failure became data with scope

The shell script printed stderr and exited. R7K1 stored structured failure:

transition: data-ready -> app-starting
attempt: att-884
category: configuration
message: required fixture version is unavailable
retryable: no without new input
external resources: database exists, app not created

The environment could remain in data-ready with a failed attempt attached. Correcting configuration created a new attempt from the same stable state.

Transient provider unavailability was retryable under backoff. Invalid manifest or incompatible migration required changed input or review. Permission failure asked for policy repair rather than endless retries.

The UI said which capability had not been reached and what remained safe. It did not mark the entire branch preview simply failed when a previous route still served reviewers.

Failure gained enough structure to support the next action.

Database migrations were especially dangerous to rerun blindly. A command could apply schema changes and lose its response.

The migration system recorded applied migration identities inside the target database. The R7K1 transition queried that ledger before and after execution. Migrations were written to tolerate the declared path or refuse incompatible state explicitly.

Environment state did not advance to data-ready merely because the migration process exited. The database had to report expected schema and fixture identity.

Rollback of application did not automatically reverse schema. Preview fixtures were disposable, so R7K1 could often remove and recreate the entire environment from declared inputs. That was safer than inventing down migrations for every experiment.

The lifecycle record named whether data could be recreated or contained review-entered state worth preserving. “Preview database” was not assumed disposable universally.

Migration receipts made one of the most consequential shell commands inspectable.

The controller sent a proxy update and timed out. It could not assume the old route remained.

Recovery queried the route generation and target environment. If it matched the intended activation, R7K1 recorded success. If it still pointed at the predecessor, retry was safe under the same generation. If it pointed somewhere else, the attempt became stale and did not overwrite newer work.

The activation request included expected current generation, acting as a compare-and-set where the routing layer allowed it. This prevented an older environment from replacing a newer one after a delayed retry.

Public smoke checks ran after route observation. A route could point correctly while the application failed under the real hostname.

The shell script's missing success message became irrelevant. The routing system itself answered what had happened.

The old script sent “preview ready” as its last command. If notification failed, retrying the whole script risked changing infrastructure. If infrastructure succeeded and the process died first, nobody heard about it.

R7K1 emitted notification work from committed lifecycle events. environment-ready created one notification record keyed by environment generation. A separate sender delivered and retried it.

Failure to send a message did not change environment readiness. Repeated delivery carried the same URL and identity. The branch page remained the authoritative status.

This separated side effects of communication from the workflow they described. It also prevented a Slack-like message from becoming the only receipt that a preview existed.

The notification system could be down while provisioning stayed correct.

Cleanup was a forward lifecycle

The original cleanup script issued remove commands best-effort and deleted the environment record. If one provider call failed, leaked resources remained with no record connecting them.

Removal used durable states:

expiry-requested
route-detached
application-stopped
data-removed
resources-verified-absent
removed

The route detached first so reviewers did not enter a disappearing environment. Each resource removal used stable identity and treated already absent as success after ownership verification.

The environment record remained as a compact tombstone after external resources disappeared. It kept identifiers, reason, receipts, and removal time for leak audits.

Cleanup failure retried from its last verified state. R7K1 did not resurrect an expired route merely because database deletion needed attention.

Provisioning and removal became two directions through one lifecycle model, not a create script plus an unrelated cron job.

Replacing the workflow did not require banning shell. Small scripts still built images, invoked provider CLIs, and ran smoke checks.

Their contracts changed:

• accept explicit resource and attempt identity,
• avoid generating hidden names,
• print machine-readable result plus human diagnostics,
• return nonzero for one scoped failure,
• support observation without mutation,
• remain safe to retry or declare why not,
• never update global lifecycle state directly.

The controller treated scripts like drivers. It recorded inputs and outputs, enforced timeouts, and verified effects through independent reads where possible.

Shell was good at composing local commands. The database-backed controller was good at remembering a workflow across process death. Each tool kept the responsibility it could support.

The postmortem was not “shell is unreliable.” It was “process memory is insufficient workflow state.”

R7K1 did not adopt a large workflow platform. A table of environments, transition attempts, leases, and receipts plus a worker loop was enough.

Allowed transitions lived in code and tests. Every handler received current record and declared input and returned a proposed next state only after verification. A transition could not skip prerequisites because a command happened to succeed.

The UI derived progress from states rather than counting shell commands. app-starting could report health attempts and expiry without inventing a percentage.

This modest implementation was easier to inspect than the shell script because its states were durable and named. Complexity had not been added from nowhere; it had moved from leftover containers and operator memory into a model.

I kept the model specific to preview environments. It did not become a generic workflow engine for every task.

Reconciliation ran even without a failure report

Controllers can crash without marking a transition failed. Provider APIs can succeed after a client timeout. R7K1 periodically scanned incomplete attempts and environments whose state had not advanced within expected bounds.

The reconciler asked:

does the declared resource exist?
does ownership match this environment?
does observed configuration match the desired receipt?
is the active lease valid?
can the transition complete, retry, or require review?

It did not restart every “stuck” environment. Restarting could duplicate or obscure the original effect. Observation came first.

Reconciliation also found drift: a manually removed container, a route pointing to an unexpected target, or a database surviving after environment tombstone.

The workflow became a loop that compared declared and observed state, not a one-time script that assumed the world remained as it left it.

The first tests stubbed every command to return zero and asserted the success message. The revised suite interrupted every boundary:

• crash after intent record but before external call,
• external success followed by timeout,
• two workers with overlapping leases,
• stale worker completing after takeover,
• route updated but response lost,
• migration committed but client disconnected,
• notification unavailable after readiness,
• removal partially complete,
• provider resource exists with wrong ownership.

Fixtures recorded expected lifecycle state, external resources, attempts, and receipts after each recovery pass.

The strongest test ran the controller repeatedly until no eligible transition remained, injecting each failure once. The final state and resource set had to match one uninterrupted run.

Idempotency became an observable convergence property, not a comment above a command.

With the shell script, an interrupted run required inspecting containers, databases, proxy files, and logs, then deciding where to restart manually. The operator held the missing state machine in their head.

R7K1's environment page showed durable state, active attempt, observed resources, last failure, and available actions. A resume action triggered reconciliation; it did not rerun the entire recipe.

Manual intervention recorded a decision and identity. Marking a resource adopted, abandoning a candidate, or forcing route rollback left an audit trail. The system did not pretend manual repair had been automatic.

This was the operational outcome I valued most. Failures still happened, but recovery became routine enough that nobody needed to be the one person who remembered the shell script's quirks.

A reconciler that always forced observed resources back to the database record could undo intentional emergency work or recreate an environment somebody was trying to remove.

R7K1 attached lifecycle generation and policy to desired state. Manual changes outside the controller appeared as drift and required one of three explicit outcomes: repair toward the current generation, adopt the observed state through a new recorded transition, or continue removal. The system did not silently choose.

Some drift was safe to repair automatically. A missing route for a still-ready, unexpired environment could be restored only if the branch route had not advanced to another generation. A missing application container required reprovisioning and full readiness, not simply marking the old instance running.

The environment page showed desired and observed values side by side. Reconciliation attempts recorded which difference triggered them.

This restraint kept the durable model from becoming a dangerous fiction enforced at all costs. Desired state was an authorized proposal with generation and lifecycle context. Observed state remained evidence that could contradict it.

Reliable automation needs both persistence and humility: remember what should exist, then verify whether changing the world toward it is still the right action now.

The script had been a prototype of a protocol

The original six commands were not wrong. They discovered the necessary capabilities in a cheap form.

They became insufficient when preview environments needed concurrency, retries, expiry, route activation, and evidence. At that point the sequence was a protocol between R7K1 and several external systems.

Giving the protocol stable identity, durable attempts, observable receipts, leases, and forward cleanup made process death ordinary. Shell adapters remained replaceable implementation details.

The key change was how R7K1 interpreted uncertainty. A missing final message no longer meant failed. A zero exit no longer meant ready. A retry no longer meant start over. The controller asked the world which named effects existed and advanced only from evidence.

The shell script knew where its instruction pointer stopped. The lifecycle model knew what the product had actually become.

That difference turned a preview from a pile of commands into a recoverable promise about one named environment and every resource it owned.