The health check that proved only a port was open

An unusable preview passed every automated gate because the check answered an operational question nobody reviewing the change had asked.

The preview was green, routed, and useless.

R7K1 had built a branch, started its container, waited for the declared port, and moved the branch URL to the new generation. The status page showed every step as successful. A reviewer opened the link and received a maintenance screen. Refreshing did not help. The application process was alive, but its database migration had not completed and its main editorial route could not read the seeded document.

Nothing in the automation had malfunctioned. The check had asked whether a TCP connection could be established. The answer was yes.

The reviewer had asked a different question: can I evaluate this change?

That gap turned a tiny health endpoint into an architectural problem. R7K1 needed to distinguish a process that existed, an application that had initialized, and a preview that was ready for its intended task. Until those meanings were separated, a green status was only a confidently formatted ambiguity.

The incident lasted minutes and invalidated the model

The candidate changed an editor toolbar and included a small database migration for saved view preferences. Its build completed normally. The container entry point started the application and launched the migration as a child process. The web server bound its port immediately so startup appeared responsive.

R7K1's controller polled that port. After three successful connections, it wrote a reverse-proxy route for the new environment and changed the branch pointer from generation 31 to generation 32.

The migration then encountered fixture data created under an older schema. It exited non-zero. The web process stayed alive and returned a maintenance response for routes that required the new table. The port continued accepting connections, so R7K1 continued reporting the candidate as healthy.

The failure had several consequences beyond a bad page:

  • The previously reviewable generation was no longer at the friendly branch URL.
  • The reviewer could not tell whether the maintenance screen belonged to the proposed change or R7K1 itself.
  • Retrying the health check produced more green evidence for the wrong claim.
  • Restarting the container repeated the same sequence because startup order had not changed.
  • Build logs looked successful because the failure occurred after artifact creation.

I restored generation 31 manually, then inspected the activation record. It contained container ID, host, port, and three connection timestamps. It contained no request through the planned hostname, no application identity, no schema version, and no evidence that the primary route could perform a useful operation.

The incident was small. The missing concepts were not.

“Healthy” had accumulated four meanings

In conversation, I used healthy to describe at least four states:

  1. The operating system process had not exited.
  2. Something accepted connections on the expected port.
  3. The application had initialized its required dependencies.
  4. A reviewer could use the candidate for its stated purpose.

Each meaning could be true while the next was false. A process could run before binding. A server could bind before connecting to its database. An application could answer a status route while its main page lacked fixtures or compatible assets. A preview could serve its main page while authentication or navigation made the proposed behavior unreachable.

Collapsing the states made recovery unsafe. If a process was dead, restarting it might help. If initialization was incomplete, a restart could interrupt useful work. If a route was not reviewable because fixtures were wrong, restarting changed nothing. If only one optional integration was degraded, replacing the whole candidate might be worse than explaining the limitation.

I stopped using a single unqualified health result inside the controller. The state model became:

started -> live -> initialized -> review-ready -> activated
             \          \              \
              dead       blocked        unsuitable

These were not decorative labels. Each transition had different evidence, timeout, ownership, and recovery.

Liveness belonged to the process supervisor

Liveness answered a narrow question: should the runtime be considered present, or has it stopped making progress in a way that justifies replacement?

The first part was simple. The container runtime reported whether the main process existed. A lightweight application endpoint then confirmed that the event loop could answer a request. That endpoint did not query the database, call integrations, or inspect every dependency. If liveness depended on a remote service, a temporary network failure could trigger restarts that increased load without repairing anything.

I made the endpoint deliberately unambitious:

{
  "process": "live",
  "artifact": "sha256:…",
  "startedAt": "2016-11-15T14:03:12Z"
}

Artifact identity mattered because a responsive process could still be the wrong process. R7K1 compared the reported digest with the generation receipt before trusting any later check.

The supervisor used consecutive failures and a bounded restart policy. One missed poll did not erase the environment. Repeated liveness loss could restart the same immutable artifact. Restart attempts were recorded under the environment rather than manufacturing new generations.

After the bound was reached, the runtime became failed and required diagnosis. An endless restart loop is not resilience; it is a machine repeatedly discarding evidence.

Liveness remained useful precisely because it did not pretend to represent the whole application.

Initialization needed dependency evidence

The application required a compatible schema, loaded configuration, asset manifest, and synthetic fixture version. Those conditions belonged to initialization.

I added a readiness endpoint that evaluated local required dependencies and returned structured reasons rather than a boolean:

{
  "state": "blocked",
  "artifact": "sha256:…",
  "schema": {
    "required": 18,
    "observed": 17
  },
  "fixtures": {
    "required": "editorial-6",
    "observed": "editorial-5"
  },
  "assets": "loaded"
}

The endpoint did not perform migrations. A probe should observe readiness, not mutate the world while a controller repeats it. Migration and fixture setup became explicit provisioning steps with their own logs and completion receipts. The endpoint only checked the state those steps were meant to establish.

This distinction solved an unpleasant retry problem. My earlier experimental endpoint attempted to create missing tables when requested. Two concurrent probes could race, and a timeout left the controller unsure whether the mutation had completed. Moving work out of the probe made repeated observation safe.

Initialization checks used a timeout long enough for declared startup work but not an unbounded wait. While blocked for a known in-progress migration, the generation page could show that phase. A schema mismatch, missing asset manifest, or failed fixture import ended activation with a specific diagnosis.

The process could remain live for inspection even when initialization failed. “Not eligible for reviewers” did not need to mean “immediately destroy the evidence.”

Calling the container directly skipped several parts of the preview that a reviewer depended on: hostname routing, forwarded headers, TLS termination, access policy, base-path configuration, and asset URLs.

R7K1 therefore checked the candidate through its planned generation hostname before moving the friendly branch route. The request traversed the same reverse proxy and policy layer as review traffic.

This caught failures the internal endpoint could not:

  • A generated route pointed to the previous container port.
  • The application constructed redirects from an internal hostname.
  • Secure cookies were missing because forwarded protocol headers were wrong.
  • Assets used an absolute path belonging to the shared environment.
  • Access middleware redirected the probe to sign-in without preserving its destination.

The generation hostname was not yet advertised as the branch's current candidate, but it existed in a restricted pending state. Only the controller and the owner could reach it during diagnosis.

The request also carried a probe identity with the smallest necessary permission. It did not borrow a developer's long-lived cookie or bypass the application's authorization layer entirely. If the preview could be used only by granting the check universal access, the check would prove a path reviewers did not have.

Route readiness made activation slightly slower. It also made the evidence correspond to the path I was about to publish.

A smoke task expressed the product gate

An HTTP 200 from the home route still did not prove that the proposed toolbar could be reviewed. The application shell could load while the seeded document failed, the editor bundle returned 404, or permissions hid the feature.

I defined a small smoke task for each previewed project. For the editorial application it did four things:

  1. Requested the main route through the planned hostname.
  2. Confirmed the response identified the expected artifact and fixture generation.
  3. Followed the link to a known synthetic document.
  4. Checked for a stable marker representing the editor's ready state.

The marker was not a sentence of interface copy likely to change. It was a versioned application contract rendered only after required data and assets were available. The smoke task did not click every toolbar control or reproduce the whole test suite. Its purpose was to prove that the review surface existed, not that the feature was correct.

The distinction mattered. A massive end-to-end suite would make activation slow and flaky. A trivial status request would repeat the original mistake. The smoke task chose one representative user path with high diagnostic value.

Its configuration lived with the previewed project and had a version. R7K1 understood a generic task result but did not hard-code what “ready to review” meant for every application.

preview contract v4
  entry: /
  destination: /documents/sample-editorial
  expected artifact: ${generation.artifact}
  ready marker: editor:v3

Changing the application's review path required changing the contract deliberately. An outdated smoke definition failed as a configuration mismatch rather than quietly approving a different task.

Artifact identity closed a subtle hole

During the first implementation, the smoke request succeeded against a cached response from the previous generation. R7K1 was about to activate a candidate based on evidence that generation 31 worked.

I had treated response success as sufficient and ignored identity.

Every built artifact already had a digest. I exposed that digest in a response header and in the readiness payload. The proxy disabled shared response caching for pending generation hosts. The controller sent a unique request identifier and required the observed artifact to equal the generation being evaluated.

Fixture and schema identities received the same treatment. A route could render successfully against the wrong synthetic database, especially when hosts were reused. The smoke receipt recorded:

generation: 32
artifact: sha256:4f…
environment: env_8d…
schema: 18
fixture: editorial-6/reset-1
hostname: g32.review.local
contract: editorial-preview-v4

The receipt made the activation decision reproducible. I could see not only that a request succeeded but what exact candidate and environment had answered it.

This was another lesson in the weakness of green indicators. Success without identity can be valid evidence attached to the wrong subject.

Activation became a compare-and-swap

Readiness checks take time. While generation 32 was being evaluated, a new push created generation 33. If completion order alone controlled the branch URL, the older candidate could become current after the newer one.

Activation therefore needed both readiness and eligibility. The controller recorded the generation as ready, then compared it with the branch's current desired generation. It moved the friendly route only if the candidate was still eligible.

Conceptually:

if generation.readyReceipt is valid
and branch.desiredGeneration == generation.id
then branch.activeGeneration = generation.id

The update was atomic in the controller's state store. The proxy configuration was derived from that committed pointer and acknowledged separately. If writing the route failed, the pointer remained on the previous ready candidate until reconciliation could complete the handoff.

A superseded generation could remain available at its immutable diagnostic URL for a short period. Its successful check was not wasted, but it did not override newer intent.

This also protected reviewers from the original incident. Generation 31 stayed active while 32 built, initialized, and failed its smoke task. The friendly URL represented the latest verified eligible candidate, not whichever container most recently opened a port.

That difference had to be visible. The branch page showed that head generation 32 had failed while 31 remained active, so nobody mistook stability for freshness.

A failing dependency did not always mean not ready

Once health had structure, the temptation was to add every integration to the required list. That would have made the preview unavailable whenever an optional sandbox service was slow.

I classified dependencies by the review task.

The synthetic database, artifact manifest, and route policy were required for the editorial smoke path. Email capture was optional for a toolbar change. An image-processing sandbox might be required for an upload proposal but irrelevant to document navigation.

Readiness contracts could name required capabilities and tolerated degradations. The generation page exposed both:

ready for: editorial review
degraded: outbound email capture unavailable
unavailable task: invitation flow

This avoided two dishonest states. I did not call the whole preview failed when the intended review remained possible, and I did not display an undifferentiated green mark while a known task was broken.

Capability-specific readiness is more work than one boolean. It is also closer to how applications behave. A service can be useful and degraded at the same time.

The boundary remained conservative. Security policy, identity mismatch, or incompatible schema could never be downgraded to a warning. Optionality was declared in the versioned contract, not improvised by a controller trying to make the dashboard green.

Checks required budgets and evidence retention

A readiness task can itself become the failure. A hung request can occupy a worker forever. Aggressive polling can overload an application during startup. Repeated browser checks can hide a race by eventually landing on a lucky response.

R7K1 gave each phase a budget:

  • Process start had a short observation window.
  • Initialization used slower polling with server-provided reasons.
  • Route readiness used bounded retries for clearly transient connection failures.
  • The smoke task ran once per artifact-environment-contract identity unless an operator explicitly requested a new attempt.

Retries retained each result. A candidate that failed twice and passed once did not appear equivalent to one that passed immediately. The activation policy could allow a transient proxy connection failure, but application assertion failures required a new environment or explicit investigation.

Logs included request identity and phase, with credentials redacted. A failure receipt linked controller timing, proxy response, application log slice, artifact identity, and contract version. These records expired later than the failed environment so cleanup did not erase the reason review never began.

I also rate-limited manual rechecks. A button that starts unlimited work is not observability; it is an incident amplifier with a friendly label.

Ongoing health was not the original readiness decision

After activation, the preview could later lose a dependency or crash. I initially considered rerunning the entire smoke task continuously. That confused two different questions.

Activation readiness asks whether a specific environment was fit to become the candidate at one moment. Ongoing observation asks whether it remains available enough to serve reviewers.

I retained the immutable activation receipt and monitored lighter signals afterward. Liveness loss could trigger bounded restart. Required dependency loss marked the preview unavailable and returned the branch page to a diagnostic state while keeping generation identity. Optional dependency loss surfaced degradation.

If a restart recreated mutable environment state, the environment received a new restart sequence in its receipt. It did not pretend the original smoke evidence covered an entirely new fixture database. For changes that could alter review meaning, R7K1 reran the contract before restoring the route.

The current status never rewrote history. A generation could say:

activated 14:08 with receipt r_42
became unavailable 16:31 after database loss
restart 1 ready 16:34 with receipt r_47

This timeline was more useful than a green dot that changed color and forgot why.

The interface stopped promising one kind of green

The branch page had previously rendered a single status beside the URL. After the incident, it showed a short progression with language a reviewer could interpret:

Build complete
Environment starting
Application blocked: fixture schema is outdated
Previous verified candidate remains available

Technical details stayed one level deeper. The reviewer did not need container terminology to understand why the new version was unavailable. The owner could open the receipt and see schema versions, logs, and request evidence.

I avoided percentage progress because the phases were not evenly timed and some had no meaningful fractional completion. Named state was more honest than a bar stuck at 83 percent.

When ready, the page said Ready for editorial review, not Healthy. That wording tied the claim to the contract. A preview with a degraded optional capability named the limitation beside it.

The interface change was not cosmetic. It forced the system to produce enough evidence to support the sentence it displayed.

The revised sequence kept failure away from reviewers

The complete activation path became:

build immutable artifact
  -> provision owned environment
  -> apply schema and synthetic fixtures
  -> confirm process liveness and artifact identity
  -> confirm required initialization state
  -> request through pending hostname
  -> execute versioned smoke task
  -> verify generation is still eligible
  -> atomically move friendly route
  -> retain activation receipt

Any failure before the route move left the current verified generation intact. The failed candidate remained inspectable under controlled access, and its resource graph still followed normal expiry.

The sequence was longer than “wait for port.” It was also easier to reason about because each step answered one question. Provisioning did not declare application success. Application initialization did not declare product readiness. Product readiness did not override branch ordering. Route activation did not erase the evidence used to justify it.

The controller could resume after a restart by examining receipts rather than repeating every side effect. It could distinguish an environment that needed observation from one that needed creation.

Reliability came from refusing to let a convenient signal stand in for several decisions.

What the port check was still good for

I did not delete the original connection check. I renamed it.

An accepting port was useful evidence that the server had bound and the proxy might connect. It helped diagnose startup and kept later application requests from beginning too early. It simply did not deserve authority to publish a candidate.

That became the lasting lesson. Weak checks are not necessarily bad checks. They become dangerous when their result is promoted beyond the question they answer.

The same caution applies to build success, container state, status codes, test totals, and dashboard color. Each can be precise evidence. None carries meaning without a subject, boundary, and decision attached.

The incident did not require a more sophisticated probe first. It required a better definition of the handoff. R7K1 was not trying to prove that a port existed. It was trying to decide whether an unaccepted change could replace the current review surface without wasting another person's time.

Once that sentence was explicit, the engineering followed. The process needed to live. The application needed compatible dependencies. The request needed to traverse the real route. A representative task needed to succeed against the exact artifact. The generation needed to remain the one I intended to publish.

Only then did the URL move.

The preview still sometimes failed. Its green state finally failed less often at telling the truth.