Green traffic did not close the incident

W93H had a button labeled Resolve incident. It changed the status, recorded the time, and turned the header green.

The button worked exactly as designed and encouraged the wrong decision.

After the connection-pool incident, interactive requests recovered while export jobs remained paused, worker configuration was still converging, and several attempts had uncertain outcomes. Pressing Resolve would have made the main record look complete while the riskiest cleanup work was beginning.

Leaving the incident visibly “active” was not right either. The public-facing capability was healthy. Treating ongoing reconciliation as continued outage exaggerated impact and made the current state harder to understand.

I had forced several kinds of recovery into one status. The interface could be green or red because the data model allowed only one question.

One incident had several completion criteria

I wrote down every statement I wanted to be true before archiving the incident:

• Interactive requests were succeeding within their useful latency window.
• Background workers were running the intended artifact and configuration.
• Every affected export attempt had a known disposition.
• The paused queue had reopened safely.
• Temporary diagnostic logging had been removed.
• The connection budget could no longer express the same invalid combination.
• The incident scenario could reproduce the failure and verify the correction.
• The causal account named its evidence and remaining uncertainty.

Only the first statement was represented by the availability graph. The others concerned configuration, data integrity, temporary operations, prevention, recoverability, and knowledge.

Calling the whole list “resolved” hid the fact that they could complete independently. Calling only the first item “resolved” overloaded the word so severely that two readers could understand it in opposite ways.

I replaced the terminal status with a set of typed recovery conditions. Current impact remained prominent, but it no longer controlled every other dimension.

Impact was a live state, not the incident's identity

The incident itself represented a bounded investigation and recovery record. Impact represented what intended capabilities were doing now.

W93H separated them:

type ImpactState =
  | 'unknown'
  | 'active'
  | 'degraded'
  | 'recovered'
 
type IncidentState =
  | 'open'
  | 'monitoring'
  | 'recovering'
  | 'reviewing'
  | 'archived'

The exact labels changed during testing. The important part was that recovered on impact did not force archived on the incident.

When requests returned to normal, W93H moved the incident from active response into recovery. The header said Impact recovered; 4 recovery conditions remain. It showed the time the capability recovered and kept the incident visible in the work queue.

If impact returned, the incident could move back to active response without creating a disconnected record. If the symptom was unrelated, a new incident could be opened and linked. The transition required a judgment about continuity rather than a timer that reopened everything automatically.

This language reduced two pressures at once: the urge to keep an outage active to preserve cleanup visibility, and the urge to close cleanup because the outage was over.

A recovery condition was a claim with evidence

My first condition implementation was a checklist. Checklists are useful, but a free checkbox reproduced the problem at a smaller scale. “Queue reconciled” could be checked without saying what reconciliation meant.

Each condition gained a type, an authored criterion, a current state, and an evidence requirement:

type RecoveryCondition = {
  kind: RecoveryKind
  criterion: string
  state: 'open' | 'blocked' | 'satisfied' | 'waived'
  evidence: EvidenceLink[]
  ownerLabel?: string
  dueAt?: string
  supersedes?: string
}

The ownerLabel did not invent an operations team. In my one-person lab it usually named a role of attention—“job reconciliation” or “configuration verification”—so the summary remained meaningful after a context switch. In a larger setting the same field could point to an actual accountable person.

Satisfying a condition required a receipt appropriate to its claim. Configuration convergence linked process runtime reports. Queue reconciliation linked job-attempt dispositions. Removing temporary logging linked the configuration event that restored the ordinary level. A maintained prevention action linked a validation test.

The evidence did not need to be produced by W93H. It could be a durable source link with a bounded excerpt, an event already in the timeline, a test result, or an authored explanation of a manual verification. The model asked the condition to show its work.

Recovery types prevented a generic checklist

I considered shipping one standard template for every incident. It quickly filled with items irrelevant to some failure modes and missed obligations unique to others.

W93H instead provided a small vocabulary of recovery kinds:

Capability recovery

The affected action works again under its defined indicator. This is the part most traffic graphs represent.

Runtime convergence

Relevant processes, configuration, dependencies, and routes match the intended post-response state. A successful control-plane action is supporting evidence, not sufficient proof.

Data and side-effect integrity

Affected operations, records, messages, or artifacts have known outcomes. Duplicate, partial, missing, or stranded work has been identified and repaired or bounded.

Temporary-action removal

Emergency switches, increased capacity, disabled jobs, elevated logging, manual routes, and temporary permissions have been reversed or intentionally adopted.

Communication and handoff

The durable record states current impact, known scope, remaining work, and safe next actions. For the personal project, this was principally a handoff to my later self.

Learned constraint

The explanation has changed something maintained: validation, code, architecture, a scenario, a runbook, or a deliberate decision to accept the risk.

An incident selected only the kinds it needed and could define multiple conditions within one kind. The taxonomy made gaps visible without pretending every event followed an identical ceremony.

Data recovery was not the same as traffic recovery

Request success is usually measured at an online boundary. Data integrity can lag behind it and fail silently.

During the pool incident, the web path recovered when connections became available. Export jobs had a different truth. Some had completed, some had never started, and some had crossed a boundary where their result might exist without a completion receipt.

W93H could not infer job safety from a green request graph. The recovery condition named the population and terminal evidence:

Every export attempt holding a lease between 14:02 and 14:18 has one of: a verified completion receipt, proof that its transaction did not begin, a deterministic output adopted as the result, or an explicit repair record.

The condition was longer than “exports checked” because it defined the set and the acceptable outcomes.

When the set was evaluated, each item received a disposition. Unknown was an allowed intermediate state and not a terminal success. A job could be deliberately abandoned if the output was unnecessary, but that decision also received a record so absence did not look accidental.

This model generalized beyond queues. A failed publication could leave an artifact built but not routed. A deletion job could remove primary data while a derivative remained. A schema change could make new writes healthy while old rows awaited repair. Traffic and stored consequence operated on separate clocks.

Temporary fixes created future obligations

Incident response rewards actions that reduce immediate harm. The same actions can become hazards when their temporary context is forgotten.

I added more database capacity once during a synthetic incident. The graph improved. If the capacity stayed, it would hide the same unbounded worker claim until load grew again. The extra headroom was useful containment and weak prevention.

Other temporary actions carried different risks:

• Disabling a queue protects a dependency and accumulates work.
• Increasing a timeout reduces visible failures and makes waiting longer.
• Raising a limit admits more work and can move pressure downstream.
• Turning on detailed logs supplies evidence and increases storage and data exposure.
• Bypassing a validation rule unblocks a repair and weakens an invariant.
• Granting broader access speeds diagnosis and expands authority.

W93H required every temporary action to name a reversal condition and review time. The related recovery condition stayed visible until the action was reversed or entered the ordinary change process.

An expiry could notify me, but automatic reversal was not always safe. Re-enabling a queue automatically after a fixed hour might recreate overload. Some changes could expire safely; others required a deliberate check. The action type recorded which behavior applied.

This turned “we will remember to put that back” into system state.

After initial recovery, I often said I was “monitoring.” The word described attention without saying what would be learned.

W93H made a monitoring interval name its hypothesis, indicators, and decision boundary. For the pool correction:

Under mixed interactive and export load, connection wait remains below the early-warning threshold, request latency stays inside its useful window, and oldest eligible export age continues decreasing for 30 minutes. Failure reopens active response; success satisfies the stability condition.

The duration came from the workload cycle needed to exercise the relevant behavior, not a default period copied into every incident. A cache problem might need expiration cycles. A scheduled job might need its next run. A memory leak could require enough sustained work to show a trend.

Monitoring without a question could continue indefinitely or end when impatience won. A defined observation made the result usable evidence.

The interface displayed which conditions the interval could satisfy. Green traffic during monitoring therefore contributed to a specific claim rather than slowly creating a feeling that things were probably fine.

Waiving a condition was a decision, not disappearance

Not every recovery condition could or should be completed. A source link might expire before evidence was captured. A low-value ambiguous job might not justify further reconstruction. A preventive change might cost more complexity than the bounded risk warranted.

Deleting the condition would rewrite the incident. Leaving it open forever would make the work queue meaningless.

W93H allowed a condition to be waived with a reason, scope, residual risk, and review decision. A waiver could say:

One synthetic export attempt remains unclassifiable because the prototype did not retain a transaction identifier. Its deterministic output key is absent and no external notification path was enabled. No repair is planned; the missing receipt is addressed in the next queue schema.

The incident could then archive without pretending the evidence had become complete.

Waivers were visible in the summary and searchable across incidents. Repeating the same waiver was a signal that an accepted limitation might actually be a neglected system property.

This was more honest than a completion percentage. Nine satisfied conditions and one waived condition were not “90 percent recovered.” Their meanings and risks differed.

Follow-up work needed a closure test

Post-incident action lists easily accumulate verbs: improve monitoring, review retries, document capacity, add tests. The verbs sound responsible and provide no proof of changed behavior.

I linked each follow-up to the failed assumption it addressed and the evidence that would close it.

For the unsafe connection allocation:

Failed assumption:
  Pool values reviewed per service cannot exceed database capacity.
 
Maintained change:
  Validate the combined maximum claim across process classes and reserve margin.
 
Closure evidence:
  The original invalid fixture fails validation; the corrected fixture passes;
  the mixed-load scenario remains within the declared budget.

For uncertain retry outcomes, the closure evidence was not “retry code refactored.” It was a scenario in which a worker disappeared after the side effect and the next attempt used a durable receipt or deterministic identity instead of repeating blindly.

Some actions improved diagnosis rather than prevention. That was acceptable if named honestly. Adding runtime configuration identity did not stop a bad value; it made desired and observed state distinguishable during recovery.

An incident could move to archived with longer-term follow-ups still planned, but the decision named which risk had been accepted in the meantime. Archive was a record state, not a declaration of perfect remediation.

Archiving was a deliberate transition

The revised archive transition asked a small set of questions:

• Is current impact accurately represented?
• Are all selected recovery conditions satisfied or explicitly waived?
• Are temporary actions gone or tracked through normal state?
• Can a later reader understand the evidence behind the accepted explanation?
• Do remaining follow-ups name their risk and completion test?

W93H generated a summary from the structured record and allowed a short authored conclusion. It preserved rejected and superseded hypotheses so the investigation did not look inevitable in retrospect.

Archiving locked some fields and required revision events for later changes. New evidence could reopen the review or create an addendum without mutating what had been known at closure.

I did not add an approval workflow to the personal project. One person pretending to be several approvers would have added ceremony without independent judgment. The state model was still useful because it made my own criteria explicit and replayable.

The archived incident remained linked to its scenario, configuration validation, job-receipt changes, and related Blog investigation. It became maintained system memory rather than a finished document stored away from the code it had changed.

The interface had to show progress without celebrating too early

The old green header offered satisfying closure. The replacement needed to stay readable without turning every incident into a wall of status chips.

The top of the page answered three questions in order:

1. What is the impact now? A concise capability state and last transition.
2. What remains for recovery? Open conditions grouped by type.
3. What should happen next? The next safe action or monitoring decision.

Completed conditions collapsed into a history. Waived conditions stayed visible in the archive summary. Temporary actions received a distinct warning because their forgotten state could create new impact.

I removed a radial completion chart. It implied that conditions had equal weight and that progress was a continuous percentage. One unresolved data-integrity condition could matter more than five completed documentation conditions.

The quieter text summary was less decorative and more actionable: Impact recovered. Runtime verified. 3 export attempts await disposition. Diagnostic logging expires in 42 minutes.

Designing incident state was a form of product design. The interface shaped which work felt real after the dramatic graph had recovered.

Recovery became a property I could test

Once recovery conditions were structured, the scenario engine could assert them.

The connection-pool replay restored interactive traffic before worker convergence and job reconciliation. The test verified that W93H did not archive automatically. It applied a temporary queue pause and checked that the incident retained a removal obligation. It delivered a late completion receipt and confirmed that the corresponding job condition could be satisfied without rewriting discovery history.

Other scenarios exercised different shapes: a false alert with no data repair, a partial deployment with healthy traffic, an expired certificate requiring replacement and verification, and a queue failure with one deliberately waived synthetic output.

The tests did not prove a human explanation was correct. They proved that the product could represent incomplete recovery without collapsing it into outage or success.

That was an important distinction. W93H was not automating judgment. It was preserving the conditions under which judgment could be made and revisited.

Green remained meaningful

Separating recovery dimensions did not diminish service restoration. Restoring an intended capability quickly is the first responsibility during active impact. The availability transition deserved a clear timestamp and prominent display.

The correction was to stop making it carry unrelated claims.

Green traffic said requests were succeeding under a defined indicator. It did not say every process ran the intended configuration, every side effect was known, every temporary action was reversed, or every lesson had entered maintained behavior.

W93H became more useful when it allowed those sentences to remain separate. The incident could leave urgent response, enter recovery, and eventually archive through evidence rather than color.

The separation also made historical reporting more accurate. Availability summaries used capability intervals. Recovery summaries used condition lifetimes. Follow-up reports used the time until a learned constraint entered maintained behavior. None borrowed the longest or shortest duration merely to produce a dramatic number. Each measure kept the definition that made it useful.

The old Resolve button had offered one satisfying moment. The new model offered something less theatrical and more durable: a system that could say exactly what had recovered, what had not, and what would make the difference.