The nine-minute outage that lasted six hours

The availability graph showed nine minutes of failed requests. I did not trust the system again for six hours.

The incident happened in the personal lab that connected several earlier projects. M31V could publish a new release. R7K1 could build a preview. Q2F8 could schedule exports and deletion work. W93H, still unfinished, collected a few deployment and alert events from all three.

I changed how a shared PostgreSQL connection budget was divided between interactive processes and background workers. The values looked conservative when read independently. Under mixed load, export workers occupied enough connections to make interactive requests wait. Latency rose. Requests timed out. Retries added more waiting requests. Health checks began failing and recycled a web process that was slow rather than dead.

I reverted the configuration nine minutes after the first capability alert. Requests recovered almost immediately.

Then the difficult questions arrived. Which worker processes had loaded the reverted values? Which export jobs had committed their result before losing an acknowledgement? Had any retries produced a duplicate output? Was the queue merely draining, or was one job stranded? Which temporary actions had I taken while moving quickly? Could I reproduce the explanation, or had the graph recovered for another reason?

The system was serving traffic. Recovery was not finished.

The change was valid in two files and unsafe as a whole

The interactive service and the export worker each had a pool-size setting. The database had a maximum connection budget, with capacity that also needed to remain available for administration, migrations, and short-lived maintenance work.

I reviewed the web value in the web configuration and the worker value in the worker configuration. Each fit comfortably below the database limit. I did not evaluate the combined claim across process counts:

interactive processes × interactive pool
+ worker processes × worker pool
+ reserved operational capacity
<= safe database budget

The missing multiplication mattered. A pool of twelve looked modest. Four worker processes could claim forty-eight connections before the interactive service, a migration, or my own diagnostic session asked for one.

The database did not immediately refuse the configuration. Pools opened connections as work arrived. The unsafe state remained latent until I replayed a set of exports beside interactive traffic. That delay made the change look healthy during activation.

The obvious correction was to lower the worker number. The maintained correction was to stop expressing the capacity decision in independent files. One typed model declared the database budget, reserved margin, process classes, replica counts, and per-process claim. Validation rejected combinations whose maximum claim exceeded the safe total.

That prevented the exact invalid state. It did not answer what had happened to work already in motion.

The rollback changed desired state first

The configuration command completed successfully. I initially treated its success as proof that the system had returned to the earlier state.

It had only changed the desired configuration and restarted the web process. The workers used leases: a process finishing an export was allowed to retain its current code and configuration until the lease ended. One worker reported the old allocation for several minutes after the rollback. Another had begun retrying a job whose database outcome was not represented clearly.

W93H showed the rollback marker near the fall in request latency. That was useful evidence, but it was not enough to write “rollback restored the system.” Several things happened in the same interval:

• New web processes loaded the lower connection claim.
• Client retry pressure fell as requests began succeeding.
• Active export transactions completed or timed out.
• Worker processes recycled after their leases.
• The queue stopped assigning new work under the unsafe allocation.
• The database released connections and cleared waiting sessions.

The rollback was an important intervention inside that sequence. The sequence, not the command receipt, explained recovery.

I added runtime configuration identity to process events. W93H could then distinguish what the controller intended from what each relevant process reported. “Reverted” became an observed convergence condition instead of a button I had clicked.

The first clock measured availability

The nine-minute number came from a user-facing capability rule. It measured the interval during which interactive requests failed or exceeded their useful latency window.

That was the right primary alert. Pool utilization helped diagnose the cause, but the person waiting for a page experienced latency and failure, not a percentage of connections. When the capability recovered, the outage clock stopped.

The number was neither false nor complete. It answered one question precisely: for how long did this defined interactive path fail its selected indicator?

It did not include degraded requests before the alert condition accumulated enough evidence. It did not describe background exports. It did not show whether the current build and configuration were uniform. It did not measure how long the explanation remained uncertain.

I had previously used “resolved” as the terminal incident state. A recovered alert transitioned the incident to resolved automatically. That model made the most visible clock stand in for every other one.

I changed the incident summary to display current impact separately from recovery work. The interactive capability could be healthy while the incident remained open for configuration verification, job reconciliation, and removal of temporary changes.

Green was an important transition. It was not a universal ending.

The second clock measured configuration convergence

The rollback record named the earlier configuration version. The deployment controller said the action succeeded. The process reports told a messier story.

The web process converged quickly. One export worker retained the newer pool allocation while completing a lease. A second restarted but briefly failed to report its configuration digest because the telemetry client initialized after job polling. The desired-state view looked correct before the observed-state view was complete.

I could have forced every worker to terminate immediately. That would have shortened configuration convergence and increased uncertainty about job side effects. A worker killed after committing an export but before writing its completion receipt could cause the job to be retried without knowing whether the output already existed.

I chose controlled worker drainage. The queue stopped assigning new exports. Active leases received a bounded interval to finish. Processes then restarted and reported the intended configuration before the queue reopened.

That choice kept a known unsafe allocation alive slightly longer in a contained path because abrupt termination posed a different correctness risk. Recovery work is full of such trades. “Make everything the old version now” sounded clean only when the in-flight work was omitted.

The configuration clock ended when all relevant processes reported the intended digest and the combined capacity model passed against the database budget. It ended after the availability clock.

The export queue contained work in four observable conditions:

1. Jobs that had not been leased.
2. Jobs whose attempt had started but had not reached the database transaction.
3. Jobs whose transaction had definitely committed and whose completion receipt existed.
4. Jobs whose worker disappeared after an attempt began but before a durable outcome was recorded.

The fourth condition was the problem. The queue represented it as failed and eligible for retry. The data layer could not prove that every failed attempt had left no output.

For idempotent calculations, a retry could replace the same derived object safely. For exports that produced a named file and a notification event, repeating the operation might create two outputs or two notifications. The retry policy treated transport failure as business-operation failure.

I paused automatic retries for the ambiguous set. I compared job attempt identifiers with database rows, storage objects, and available log evidence. The evidence was not elegant because the system had not been designed to answer the question. Some work could be classified from a transaction identifier. Some required checking a deterministic output key. One attempt remained uncertain until I replayed its input in a non-notifying path and compared the result.

This reconciliation consumed most of the six hours and left a harder question behind: what durable receipt would allow an operation to be repeated without asking a person to infer its side effects from fragments?

The immediate changes were smaller. Every export gained a stable operation identifier. Output creation used a deterministic key or an explicit uniqueness constraint. Completion wrote a durable receipt before acknowledgement. Retry policy distinguished known-not-started, known-completed, safely-repeatable, and unknown-outcome attempts.

The job-recovery clock ended only when every affected attempt had a defensible disposition. A healthy queue-depth graph could not end it.

The fourth clock measured temporary actions

During the incident, I made several changes that were useful precisely because they departed from normal operation.

I paused the export queue. I lowered client retry attempts. I increased diagnostic logging for pool acquisition. I disabled an automated cleanup that might have removed intermediate evidence. I ran a read-only database query from an administrative connection.

My first notes recorded the actions in shell history and a scratch document. They did not reliably record their intended duration or reversal condition.

Temporary response changes are easy to forget after visible impact ends. A paused queue can become tomorrow's backlog. Extra logging can consume storage or retain data outside its ordinary policy. A reduced retry count can silently remain as a new product behavior. A disabled cleanup can turn incident preservation into indefinite retention.

W93H gained a temporary-action record with four fields that mattered: what changed, why it was safe enough under the incident conditions, what evidence would allow reversal, and when it would expire or demand review.

An action did not disappear when reversed. It received a receipt linking the prior and restored state. If I decided to keep the incident value, it had to travel through the normal configuration path as an intentional maintained change instead of becoming permanent through neglect.

The temporary-action clock ended when the queue reopened under the validated capacity model, retry behavior returned through a reviewed configuration, extra logging was disabled, and cleanup resumed after the evidence set was preserved appropriately.

The fifth clock measured explanation

The graph sequence suggested a simple story:

Configuration deployed, latency rose, configuration rolled back, latency recovered.

Every clause was observable. The implied causal completeness was wrong.

The useful explanation needed the mechanism. The combined web and worker pool claims could exceed the safe database budget under mixed load. Export workers occupied connections. Interactive requests waited. Their timeouts triggered retries, which increased the number of waiting requests. Health checks judged slow processes unhealthy and added restart churn. Rolling back reduced new claims, while worker drainage and falling retry pressure allowed the database to recover.

It also needed contributing conditions. The capacity values were reviewed separately. Staging load did not combine realistic interactive and export concurrency. Runtime configuration identity was incomplete. The alert detected user-facing latency correctly but only after pool wait had already grown. The queue could not distinguish an unknown side-effect outcome from a safe retry.

“Bad configuration” was true at too low a resolution. If that phrase became the whole cause, the corrective action would be “choose better numbers.” The system would still make an unsafe combination easy to express and difficult to inspect.

I replayed the incident with recorded request and job shapes. The original configuration reproduced pool wait and latency. The validated lower allocation did not under the same bounded workload. Forcing a worker to disappear at the ambiguous point reproduced the retry uncertainty independently of the pool failure.

Reproduction did not turn a personal lab into a statistically rich production environment. It made the claimed mechanism more credible and converted it into maintained tests.

The explanation clock ended when the incident record could connect impact, change, mechanism, actions, recovery, and remaining limits without relying on the timeline's visual suggestion alone.

The evidence lived on incompatible clocks

Constructing that account exposed a more literal time problem.

Deployment events used controller time. Application logs used process time. PostgreSQL recorded database time. Queue transitions carried worker timestamps. Metrics arrived with sample timestamps and evaluation delay. My notes used the laptop clock. One worker had drifted far enough to place a symptom before the action that likely preceded it.

The first W93H importer normalized every event to a corrected timestamp and sorted the result. The output looked authoritative. It had erased the fact that correction had occurred.

I changed the event envelope to preserve:

sourceTime
ingestedAt
sourceSequence
knownOffset
offsetObservedAt

Display time could use a known correction, but original time remained evidence. Ingestion time showed when W93H learned about the event. A source sequence could establish order within one producer even when its wall clock drifted.

Where two events from different sources fell inside unresolved clock uncertainty, W93H grouped them into an interval instead of forcing exact order. The reconstructed timeline became less visually neat and more defensible.

Late evidence carried both positions: where its source time placed it in the incident and when it became available to the investigation. The final record could improve without pretending I had used evidence I did not possess at the time.

The clock problem was not an observability detail. It limited which causal sentences the incident could support.

The web process remained capable of serving work if it acquired a database connection. The health endpoint exercised a path that needed the same constrained pool. When the check timed out, the process was restarted.

Restarting did not create database capacity. It closed and reopened connections, dropped in-flight requests, and temporarily increased coordination work. The health mechanism interpreted dependency-induced slowness as local process failure.

I separated liveness from readiness. Liveness answered whether the process event loop and essential internal machinery could continue. Readiness answered whether the process should receive new traffic given its current dependencies and startup state. Neither check became a full synthetic transaction.

A process that temporarily could not serve database-backed work could become unready without being killed immediately. The traffic layer stopped assigning new requests while existing work received a bounded chance to finish. A genuinely wedged process still failed liveness and restarted.

This did not solve pool starvation. It stopped recovery machinery from adding churn to the same constrained resource. The capability alert remained the measure of user impact; health checks controlled process participation.

I added the distinction to the incident scenario. Under pool exhaustion, readiness could fall, but liveness should remain stable. If the process became internally unresponsive, the liveness path still had to detect it without needing the database it was trying to diagnose.

Interactive clients retried selected timeouts automatically. The policy had been reasonable under intermittent transport failure: wait briefly, add jitter, and try again within a bounded count.

Under connection starvation, every retry joined the same bottleneck. The original request might still be waiting or might have reached an unknown outcome. A retry increased pressure while reducing the remaining time in which a useful response could arrive.

I modeled a retry budget across the request path. A client attempt, gateway retry, and downstream database retry could not each behave as if it were the only retrying layer. The overall operation had one time budget and a bounded amplification allowance.

Interactive reads that were safe to repeat used exponential backoff with jitter and a deadline derived from the useful response window. Operations with side effects required stable identities and server semantics before automatic retry. Some failures became immediately visible rather than being hidden by retries that had little chance of success.

The change reduced load amplification, but it also exposed more first-attempt failures to the interface. That was honest. A retry policy is not successful because it hides an error counter; it is successful when it improves the chance of a useful, correct outcome within the operation's constraints.

The incident made retries part of capacity planning. Maximum concurrent work included the attempts the system could create when it began failing, not only the requests it received while healthy.

Communication existed even with one operator

There was no fictional incident team to update. There was still a communication problem.

During the response, I moved between code, configuration, dashboards, database queries, and queue inspection. Each context switch spent working memory. After a break, I no longer remembered which hypothesis a command had tested or whether an observed value came before or after the rollback.

I treated notes as a handoff to my future self. The incident summary named current impact, verified observations, active hypotheses, consequential actions, outstanding recovery conditions, and the next safe investigation step.

Free-form narrative remained useful, but it linked to structured evidence. “Workers still mixed” pointed to runtime configuration reports. “Exports reconciled” pointed to a set of job receipts and explicit dispositions. “Retry change reversed” pointed to the configuration event that restored the maintained policy.

This was not bureaucracy imposed on a hobby system. It was a way to make interrupted reasoning resumable, even when the next collaborator was simply a later version of me.

W93H's revised incident model separated conditions that had previously collapsed into resolved:

Current impact

Were intended interactive and background capabilities behaving within their useful windows? This could return to healthy while other work remained.

Runtime convergence

Were relevant processes running the intended artifact and validated configuration? Did desired and observed state agree?

Data and job integrity

Did every affected operation have a known outcome? Were duplicates, partial outputs, and stranded work identified and repaired?

Temporary-change removal

Had emergency limits, pauses, logging, and cleanup exceptions been reversed or deliberately adopted?

Evidence and learning

Did the record support its causal account? Had the learned constraint entered validation, code, a runbook, or a maintained scenario?

Each condition required a receipt rather than a hopeful checkbox. The incident summary could show “impact recovered” prominently without presenting the whole event as closed.

I resisted turning the condition list into a universal ceremony. Different incidents needed different recovery obligations. W93H supplied a small vocabulary and let the scenario define what evidence completion required.

“Be more careful” failed the action test

The easiest postmortem conclusion was that I should have reviewed the pool numbers more carefully. It was also nearly useless.

The configuration had distributed one capacity decision across files and processes. A reviewer had to know the database limit, reserve enough headroom, multiply per-process pools by replica counts, include retry amplification, and anticipate mixed load. The unsafe state was syntactically valid and operationally obscure.

The actions changed those conditions:

1. One capacity model derived or validated the connection claims across process classes.
2. Mixed interactive and export load became a repeatable scenario.
3. Pool wait and oldest eligible job age appeared before request timeouts in the evidence set.
4. Processes reported artifact and configuration identity at runtime.
5. Job attempts gained stable operation identifiers and durable outcome receipts.
6. Retry policy was bounded across layers rather than configured independently.
7. Recovery conditions stayed open after availability returned.

Not all of these changes prevented the first symptom. Some made recurrence easier to detect, contain, explain, or repair. Prevention was one part of reliability; recoverability was another.

Six hours was not a second outage metric

I did not replace the nine-minute figure with six hours on the availability chart. That would have been another category error.

Nine minutes measured failed interactive service under a defined indicator. Six hours described the time until I had verified configuration convergence, classified affected jobs, reversed temporary changes, reproduced the mechanism, and recorded maintained corrections. The two durations belonged to different questions.

Keeping them separate prevented exaggeration and erasure. The service was not unavailable for six hours. The incident was not responsibly complete after nine.

The most important change to W93H was not a new graph. It was permission for several clocks to coexist without one claiming to summarize the others. Impact, configuration, data integrity, temporary operations, and understanding could recover at different rates.

That model made the incident feel less tidy. It also made the next action visible. A green graph no longer invited me to close the browser and trust that everything behind it had recovered too.

The outage ended quickly. The system earned confidence back one receipt at a time.