W93H

W93H was a 2019 personal project for reconstructing incidents across the growing collection of services behind my earlier experiments. R7K1 had build workers and routes. Q2F8 had collection, retention, and erasure jobs. M31V had publishing callbacks and immutable releases. I had enough moving parts to produce realistic failures and no coherent way to explain one afterward.

The outage was short; the explanation was not

The triggering incident happened in the lab environment. A configuration change gave export workers too much of a shared PostgreSQL connection pool. Interactive requests began waiting, health checks failed, and the web service restarted. Rolling back the configuration restored availability in about nine minutes.

Understanding the incident took most of a day. Deploy markers were in one interface, alerts in another, worker progress in logs, database signals on a graph, and my actions in shell history and rough notes. Several clocks disagreed. A nearby deployment looked causal before the pool evidence arrived. The service was green long before I had checked delayed exports and possible duplicate work.

W93H grew from that mismatch. It was not another metrics warehouse. It assembled the smallest durable narrative needed to distinguish observation, action, hypothesis, and recovery.

I built W93H around synthetic and self-hosted services, then replayed the connection-pool failure and several other scenarios. There was no fictional operations department or professional on-call role. I was designing for the person who had created the system and still could not reliably reconstruct what it had done.

The project’s questions were:

• What was observed, by which source, and when?
• What changed in deployed code or configuration?
• What action did a person or automation take?
• Which proposed explanation remained only a hypothesis?
• What evidence supported or contradicted it?
• Which recovery conditions remained open after traffic returned?

W93H preserved links back to specialist tools. It did not copy every log line or metric sample into a second, worse search system.

Compared with Q2F8, the difficulty was no longer enforcing a known policy on one event. It was joining incomplete evidence from several systems without manufacturing certainty.

A typed operational event

Small Go ingestors accepted a deliberately narrow set of events:

• Deploy and rollback markers.
• Configuration-version changes.
• Alert state transitions.
• Service starts, stops, and readiness changes.
• Queue depth summaries and job state transitions.
• Selected resource-threshold crossings.
• Human actions, annotations, and handoffs.

Every event received a common envelope:

source
sourceEventId
kind
sourceTime
ingestedAt
correlationKeys
payloadVersion
payload

The source event identifier made repeated delivery idempotent. sourceTime remained untouched. ingestedAt showed when W93H learned about the event. Known clock offsets could influence display without rewriting the evidence.

Payload schemas stayed specific to their event kinds. A deploy marker named commit, artifact, environment, actor class, and configuration digest. An alert transition named rule, prior state, new state, observed value, and evaluation window. A human action named what was attempted and linked its available receipt.

This resembled Q2F8’s catalogue but served a different purpose. The schema established what a source claimed. It did not decide whether the event caused anything else.

W93H stored normalized events, incident membership, annotations, hypotheses, and recovery conditions in PostgreSQL. Raw high-volume logs and metrics remained in their original tools.

An ingestor retained a bounded excerpt only when an event needed immediate context, along with a source link and query window. This kept the incident narrative small enough to read and preserved the richer source for investigation.

The first ingestion prototype copied broad log streams. Storage grew quickly, search became noisy, and W93H started competing with the tools it was meant to connect. I removed generic log ingestion and required a typed operational reason for each event kind.

Late events were normal. A job service could reconnect and deliver a transition after the incident had been opened. W93H inserted it according to its source time while also marking when it became visible to the investigator. The reconstructed timeline could therefore differ from the sequence a responder had seen live.

That difference was important. An event discovered later should improve the explanation without pretending it had informed an earlier decision.

The connection-pool incident involved service time, database time, monitoring evaluation time, deployment-controller time, and my laptop clock. Sorting their timestamps produced a clean sequence and an unjustified one.

W93H preserved original source time, ingestion time, and any known display correction. Sources with monotonic sequence numbers retained them. When order across two sources could not be established within the clock uncertainty, the interface grouped them into the same interval rather than forcing one above the other as fact.

A clock-health panel showed recent offset estimates and abrupt changes. If a source clock shifted during an incident, events remained searchable by both original and adjusted windows.

The timeline displayed a small marker on corrected or uncertain timestamps. Technical detail stayed available without turning every row into a timestamp lecture. The important behavior was that the visual order did not silently claim more precision than the sources supplied.

T04P had introduced this distinction for freshness. W93H extended it to historical explanation: chronological neatness is not the same as evidentiary accuracy.

Observation, action, and hypothesis

The interface used three distinct forms.

Observed events were statements received from sources: an alert entered firing, a release became active, a worker lease expired. Human actions represented attempts: configuration rolled back, queue paused, service restarted. Hypotheses connected events through an explanation with an author, status, and evidence links.

A deployment near an alert did not automatically become the root cause. The first version drew vertical deployment lines across metric charts, which made proximity look causal. W93H instead let me propose:

The worker allocation introduced in release 7f3 exhausted the shared connection pool and delayed interactive requests.

The hypothesis linked the configuration diff, pool saturation interval, request-wait signal, worker activity, and recovery after rollback. It could be proposed, supported, contradicted, confirmed for the incident record, or superseded.

Confirmation did not mean mathematical proof. It meant the incident record had enough evidence for the stated operational conclusion, with alternatives and remaining uncertainty preserved.

This structure prevented annotations from becoming an unsearchable chat transcript. Every note either added an observation, recorded an action, changed a hypothesis, or updated a recovery condition.

Configuration as an incident artifact

The incident could not be explained from source commit alone. The same code ran with different web and worker allocations. W93H added a digest of operationally relevant configuration to deployment events and retained a redacted, structured diff between versions.

The underlying services moved connection allocations into one capacity model. A validation rule checked that reserved worker and interactive limits did not exceed the database budget with a safety margin.

W93H did not become the configuration authority. It recorded which validated configuration each process reported at runtime. That distinction caught a partial rollout where the desired configuration and observed process state differed.

Secret values never entered the event. The schema stored presence, version, or redacted classification where useful. A configuration diff that leaked credentials would have turned incident evidence into a new incident.

The deployment view linked commit, artifact, desired configuration, observed runtime versions, and activation result. It became possible to say not only what should have run but what the system reported actually running.

Recovery had more than one clock

The first incident state machine ended at resolved, triggered when availability returned. The connection-pool incident showed at least four recovery clocks:

• Requests served normally again.
• Intended code and configuration verified across processes.
• Delayed jobs reconciled and possible duplicate outputs checked.
• Temporary mitigations removed and the explanation recorded.

W93H replaced the single terminal state with explicit recovery conditions. A service graph could be green while the incident remained open for data repair or follow-up.

Each condition had an owner label, evidence requirement, state, and optional due time. Completion linked the observation or receipt that satisfied it. “Queue reconciled” therefore pointed to counts and job receipts rather than a checked box alone.

The incident summary showed current impact separately from recovery completeness. This reduced the pressure to keep an outage banner active merely because the record was unfinished, and prevented availability from erasing work that remained.

I later added a “temporary change” condition. Rate limits, feature disables, and extra capacity applied during an incident created a removal obligation. W93H kept them visible until reversed or deliberately adopted through the normal configuration path.

Handoff to my future self

Even in a one-person system, handoff mattered. An incident could stretch past the time I could keep every detail in working memory. Returning the next morning was a handoff between two states of the same person.

The handoff view summarized:

• Current impact and affected capabilities.
• Established observations.
• Active and rejected hypotheses.
• Last consequential actions and their receipts.
• Outstanding recovery conditions.
• Known gaps in evidence.
• The next safe action and what would make it safe.

The summary was generated from structured incident state and allowed a short authored narrative. It did not attempt to synthesize certainty automatically.

External conversations and scratch notes were represented by concise linked summaries rather than copied walls of text. If a decision had been made elsewhere, W93H required a durable account of it to enter the incident record.

This work anticipated D4U7. Distributed collaboration is not only many people in different places; it is any important work whose context must survive delayed attention.

An incident scenario engine

I converted the original failure into a replayable scenario. It scheduled deploy events, configuration observations, pool saturation, request latency, job progress, an alert delay, rollback, and late-arriving worker evidence.

Scenarios did not replay real side effects. They delivered recorded event fixtures through the same ingestion contracts as live lab services. A controllable clock allowed tests to advance through thresholds and late arrival.

The scenario engine asserted narrative invariants:

• A nearby deployment remained an observation until linked by a hypothesis.
• Late evidence showed both source position and discovery time.
• Availability recovery did not complete job-reconciliation conditions.
• Duplicate source delivery produced one event.
• A rejected hypothesis remained in history.
• A clock correction did not destroy original time.

I added alternative scenarios: a false-positive alert, a slow dependency with no local deploy, a partial configuration rollout, and a queue retry that risked duplicate output.

The scenarios made incident-interface review repeatable. I could inspect how the timeline behaved under uncertainty without waiting for a real failure or turning every test into an ad hoc story.

Correlation without automatic incident truth

W93H used correlation keys to suggest related events: environment, service, release, request class, job family, and bounded time window. Suggestions entered a review queue and did not become incident members silently.

The first automatic grouping pulled every alert in the environment into one incident. Background noise expanded the timeline and made a broad outage look more certain than it was. Narrower keys and manual confirmation kept the narrative legible.

An event could belong to more than one incident record when the same underlying condition had separate consequences. Membership carried the reason it had been included.

The project did not calculate a root cause score. It surfaced candidate relationships and preserved enough source evidence for a person to decide. This was a deliberate boundary against “smart” observability that converts correlation into confidence.

The same boundary later shaped K81R: retrieval can find plausible supporting material, while answerability and evidence still need separate evaluation.

Operational use of W93H itself

An incident tool that fails during an incident is an obvious trap. W93H ran with a small dependency footprint and could accept events through a bounded local buffer when its database was briefly unavailable.

Ingestors persisted their cursor and retried with the same source event ID. The buffer had a visible capacity and age. If it filled, the system reported evidence loss explicitly rather than claiming completeness.

The interface was server-rendered with light live updates. A page refresh recovered the current incident projection. It did not require a large client state store to reconstruct truth from a stream.

Backups and restoration focused on typed events, incident relations, annotations, hypotheses, and recovery conditions. Source links could expire, so important bounded excerpts and configuration diffs were retained according to a documented policy.

A canary incident passed through open, observation, hypothesis, action, recovery, and archive states on a schedule. It verified that W93H could record its own deployment and link its current build identity.

I avoided adding W93H as a dependency to every service’s critical path. Event delivery was asynchronous and failure-tolerant. The incident narrative should observe the systems, not become the gate through which their primary work must pass.

The lab initially produced alerts for individual resource thresholds: CPU high, pool utilization high, queue depth high, and restart count high. They were easy to configure and often described a busy but functioning system.

I reorganized the primary alerts around a few user-facing capabilities: interactive requests completed within their useful window, publications became visible after approval, and background exports finished within their expected period. Resource signals remained diagnostic evidence and did not all page the imaginary operator.

Each alert definition named the capability, observation window, threshold rationale, known blind spots, and runbook entry. W93H ingested state transitions rather than every evaluation sample. The incident view could show when an alert began pending, fired, and recovered, along with the rule version active at the time.

Rule changes were deployment events. Tightening a threshold during an incident could otherwise make the timeline look as if impact had changed when only measurement policy had. The versioned definition preserved that distinction.

I used error-budget thinking as a planning tool without inventing contractual service-level claims. A small historical report showed how much of a chosen reliability target recent test runs would have consumed and which failure modes contributed. Its purpose was to prioritize work, not advertise a percentage unsupported by the personal environment.

The connection-pool incident demonstrated why impact and resource signals both mattered. Request latency identified the failing capability. Pool saturation and worker activity helped explain it. Reversing those roles would have made an internal threshold the product and the person waiting for a page a detail.

The first post-incident notes ended with broad actions such as “improve monitoring” and “add capacity.” They sounded responsible and had no completion test.

W93H linked follow-up work to the failed assumption it addressed. The connection-pool incident produced four concrete changes:

1. One validated capacity model allocated database connections across interactive and worker processes.
2. Deploy events retained desired and observed configuration digests.
3. Export jobs used bounded concurrency and exposed their active connection claim.
4. The incident scenario became a recurring regression test.

Each change had evidence and a review date. A completed item linked its release or test. A decision not to proceed kept the reason rather than disappearing from an unchecked list.

The incident record also separated contributing conditions from root cause. Shared capacity, a missing validation rule, weak deployment evidence, and alert delay all mattered. Compressing them into “bad config” would have produced one patch and preserved the system that allowed the failure.

I scheduled a short retrospective against the scenario after the fixes. The replay needed to reject the invalid capacity model before deployment; if forced past validation, the user-facing alert and evidence still needed to behave coherently. Prevention and diagnosis were tested separately.

This closed a loop that earlier projects had only partially captured. An incident was not complete when a narrative existed. The learned constraint had to enter code, policy, or a maintained test. Otherwise the explanation remained archival rather than operational.

The stories the first interface invented

The initial deploy lines visually implied causation. Separating observed events from hypotheses corrected the product claim.

Copying broad logs created an expensive second search surface. Typed events and bounded source excerpts restored W93H’s narrow purpose.

Normalizing timestamps originally replaced source time. Keeping original, ingestion, and display-corrected time made uncertainty auditable.

The single resolved state equated restored traffic with complete recovery. Explicit conditions represented the remaining work.

Automatic event grouping pulled irrelevant alerts into the narrative. Correlation became a suggestion with a recorded inclusion reason.

The first annotations were free-form comments. Structuring observation, action, hypothesis, and recovery updates made the record reconstructable without eliminating authored explanation.

W93H was my first personal project centered on evidence across several independent systems. It joined typed ingestion, clock uncertainty, configuration identity, human actions, hypotheses, recovery state, and scenario replay without claiming to automate causation.

The difficulty was not collecting more telemetry. It was preserving the epistemic status of each piece. An alert was an observation. A rollback was an action. A root-cause statement was a hypothesis supported by linked evidence. A green graph was one recovery condition, not the end of the incident.

The project gathered earlier lessons into a new shape. T04P contributed freshness and source distinction. X8B6 contributed stable event identity and retries. M31V and R7K1 contributed artifact and configuration identity. Q2F8 contributed typed catalogues and finite retention.

D4U7 would take W93H’s handoff lesson into a product designed for delayed human attention. N17Q would later reuse its semantic event timeline and hypothesis discipline for agent traces. In both cases, the hard problem was making a complex run explainable without manufacturing a story the evidence could not support.

Evidence the incident tool could not create

W93H used personal and simulated systems. It could not reproduce the organizational stress, scale, or incomplete access of a real production incident. It could not prove causation automatically, and a human-authored note could still be wrong.

Missing instrumentation remained missing evidence. Source links could decay. Clock correction remained an estimate. W93H’s contribution was a defensible narrative structure with visible uncertainty, not a universal truth machine.

Technology: Go ingestors, PostgreSQL, typed operational events, configuration digests, bounded source excerpts, server-rendered incident views with live updates, controllable scenario clocks, and durable recovery conditions.