A deployment marker is not a cause / Bob's Site

The first useful feature I built in W93H was also the first one I had to make less persuasive.

W93H placed deployment markers beside alerts and operational events. Before that view existed, investigating a failure meant opening the release history in one tab, a metrics dashboard in another, and logs in a third. The marker answered a high-value question quickly: what changed near the beginning of the symptom?

On the timeline, a vertical release line crossed every graph. An alert began two minutes later. The visual relationship was so strong that I read it as an explanation before inspecting the diff.

The release was not the cause. A scheduled cleanup job had started under the previous version and exhausted its own concurrency allowance. Rolling the release back did nothing. The job finished, the resource graph recovered, and the release line received credit for both the failure and the recovery because it happened to stand nearby.

I had built an interface for correlation and styled it like causation.

“What changed?” is a search strategy

Recent change is a reasonable place to begin an investigation. Software behavior is not spontaneous. A code release, configuration edit, dependency response, data-shape change, traffic shift, expired certificate, or operator action often explains why a previously stable path behaves differently.

The heuristic becomes dangerous when it turns into a conclusion. “What changed?” narrows the search. It does not establish which change produced the observed outcome.

Several ordinary conditions weaken the inference:

Systems change even when application code does not. Data volume, clock time, dependency behavior, caches, certificates, and workload mix continue moving.
A release may activate gradually. The marker's timestamp may describe when deployment started, when an artifact was accepted, or when the last process reported ready.
A failure may have a long delay. A new leak, scheduled path, or accumulating queue can appear hours after the change that enabled it.
Multiple changes may overlap. Code, configuration, schema, and infrastructure can move under different mechanisms.
Rollback may coincide with natural recovery. A queue drains, load falls, or a dependency returns while the rollback is still in progress.

The release marker was valuable because it indexed a candidate. It became misleading when the display implied the candidate had already won.

A deployment was not one timestamp

My original event schema stored a commit identifier and deployedAt. That field compressed a process into an instant.

R7K1 and M31V had already taught me that an artifact could be built, routed, and retired at different times. W93H forced the same precision into operational evidence. A release event needed at least these stages:

A deployment was requested for an intended artifact and configuration.
The artifact was fetched or made available to the target.
A process started with a reported build identity.
Readiness checks allowed it to receive traffic or work.
The desired set of processes reported the intended identity.
The deployment controller declared completion, failure, or rollback.

For a one-process service, those stages could occur within seconds. They still described different claims. An alert beginning after the request but before activation had a different relationship to the release than one beginning only after the new process served traffic.

W93H stored stage transitions instead of drawing one authoritative line. The summary marker remained, but expanding it revealed the interval and any mixed state. If some worker still reported the earlier artifact, the timeline did not claim a clean cutover.

That detail mattered during background processing. A web process could activate immediately while a worker completed its current lease before restarting. “The deployment is done” from the controller did not guarantee every process performing consequential work had observed the new configuration.

Commit identity was necessary and insufficient

The next mistake was assuming a commit described the running system.

The same source could produce different behavior under a different dependency lockfile, build recipe, base image, environment variable, feature switch, database schema, or external service response. Conversely, two commits could produce operationally equivalent artifacts if only documentation changed.

W93H connected several identities:

sourceCommit
artifactDigest
buildManifestDigest
configurationDigest
schemaVersion
runtimeReport

The build manifest came from R7K1's earlier work. It named the inputs that made artifact reuse safe: source, dependency lockfile, build recipe, and base image. The artifact digest identified the immutable output. The configuration digest identified a redacted, typed set of runtime decisions. A process reported what it actually loaded rather than relying only on what the deployment controller intended.

No single identifier explained the system. Together they let me ask a specific question: which executable bits and operational rules did this process claim to be using at this moment?

W93H did not ingest secret values. Configuration entries were classified, redacted, or represented by version and presence where their raw contents were unsafe. An incident tool that copied credentials into a convenient diff would create a larger failure than the one it was investigating.

Desired state and observed state could disagree

The deployment controller knew what it had asked the system to run. The processes knew what they had loaded. I initially trusted the first source because it was easier to collect.

A synthetic partial rollout broke that assumption. The controller advanced the desired configuration, replaced the web process, and recorded success. A worker remained alive because it was completing a long job. Its runtime report still named the previous configuration digest.

The release appeared uniform in deployment history and mixed in reality.

W93H displayed desired and observed state separately. A deployment could be complete according to its orchestrating mechanism while runtime convergence remained open. That did not mean the controller was wrong; it meant its completion condition had a narrower definition.

The distinction was particularly important for rollback. Reversing desired state was an action. Recovery required evidence that relevant processes had observed the reversal and that the affected capability had improved. A rollback command returning success established neither on its own.

This made the timeline a little more complicated and the incident narrative much safer. It replaced “version B caused the failure; we rolled back to A” with testable statements:

Version B with configuration C became active on the web process at 14:03.
Interactive latency rose above its capability threshold at 14:05.
The worker continued reporting configuration B0 until 14:13.
Desired state returned to version A at 14:09.
All relevant processes reported A and configuration A0 by 14:14.
Latency recovered during the interval, while queue age recovered later.

The longer account left room for the actual mechanism.

Configuration deserved first-class change evidence

Code review had trained me to look for source changes. Some of the most consequential operational changes lived elsewhere.

Connection limits, worker concurrency, timeouts, retry counts, routing weights, and feature switches could all alter behavior without a new source commit. If W93H drew only code releases, those changes appeared as spontaneous incidents.

I created typed configuration events. Each event named the policy field, prior version, next version, validation result, intended scope, and activation observations. A structured redacted diff could be linked without storing unsafe raw values.

The types mattered. Comparing two opaque environment-file hashes could prove that something differed, but not whether the difference affected database capacity or only a harmless display option. A typed capacity model could explain that interactive processes claimed 24 connections, workers claimed 18, and the reserved total exceeded the safe database budget.

The configuration system remained the authority. W93H was evidence and history, not a second place to edit values. It accepted signed or authenticated events from the change path and runtime reports from services.

Separating observation from control reduced a dangerous temptation: fixing an incident inside the interface that was also trying to preserve its history. Operator actions could be linked from W93H, but their actual execution stayed in the system responsible for enforcing permissions and recording receipts.

Rollback was an experiment with weak controls

Rollback is powerful evidence when the change is reversible, the prior state is actually restored, other relevant conditions remain stable, and the symptom responds in the expected way. Real incidents rarely provide all four controls.

During the cleanup-job scenario, the release rollback overlapped with job completion. The graph recovered after the rollback command, but the relevant process had never been running the new artifact. The action and recovery were related only by time.

During a later connection-pool scenario, rollback did contribute to recovery, but not alone. The desired configuration changed, web processes recycled, existing worker leases continued, client retry pressure fell, and the queue drained. Saying “rollback fixed it” discarded the sequence needed to prevent recurrence.

W93H recorded rollback as an action with an intended effect. The resulting hypothesis could be strengthened or weakened by observations:

Did relevant processes report the prior artifact and configuration?
Did the affected indicator change after activation rather than after request?
Did untreated instances behave differently from reverted instances?
Did a replay of the triggering workload reproduce the difference?
Did another concurrent variable change during the same interval?

In a small personal environment, I rarely had enough parallel traffic for strong comparisons. I therefore wrote conclusions with proportional confidence. “The rollback preceded recovery” was an observation. “The new worker allocation was the primary cause” needed the configuration diff, saturation evidence, process identities, and later reproduction.

That wording was not timid. It was accurate about the experiment I had actually run.

The interface needed uncertainty, not neutrality theater

My first correction was visual: deploy markers no longer stretched as dominant red lines across every chart. They became compact change bands with neutral styling, expandable details, and a clearly labeled relationship to incident hypotheses.

Visual restraint helped, but the data model had to carry the distinction. W93H had separate records for:

Observation: a source reported an event or measurement.
Action: a person or automation attempted a change.
Hypothesis: an explanation connected observations and actions.
Evidence link: an item supported, contradicted, or bounded a hypothesis.

A deployment marker was an observation about change. It could be linked as supporting evidence for a hypothesis, but proximity alone did not create that link.

Hypotheses had states such as proposed, supported, contradicted, superseded, and accepted for the incident record. “Accepted” did not claim universal proof. It meant the available evidence supported the operational conclusion strongly enough to guide the maintained changes, while recorded limitations remained visible.

I considered assigning numerical confidence. The apparent precision was not justified. A status, authored rationale, and evidence set were more legible. The interface could say why a theory remained weak rather than converting judgment into 73 percent.

Competing explanations made the primary one stronger

Once the interface allowed hypotheses, I noticed that I tended to create only the explanation I already favored. That reproduced the same confirmation bias in a more structured form.

For scenarios with meaningful uncertainty, I required at least one plausible alternative during review. If a release looked causal, alternatives might include traffic mix, a scheduled job, dependency degradation, data-shape change, or observation failure. The purpose was not ritual doubt. It was to ask what evidence would distinguish them.

In the cleanup-job incident, the competing explanation predicted that the resource curve should correlate with job lifecycle rather than release activation. Adding job start and completion events made that visible. In the connection-pool scenario, an external database slowdown would predict broader query latency without the same allocation pattern. Pool wait time and worker concurrency supported the allocation explanation more directly.

An alternative could be rejected with a recorded reason. It remained in history so the final narrative did not pretend the investigation had been obvious from the start.

This also improved follow-up work. A fix aimed only at the chosen cause could be paired with a test for the closest alternative. If both produced similar symptoms, the alert and runbook needed discriminating evidence rather than another generic graph.

Deployment annotations needed consistent clocks

Temporal reasoning depends on time, and W93H's sources did not share one perfect clock.

The deployment system recorded controller time. Processes logged local time. Metrics arrived with sample timestamps. The database used its own clock. My manual notes used the laptop. Sorting those values as exact could place an alert before activation or make a rollback appear faster than physically possible.

W93H preserved source time and ingestion time for every event. Known clock offset could adjust the display, but the original remained available. Sources with sequence identifiers retained them. Where ordering across sources could not be established inside the uncertainty, the interface grouped events into an interval instead of inventing precision.

This limited causal claims. If release activation and the first failed request were within an uncertain two-second window, the timeline could not honestly say which occurred first. Broader sustained behavior might still support the theory, but the neat visual sequence did not.

The lesson was uncomfortable for a timeline product: the most truthful ordering sometimes looked less orderly.

A marker also needed absence detection

If deployment events failed to arrive, the timeline could make a changing system appear stable. The absence of markers was not evidence that nothing had changed.

Each service periodically reported its current artifact and configuration identity. W93H compared those reports with recent events. An identity transition without a corresponding deployment or configuration event created an evidence-gap warning. A deployment event without later runtime confirmation remained unconverged.

This was not a second deployment controller. W93H did not repair state. It exposed disagreement between the change narrative and process observation.

I tested the path with deliberately lost events. Duplicate delivery collapsed through stable event identifiers. Out-of-order delivery retained both source and ingestion order. A late deploy event could improve the reconstructed history without pretending it had been available during the original investigation.

The same distinction later shaped the incident review: what I know now may differ from what I could reasonably have known then.

The postmortem should not begin with the release name

Early postmortem drafts opened with the suspicious deployment: “Release 7f3 introduced…” That sentence made the cause feel settled before the reader saw the impact or evidence.

I changed the narrative order. The record began with the failed capability and the observation window. It then described relevant system state, candidate changes, investigation, actions, recovery dimensions, and the evidence supporting the final explanation.

The release appeared where the investigation encountered it. If it became causal, the mechanism had to be stated: which behavior changed, under which input, through which resource or state transition, and how the evidence distinguished it from alternatives.

That structure prevented a nearby commit from becoming the protagonist by default. It also made incidents without recent deployments easier to explain. A scheduled expiration, data threshold, or dependency change did not have to compete with an absent release narrative.

Change visibility was still worth building

Making deployment markers less authoritative did not make them less useful.

They reduced search time, connected an artifact to its manifest and configuration, exposed partial activation, and preserved actions that would otherwise live only in shell history. The marker often identified the right branch of investigation. It simply stopped pretending to be the answer.

The redesign also improved deployment engineering. Services reported runtime identity. Configuration became typed and versioned. Desired and observed state were compared. Rollback receipts distinguished a requested reversal from verified convergence. Those capabilities made releases more understandable even outside incidents.

It changed how I reviewed small changes too. I stopped asking only whether a release mechanism had succeeded and asked whether the intended behavior had become observable at the relevant boundary. A completed file copy was not a visible publication. A restarted worker was not a reconciled queue. A new process accepting connections was not proof that its useful request class remained healthy. Deployment evidence became a chain of increasingly meaningful receipts rather than one celebratory timestamp.

The central correction was about epistemic status. A change event told me that something changed. A symptom event told me that a capability failed. A causal account had to connect them through a mechanism and evidence.

On a timeline, two marks can be one pixel apart and still have no causal relationship. The interface should help investigate that distance, not erase it.