The clock correction must not erase the clock

W93H's first incident timeline was wrong in a particularly convincing way: every event was in perfect chronological order.

Deployments came from one controller, alerts from a monitoring process, application events from several hosts, job transitions from PostgreSQL, and my annotations from a browser. Each source supplied a timestamp. W93H converted them to UTC, sorted them, and drew one clean line.

During a replay of the connection-pool incident, the line showed a worker retry beginning before the worker had lost its lease. A deployment activation appeared after a request served by the new artifact. The sequence violated facts carried by source-local identifiers.

One worker clock was behind. The deployment controller and database were not perfectly aligned. Monitoring timestamps described sample and evaluation at different moments. My transformation had made the formats comparable and the claims falsely precise.

I could estimate clock offsets and make the display more useful. I could not let the estimate replace what each source had actually said.

UTC solved representation, not agreement

The original importer treated timezone normalization as clock normalization. It parsed local timestamps, converted them to UTC, and assumed the resulting instants belonged on one exact axis.

UTC removed ambiguity about timezone and daylight-saving interpretation. It did not make independent clocks agree.

Two machines could both emit ISO 8601 timestamps ending in Z and still disagree by seconds. A browser could inherit a laptop clock that had slept and resumed. A virtual machine could correct drift gradually or step after a larger error. A monitoring system could attach the sample time, the end of an evaluation window, or the time it delivered a notification. All were valid timestamp shapes with different semantics.

The event schema needed to answer two separate questions:

1. What time did the source claim?
2. What does W93H know about how that clock related to others?

Converting to UTC answered neither by itself. It only gave the claims a shared notation.

I kept UTC for storage and display because it simplified comparison and avoided locale surprises. I stopped describing conversion as correction.

The source timestamp was immutable evidence

My first correction pass subtracted a known host offset and wrote the adjusted value into the event's occurredAt field. The timeline improved immediately. The audit trail became worse.

If the offset estimate changed later, I could not reconstruct the original claim. If two investigators disagreed about the correction, the data model had already chosen one. A late clock-health event could silently move historical evidence without showing why.

I replaced the single field with an envelope:

type EventTime = {
  sourceTime: string
  sourceClock: string
  ingestedAt: string
  sourceSequence?: string
  precision?: 'millisecond' | 'second' | 'minute'
}

sourceTime was immutable. sourceClock identified the clock domain rather than merely the host name; a database timestamp and an application timestamp on the same host could still have different production paths. ingestedAt recorded when W93H accepted the event. A source sequence captured stronger local ordering when available. Precision prevented a minute-level manual note from masquerading as a millisecond observation.

Clock correction lived in a separate versioned record. The displayed position was a projection of source evidence through the best available estimate, not a rewritten fact.

That distinction let the interface say, “shown at 14:03:12 after applying the offset estimate observed at 14:10; source reported 14:03:05.” The extra detail stayed behind an expansion, but the model could support it.

Ingestion time answered a different question

It was tempting to use ingestedAt as the authoritative order. W93H assigned it on one server, so it avoided cross-host clock disagreement.

Ingestion order described when W93H learned about events, not necessarily when they occurred. A disconnected worker could buffer five transitions and deliver them after reconnecting. A monitoring rule might wait through a five-minute evaluation window before emitting a firing transition. A deployment event could be retried behind a network failure. The last event to arrive could be the earliest event in the incident.

Both orders mattered.

Source order helped reconstruct system behavior. Discovery order helped reconstruct the investigation. If late evidence changed the likely cause, the final postmortem should include it while preserving that I had not possessed it during the response.

W93H's live view therefore defaulted to “known by this time” during an active incident. A reconstructed view could place late events by corrected source time and mark their delayed discovery. Switching views made a subtle difference visible: the system's eventual story and the operator's contemporary story were not identical.

This stopped retrospective knowledge from making earlier actions look irrational. A decision could be reasonable under the evidence available then even if a late event later contradicted its hypothesis.

Sequence numbers outranked a drifting wall clock locally

Several sources already carried ordering evidence stronger than timestamps.

The export queue had attempt revisions. The database had transaction and row versions. The deployment controller emitted an ordered transition identifier. A process could attach a monotonic counter to its events even if its wall clock moved.

If worker event 104 claimed 14:02:10 and event 105 claimed 14:02:08, I did not reorder them by wall time. The source sequence established that 104 preceded 105. The backwards timestamp became clock evidence.

W93H retained both values and flagged the contradiction. The display could order the pair by sequence while showing an approximate time interval. Across two sources, sequence numbers could not be compared unless the protocol explicitly connected them. A database revision did not magically order a deployment event.

I built a small partial-order graph rather than treating the archive as a simple sorted list. Edges came from trustworthy relations:

• Consecutive sequence values from one source.
• A request receiving a response.
• A job lease preceding its terminal receipt.
• A deployment request preceding activation under the same operation ID.
• An annotation explicitly created after observing another event.

Clock estimates could suggest additional display order, but they did not override a contradictory causal edge. When the graph and corrected timestamps disagreed, the disagreement was the useful output.

Monotonic duration and wall-clock position were different data

Application logs often used wall-clock timestamps to calculate duration by subtraction. That failed when a clock adjusted between start and end.

Inside one process, I measured elapsed work with a monotonic clock where the runtime exposed it. The event could report durationMs independently of the wall-clock timestamps used to position it in the broader incident.

A request might say:

{
  "startedAt": "2019-05-21T14:02:03.412Z",
  "durationMs": 1840,
  "clockDomain": "web-3/process-812",
  "sequence": 48921
}

W93H did not infer that the end wall time was exactly startedAt + durationMs across clock domains. It could display the duration as a property of the operation and use the wall timestamp, offset estimate, and precision to position its start.

This mattered when visualizing latency beside alerts. A monotonic duration of 1.84 seconds was credible even if the host's wall clock was correcting. The apparent relationship to a database event on another host remained bounded by cross-clock uncertainty.

Separating the two prevented a clock adjustment from creating negative request durations or impossibly fast work.

Offset was an estimate with a validity interval

I initially stored one offset per host: worker-2 was seven seconds behind. That turned a moving relationship into a permanent property.

Clock offset could change. A machine might drift, resynchronize, restart with a corrected clock, or move to different infrastructure while retaining a logical source name. Network delay also affected how precisely W93H could estimate the difference.

Offset observations became versioned intervals:

sourceClock
observedAt
estimatedOffset
uncertainty
method
validFrom
validUntil

For a simple heartbeat, W93H knew when it sent a challenge, when the source stamped a response, and when the response returned. Assuming symmetric delay would be too strong, but the round trip gave a bound. Database comparisons and deployment-controller handshakes supplied other observations. I recorded the method because estimates with different assumptions should not look interchangeable.

Between observations, the display interpolated cautiously only within a configured drift bound. Across a restart or a large step, it opened a new clock segment. Events near the boundary carried wider uncertainty.

I did not attempt to build a precision time service. W93H needed enough honesty to avoid false ordering, not nanosecond agreement. A seconds-wide uncertainty interval was acceptable if the incident question operated over minutes. It was decisive when two events were only one second apart.

Clock steps deserved events of their own

When a source clock changed abruptly, the early model merely produced a new offset estimate. The incident timeline appeared to shift without explanation.

I made clock discontinuities visible operational events. A source could report that its wall clock stepped or that it restarted. W93H could also infer a likely discontinuity when ordered events moved backwards beyond expected drift, then mark the inference as such.

The event did not imply the time service was the cause of the incident. It said that chronological claims involving that source had changed reliability.

This was useful beyond ordering. Certificate validation, cache expiration, scheduled work, leases, and retention policies could all behave differently under a bad clock. A clock correction during an incident might be relevant to the failure mechanism, not only to its documentation.

I resisted automatically “fixing” application records after a clock step. A lease protocol should use its own authority and safety rules. W93H observed the discontinuity and linked affected evidence; it did not mutate source systems from the timeline.

Precision was part of the claim

Manual annotations were often written after an action: “paused queue around 14:12.” Importing that sentence as 14:12:00.000 invented precision.

W93H stored explicit precision or a time range. A note could represent the minute from 14:12:00 through 14:12:59. An alert evaluation covering five minutes could carry the window rather than only its end. A deployment activation observed through a periodic poll could say it occurred between the prior and current poll.

The interface rendered ranges as bands and point events as marks. Exact-looking millisecond timestamps were reserved for sources that actually supplied that resolution; resolution still did not guarantee accuracy.

This changed how hypotheses were written. If a configuration activation interval overlapped the beginning of a latency window, the record said the order was unresolved. It did not choose the midpoint and narrate certainty.

The loss of a crisp before-and-after story was often the evidence needed to design a better event. A process could emit its own activation receipt with artifact and configuration identity rather than relying on a minute polling interval.

Time uncertainty became feedback on observability quality.

Sorting needed stable, explainable tie-breaking

Even after correction, many events shared the same displayed time. A burst of log-derived transitions might have only second precision. Database updates in one transaction could share a timestamp. I needed deterministic rendering without implying that the tie-breaker represented causality.

W93H sorted first by the lower bound of the display interval, then grouped overlapping intervals. Within a group, source-local sequence edges determined what they could. Remaining ties used ingestion order and stable event ID for repeatability.

The group label explained that order within it was partially unresolved. Expanding an event showed why it occupied its position: corrected source time, causal relation, or deterministic display tie-breaker.

This detail prevented screenshots and incident links from rearranging on each query. It also stopped deterministic output from being mistaken for established temporal order.

I considered hiding the complexity and showing one flat sequence everywhere. That would optimize for visual fluency at the cost of the timeline's central claim. W93H existed to preserve evidence under uncertainty; it could not quietly make uncertainty disappear at the rendering layer.

Corrections needed versions and replay

A new offset observation could improve the reconstructed position of old events. If W93H silently moved them, a saved incident view or authored explanation might no longer match.

I versioned the correction model. The incident record named which time projection it used when a hypothesis or review was accepted. A later recalculation could show a diff: these events moved, these previously ordered intervals now overlap, and this causal sentence may need review.

Original event data never changed. Replaying projection version 3 reproduced the timeline used during the original review. Version 4 could incorporate a newly discovered clock step.

This was the same lesson R7K1's manifests had taught in another domain. Derived output should name the inputs and rules that produced it. A timeline was a derived artifact, not a transparent window into reality.

The versioning also made tests meaningful. Fixtures included drift, steps, late delivery, duplicate delivery, mixed precision, and contradictory sequence evidence. Expected output asserted both placement and uncertainty annotations.

Database time was useful, not universal

PostgreSQL supplied a shared time source for operations that reached it. I briefly considered stamping every event through the database and treating that timestamp as authoritative.

That would align ingestion, not occurrence. An application event delayed in a buffer would receive a precise database time when it was inserted, long after the action. It would also make the observability path depend more heavily on the database being investigated.

Database-generated timestamps were valuable for durable transitions performed inside a transaction. A job receipt written with its state change could use database order to establish the relation between those records. They could not order a browser action that had not yet reached the server or a worker event waiting offline.

I treated database time as one strong clock domain with clear scope. It did not become universal time by convenience.

The same boundary applied to the W93H ingestion server. Centralization made one ordering easy to query. It did not retroactively centralize the systems being observed.

The final interface exposed two related views.

The response view showed events in the order W93H knew them, with source-time hints. It answered what evidence was available when an action or hypothesis changed.

The reconstructed view positioned events using source time, clock estimates, causal edges, and explicit uncertainty. It answered what sequence the available evidence now supported.

Neither was “the real timeline.” Each had a defined question. The response view could explain a decision. The reconstructed view could explain the system. Late evidence appeared differently in both and linked them.

For ordinary incidents, the views looked similar. That was useful reassurance. When they differed, the difference became part of the review instead of a hidden implementation detail.

I added a compact clock-health panel rather than decorating every row heavily. Corrected, low-precision, late, or ambiguously ordered events received small markers. Detailed provenance appeared on demand. Honesty did not require making the default view unreadable.

Correction should increase knowledge without rewriting history

Clock correction is necessary. Refusing to correct known drift would knowingly preserve a misleading display. The error was not correction; it was treating the corrected value as if the source had emitted it and as if the estimate were timeless.

W93H ended up preserving three histories:

• The source's claim about when an event occurred.
• W93H's evolving estimate of where that claim belonged relative to other clocks.
• The moment the evidence became available to the investigation.

Those histories supported different kinds of reasoning. A source-local sequence could prove that one worker transition preceded another. A corrected interval could suggest relationship to a deployment. Discovery time could show why an early hypothesis ignored evidence found later.

They also changed how I spoke about absence. If W93H had no event inside a reconstructed interval, that did not prove nothing happened; it proved no accepted source had supplied an event for that interval by the chosen discovery cutoff. A missing heartbeat might strengthen a telemetry-failure hypothesis, while a missing deploy marker might expose a gap in the change path. The time model made the scope of “nothing” explicit instead of letting an empty stretch of pixels become a factual claim.

The clean original timeline had offered one answer with no visible assumptions. The revised one offered bounded answers and showed their basis.

Time in a distributed system is not absent. It is plural. Correcting a clock should help those claims meet without allowing one estimate to erase where any of them came from.