An alert is a claim about user impact

The first version of W93H could tell me that a machine was busy. It could not tell me whether anything a person cared about had stopped working.

I had added alerts the way I had seen monitoring examples teach them: CPU above a threshold, memory above a threshold, database connections above a threshold, queue depth above a threshold. Each rule was simple to express and easy to demonstrate. I could make the graph cross a line, receive a notification, and feel that the system was being watched.

Then I ran the alerts against the collection of personal services behind M31V, R7K1, and Q2F8. A build compressed images and held the CPU near its limit without delaying a preview. A cleanup job increased database activity while the public pages remained fast. A quiet worker retained more memory after processing a large document but had enough room to continue. The alerts described resource consumption accurately. They described impact poorly.

The opposite failure was worse. An export worker could stop making progress while CPU, memory, and database connections all looked calm. A publication callback could fail for one route while the aggregate request-success graph remained reassuringly green. My most confident alerts sometimes fired during healthy work and stayed silent during broken work.

I had treated an alert as a statement about a number. It needed to be a claim about a capability.

A threshold is not yet a reason to interrupt

Every alert compresses a complicated system into a binary transition: something deserves attention now, or it does not. The threshold is only the last line of that argument.

For the CPU rule, I could say what the number meant technically. A five-minute average above 85 percent meant the host had spent most of the window with little unused processing capacity. I could not answer the more important questions:

• Which operation had become unavailable or unacceptably slow?
• What action should I take if the rule fired?
• How long could the condition continue before it mattered?
• Which healthy workloads were expected to produce the same reading?
• Which harmful failures could occur without producing it?

Without those answers, the alert asked me to investigate a possibility rather than respond to known impact. That may be appropriate for a dashboard or a capacity review. It is a poor basis for an interruption.

This distinction mattered even in a personal laboratory. There was no staffed rotation and no contractual availability promise. My attention was still finite. If a notification interrupted other work, it needed to justify the interruption. If it did not, I would eventually teach myself to ignore the channel.

I started writing a short argument beside every alert definition. A rule had to name the capability it represented, the observation that supported the claim, the chosen time window, the immediate response, and the blind spots that remained. Rules that could not make the argument became diagnostic signals instead.

Capabilities were smaller than services

The first replacement idea was to alert on service availability. That was better than alerting on host resources, but the word service still hid too much.

M31V's publishing process exposed a web interface, rendered content, generated assets, moved a release into place, and sent a completion callback. The site could continue serving the previous release while publishing was broken. A single service-level status would either claim the whole system was down or conceal the failed path.

I instead named capabilities from the point of view of an intended action:

• A reader can load a published page within a useful time.
• An approved publication becomes visible with a verifiable release identity.
• A requested export reaches a terminal state within its expected period.
• An authorized deletion request completes or exposes a recoverable reason it did not.
• A preview URL serves the artifact associated with its branch and manifest.

These statements were deliberately more specific than “the app is healthy.” They also crossed implementation boundaries. Publishing involved a request handler, a build worker, storage, and a release switch. Its success could not be inferred from one process being alive.

The capability list became a product inventory. It revealed paths I had implemented but never defined operationally. I knew how a deletion job executed, for example, but I had not decided how long a pending deletion could remain pending before it represented a failure. Monitoring could not supply that product decision for me.

Symptoms and causes deserved separate channels

Once capabilities were named, I divided signals into symptoms and possible causes.

A symptom described the capability from outside its implementation boundary: response latency, failed publication visibility, an export missing its completion window, or a deletion request stuck outside a valid state transition. A cause signal described an internal condition that might explain the symptom: pool saturation, queue growth, dependency errors, disk pressure, or a process restart.

The separation was not absolute. A queue is part of a system, and its depth may be visible to nobody using the product. But the distinction forced a useful ordering. The primary alert said what was failing. The supporting signals helped explain why.

That changed the first minute of an investigation. Instead of opening with “CPU is high; find out whether that matters,” W93H could open an incident with “publication completion has exceeded its useful window” and place CPU, queue age, worker activity, and recent deployments beside it as evidence.

Resource thresholds did not disappear. They became dashboard panels, capacity warnings, or secondary incident signals. Some still justified an urgent notification when they predicted irreversible harm—a nearly full filesystem that would corrupt an imminent write, for example—but the alert definition had to state that mechanism. “High” by itself was not a mechanism.

The most direct way to test a capability seemed to be a synthetic transaction. For published pages, W93H could request a known URL and measure the response. For preview routing, it could create or use a fixture branch and verify the artifact identity returned at its URL. For an export, it could submit a tiny job and wait for its receipt.

These checks were valuable, but I initially gave them more authority than they deserved.

The page check used a small static document that stayed hot in every cache. It missed slow uncached pages. The publication check used a fixture with one image and did not exercise the larger asset path that had failed before. The export check used a trivial dataset and could finish even when normal exports were starved behind a different queue class.

A passing synthetic proved that one carefully chosen route worked under the check's conditions. It did not prove that all representative work was healthy.

I kept the checks and documented their sampling boundaries. Where possible, I paired them with measurements from actual requests or jobs. A known page check caught total reachability failures quickly; a distribution of real request latency showed whether ordinary pages remained useful. A fixture publication verified the mechanism end to end; age and completion state across real lab publications showed whether work was accumulating.

This produced a less satisfying status page and a more honest one. The capability could be healthy, degraded, failing, or unknown. Unknown was not a euphemism for success. It meant the evidence was absent, delayed, or outside the check's coverage.

Windows encode patience

My first rules used five-minute windows because monitoring examples often did. Five minutes looked reasonable on a graph. It had no consistent relationship to the capabilities.

A page that normally returned in a fraction of a second should not remain unusable for five minutes before the system acknowledged it. An export expected to run for twenty minutes should not alert because it had been active for six. A deployment check might tolerate a short transition while processes recycled, whereas a deletion request might have a longer completion target but stricter evidence requirements.

I began treating the evaluation window as a statement about patience. How much bad experience could occur before an interruption was justified? How much sampling was needed to avoid reacting to one unrepresentative event? What was the cost of waiting compared with the cost of a false alarm?

For interactive requests, I combined a short, severe condition with a longer, lower-rate condition. A near-total failure should surface quickly. A smaller but persistent failure rate should surface before it consumed an unacceptable share of the period I had chosen as a reliability target. I was borrowing the logic behind multi-window error-budget alerts without pretending my personal projects had formal customer commitments.

For background work, elapsed age mattered more than raw failure rate. A queue could contain many quick successful jobs and one stranded important job. I monitored the oldest eligible item by job class, the rate of progress, and the terminal receipts. Queue depth alone could not distinguish useful throughput from a pile that was no longer moving.

The configuration expressed the reasoning rather than only the threshold:

capability: publication-visible
claim: approved content is not becoming visible in its expected window
observe:
  completion_receipts: publication.release.visible
  expected_within: 4m
evaluate:
  severe: 2 consecutive fixture failures
  sustained: 3 overdue publications within 20m
respond_with: runbooks/publication-stalled.md
blind_spots:
  - fixture assets are intentionally small
  - a cached public route does not exercise release creation

The exact syntax was unimportant. Writing the claim and blind spots beside the numbers made review possible.

One of my oldest monitoring bugs came from the familiar comparison value > threshold. When the source stopped reporting, there was no value above the threshold. The rule remained green.

For resource metrics, missing data might mean an idle process, a failed collector, a network partition, a renamed label, or a service that no longer existed. For a capability check, it might mean the checker itself was unable to run. Those outcomes could not share one interpretation.

W93H represented observation health separately from capability health. Each rule recorded when it had last evaluated, which inputs it expected, and whether enough evidence existed to make its claim. A missing source could place the result in unknown; an explicit service-discovery event could mark a retired target as not_applicable; an empty queue could make a job-latency signal validly quiet.

This was not cosmetic status vocabulary. It changed escalation. If the public-page synthetic stopped reporting while independent request data showed success, I needed to repair the checker without declaring the site unavailable. If both the checker and request telemetry disappeared, I had lost the ability to assess the capability and treated that loss seriously.

I also added a dead-man check for the alert pipeline. A small scheduled event had to travel through evaluation, routing, receipt, and W93H ingestion. Its absence did not say a product capability was broken. It said the system responsible for making claims had stopped supplying evidence.

Labels could erase the affected minority

Aggregate success rates made another reassuring lie possible. If a high-volume cached route remained healthy, it could outweigh complete failure in a lower-volume but important path. The overall graph might show 99 percent success while every deletion request failed.

Splitting every measurement by every label was not an answer. High-cardinality dimensions made queries expensive and created noisy rule sets with too little traffic in each group. I needed a deliberate capability taxonomy, not arbitrary slicing.

I defined a bounded set of route and job classes based on different user intent and failure behavior. Interactive reading, publication, export, deletion, and preview routing each deserved separate evidence. Within those classes, I preserved useful dimensions for investigation but did not automatically turn every dimension into an alert.

Low-volume capabilities required different detection methods. A rate over five minutes was meaningless when one operation occurred in an hour. Durable state and overdue age worked better. If a deletion request entered accepted, it should eventually produce a terminal receipt or a visible recoverable failure. The absence of that transition was evidence even without a statistically interesting sample.

The design made quiet paths first-class. Popularity was no longer a prerequisite for reliability.

Alert state needed a history

The first alert events stored only firing and resolved. That was enough to color a row and not enough to reconstruct why it changed.

Rules evolved. Thresholds were tuned, windows changed, and labels were reorganized. If W93H displayed an old incident using the current rule definition, it could attribute a historical state transition to logic that had not existed yet.

I versioned each definition and attached the active version to every evaluation transition. The event included the observation window, summarized values, prior state, new state, and reason. A rule change was recorded like a deployment marker. Silence caused by a disabled rule therefore looked different from recovery caused by improved behavior.

The state machine gained pending. A brief threshold crossing could enter pending without opening an incident, then return to normal if it failed to satisfy the required duration. This preserved evidence of near misses without sending every fluctuation through the urgent path.

Acknowledgement did not change the underlying state. It recorded that I had seen the notification and intended to investigate. Silencing did not turn a failure green either. A silence had a scope, reason, creator, and expiry. When it ended, the current condition was evaluated rather than replaying every suppressed notification.

Those distinctions let the timeline answer a basic question later: did the system recover, did the rule change, or did I merely stop listening?

An alert without a plausible next action was an invitation to browse graphs. I wrote a short runbook for each interrupting rule and used the alert fixture to test it.

The runbook began with the capability claim and the evidence that caused it to fire. It then listed fast discriminators rather than an exhaustive tour of the system. Was the failure limited to one route class? Did it begin near a deploy or configuration change? Was work accumulating or only completing slowly? Were terminal receipts absent, duplicated, or delayed? Which action was safe without knowing the cause?

I resisted instructions such as “restart the worker” unless the restart semantics were understood. A restart might re-run an operation with an unknown outcome. The runbook linked to the job receipt and retry policy before suggesting it.

Testing a runbook exposed gaps in the alert. If the first step required a label or timestamp that the notification did not carry, I added it. If the runbook always concluded that no action was needed, the rule lost its interrupting status. If the safe response depended on information W93H did not retain, I changed the event schema or accepted the documented limitation.

The notification became a compact handoff from automated observation to human judgment. It needed enough context to begin responsibly, not enough raw data to contain the whole investigation.

Error budgets were a lens, not a badge

In 2019, reliability conversations increasingly used service-level objectives and error budgets to connect technical work with user experience. The model was useful, but it was easy to copy the vocabulary without the responsibility behind it.

My projects did not have negotiated external objectives, a staffed reliability function, or enough traffic for every statistical assumption. Publishing a many-decimal availability number would have implied a maturity the environment did not possess.

I used a modest internal target as a planning lens. For a capability and a period, how much unsuccessful or excessively slow experience was I willing to accept before reliability work displaced feature work? Which incidents consumed that allowance? Was the chosen indicator actually representative?

The exercise improved priorities. Frequent brief publication delays consumed more of the chosen allowance than dramatic CPU spikes that affected nobody. A missing receipt on a rare deletion operation did not dominate a percentage, but its consequence justified a separate correctness target. Not every promise fit one availability ratio.

I displayed the target and its assumptions only inside the lab. It was a decision aid, not a marketing statistic. The most useful output was often the disagreement it caused with my own model: if I could not defend the indicator or target, I had discovered an unfinished product decision.

The alert review removed more than it added

After rebuilding the rules around capabilities, I expected a larger monitoring system. The interrupting set became smaller.

CPU, memory, pool utilization, queue depth, restart count, and several host-level thresholds remained available for diagnosis and capacity work. They did not all notify me. Primary alerts covered a handful of intended actions, and each carried a tested route into W93H.

I removed rules that duplicated the same impact at different layers. If both page failure and upstream dependency failure fired, the capability alert opened the incident and the dependency event joined as evidence. I also removed a broad “any error log” alert. Error logs were neither consistently user-affecting nor consistently exceptional; the rule mostly measured logging style.

Fewer notifications made each transition more credible. That credibility mattered during the later connection-pool incident. Request latency made the user-facing failure visible. Pool saturation and worker activity explained the mechanism. Had I kept their roles reversed, the incident would have begun with an internal number and left the affected action implicit.

A claim should remain falsifiable

The final alert template ended with two uncomfortable fields: blind_spots and evidence_that_would_disprove_this.

The first prevented a synthetic or aggregate from becoming universal through omission. The second kept the claim open to correction. A deployment near a failure did not prove causality. A recovered metric after rollback did not prove the rollback was the only recovery mechanism. An alert established that its defined evidence crossed a condition; it did not get to write the postmortem.

I am carrying that boundary into W93H's incident model. Observations, actions, and hypotheses occupy different forms because a monitoring transition should not silently promote itself into an explanation.

I had begun with a set of thresholds and ended with a discipline for making limited claims. An alert said: this capability appears to be failing, according to these observations, over this window, with these known gaps, and this is why attention is justified now.

The sentence was longer than “CPU high.” It was also useful.