The canary incident tested the incident tool

W93H's database was reachable, its event endpoint returned 200, and its interface loaded. None of those checks proved that W93H could help during an incident.

The product depended on a chain: a source had to emit a typed event, delivery had to survive retries, ingestion had to preserve identity and time, correlation had to attach the event to the right incident, the interface had to show its evidentiary status, a recovery condition had to remain open, and the archive had to retain the result.

Every component could look healthy while the chain was broken. An ingestor could accept events under the wrong schema version. A correlator could drop them into an unrelated incident. A delayed event could appear in reconstructed history but disappear from the live response view. An interface regression could hide open temporary actions behind a green impact state.

I needed a check that proved more than the parts were alive. I needed a small incident whose purpose was to test the incident tool.

A synthetic incident was not a fake outage notification

The easiest implementation would have sent a test alert through the notification channel. That would verify routing and little else. It would not create a deployment marker, exercise an affected capability, introduce late evidence, record an action, or require recovery.

I designed a bounded scenario called Canary Publication. It used an isolated fixture path in M31V:

1. Publish a small fixture artifact under a canary-only route.
2. Verify that the route serves the expected release identity.
3. Activate a scenario flag that prevents the fixture release from becoming visible.
4. Let the capability rule enter pending and firing.
5. Open or associate a canary incident in W93H.
6. Record a hypothesis and a reversible operator action.
7. Remove the fault, verify capability recovery, and satisfy remaining conditions.
8. Archive the incident and confirm that evidence remained queryable.

No ordinary route or background job depended on the fixture. The failure was real inside its boundary: the canary publication did not become visible, and the same observation path used for ordinary publication detected it. The blast radius was intentionally one synthetic capability.

This was not security theater or a status-page demonstration. It was a maintained end-to-end test with an explicit failure budget of its own.

The canary needed an identity that could not collide

The scenario generated events resembling real operational events. If its identifiers overlapped with normal work, W93H could contaminate an actual incident record or teach alert summaries to discount a real failure as a test.

Every canary run received a stable namespace and a unique run ID. Source events carried both. The fixture route, publication ID, artifact, configuration change, and incident membership all derived from that run identity.

scenario: canary-publication/v2
runId: 2019-08-13T14:00Z-7c2a
scope: fixture/incident-canary
expectedFault: release-visibility-blocked

The namespace was not a generic test=true label that arbitrary producers could add. W93H accepted canary control events only from the scenario runner's authenticated source, and the target services allowed the fault flag only inside the reserved fixture scope.

The normal alert evaluator still observed the capability. It did not receive a special “pretend to fire” command. A separate expectation record told the verifier which alert transition should result and by when.

That separation kept the test honest. The path under test did not get to declare its own success.

Fault injection stopped before shared state

I considered several canary failures: consume database connections, delay all publishing jobs, return errors from a shared dependency, or corrupt a fixture payload. The first three could escape the synthetic boundary. The fourth risked testing validation rather than incident handling.

The chosen fault acted at the release-visibility decision for one namespaced artifact. The build still ran. The artifact still carried a digest. The routing step deliberately withheld activation and emitted the same typed failure it would use for an ordinary release problem.

This tested a meaningful capability without imposing load or mutating shared data. The scenario had a hard duration limit and an independent cleanup path. If W93H failed completely, the runner could still remove the flag and delete the fixture through M31V's own control boundary.

A reliable chaos experiment should not require the system being tested in order to stop the experiment. Even this tiny canary followed that rule.

I documented what it did not prove. It did not exercise database saturation, high event volume, or ambiguous export side effects. Those belonged to replayable scenarios and focused failure tests. The canary proved one narrow vertical path continuously.

If W93H reported “canary succeeded,” that statement alone would be circular. A broken product could congratulate itself.

The scenario runner kept an external expectation log. For each run it recorded milestones and deadlines:

• Fault activation receipt exists.
• Publication capability enters firing with the active rule version.
• W93H ingests the alert and change events without duplicate membership.
• The incident shows impact active and at least one open recovery condition.
• Fault removal produces a source receipt.
• Capability returns to healthy after observation, not merely after command.
• Temporary action is reversed.
• Incident archives with the expected event set.

The verifier queried W93H through the same read API the interface used, but it retained its own run state. If W93H was unreachable, the check failed externally. If W93H claimed archive success while missing the original fault event, the expected event-set assertion failed.

I deliberately did not make the verifier a second full incident platform. It stored compact milestones, hashes, and deadlines. Its job was to say whether the vertical slice behaved as expected and provide enough evidence to repair W93H when it did not.

An independent scheduled dead-man signal confirmed that the verifier itself still ran. No finite design eliminated the need for an outside reference; it could only make the trust boundary explicit.

The scenario tested state transitions, not screenshots

The first canary assertion checked that the incident page contained a red badge. That test broke when I changed the label and could pass while the underlying state was wrong.

I moved assertions to product contracts:

expect(incident.impact).toEqual({
  capability: 'publication-visible',
  state: 'active',
  ruleVersion: expectedRuleVersion,
})
 
expect(incident.recoveryConditions).toContainEqual(
  expect.objectContaining({
    kind: 'temporary-action-removal',
    state: 'open',
  }),
)

Browser tests still verified a small set of essential interface behaviors: the current impact was perceivable without color, keyboard focus reached the open condition, late evidence carried a marker, and archive controls did not appear while required conditions remained open.

The two levels protected different contracts. API assertions proved the incident model. Browser assertions proved that a person could understand and operate it.

I avoided snapshotting the whole timeline. Event ordering could legitimately change when the correction model improved, and visual snapshots did not explain whether uncertainty had been preserved. Focused assertions named the semantics that mattered.

It needed to fail when evidence was late

A perfectly ordered canary would exercise W93H's easiest path. Real evidence could arrive late, duplicated, or briefly unavailable.

The scenario delayed one process-runtime event until after impact recovery. It delivered the event twice with the same source ID. The first delivery had to appear at its source-time position in reconstructed view and at its discovery-time position in response view. The second had to collapse idempotently.

The delayed event prevented the runtime-convergence condition from being satisfied early. Traffic could return to green, but the incident remained in recovery until the process identity arrived.

This single fixture tested several important properties:

• Ingestion acknowledged duplicate delivery without creating duplicate evidence.
• Late evidence improved reconstruction without rewriting what had been known live.
• Recovery state did not follow impact state automatically.
• The archive included source time, ingestion time, and event identity.
• The interface exposed delay without making the timeline unreadable.

I later varied delay within a safe range so the test did not depend on one hard-coded interleaving. The expected partial order remained stable even when wall-clock positions shifted.

The tool had to observe its own release carefully

W93H also needed to identify which version of itself processed and displayed the canary. Recording only the target service's release left a gap in the evidence path.

The ingestor attached its build identity and schema support range to the acceptance receipt. Derived incident projections recorded the projector version. The interface exposed its build identity in diagnostic details. A canary archive therefore named the versions that produced its interpretation.

This did not mean W93H should open an incident for every W93H deployment. Its release marker joined the canary record when the run crossed that change window or when projection behavior differed.

Deploying W93H during an active canary tested compatibility. An older ingestor could accept an event that a newer projector understood, provided the schema contract allowed it. Unsupported payloads went to a visible quarantine with a reason; they did not disappear behind a successful HTTP response.

The canary caught exactly that problem once. I had added a required recovery-condition field in the projector before the ingestor populated it. Component tests passed against matching fixtures. A canary spanning the mixed versions produced an incomplete condition and failed externally.

The fix was version-tolerant reading and an explicit migration interval, not tighter deployment timing.

An observability tool becomes dangerous when every observed service needs it to complete ordinary work.

W93H event delivery was asynchronous. Producers wrote to a bounded local outbox or emitted through a failure-tolerant transport path. If W93H was unavailable, the publication still succeeded or failed according to M31V's own rules. The event path could retry with a stable source identifier later.

The canary explicitly stopped W93H ingestion during one run. M31V's fixture failure and recovery completed independently. When ingestion resumed, events arrived late and the verifier confirmed that W93H reconstructed them while clearly showing the discovery delay.

The local buffer had visible limits. If it filled, the producer recorded evidence loss rather than blocking the primary operation forever or claiming complete delivery. W93H displayed a gap condition for the affected source interval.

This was an important architectural boundary. Incident evidence should be durable enough to survive common failures, but the evidence system should not become a synchronous dependency of every product capability it watches.

The canary tested that absence as deliberately as presence.

Notifications required their own bounded test

I did not want a recurring canary to train me to ignore normal incident notifications. If every scheduled run looked like a real emergency, the test would damage the behavior it was supposed to validate.

Canary alerts used the same routing engine and a distinct destination policy. A quiet receipt channel verified payload, grouping, acknowledgement link, and resolution update. Less frequent attended drills used the ordinary visible path at an announced time.

The routine run asserted that no normal urgent route was selected. The attended drill asserted that one was. Both carried unmistakable canary context without changing the underlying capability evaluation.

I tested silence behavior separately. A maintenance silence scoped to the fixture could suppress notification while the alert state still changed and entered W93H. The verifier confirmed the distinction: notification policy had changed; capability evidence had not turned healthy.

This preserved the lesson from alert design. Delivery is an action taken in response to an alert state. It is not the alert's truth.

The first canary report declared success when the incident archived. It left three fixture artifacts and an expired fault flag row in the database.

Synthetic systems create real debris. If cleanup is not tested, the canary eventually changes the environment it measures. Old fixtures can make names collide, fill storage, distort queue statistics, or satisfy checks accidentally.

I added cleanup conditions:

• The fault flag is absent or explicitly inactive.
• The fixture route no longer resolves outside its retention window.
• The artifact follows the documented short retention policy.
• The incident archive remains while disposable source material expires predictably.
• The next run begins from a verified baseline.

Cleanup happened through ordinary interfaces where possible. A hidden database deletion would have bypassed the same lifecycle the canary was meant to test. Emergency cleanup remained available outside W93H for containment.

The verifier distinguished cleanup failure from incident-path failure. Both mattered, but they led to different repairs. A successful archive with failed fixture removal did not become a fully successful run.

Flaky canaries were product defects

The first scheduled version failed intermittently because its deadline assumed that alert evaluation, ingestion, and browser projection would complete in a fixed number of seconds. Under ordinary laptop sleep and background load, the interval varied.

Increasing every timeout would have made the test slower and hidden actual stalls. I replaced sleeps with milestone waits and bounded deadlines derived from the rule windows. The runner advanced only when the previous receipt existed. The alert still had to fire within its contractual maximum, but the test did not assume a particular polling phase.

Clock control was available in local replay, not in the live vertical check. The live canary needed to experience real scheduling enough to test delivery and time semantics.

When a run failed, the external log recorded the last completed milestone and relevant identities. W93H itself could be unavailable, so the diagnostic record could not depend on opening the failed incident page.

I tracked repeated failure categories rather than a pass-rate vanity metric. Schema incompatibility, missed deadline, duplicate membership, cleanup failure, and verifier failure each pointed to a different property. A flaky check spent trust just like a noisy alert.

The scenario runner needed permission to create a fixture publication, activate one scoped fault, read the resulting incident, and clean up. Giving it general deployment or database authority would have made a small test an attractive path to broader change.

I created narrow credentials tied to the fixture namespace. The fault endpoint accepted only an enumerated fault and a bounded expiry. It could not accept arbitrary delay values, code, or route names. Every activation received an immutable receipt.

The verifier used read-only W93H access. Cleanup authority belonged to the scenario runner and targeted only resources bearing the run identity. Secrets never entered incident payloads; events carried credential version or actor class where relevant.

The canary also tested unauthorized activation. A request outside the namespace had to fail without creating a partial fault record. This was not a full security audit. It made the high-risk testing control obey the same evidence principles as the product.

Safe fault injection is a feature, not a comment saying “test only.”

Before the vertical scenario, I could prove that individual parsers, projectors, and components passed their tests. I could not prove that a real typed event became a usable incident under the deployed combination.

The canary found boundaries the component suites missed: mixed schema versions, lost incident membership, a green impact state hiding an open temporary action, duplicate events after retry, and cleanup that never ran.

It did not replace unit, integration, replay, or browser tests. It sampled one narrow path through the assembled system. Its value came from crossing ownership boundaries with the same contracts a real incident would use.

I ran it after meaningful W93H releases and on a modest schedule when the lab was active. An attended drill periodically verified the human notification and runbook path. The frequency respected the fact that the environment was personal and not continuously operated.

The outcome was not “W93H never fails.” It was a maintained signal that the most important narrative path had recently completed, with known limits.

An incident tool needs evidence about itself

W93H existed to tell a defensible story when other systems behaved unexpectedly. It could not earn that role through a health endpoint alone.

The canary incident proved a bounded chain: meaningful failure, capability observation, typed evidence, incident state, human-action receipt, separate recovery conditions, archive, and cleanup. An external verifier kept the proof from being self-awarded.

The exercise also clarified the project's boundary. W93H observed and reconstructed; it did not control ordinary services, automatically declare causes, or require every system to wait for its acknowledgement.

A green /health response said the process could answer one request. The canary said the product could still perform its reason for existing.