A small source simulator taught me more than production-like data

A deterministic feed generator made delayed, duplicated, skipped, and out-of-order events ordinary enough to design T04P around them.

My first T04P test feed tried to look realistic. It generated entry counts with random bursts, changed equipment states occasionally, and emitted notes from a sample list. The dashboard moved convincingly.

It was poor at explaining failures.

When a count looked wrong, I could not reproduce the exact event order. Random timing created noise around the behavior I wanted to inspect. The generator resembled production activity while avoiding the rare, structured conditions that made the product difficult.

I replaced it with a much smaller simulator. It had fewer data fields, no elaborate model of a venue, and a deliberately explicit script. It taught me more because it could produce one uncomfortable sequence on demand.

Realistic volume was not realistic behavior

The random generator varied values but kept the transport clean. Events arrived once, in order, with current timestamps. Reconnect started a new neat stream. Every source either ran normally or stopped completely.

Those defaults encoded a perfect system beneath messy-looking numbers.

The failures T04P needed to understand had shapes:

  • A source paused while its adapter and browser connection stayed open.
  • A delayed batch arrived with old observation times.
  • One revision was skipped and a later revision arrived.
  • An event was delivered twice.
  • Two independent sources arrived in an order different from their observations.
  • A source restarted and reset its sequence.
  • A browser reconnected while the service continued receiving.

Adding more random traffic made these cases harder to see. I needed control over order, time, identity, and boundaries.

The simulator stopped trying to be a miniature venue. It became a language for feed behavior.

A scenario was a readable timeline

I represented each scenario as a sequence of commands with relative time. The first version used a small JavaScript array rather than inventing a file format:

var staleAfterReconnect = [
  at('00:00', observe('entry', 120, 40)),
  at('00:10', observe('entry', 124, 41)),
  at('00:20', disconnectPath('source-to-adapter')),
  at('00:50', buffer(observe('entry', 131, 42))),
  at('01:20', reconnectPath('source-to-adapter')),
  at('01:21', releaseBuffered()),
  at('01:22', observe('entry', 136, 43))
]

The arguments named source, value, and source sequence. The timeline runner supplied observation and receipt clocks according to the path state. A scenario could override either when testing clock errors.

I valued readability over clever composition. Someone inspecting a failed test should be able to trace which path changed and which observation crossed it. Helpers reduced repetition but did not hide the event identity.

The scenario name described the behavior under examination, not the feature it happened to touch. “Buffered observation remains old after delivery” was more useful than “dashboard test 7.”

The first simulator used real setTimeout calls. A ninety-second freshness case took ninety seconds, and scheduling jitter made event order unreliable under load.

I introduced a simulation clock. Advancing to 01:20 ran every scheduled command through that point without waiting in real time. The dashboard's classification logic already accepted an explicit now, so the same clock could drive source observation, adapter receipt, and view updates in the test environment.

Not every part of the browser could be virtualized. Socket.IO and DOM callbacks still used the event loop. I kept the simulation runner separate from assertions that required an actual transport. Most feed and projection tests used the fake clock; a smaller set verified the integration with real scheduling.

The runner maintained distinct values for source time and service time. A scenario could drift one clock or freeze it while the other advanced. This prevented the fake clock from recreating the single-timestamp assumption I was trying to test.

Deterministic time turned freshness thresholds into immediate boundary cases. It also made failures reproducible: a log at simulated 01:21 meant the same position on every run.

Faults attached to paths

My earliest command was simply offline(). It stopped everything and produced an easy red screen. The real system had several paths that could fail independently.

The simulator modeled a small topology:

source -> adapter -> service -> browser

Each edge could be open, delayed, buffering, dropping, or duplicating. Nodes could be active, paused, or restarted. A path fault affected only messages that crossed that boundary.

This distinction produced revealing combinations. Pausing the source left adapter polling and socket heartbeats active. Closing the browser path let the service continue building a current projection. Buffering source-to-adapter delivery preserved old source times while creating recent receipt times on release.

I did not simulate every property of TCP or a physical network. The states described application-visible effects. dropNext(2) was enough to test a revision gap. bufferFor('60s') was enough to test delayed history.

Naming the boundary kept expected behavior precise. A source outage and a browser disconnect both stopped visible updates, but they required different explanations and recovery.

Identity made duplicates testable

Randomly producing the same value twice did not create a duplicate event. Two legitimate observations can share a value. A duplicate repeats one event identity.

Every simulated observation carried stable source identity and sequence. The delivery layer could emit the exact object twice. The service could then prove whether normalization and browser application were idempotent.

event source: entry
source instance: A
source sequence: 42
value: 131
deliveries: 2

The expected result was one applied source observation, a duplicate counter incremented in diagnostics, and no second visual notification.

A separate scenario emitted two observations with the same value and sequences 42 and 43. Both were meaningful because the second proved continued source progress. This kept deduplication from collapsing stable observations merely because their payloads matched.

The simulator forced the product model to say what made an event the same. Payload equality was insufficient. Receipt time was unstable. Source identity plus scoped sequence was the best evidence available for this feed.

Gaps were assertions, not surprises

To test continuity, the runner could generate revisions 10, 11, and 12, then drop delivery of 11 to the browser. The client should hold 10 as its last contiguous point, buffer 12 within a limit, and request recovery.

The scenario declared expected transitions beside the commands:

expectAt('00:12', {
  transport: 'open',
  continuity: 'recovering',
  lastContiguousRevision: 10,
  bufferedRevisions: [12]
})

An assertion did not merely compare the final count. Several incorrect paths could converge on the same number. The intermediate state proved that the client recognized uncertainty instead of accidentally arriving at the right value.

The runner captured a transition log, so a mismatch showed both expected and actual sequences. “current → recovering at revision gap” was easier to reason about than a screenshot of a stale badge.

I kept final projection assertions as well. State-machine correctness without data correctness would be a hollow victory. The combination showed what the browser knew and what it displayed.

Out-of-order did not mean reverse everything

The first out-of-order command shuffled a list randomly. That again sacrificed explanation for chaos.

I defined exact permutations. A scenario delivered source sequence 52, then 54, then 53. Another delivered independent equipment and entry events in opposite receipt and observation order. A third delayed an old note until after its resolution had arrived.

Each represented a different question:

  • Can one feed close a small gap from its buffer?
  • Does the timeline preserve source and receipt chronology separately?
  • Can a linked record appear before the record it references?

The third case changed the model. A resolution referencing an unknown note could be held briefly, then reconciled when the note arrived. After a bound, it became an orphan diagnostic rather than crashing the renderer.

Exact order let the scenario state the intended policy. “Messages may arrive randomly” was true and too broad to guide a design decision.

The simulator did not make disorder realistic by being unpredictable. It made disorder useful by making each permutation repeatable.

A simulated source restart reset its local sequence from 87 to 1. The first client treated every new event as an old duplicate.

I added source instance identity. Restart created instance B, sequence 1. The adapter recognized that the sequence was monotonic within an instance, not across all time. It requested or emitted a snapshot and recorded the discontinuity.

The scenario also tested a dangerous variation: instance identity accidentally remained A after the sequence reset. In that case the adapter did not guess that a restart occurred. It marked the source sequence invalid and withheld current state until a full observation established a new contract.

This distinction mattered because automatically accepting any lower number as a restart would allow genuinely delayed old events to replace newer state.

The simulator made a restart cheap to repeat. Without it, I would have tested by manually restarting processes and hoped the timing produced the intended sequence. With it, the scope rule lived in an executable example.

One button ran a scenario slowly

Fast deterministic tests were necessary and not sufficient. I added a simple control page where I could choose a scenario, pause it, step one command, or play it at a visible speed.

The page showed four synchronized views:

  • The scenario command about to run.
  • The simulated topology and path states.
  • The raw event with observation and receipt fields.
  • T04P's resulting projection and visible status.

Stepping made design review concrete. I could stop after the browser received a delayed message and inspect whether the label said “just now” or preserved its age. I could pause at a revision gap and verify that the last known count remained qualified.

The control page was intentionally plain. It existed to reveal causality, not to imitate the final dashboard. A copyable scenario trace made it easy to turn a visual discovery into an automated assertion.

This bridged code tests and interface judgment. The same scenario that asserted state transitions could be played slowly enough to evaluate wording, hierarchy, and motion.

When a real test run produced surprising timing, I did not import a giant log and call it a test. Large recorded streams contained irrelevant activity, changing identifiers, and private note content.

I reduced the behavior to the smallest synthetic fixture that preserved the failure. If the important sequence was “old batch delivered after reconnect, followed by a fresh snapshot,” the scenario contained those events and invented safe values.

The reduction process was diagnostic. Removing an event and watching the bug disappear identified dependencies that a raw replay would hide. It also made the resulting test stable as unrelated schemas evolved.

I recorded the source of the scenario in a private development note—manual experiment, endurance run, or model review—without copying real content into the repository. The public-facing simulator remained a personal lab with invented subjects.

This kept the tool historically useful. A scenario represented a system rule, not one opaque afternoon of traffic.

A testing tool can prove the wrong thing confidently. The first runner delivered commands synchronously and accidentally guaranteed that the dashboard finished one event before the next began. A real transport could schedule callbacks between rendering and recovery work.

I added explicit delivery steps and a way to yield between them. Integration scenarios could choose same-turn, next-turn, or delayed delivery. I did not randomize the choice; the fixture named it.

The runner validated its own script. Source sequences had to be unique unless a duplicate command intentionally reused one. A released buffer preserved original observation identity. A path marked dropped could not silently deliver.

I also kept the simulator out of the production bundle. The dashboard shared normalization types and pure reducers, but the control interface and fault commands were development tools. A build check made accidental inclusion visible.

These constraints prevented the simulator from becoming a parallel implementation that always agreed with itself. It produced inputs and controlled boundaries; T04P still executed its real state logic.

Scenarios became design documents

The most valuable output was not code coverage. It was a vocabulary for discussing difficult states.

“What should the page do when the transport is open, source receipt is active, observation time is old, and continuity is complete?” could become a small timeline everyone could step through. The expected interface text sat beside the model assertion.

I grouped scenarios by product promise rather than code module:

freshness
  source pauses without disconnecting
  delayed batch arrives recently
continuity
  one revision is missing
  delta window expires
identity
  duplicate delivery
  source instance restarts
lifecycle
  browser sleeps during recovery
  configuration changes before response

That organization survived refactoring. Socket code, adapters, and renderers could change while the scenario still expressed the user's situation.

The list also revealed missing interface states. If a scenario required an expected result I could not name without combining several booleans, the product model was probably incomplete.

Once the topology and commands existed, it was tempting to combine every path state, source state, clock condition, and browser lifecycle. The matrix grew faster than its meaning. Many combinations were impossible under the protocol, and others exercised the same product invariant through different noise.

I selected scenarios from promises and past mistakes. One proved that recent receipt could not rejuvenate an old observation. One proved that a gap withheld current status. One proved that a late recovery response lacked authority after a newer attempt. A new scenario needed to protect a distinct rule or a materially different consequence.

Pairs of faults were useful when their interaction created a new ownership problem: a service restart during browser recovery, for example. Combining source drift, duplicate delivery, background throttling, configuration replacement, and an expired delta window made a dramatic demo and a poor explanation.

This kept failure testing maintainable. The suite was not a claim that every possible world had been enumerated. It was a curated set of counterexamples to the system's most important assumptions.

When a new bug appeared, I first asked whether an existing promise should have caught it. Sometimes the right fix was strengthening one scenario rather than adding another nearly identical one.

I did not abandon generated streams entirely. After the deterministic scenarios passed, a seeded stress mode varied event values, short delays, and harmless interleavings within declared bounds. A failing seed printed the exact command sequence so it could be reduced into a permanent fixture.

The seed made a failure repeatable, but repeatability alone did not make it understandable. I treated the generated sequence as evidence for investigation, not as the final test. The smallest meaningful order became the regression scenario.

Generated runs checked broad invariants: contiguous revision never moved past a gap, retained collections stayed within bounds, current status required fresh evidence, and one event identity produced at most one projection effect. These properties could be evaluated across many sequences without asserting one fixed final count.

Deterministic examples and seeded exploration complemented each other. The examples explained known hazards. The generator searched nearby spaces for combinations I had not written. Neither needed to masquerade as production traffic.

Small and adversarial beat large and plausible

The realistic generator remained useful for visual rhythm and sustained volume. It helped answer whether the wallboard felt noisy during an ordinary event. It was not the main tool for correctness.

The small simulator owned the hard questions. It could freeze one clock, duplicate one identity, skip one revision, restart one source, or interrupt one path. It ran quickly, repeated exactly, and produced a trace a person could understand.

Its simplification was a strength. Production-like data often carries enough incidental complexity to make any result seem plausible. A minimal scenario denies the implementation that cover.

T04P improved when failure stopped being an accident I waited to witness and became a state I could request by name.

I have built some version of that tool many times since: not a grand digital twin, but a small, deterministic world that knows how to be inconvenient on command.

Its best feature was that every inconvenience arrived with a name and a reason.