Containers changed what I could reproduce

X8B6's hardest synchronization failure required a particular server build, database schema, and seeded receipt history. I reproduced it once on my laptop, upgraded a local package while investigating, and could not reproduce it again.

The test instructions were a page of setup steps: install the right Node.js version, configure the database, run migrations through version four, import a fixture, change two environment variables, and start the service in a particular order.

Docker reached its 1.0 period in 2014 with an appealing promise: describe an image, run a container from it, and move more of that setup from memory into versioned files.

For X8B6, containers changed what I could reproduce. They did not remove the need to define what “ready” meant or what state the test required.

The environment was part of the failure

The bug involved an old operation receipt stored under a previous schema. The current service should have returned that receipt during retry. One local database had a different collation and one migration had been run manually rather than by the project script.

Code and test data alone were insufficient. The runtime version, system packages, database version, configuration, and migration order influenced the result.

I wrote the environment as an inventory:

base operating-system image
Node.js runtime and package install
X8B6 service source at one commit
database server version
schema migration command
fixture import command
published service port
explicit configuration values

This mirrored the performance inventory from C62Y: vague categories became decisions when named. “Works on my machine” compressed a long list of environmental facts.

The container experiment aimed to make enough of those facts buildable that another clean run could produce the same failure.

The Dockerfile became an executable setup record

The first Dockerfile was small and unsurprising:

FROM a-pinned-node-base
WORKDIR /srv/pocket-ledger
COPY package.json /srv/pocket-ledger/package.json
RUN npm install
COPY . /srv/pocket-ledger
EXPOSE 3000
CMD ["node", "server.js"]

The actual historical base tag and package commands were pinned in the project; the important structure was that the image declared its starting environment, installed dependencies, copied the application, and named the startup process.

Building produced an image. Running that image produced a container instance. Deleting the container did not delete the recipe or change the image it came from.

The file replaced most of the setup page with instructions the build actually executed. When a system dependency changed, the diff showed it. A clean image build caught undeclared assumptions that my long-lived laptop already satisfied.

The Dockerfile did not guarantee reproducibility by existing. Floating base tags and unpinned application dependencies could still change underneath it. It gave the project a place to make those choices explicit.

Docker images were built in layers. If early instructions and their inputs remained unchanged, rebuilding could reuse cached work. Copying the entire source tree before installing packages invalidated the dependency layer after every code edit.

I copied the package manifest first, installed dependencies, and copied frequently changing source afterward. This made development builds faster without changing the runtime result.

The optimization exposed an integrity question. If the manifest did not fully pin dependency versions, a cache hit could preserve one dependency graph while a clean build installed another. Fast local builds and reproducible clean builds were not automatically the same.

I kept a clean-build check for release fixtures and recorded the resolved dependency state available to the tooling at the time. Cached layers were an acceleration, not evidence that the environment could be rebuilt from declared inputs.

Layer order also affected image size. Temporary package files left in a prior layer remained part of the image even if removed later. I kept installation steps deliberate and inspected the resulting artifact rather than assuming the final visible filesystem told the whole size story.

The cache rewarded stable inputs. It also made hidden dependency drift easier to overlook unless clean builds remained routine.

I first called the service image pocket-ledger:latest. Rebuilding moved that tag to new contents. A bug report saying “run latest” became unreproducible as soon as the next image replaced it.

Candidate images received build-specific tags tied to the source revision and fixture version. The test record captured the exact image identity used. A friendly tag could point to a candidate, but it was not the only evidence.

The same principle applied to base images. A convenient runtime tag could be republished or advance. Pinning as precisely as the tooling allowed reduced drift, and recording the built image identifier preserved what had actually run.

I did not claim bit-for-bit reproducible builds. External package repositories, timestamps, and early ecosystem practices made that a larger problem. The improvement was narrower and real: one named image could be run repeatedly without reinstalling the environment by hand.

Tags were labels. Reproduction needed an immutable-enough artifact and the inputs that produced it.

The database state did not belong inside the service image

My first attempt baked a prepared database directory into an image. It made one test start quickly and coupled durable mutable state to an application artifact.

Containers have writable layers, but that layer follows the container's lifecycle and is a poor place to pretend important database state is automatically safe. X8B6 used a separate database container and an explicit test volume or seeded disposable data directory.

For a reproducible failure fixture, the database began empty, ran migrations through a declared version, and imported a small seed of operations and receipts. The fixture could be destroyed and recreated. For development data worth keeping, a named external location survived container replacement and had its own backup expectations.

The distinction mirrored the offline cache lesson:

service image: replaceable executable environment
container writable layer: disposable runtime changes
fixture volume: intentionally resettable test state
development volume: persistent data with explicit custody

Docker did not decide which category data belonged to. The project had to assign lifecycle and consequence.

Starting the database container before the service container did not mean the database was ready to accept connections. X8B6 sometimes launched while the database was still initializing and exited with a misleading configuration error.

I added a bounded startup script that attempted the actual database connection and migration-status query before launching the service. It distinguished connection refused, authentication failure, missing schema, and migration failure.

The script stopped after a practical bound and exited with an explanation. It did not loop forever while the container appeared “up.”

The service itself retained readiness semantics. A running Node.js process could be alive while its schema was incompatible. The container runtime could restart a process; it could not infer that returning an honest maintenance response was preferable to endless restart.

This corrected another green-light assumption. Container running was transport-level evidence about a process, not proof that X8B6 could safely accept operations.

Reproducible startup included the states between process creation and application readiness.

Hard-coding database credentials and environment-specific URLs into an image would make it less portable and leak secrets into layers. The service read configuration from explicit environment variables or mounted development files with safe local values.

The image declared which values were required and failed clearly when one was missing. A sample environment file contained invented credentials for the disposable test fixture, not reusable production secrets.

I avoided making every difference configurable. Runtime version and system dependencies belonged in the image. Database endpoint, port mapping, and fixture mode belonged to the run. Too much configuration recreates an undocumented environment through flags.

The test script printed non-sensitive configuration names and image identity at startup. It redacted credential values. This made a captured run explainable without turning logs into secret storage.

The boundary was practical: bake the executable environment, inject deployment-specific connection and policy values, and version the list of required inputs.

Publishing the service port made the container reachable from the host. It did not make the service safely isolated from everything else on the machine.

For local tests, I bound ports to the local interface and exposed only what the fixture required. The database did not need a broadly reachable host port when the service could connect through the container network arrangement.

I treated containers as a useful process and filesystem boundary, not a perfect security sandbox. The service still ran with minimum necessary privileges, validated input, and kept test credentials distinct from anything valuable.

Mounting the source tree into a development container also granted the container access to those files. Convenience did not erase that capability. The run script mounted only the project paths needed.

The early excitement around containers made it tempting to translate “isolated” into “secure.” X8B6 documented the narrower claim: the test process ran in a separately described environment with controlled mounts and ports. Security remained a set of explicit policies.

Source mounts and built images served different loops

During development, rebuilding an image after every JavaScript edit was slow. A bind mount let the container use the working source while retaining the declared runtime and dependencies.

That loop was useful and less reproducible than the candidate image. The mounted source could include uncommitted changes, generated files, or host-specific permissions. A bug reproduced under a mount needed a built-image confirmation before being called captured.

I named the modes:

development run — mounted source, rapid edits
candidate run — source copied at build, tagged image
fixture run — candidate image plus versioned database seed

The commands made the difference visible. I did not let a developer-friendly mount quietly replace the artifact used in the test record.

This distinction preserved both speed and evidence. Containers improved the inner loop without making every inner-loop run a release artifact.

Logs needed to leave the container

Deleting a failed container could delete its writable filesystem and any log file stored there. The process wrote structured logs to standard output, where the test runner could capture them beside the scenario trace.

Each run included image identity, schema version, fixture version, and a generated run identifier. Operation IDs appeared where relevant. Secrets and note bodies did not.

The captured bundle contained:

run manifest
service log
database migration log
scenario command trace
assertion result

This made the environment repeatable and the evidence portable. A container that failed and vanished was still diagnosable.

I kept logging separate from persistence. Standard output simplified disposable fixtures; durable operational deployments would need an external log policy. Docker did not supply retention merely because it captured a stream.

The decisive test command removed the disposable database state, started the database, waited for readiness, ran the exact migrations, imported the seed, launched the service image, and executed the timeout-retry scenario.

It then asserted that one operation identity produced one adjustment and one receipt. Cleanup removed containers and the disposable volume after saving the result bundle.

Running from zero mattered. Reusing a development volume could hide a missing migration or leave a receipt from the previous test. The fixture made its reset destructive by design because its data was generated and recoverable.

The command refused to point at an unmarked persistent volume. A guard checked the fixture label before deletion. Tooling convenience should not turn a typo into loss of development data.

The same lifecycle—create, seed, test, capture, destroy—made failures repeatable without accumulating mysterious local databases.

Containers exposed architecture assumptions

The service originally read a configuration file from a hard-coded path on my laptop. A migration script expected a globally installed tool. One test connected to localhost and accidentally reached the host database instead of the fixture.

Containerizing did not create those couplings. It removed the background environment that had concealed them.

Fixing the assumptions improved the non-container path too. Configuration became explicit. Project-local tools replaced global dependencies. Service endpoints were injected. Migrations ran through one owned command.

This was the largest benefit after reproducibility. An image build asked, “What does this program actually require to exist?” A clean container run asked, “Which dependencies does it assume are nearby?”

The answers became part of the repository instead of lore attached to one machine.

Once the service and database ran in containers, it was tempting to treat deployment as solved. The project still lacked automated failover, rolling updates, secret distribution, durable backup, capacity policy, and coordinated schema compatibility across versions.

I used the containers for development, repeatable tests, and a small controlled host experiment. I did not claim that packaging two processes defined a production platform.

Even lifecycle within the fixture needed explicit scripts: start order, readiness, migration, seed, test, evidence capture, and cleanup. Multiple hosts and long-lived operations would multiply those responsibilities.

This restraint kept the article's breakthrough proportional. Docker packaged an environment and made a named artifact runnable. It did not eliminate distributed systems or operational ownership.

The project became easier to reproduce before it became easier to operate.

A new service image could start against a database created by an older version. X8B6's server and local offline clients already carried versioned operations, so container replacement joined an existing compatibility story.

The candidate run tested current image against supported database schema and operation payload versions. A migration ran deliberately before the service advertised readiness. Rolling backward after a destructive schema change was not assumed safe.

I kept database changes additive during the experiment where practical and captured a backup of non-fixture data before migration. Replacing a container was easy; reversing durable state was not.

This corrected a subtle psychological risk. Immutable application images make rollback feel universal. They can restore executable code quickly and cannot automatically undo external database effects created by the newer code.

Artifact immutability and data reversibility were separate properties.

Reproduction gained a manifest

A passing or failing run produced a small manifest:

source revision
service image identity
base image reference
database image reference
schema version
fixture version
scenario name
configuration profile
result

The manifest did not guarantee that every remote package remained available forever. It dramatically improved the ability to answer what had run.

When a teammate in an imagined larger project said the test passed locally, the next question could be “with which manifest?” rather than a long comparison of installed tools.

I stored the Dockerfile, fixture scripts, and seed beside the application. The image itself could live in a registry or be rebuilt. The project retained both artifact identity and recipe.

Reproduction became a property with evidence, not a feeling produced by matching laptops.

The useful boundary was smaller than the hype

Docker arrived at an exciting moment and invited very large architectural predictions. X8B6 benefited from a narrower use.

It placed the service runtime, system dependencies, and startup command into a buildable image. It separated disposable test containers from persistent data. It made clean fixtures cheap enough to run repeatedly. It revealed hidden host assumptions and gave failing scenarios an environment manifest.

The project still had to pin inputs, define readiness, preserve logs, protect volumes, validate configuration, manage schema changes, and design security boundaries. A running container did not prove a current application. A tag did not guarantee immutable identity. A writable layer was not a backup.

Before the experiment, reproducing the receipt bug required nursing one laptop into a remembered state. Afterward, a named image and fixture could rebuild the conditions, run the dropped-response scenario, and disappear without taking the evidence with it.

Containers did not make X8B6 correct. They made one version of its incorrectness portable enough to understand and keep fixed.

That was enough of a breakthrough for 2014. The image replaced an increasingly fictional setup document with something the machine could execute, and the fixture made destructive experimentation safe because its lifecycle, boundaries, and evidence were designed together from the start.

The more interesting possibility is not one containerized production service. It is making a complete disposable environment cheap enough to attach to a specific change. A reviewer could receive the proposed revision, its exact image identities, representative data, and a URL, then discard the whole environment after the decision. That would turn environment setup from shared infrastructure into part of the review artifact. Containers do not supply the scheduler, routing, secrets, expiry, or evidence for that system, but they make the unit small enough to imagine building it.