Atomic deployment in four boring steps

M31V's deployment script once generated the site and copied files directly into the live document root. It was short, understandable, and capable of publishing a half-new site.

If transfer stopped after the archive but before the article, readers received a new link to a missing page. If an asset filename changed before the matching HTML arrived, the active release pointed at a file not yet present. A second run often repaired the mixture, which made the process feel more reliable than it was.

I replaced the one clever command with four boring states:

build
upload
verify
activate

Each state produced evidence the next one required. None except activation could change what readers saw.

Build created a sealed candidate

The build ran in a clean workspace with an explicit publication clock and pinned project toolchain. It normalized source, rendered every public surface, generated assets and redirects, and wrote a manifest.

The result lived in a directory named by build identity. After sealing, deployment did not edit it. A source change required another build.

The manifest included:

build ID and source revision
publication clock
generated path and asset inventory
content and redirect counts
file size and checksum records
validation result
tool and configuration versions

The candidate passed local artifact checks before upload. That prevented using the release host as a slow validator.

Build answered one question: do these inputs produce a complete artifact eligible for transfer? It did not claim the host had received or could serve it.

Upload targeted an inactive directory

Every candidate uploaded to a new release path:

/srv/pressroom/releases/build-b071/

The active document root pointed elsewhere. A partial directory was invisible to ordinary requests.

The uploader created a transfer marker, copied files, then wrote the final manifest last. Presence of a directory did not mean transfer was complete. A retry resumed or replaced only that inactive candidate under strict identity checks.

The script refused to upload one build ID over different existing contents. If the directory already held a verified matching manifest, upload could be skipped. If identity matched but bytes did not, the candidate was quarantined and recopied under a clean path.

Upload answered: did the release host receive the candidate bytes? It did not turn those bytes into the current site.

Verification read from the host

Local validation could not prove the transfer was complete, permissions were correct, or server routing matched the artifact.

Verification ran against the inactive release through a host-specific candidate URL or local virtual-host mapping. It checked file inventory against the manifest and requested representative routes.

home page and one article
author and topic archives
feed and sitemap parsing
fingerprinted image and stylesheet
not-found response
redirect configuration in candidate mode
build identity response

The checks verified content type, encoding, expected canonical metadata, and body identity where useful. They did not rely only on a directory checksum; the web server had to interpret paths correctly.

A failed check left the current release untouched. The candidate remained available for diagnosis or could be discarded.

Verification answered: can this host serve this inactive artifact under the required contract?

Activation was one pointer change

The host exposed a stable active path through a symbolic link or equivalent release pointer:

/srv/pressroom/current -> /srv/pressroom/releases/build-b071

Activation created a new link and renamed it into place atomically on the same filesystem. Requests saw either the previous target or the new target, not a pointer edited character by character.

The script recorded the previous build ID before switching. It then requested public routes and confirmed that the active identity matched the candidate.

Activation did not copy article files, run transformations, or rewrite configuration. All expensive and fallible work had happened while inactive.

This made the public transition tiny enough to reason about. It did not make every cache update instantaneous, but the origin's active release changed as one named state.

Rollback reused the same operation

The old deployment plan imagined rollback as uploading old files again. That repeated the risky transfer while the site was already degraded.

M31V retained the prior verified release. Rollback changed the active pointer back and ran the same public smoke checks.

current: b071
previous verified: b070
rollback -> activate b070

The old artifact had already been served and verified. It remained immutable after deactivation. Rollback did not rebuild from source under a potentially broken new toolchain.

This had a limit. A static artifact could roll back cleanly only if external services, cache policy, and host configuration remained compatible. M31V recorded those dependencies and avoided coupling article releases to irreversible state changes.

Fast rollback was a property of retained artifacts and small activation, not a special reverse script.

Two deployments starting together could each upload safely and then race to activate. The later pointer change might publish an older source revision after a newer one.

Upload and verification ran concurrently for distinct candidates if desired. Activation acquired a short host lock, read current identity, checked the candidate's expected predecessor or release ordering policy, and switched.

The lock did not cover a long transfer. That would turn an abandoned session into a prolonged deployment outage. It protected the critical state transition only.

If the current release changed after candidate verification, the deployment reevaluated whether activation was still allowed. A candidate could be rejected as stale or require renewed smoke checks against shared redirect configuration.

The lock file carried owner, start time, and command identity for diagnosis. Force removal was deliberate and recorded.

Concurrency became a release-order decision instead of whichever upload finished last.

Activation identity was observable

Every public response could be connected to the active build through a safe diagnostic header or protected metadata endpoint. The footer did not need a visible hash, but deployment checks could ask the host which release it served.

This caught a configuration mistake where the pointer changed correctly but the server still used a cached document root from another process.

The deployment log recorded:

candidate identity
previous active identity
activation time
operator or automation identity
public verification result
rollback target

When a reader reported an issue, the build identity narrowed investigation to one immutable artifact. Source revision alone was insufficient if build configuration or publication clock differed.

Observability made activation a fact the system could prove, not a command that probably ran.

The origin pointer could switch atomically while browsers or intermediaries retained documents from the prior release. Fingerprinted assets were safe because old HTML continued referencing old asset names retained with the old release or in shared immutable storage.

HTML used a shorter revalidation policy. Activation did not purge every cache blindly; it used targeted policy where the hosting layer supported it and tolerated a documented transition window.

The critical invariant was cross-release compatibility:

• Old HTML assets remained available while old HTML could be cached.
• New HTML referenced only assets uploaded before activation.
• Redirect changes did not remove destinations still needed by cached pages.

M31V did not claim every reader changed versions in one instant. “Atomic deployment” meant the origin never exposed a partially assembled candidate. Cache coherence required its own design.

Naming that limit prevented a strong local filesystem guarantee from being exaggerated into global simultaneity.

Keeping each release fully self-contained made rollback simple but duplicated large assets. Moving fingerprinted assets into shared storage saved space and introduced cleanup risk.

M31V allowed shared immutable assets only when their names contained content identity. Upload placed them before candidate verification. Existing same-name files had to match bytes.

A retention scanner counted references from the active release, rollback set, and cached-document window before deleting an unreferenced asset. It never removed assets merely because the newest artifact did not use them.

The simpler default remained release-local ownership for small files. Optimization followed actual storage pressure.

Asset cleanup was treated as a separate maintenance job, not part of activation. A failed cleanup could waste storage without damaging the release; a mistaken cleanup inside deployment could break it.

Separating lifecycle again made failure less consequential.

The first deployment activated HTML with the new build and updated server redirects in a separate command. A route could briefly have new content and old URL behavior.

M31V generated a redirect map as part of the artifact and verified it in candidate mode. Activation switched the content and the server's selected redirect configuration through one release identity where the host allowed it.

If host configuration required a reload, the process staged the new config, syntax-checked it, changed the pointer, performed the bounded reload, and verified both active identity and redirects. Failure triggered the known recovery path.

The operation was no longer a pure filesystem rename, so the deployment report stated that honestly. Content activation remained one pointer; host adoption had an additional checked boundary.

URLs were part of the publication. Their release could not be left to an unrelated manual step.

A transferred file could exist and remain unreadable to the server process. A directory could lack traversal permission. Local checks would pass.

Upload applied permissions according to a narrow policy and refused unexpected executable bits. Verification requested through the server rather than reading only as the deployment user.

The release directory became read-only to ordinary deployment operations after verification. Activation did not require mutating it. Logs and runtime state lived elsewhere.

This prevented a public process from writing into article files and kept later manual fixes from quietly changing an identified artifact. An emergency edit required a new build or produced an explicitly tainted release that could not masquerade as the original manifest.

Immutability was partly social convention and partly file permissions, backed by checksum verification.

Failure at each step had a different recovery

The four-step model earned its value when something went wrong:

The script persisted its state and candidate identity. Rerunning did not blindly start from scratch or assume the previous step finished.

If activation command lost its connection after switching, the result was outcome unknown. The recovery first queried active identity. It did not switch again and risk undoing success.

The same distributed-systems lesson from X8B6 applied: a timed-out deployment command was evidence about the command session, not proof the pointer had not changed.

A partial upload became ordinary debris

Under direct-to-live deployment, partial upload was an incident. Under release directories, it was inactive debris with storage cost.

Candidate directories carried markers:

uploading
uploaded
verified
active
retained
rejected

A cleanup job removed abandoned uploading candidates after a safe age and lock check. It never inferred state from directory modification time alone.

Failed verified candidates could be retained temporarily for diagnosis. Their manifest and logs made them explainable. They had no route to activation without passing the gate again.

This was a design objective: make common failure boring by ensuring it happens outside the public pointer.

Reliability often comes from reducing the consequence of failure rather than preventing every interrupted copy.

Smoke checks represented reader tasks

The first post-deploy check requested / and looked for status 200. M31V could serve the old home from a cache while the new article path failed.

Candidate and public smoke suites followed representative tasks:

open home and follow newest article
open topic archive and follow article
fetch feed and resolve newest entry URL
request one old URL and follow redirect
request fingerprinted image referenced by article
request a missing path and inspect not-found behavior

Checks asserted build identity where possible and parsed links from the actual response. They remained small enough to run during activation.

The complete link and schema suite ran before upload. Smoke checks verified the host path and public transition, not every editorial rule again.

Different gates supplied evidence at the boundary they could observe.

Manual steps became explicit approvals

The early deployment required remembering to update a pointer, clear a cache, and preserve the old folder. Automation removed memory work but retained a deliberate activation approval while the project was still operated manually.

The command displayed candidate summary, changed surfaces, verification status, current build, and rollback target. Confirmation named the candidate ID:

Activate build b071, replacing verified build b070? [type b071]

This was not security against a malicious operator. It protected against activating the wrong terminal's candidate.

Later automation could replace human confirmation with a signed policy decision. The state machine remained the same. Approval authorized one identified candidate; it did not authorize “whatever the directory contains when the command runs.”

Intent received identity just as the artifact did.

M31V retained the active build, immediate rollback, several recent verified releases, and candidates pinned to incidents or scheduled work.

Cleanup checked the active pointer and deployment locks before deletion. It removed preview-only and failed candidates under shorter policies. Manifests could be archived longer than full bytes.

Disk pressure did not permit deleting the only rollback target automatically. The system warned and stopped accepting new uploads before sacrificing recovery invisibly.

Retention numbers were modest for a small publication. Writing them down prevented an old cron job from deleting directories according to age while one happened to be active.

Release storage was not a random cache. It carried recovery value that changed with state.

Build trusted source and toolchain enough to produce a candidate. Upload trusted transport enough to copy bytes. Verification trusted the release host enough to serve the inactive candidate. Activation trusted a tiny atomic operation enough to change public state.

Combining them into one command line was fine for convenience as long as the internal states, evidence, and stop conditions remained visible. A deploy command could orchestrate the four without flattening them into “success” or “failed.”

The deployment log could end at verified and wait for approval. It could retry upload without rebuilding. It could recover an uncertain activation by reading the pointer.

This made the workflow composable because every step had a durable product and a narrow claim.

Boring was the breakthrough

Nothing in M31V's deployment was conceptually exotic. Build a complete directory. Copy it somewhere inactive. Prove the host can serve it. Change one pointer. Keep the old pointer target.

The improvement came from refusing to let intermediate states become public. An interrupted upload became cleanup. A failed verification became a rejected candidate. A rollback became another activation of known bytes. A timed-out command became a query for active identity.

“Atomic” described the smallest critical transition, not the whole distributed experience. Browsers could cache old HTML, host configuration could need reload, and public smoke checks could still fail. Those limits received explicit compatibility and recovery policy.

The four steps made each problem occur in a place where it could be named and contained.

I trust boring deployment sequences because they leave little room for optimism. Build, upload, verify, activate. If a process cannot tell me which of those happened, it is not yet ready to publish.

The sequence also scaled down well. It required no elaborate platform: a generator, an inactive releases directory, a manifest, a way to serve a candidate, and one carefully controlled pointer. The discipline came from state boundaries, not infrastructure prestige. That made it appropriate for M31V and durable enough to influence every later workflow in which proposed work had to become current without exposing the proposal halfway through to people who depended on its integrity every single day.