HTTPS was the easy part

Automating certificates for a publishing system exposed the larger migration hiding behind one change of scheme.

In December 2015, obtaining a certificate stopped being the part of HTTPS I feared most. That was fortunate, because the certificate turned out to be the easy part.

M31V was a small publishing system built around Markdown, server-rendered templates, and atomic releases. Its previous deployment process involved manually moving files to a server and checking a handful of pages. We had already replaced that with versioned release directories and a single symlink switch. When Let's Encrypt entered public beta, automated certificates seemed like the final piece of a cleaner delivery path.

An early ACME client proved domain control, requested a certificate, and installed it. A test request returned a lock in the browser. The first checklist item was complete.

The site was not migrated.

The hostname inventory came before the certificate

M31V was not one hostname in practice. The public site used the bare domain and www. Images had once lived under an asset host. Preview used another name. Administrative publishing, feed callbacks, and monitoring each carried their own configuration.

I made a hostname inventory with ownership and purpose:

public canonical origin
legacy public alias
asset origin
protected preview
publishing administration
incoming callback endpoint
monitoring targets

For each, I asked whether it should receive a certificate, redirect to another host, remain private, or be retired. A certificate covering only the address I typed into the browser would leave alternate entry points broken or insecure.

The inventory also prevented automatically issuing certificates for forgotten names without understanding where DNS pointed. Domain validation proves control at one moment; it does not establish that the hostname belongs in the product forever.

I reduced the public set and made the canonical host explicit before broad redirects. Smaller identity surface made renewal, monitoring, and future strict transport policy easier to reason about.

Issuance had its own failure path

The ACME client needed to prove control of each hostname. That challenge path had to remain reachable while redirects, release routing, and maintenance pages changed around it.

I isolated the challenge location from ordinary publication releases and tested it through the real public route. A content deployment could not accidentally remove it. The validation response contained only the required token and did not expose a general writable directory.

The client ran with narrowly scoped file and process permissions. Private keys were created and stored outside the static release artifact, readable only by the server identity that needed them. Backups and transfer treated keys differently from public certificates.

I avoided copying key material into deployment logs or candidate manifests. The manifest recorded certificate identity and expiry for observation, not the secret.

Automating issuance reduced manual work and introduced a privileged automation path. The path needed the same questions as any other: what can it write, how does it fail, and what evidence proves it succeeded?

Years of content contained absolute http:// URLs. Some images came from services that did not offer HTTPS. Feed templates, canonical links, social metadata, and email previews each constructed URLs independently. A few scripts were protocol-relative, which had once felt flexible and now made ownership harder to find.

The browser reported mixed content, but the deeper problem was that the application's origin had been copied into too many layers.

We introduced one canonical site origin in configuration and made templates build internal URLs from paths. A content scan found absolute internal links and rewrote them carefully. External assets were classified rather than blindly replaced: migrate to a secure source, serve a local copy when licensing allowed, remove the dependency, or accept that the page could not yet be fully secure.

That inventory took longer than issuing the certificate. It also improved the system independently of HTTPS by removing several conflicting sources of truth.

Mixed content was a dependency map

The first secure page loaded with a warning because one image remained on HTTP. A script requested insecure data and was blocked entirely. The two failures looked similar in the browser and had different consequences.

I scanned source content, generated artifacts, stylesheets, scripts, feed templates, and runtime request code for insecure URLs. Static text search found obvious cases. Browser testing found URLs assembled at runtime and redirects that began secure but landed on an insecure asset.

Every dependency entered one of four states:

secure upstream available — update and verify
safe to host locally — copy with provenance and update policy
not essential — remove
no secure replacement — block migration for the affected surface

Blindly replacing http: with https: could point at a server that did not support TLS or served a different certificate. Protocol-relative URLs also delegated the decision without proving the upstream worked securely.

The process became a supply-chain inventory for the rendered page. M31V learned which third-party resources it depended on and which could disappear without harming the article.

Static scans could not exercise every old article and interaction. I introduced a Content Security Policy in report-only form focused on insecure resource behavior and unexpected origins, then reviewed violations on representative routes.

The reports were signals, not an automatic list of safe policy. Browser extensions and unusual clients could add noise. The collection endpoint received a bounded, privacy-conscious payload and did not become a permanent excuse to gather navigation history.

Once important dependencies were understood, enforcement could prevent insecure loads rather than relying on every template to remain perfect. The policy was tested against preview, public articles, admin, and callback responses separately because they had different needs.

I kept the directive set proportional to the migration. HTTPS did not require solving every content-security concern in one release, and a broad untested policy could break publication while appearing protective.

Report-only deployment helped reveal architecture. Enforcement followed after the allowed origins and recovery plan were credible.

Redirects are production behavior

The first redirect rule sent every HTTP request directly to its HTTPS equivalent. It worked until an old publishing callback reached a hostname the certificate did not cover. The callback followed the redirect, failed TLS validation, and silently stopped delivering updates.

We changed the rollout into stages:

  1. Serve both schemes while auditing application behavior.
  2. Update generated links and callbacks to prefer HTTPS.
  3. Redirect a limited set of paths and observe errors.
  4. Expand the redirect after dependent systems had migrated.
  5. Consider strict transport policy only after subdomains and recovery paths were understood.

This felt less elegant than one universal rule. It was operationally honest. A permanent redirect can outlive the deployment that created it through browser and intermediary caches. Security work deserved a rollback plan before it deserved a tidy configuration file.

We preserved request paths and query strings, used permanent status only when the mapping was truly permanent, and tested non-GET behavior instead of assuming every client handled redirects identically.

Public article requests were safe to redirect. Administrative form submissions and webhooks could carry bodies, signatures, and clients with different redirect behavior.

I updated those callers to use the secure endpoint directly before enforcing redirects. A callback test sent the actual method, body, headers, and signature through the route and verified what arrived. I did not rely on a browser address-bar test to prove an API migration.

For old HTTP mutation endpoints, the preferred outcome was refusal with an explicit migration response once known callers moved, not silently transforming requests under a status code every client might interpret differently.

Query preservation also received tests because feed and campaign parameters mattered to some read routes. Canonical metadata removed tracking from identity while the redirect preserved the reader's request according to policy.

The staged rollout separated public navigation, static assets, administrative sessions, and machine callbacks. “Redirect all traffic” became the final state, not the first experiment.

HTTP Strict Transport Security could tell browsers to use HTTPS for future requests without trying HTTP first. It also made a certificate or subdomain mistake harder to recover from because compliant browsers remembered the policy.

I delayed HSTS until every required hostname served valid TLS, renewal monitoring worked, redirects were stable, and the emergency certificate path had been rehearsed. I began with a short duration and did not include all subdomains automatically.

Preload-style permanence was outside the first migration. M31V had legacy and experimental subdomains that needed retirement or explicit secure ownership before inheriting a broad rule.

This was a good example of security strength requiring operational maturity. A stronger client promise is valuable only when the service can keep it through renewal failure, rollback, and personnel absence.

The lock icon appeared without HSTS. Strict transport behavior arrived after the supporting system earned it.

Secure cookies changed application assumptions

M31V's administrative session cookie gained the Secure flag so it would never travel over HTTP. We also tightened its scope and reviewed how the login redirect worked. That exposed a development habit: local environments depended on behavior unlike production, and developers sometimes disabled the security attributes to make the application convenient.

We created a local TLS path and kept the production cookie contract intact. The alternative—shipping different authentication semantics depending on environment—would have made the most sensitive path the least faithfully tested one.

The public site was mostly static, but the publishing service also accepted webhook requests. Those endpoints needed their own verification. HTTPS protected transport; it did not prove the sender. Signed payloads and replay protection remained separate responsibilities.

That distinction sounds obvious written down. The lock icon has a way of making several different questions feel answered at once.

Cookie review reached beyond Secure

The administrative session already had assumptions about path, host, expiry, and cross-site requests. Moving to HTTPS was an opportunity to narrow them rather than adding one attribute and declaring completion.

The cookie was host-scoped where possible, limited to the administrative path, inaccessible to client script when the application did not require it, and expired according to the actual session policy. Logout invalidated server state as well as asking the browser to remove a value.

The login flow regenerated session identity after authentication. Redirect destinations were allow-listed rather than accepted from an arbitrary parameter. Forms retained request-forgery protection; TLS did not prove that an authenticated browser intended a cross-site submission.

Local development used trusted local TLS and the same cookie semantics. Where that was impossible, tests exercised production policy independently instead of weakening the deployed cookie to match convenience.

HTTPS protected the cookie in transit. Session design still determined where it traveled and what possession allowed.

The certificate chain was part of compatibility

A modern browser on my laptop accepted the first configuration. Older clients and command-line tools exposed an incomplete intermediate chain. The server certificate was valid, but some clients could not construct trust from what the host supplied.

I tested the served chain from outside the server, not only the certificate files on disk. Verification included hostname coverage, expiry, signature chain, protocol negotiation, and representative client compatibility appropriate to the publication's audience in 2015.

I avoided maximizing compatibility by enabling old protocols and weak cipher choices without policy. Supporting a client was not useful if the connection no longer met the security goal. The accepted configuration followed contemporary server guidance and was recorded so later changes were reviewable.

This was another boundary where “the browser shows a lock” was indirect evidence. Different clients carried different trust stores and capabilities. The server had to present the intended chain consistently.

Certificate installation meant more than pointing at one PEM file.

A certificate that works today is a demo. Renewal is the product.

The renewal job ran automatically and wrote structured output. A failed attempt alerted before expiration became urgent. We tested the command against the staging service to avoid rate limits and performed a dry run after server changes. The web server reloaded only after the new files passed validation, leaving the current certificate active if installation failed.

The challenge was not writing a scheduled task. It was deciding who would notice it three months later. Ownership, alert routing, logs, and a recovery command were part of the certificate system.

M31V's atomic release pattern helped. Application deployments and certificate renewal were independent, so rolling back content did not roll back keys. A shared “deploy everything” script would have coupled two systems with different lifecycles.

Renewal wrote beside the current certificate

The renewal job never overwrote active key and certificate files piece by piece. It wrote a complete new set to a versioned location, checked permissions, parsed validity and hostnames, verified key correspondence, then changed the server's selected certificate through a controlled reload.

If any check failed, the current known certificate remained active. A failed reload left the old process serving rather than stopping the site to adopt bad files.

The job recorded:

renewal attempt identity
certificate names and expiry
challenge result
validation result
reload result
externally observed certificate identity
next scheduled check

An external probe confirmed what the public host actually served. Successful file generation on disk did not prove the server process had adopted it.

This reused M31V's release instinct: prepare an inactive candidate, verify it, then make the smallest controlled activation.

Alerting one day before expiry would turn an automated failure into an emergency. M31V monitored remaining lifetime with several thresholds and treated repeated renewal failure as urgent while the current certificate still had time.

The alert named affected hostnames, last successful renewal, recent failure reason, current served expiry, and the safe manual command. It did not send only “certificate error,” which would force the responder to reconstruct the system under time pressure.

I tested the alert path independently. A renewal log stored on the same failed host was not sufficient notification. The external monitor came from another boundary and checked the public endpoint.

Staging-service tests exercised the ACME flow without consuming production limits. Periodic dry runs detected broken permissions or challenge routing before the real renewal window.

Automation became trustworthy when failure was expected early enough to recover calmly.

The migration found old architecture

HTTPS forced us to inspect places the normal publishing flow did not exercise:

  • RSS readers with cached feed addresses.
  • Social cards built by a separate template.
  • Old redirects left from a previous URL structure.
  • Images embedded in archived posts.
  • Administrative bookmarks and callbacks.
  • Monitoring that checked only the HTTP endpoint.

Each one was a small piece of forgotten architecture. Security migrations are good at finding those pieces because they change an assumption beneath the entire product.

Changing the feed's self URL and article links could make some readers interpret existing entries as new if entry identity was tied incorrectly to the URL. M31V kept stable entry IDs while updating canonical links to HTTPS.

The HTTP feed URL redirected permanently to its HTTPS equivalent after reader tests. The generated feed declared the secure self URL, and every article link used the normalized canonical origin.

Email templates and social previews were separate renderers with their own absolute URL requirements. Artifact validation compared their hosts and schemes with the content graph. An image used by an email client needed public HTTPS reachability even if the browser page no longer referenced it.

This work justified the earlier one-source content graph. The migration changed one origin input and produced a report of every affected output rather than searching several independent templates blindly.

Identity survived the scheme change because IDs and content relationships did not depend on copied URL strings.

Monitoring had to follow both schemes during rollout

The old monitor requested the HTTP home page and considered a redirect healthy because it received a response. It never completed TLS or checked the final page.

During migration I kept separate probes:

HTTP entry -> expected redirect and preserved path
HTTPS origin -> valid chain and expected build identity
canonical article -> secure assets and metadata
administrative login -> secure cookie behavior
callback endpoint -> direct secure delivery and signature verification

Once enforcement completed, an unexpected successful HTTP content response became a failure rather than a sign of availability.

The external probe recorded certificate identity and expiry. Application health and certificate health remained related but distinct: a valid certificate could front a broken release, and a healthy process could serve an expired certificate.

Monitoring evolved with the product contract instead of continuing to ask the easiest old question.

Recovery was rehearsed before strictness

I wrote and tested the emergency paths:

  • Restore the last known certificate selection after a bad renewal.
  • Correct challenge routing without changing the active content release.
  • Disable an HTTP/2 configuration problem while keeping HTTPS.
  • Roll back redirect expansion for a dependent callback before permanent caching spread.
  • Serve a minimal secure maintenance response if the publishing application failed.

Not every recovery could undo client state. A cached permanent redirect or HSTS policy might persist after the server changed. That was why rollout started with limited paths and short-lived policies.

The runbook listed which actions were reversible at the origin and which required waiting for client caches or issuing a new valid certificate. It named who controlled DNS and where recovery credentials lived without placing secrets in the document.

Security controls became safer to strengthen after the recovery boundary was understood.

I had expected cryptography to be the difficult subject. The cryptography was implemented by people who understood it far better than I did. My work was the system around it: configuration, content, dependency mapping, rollout, renewal, observation, and recovery.

The migration changed the build contract

M31V's candidate verifier gained security invariants:

all internal canonical URLs use the declared secure origin
public artifacts contain no insecure active-resource references
redirect sources and destinations preserve the HTTPS policy
feed, sitemap, email, and social metadata agree on origin
preview-only hosts do not leak into public output
forms and callbacks use explicit secure endpoints

Host verification added certificate and response checks the static build could not perform. Keeping the layers separate made failures actionable. A source URL error belonged to the artifact; an incomplete chain belonged to deployment.

The build failed rather than silently upgrading arbitrary external URLs. Every secure dependency was either verified, locally owned, or deliberately removed.

The scheme became one controlled input instead of a string repeated across templates. Future host migrations would benefit from the same discipline.

HTTPS work left the publication architecture cleaner because it turned origin assumptions into declared contracts.

That was a useful correction. “Enable HTTPS” sounds like changing a server setting. In a living product, a scheme participates in identity, caching, cookies, links, integrations, and operations. The certificate makes secure transport possible. The migration makes the product consistently use it.

The limitations remained worth stating. TLS did not make published content correct, authenticate webhook senders, prevent application authorization mistakes, secure compromised reader devices, or make third-party scripts trustworthy. It protected communication to an authenticated host under the certificate and protocol policy.

M31V also depended on automated issuance infrastructure, DNS control, server configuration, and someone responding before expiry. Automation reduced recurring labor and concentrated the need for good monitoring and narrow privileges.

The project succeeded when HTTP became only an intentional entry point to the secure origin, public artifacts agreed on HTTPS identity, administrative sessions retained their security contract, and renewal could fail visibly without immediate outage.

The lock appeared on the first afternoon. Confidence took longer.