Host Protection Blocked Our Deploy: Why a Healthy Container Failed Readiness

The container was healthy. The deployment still stopped.

Our deployment log looked contradictory: the new containers had started, Nginx accepted its configuration, and then the release was rejected:

Candidate blue did not become ready; the public upstream was not changed.
Candidate was not activated; the previous HTTP release is still serving.

This happened after we tightened hostname validation to stop unfamiliar domains from displaying DeployWatchers. The protection did its job. An internal deployment request was now using a hostname the application was supposed to reject.

The fix was to keep connecting directly to the candidate container while sending the application's configured public HTTP Host. We kept the allowlist intact and corrected the probe.

This is the deployment side of our earlier investigation into a foreign domain serving our website. That article covers the origin protection; this one follows the readiness failure it exposed.

What our two checks actually proved

Our application container runs PHP-FPM. Its Docker healthcheck validates the FPM configuration and opens a socket to its local listener. That is useful evidence that the process can start and accept a connection. It does not send a complete HTTP request through the candidate web server and Laravel.

The deployment script performs another check before changing public traffic: it requests /health through the candidate web container and expects HTTP 200 with JSON containing status: "ok".

Check in our deployment	What it exercises	What it does not establish
Application container healthcheck	FPM configuration and a local TCP listener	HTTP routing, Host validation or application response
Candidate readiness request	Candidate web server and application health route	The public CDN/TLS path or every customer workflow
Check after switching	The gateway's selected release and its health response	Complete correctness of the release

These are properties of our chosen checks, not fixed meanings imposed by Docker. A Docker healthcheck can itself make an HTTP request if you configure it that way. Docker runs the configured test; its healthy label is only as comprehensive as that test. Docker Compose healthcheck reference ↗ (opens in a new tab)

Consequently, healthy plus failed readiness was entirely possible. The two checks were asking different questions.

The hidden assumption in the internal URL

The candidate lived behind a Docker service name such as web-blue. A request to this address reached the right container:

# Run inside a container attached to the candidate's Docker network.
curl --noproxy '*' --connect-timeout 2 --max-time 5 \
  -sS -o /dev/null -w '%{http_code}\n' \
  http://web-blue/health

But the request also identified its HTTP host as web-blue. After the Host guard was enabled, that name was no longer acceptable to the application.

The network destination and the HTTP hostname describe different things. Docker DNS can resolve a service name without making that name a valid public identity for the application. Nginx also uses the request hostname when selecting a virtual server; reaching an IP address or port does not establish which site will handle the request. Nginx request processing ↗ (opens in a new tab)

In our deployment, the hostname restriction existed in Laravel as well as at the public origin boundary. Correcting only the host-level Nginx configuration would not fix a direct candidate request that Laravel rejected.

The narrow symptom matters: a fast HTTP 421 points toward a hostname/routing rejection in this setup. It calls for a different investigation from a connection refusal, timeout, HTTP 401 or upstream 502. Those responses can have other causes; none should be diagnosed solely from the status number.

Keep the destination; correct the request identity

For a private HTTP hop to a candidate, the diagnostic correction is small:

# Example public hostname: app.example.test.
# Run from the same Docker network as web-blue.
curl --noproxy '*' --connect-timeout 2 --max-time 5 \
  -sS -o /dev/null -w '%{http_code}\n' \
  -H 'Host: app.example.test' \
  http://web-blue/health

This still connects to web-blue; it cannot accidentally test the currently active public release through DNS. The changed header lets the candidate receive the hostname that the real application expects.

The status-only command is for diagnosis. Our release gate also parses the response body. HTTP 200 containing a generic web-server page must not count as application readiness.

For our implementation, the hostname comes from server-controlled APP_URL, rather than an arbitrary request header or user input. The helper validates the candidate name against the two known services, parses the configured hostname and optional port, and constructs the Host header. A nonstandard public port must be handled deliberately, not dropped accidentally.

Another way to express the same separation with curl is --connect-to. Keep the intended hostname in the URL and override only the connection target:

# Private HTTP candidate probe; no public DNS lookup for app.example.test.
curl --noproxy '*' --connect-timeout 2 --max-time 5 \
  -sS \
  --connect-to app.example.test:80:web-blue:80 \
  http://app.example.test/health

For a candidate that terminates HTTPS itself, keep the public hostname in an HTTPS URL and map its connection to the candidate's TLS port. That preserves the intended TLS name as well as the HTTP hostname. Merely overriding Host on an HTTPS URL containing an internal service name does not fix TLS SNI or certificate-name validation. Do not add -k to make a production readiness gate pass. curl connection target reference ↗ (opens in a new tab)

The HTTP examples above describe an internal hop in our architecture. They do not replace a separate check of the public HTTPS endpoint.

What we required before accepting a candidate

Changing the header solved this specific rejection. The surrounding gate still needed to reject other bad responses.

Our candidate helper requires an exact HTTP 200 response and parsed JSON with status equal to ok. It disables automatic redirects and uses a bounded request timeout. A redirect can otherwise conceal a wrong hostname by sending the probe to the active release or a login page.

On the test environment, Basic Auth is another boundary. Where the probe legitimately needs it, our helper obtains its credentials from server-side environment configuration. We do not place passwords in the URL, publish them in examples or print authorization headers in deployment logs. A 401 should lead to checking that boundary, not weakening the Host policy.

We retained three constraints:

• The connection must reach the candidate being evaluated.
• The request must use the configured application identity and required authentication.
• The response must satisfy the application's health contract, within the deployment deadline.

Adding every Docker service name to the production Host allowlist would blur these boundaries. Making a health URL return success before it exercises the relevant application path would conceal the failure instead.

Likewise, increasing the readiness timeout would not solve a deterministic Host rejection. Repeating the same invalid request for another minute still tests the wrong request.

Why the rejected release did not replace the working one

The most reassuring line in the log was that the public upstream had not changed. The candidate check occurred before activation, so the previous HTTP release remained in service.

Our intended sequence is:

1. Start the candidate slot while the existing slot continues serving.
2. Wait for container health and candidate HTTP readiness.
3. Prepare the gateway configuration for the candidate.
4. Validate and gracefully reload the gateway.
5. Confirm the selected slot through the gateway before retiring the previous slot.

Nginx reload can start new workers while existing workers finish serving their clients. That behavior supports the switch, but it does not establish that a new application release is ready or make incompatible database migrations safe. Nginx reconfiguration documentation ↗ (opens in a new tab)

In the broader deployment regression fixture, we also check a request that spans a switch, a rejected candidate, rollback and retained static assets. These are separate failure modes from the Host header. Passing this readiness check alone is not a promise of zero downtime under every production condition.

How we reproduced the failure and tested the correction

We used two local experiments, with no production configuration changes.

First, a small Nginx fixture isolated the HTTP identity problem. It returned a candidate JSON marker only for the configured hostname and rejected other hosts with 421. A synthetic container healthcheck remained healthy throughout. This fixture models the request boundary; it does not run Laravel or pretend to reproduce PHP-FPM health.

Local probe	Observed result
Container's synthetic healthcheck	Healthy
HTTP Host web-blue	421
HTTP Host app.example.test	200 with status: "ok"
Allowed hostname with an explicit port	200 with status: "ok"
Unrelated hostname	421
Canonical URL with a mapped connection target	200 with status: "ok"

Second, our repository's isolated deployment regression uses real Nginx and PHP-FPM containers and the actual blue-green deployment helpers. Its PHP fixture enforces the configured hostname and supplies versioned health responses. It is deliberately smaller than the full Laravel application: there is no production database, customer traffic or external payment integration. In this run, all 502 sampled traffic requests succeeded; the spanning request, rejected candidate, rollback and asset-retention assertions also passed. This is a local test result, not a production availability measurement.

Together, the tests answer two useful questions: can a healthy process still reject a probe's HTTP identity, and does the deployment machinery accept a correctly addressed candidate while keeping a failing candidate out of the active route?

What to inspect when healthy containers fail readiness

Start by capturing the candidate's HTTP status without following redirects. Then compare the probe's destination, hostname, scheme, authentication and path with the route you intended to exercise. Do this from the same network context as the deployment runner; a Docker service name usually will not resolve from your laptop.

Record the candidate slot, failure category, HTTP status, attempt count and elapsed time. Keep response bodies, cookies, authorization headers and environment dumps out of routine deployment logs. If you need a body for investigation, inspect it privately with appropriate redaction.

Finally, test both sides of the policy: the canonical hostname must succeed and an unrelated hostname must still fail. A readiness correction that silently disables hostname validation is a security regression.

For us, the stopped deployment exposed a stale assumption in the probe. Host validation stayed enabled, the candidate check learned to send the right identity, and the gate continued to protect the working release from a candidate it could not verify.