The Fix That Never Shipped | The Foundry

For six days this month, our website answered every request with a 404 Not Found.

The server was healthy. The TLS certificate was valid. The application container was running. The code that fixed the underlying bug had already been written, reviewed and merged, six days before anyone noticed the outage. It just never reached the cluster.

This is the full write-up. We have written it so that it teaches the Kubernetes concepts as it goes, because the failure only makes sense once you understand how the pieces are supposed to fit together. If you run Kubernetes, we think you will recognize the shape of this one.

First, how a request is supposed to reach an application

Kubernetes does not route traffic the way a traditional web server does. Four separate objects have to agree before a visitor sees your page, and each one can be individually healthy while the chain as a whole is broken.

Kubernetes primer: the request path

Pod: The smallest unit you can run. A Pod holds one or more containers that are always scheduled together on the same machine. Critically, containers in a Pod share one network namespace, so they reach each other on localhost, exactly as if they were two processes on a single server.
Service: A stable internal name and virtual IP that load balances across a changing set of Pods. Pods are disposable and their IPs change constantly. A Service is the fixed address that other things talk to.
Ingress: A rule, not a program. It says "requests for this hostname and path go to that Service." On its own it does absolutely nothing.
Ingress controller: The program that actually watches for Ingress rules and turns them into a real proxy configuration. This is the piece that receives traffic from the internet. Without a controller reading your Ingress, the rule is just a row in a database.

That last distinction is the one that cost us six days, so it is worth stating plainly. In Kubernetes you do not configure a load balancer directly. You write down what you want and hope that something is listening. Usually something is.

A 404 is an answer, not a silence

When a site returns 404, the instinct is to ask what is missing. That is usually the wrong question. A 404 means something received the request, made a decision and declined. The better question is who answered?

Our application serves a small health endpoint that returns the literal text ok. We asked the live site for it and got back a 200 with an empty body.

That mismatch was the whole investigation compressed into one request. A 200 meant something was alive and willing to respond. An empty body meant it was not our application, because ours would have said ok. What replied was the ingress controller's own built-in default backend: the answer a request gets when it reaches the front door and no rule inside claims it.

Our application was never asked. For six days every request had been stopping one layer short of it, at a layer healthy enough to answer politely.

This is a technique worth stealing. Give every service a health endpoint that returns a distinctive body, not just a 200. When something in the chain answers on its behalf, the wrong body tells you instantly which layer you are actually talking to. A status code alone would not have.

Three failures, stacked

Behind that 404 sat three separate problems. Any one of them alone would have been a routine afternoon.

A container that could never start.
Our web tier and our API run as two containers in a single Pod, so they share a network namespace and should reach each other on localhost. The web server's configuration proxied to the API by Service name instead. That name does not resolve from inside the Pod, and an unresolvable upstream is a fatal startup error for a web server, not a degraded state. It refused to start, then refused again on every restart, thousands of times over six days.
```
proxy_pass http://api:8000/;        # wrong: a Service name,
                                    # resolved from outside the Pod

proxy_pass http://127.0.0.1:8000/;  # right: same Pod,
                                    # same network namespace
```
The lesson generalizes. Containers in the same Pod talk over localhost and must use distinct ports, because they share one port space. Containers in different Pods talk over Service names. Mixing up which situation you are in produces an error that looks like DNS but is really about topology.
An Ingress rule that nobody owned.
Every Ingress carries an ingressClassName that says which controller should handle it. Ours named a controller we do not run.

Nothing catches this for you. The API server accepted the resource because it is structurally valid. It appeared in listings looking completely normal, with a hostname and an address. The controller we do run read it, saw it was addressed to a different controller and correctly ignored it. So traffic arrived at a front door with no rules behind it, and the default backend answered, exactly as designed.
A rollout that gave up.
A Deployment tracks its own progress and gives up after a deadline, marking itself with a ProgressDeadlineExceeded condition. Because the container in failure one could never become healthy, our Deployment carried that marker permanently.

Our delivery controller checks for exactly that condition before it commits an update. It found it, concluded the rollout had stalled and aborted, in a fraction of a second. After exhausting its retry budget it stopped trying altogether. This is the deadlock worth noticing: the broken state was itself the reason the fix for that state could not be applied.

Here is the uncomfortable part. The first two had already been fixed. Both corrections were sitting in our main branch, merged and green, before the outage even began.

The root cause was a default

We deploy with GitOps. Rather than pushing changes into the cluster from a pipeline, a controller inside the cluster continuously pulls from git and works to make reality match what it finds there.

Kubernetes primer: reconciliation

Almost everything in Kubernetes is a control loop. A controller compares desired state against observed state, takes one step to close the gap, then repeats forever. This is why Kubernetes self heals: kill a Pod and the loop notices the gap and makes another one.

GitOps extends that loop past the cluster boundary so git becomes the desired state. The property you are buying is convergence. Whatever happens, the system should drift back toward what the repository says. That guarantee is only as good as the controller's definition of "changed."

Our application is packaged as a Helm chart that lives in the same repository as the application code. The setting that governs when the controller rebuilds that package was left at its default:

chart:
  spec:
    chart: ./charts/app
    # reconcileStrategy not set,
    # so it defaults to: ChartVersion

ChartVersion means rebuild the package only when the package's own version number changes.

Ours was set to 0.1.0 the day we bootstrapped the platform and never moved. Bumping it was not part of our release process, because our release process identifies builds by image tag instead. Two conventions, each reasonable alone, quietly incompatible together.

So the controller did exactly what we asked. On every commit it fetched the repository and compared the chart version it found against the version it had already built. Both said 0.1.0. It concluded there was nothing to do and logged success. Every change we made after the initial deploy was fetched, stored and discarded.

This is the part worth internalizing. Nothing was broken. Source control was correct. The build was correct. The pipeline was correct and reported success, truthfully, every single time. Git described one system, the cluster ran a different one, and both were internally consistent about it. There was no contradiction anywhere for a human to trip over.

The correction was one line:

chart:
  spec:
    chart: ./charts/app
    reconcileStrategy: Revision

Revision keys the rebuild on the git commit rather than the version number. With that in place, the package rebuilt against current source. The two fixes that had waited six days finally applied, the containers started, and the Ingress was claimed by the controller that actually exists. The site recovered on the first attempt.

The signal that pointed the wrong way

We were not flying blind. The delivery system had been reporting a failed release the whole time. It reported it as an application problem, though: a Deployment that would not become healthy.

That was true, and it was the wrong place to look. The reason it would not become healthy was that it was still running a build from before the fix. An accurate error describing a real symptom held attention on the symptom, while the layer that was actually wrong kept reporting success.

When a system tells you something is broken, confirm which layer is speaking. The loudest failure is not always the causal one.

Two things we found on the way

A credential had expired, and caching hid it. One credential used to fetch our own build artifacts had stopped working. We had not noticed, because the previously fetched artifacts were still cached on the node and the system was configured to prefer the cache. Nothing needed the credential, so nothing reported it broken. It would have surfaced as a hard failure at the worst possible moment, the first time we tried to ship something new, during an incident. We replaced it before rolling forward.

If you set an image pull policy that prefers a local cache, you have also disabled a continuous test of your registry credentials. That is usually a fine trade. It is worth making on purpose.

Recovery tooling deserves rehearsal. Restoring access to the machine, confirming what was actually running on it and comparing that against what git claimed took longer than the fix did. That ratio is normal. It is also the ratio worth attacking, because it is pure overhead on every future incident.

What we took away

A green pipeline is not a shipped change. "The job succeeded" and "the change is running in production" are different claims. Only one of them matters, and it is the one you have to go and verify.
Defaults encode someone else's assumptions. Versioning on every change is right for charts published to a registry. It is wrong for a chart living beside the application it deploys. The default was not broken. It was answering a question we were not asking.
Ask who answered, not what is missing. Identifying which component produced a response narrows an outage faster than any amount of reading the response itself.
Configuration that validates is the dangerous kind. Naming a controller that does not exist was legal, accepted and displayed as healthy. Errors that fail loudly get fixed in minutes. Errors that pass review and then do nothing last six days.
A resource existing is not the same as a resource working. This is the Kubernetes lesson underneath all of it. Listings show you what has been declared, not what is acting on it. Get comfortable asking which controller owns a given object, and what it has done about it lately.
Diff running state against declared state deliberately. One manual comparison of what the cluster was running against what the repository said would have exposed the drift immediately. We now check.

Why we published this

The Foundry is where we keep our prototypes, our testing, our documentation and our lessons learned. That includes the lessons that cost us something.

We build and operate infrastructure for organizations running systems they cannot afford to have quietly serving a six day old snapshot. We would rather show you how we reason through a failure than pretend we do not have any. A delivery pipeline reporting success while shipping nothing is not an exotic failure. If you run continuous delivery, it is worth twenty minutes today to confirm that what is running is what you think you shipped.

Book a Technical Discovery Call See Our Services

A.M. Jones Founder, Maxtin Engineering