Why the migration ran before the database was up

A project that has been running for months never shows this problem, because nothing in it starts at once. The database was created in March, the API in April, and every deploy since has replaced one app next to services that were already answering. Start order was solved by the calendar.

Then someone duplicates the project for a load test, or a pull request opens a preview, or a template stands up a fresh copy, and every piece arrives in the same second. The API's pre-deploy migration dials Postgres, which is still initialising its data directory, gets connection refused, and exits 1. The rollout is blocked on a failed migration. Nothing will retry it until something about the app changes. The preview comment says Building for every app, which was true for about a minute.

Kubernetes does not order anything

Docker Compose has depends_on, with a condition: service_healthy that waits for a health check. Kubernetes has no equivalent, deliberately. Its answer is that every process should tolerate its dependencies being absent: crash, restart with backoff, and eventually find them there. For a long-running server that is roughly true. It costs a few restarts and a red badge that goes green.

It is not true for the two kinds of work a fresh project runs first. A migration is a Job, and a Job that fails once has failed, because retrying half a migration is worse than not running it. And a platform that watches health will, quite correctly, report the crash-looping app as degraded, so a project seconds old pages someone about an outage that is really just a startup.

The usual workaround is an init container that loops on nc -z db 5432. That is closer, and it has to be written into every app, it knows a host and a port that were copied from the connection string and drift from it, and it answers something is listening, which is not the same question as is the database the app is about to use ready.

Derive the edge, do not declare it

The first design decision is where the dependency comes from. Asking people to declare it, a dependsOn list per app, produces a second description of something the configuration already says. The API's environment already contains DATABASE_URL from the Postgres service, or ${{ db.HOST }}. That reference is the dependency. Deriving it on every save means the order is never stale, never forgotten on the new app, and never a field anyone has to learn.

Not every reference means the same thing. A database the app reads on boot is required: starting without it is starting into a certain crash. A sibling app it calls on some requests is optional: worth waiting for briefly, not worth waiting for forever. Two apps that reference each other cannot both wait, so they start alongside one another and the cycle is broken rather than deadlocked.

Ready means the thing you will dial

The second decision is what ready means, and the useful answer is narrow: the Kubernetes Service that the app's connection string names has a ready endpoint. That is literally the address the app is about to dial, and it is the one signal common to every database operator and every chart, because they all end in a Service. It is also sometimes not enough. A Postgres that answers but has not yet created the database or the extension the app needs is up and useless, and an operator that creates those a few seconds later will lose a race to a fast migration. Where the operator reports that state, it is worth waiting for as well.

There is a subtler race after that, and it cost us an afternoon. On clusters that enforce network policy, a brand-new pod can be blocked for a second or so while its policy is programmed, and a connection in that second is refused even though the database is ready. A long-running app shrugs it off with one restart. A migration that dials at the first instant fails. So the wait has to be at the point of dialing, from inside the pod that will dial, not only on the platform's side.

Hold, never take down

The last decision is what waiting looks like. The rule we settled on: a held app never scales down what is already running. On a fresh project there is nothing running, so waiting is simply not starting. On an existing app whose database has a problem, the previous version keeps serving while the new one waits, which is the same promise a rollout already makes.

And waiting is said out loud. Waiting for orders-db to be ready is a status a person can act on. After long enough, fifteen minutes in our case, it stops being a wait and becomes a problem that names the dependency, so the red badge lands on the database that is broken and not on the five apps queued behind it.

How this works in Anyport

Every app's dependencies are derived from what it references: a managed service it reads from is required, a sibling app is optional and waited on for up to two minutes. The agent on the cluster holds the app's rollout, and its pre-deploy command, until each required service's Service has a ready endpoint, and for Postgres until the database and its extensions exist. Inside the pod, the pre-deploy command and task runs check each required host and port before the command starts. Nothing running is scaled down to wait.

A waiting app reads Waiting for db to be ready, and asks for attention after fifteen minutes. The project's setup card shows the stages the graph produces, a duplicate says the order it will start in, postgres, redis → api → web, and a preview comment says Waiting for api rather than Ready while an app is not answering. There is nothing to configure. The documentation has the rule in two paragraphs, which is about as much as start order should ever ask of you.