A worker is not a web app that nobody calls

Give every app a port, a readiness check and a URL, and a queue consumer gets three things it cannot use and one it cannot pass. Why serving traffic is one decision rather than three settings, and the start command bug that tends to arrive with it.

A deploy platform built around HTTP assumes every container answers on a port. For a web app that assumption is the whole product: a port to route to, a readiness check that dials it, and a URL that reaches it. Then somebody deploys a queue consumer, a scheduler loop or a chat bot, a process that only ever makes outbound connections, and the assumption starts to cost something.

Until recently every app in Anyport got port 8080, a readiness check on that port, and a public URL, whether or not anything inside listened. This post is about why that was wrong in a more interesting way than "the URL answers nothing".

A readiness check that can never pass

A readiness probe is Kubernetes asking a pod whether it should receive traffic. A TCP probe dials the port, and until the dial succeeds the pod is not ready. For a worker the dial is refused every time, because nothing in the process opened the port, and the kubelet records it every ten seconds: Readiness probe failed: dial tcp 10.42.0.17:8080: connect: connection refused.

The readiness check dials port 8080 and is refused, because the worker never opens a port. Switching off Serves web traffic removes the port, the check and the URL together, and the worker goes live.

The worker, meanwhile, is fine. It is consuming the queue, doing its job, and every surface that reports health says it is broken. The first deploy never finishes rolling out, because a Deployment waits for its new pods to be ready, and after ten minutes it gives up with ProgressDeadlineExceeded.

The second deploy is worse. A rolling update of a single replica starts the new pod before stopping the old one, and only stops the old one once the new one is ready. The new one never is. So the old version and the new version both run, both consume the same queue, and the rollout sits half finished until the deadline. For a worker that processes payments or sends email, two versions reading one queue is a real bug, produced entirely by a health check that did not apply.

Three settings that are one answer

The usual workarounds each fix one symptom. Open a tiny HTTP server inside the worker so the probe has something to dial, which makes the health check report that a socket is open rather than that the work is happening. Remove the probe, and keep the port and the URL that still answer nothing. Point the port at whatever the process does listen on, if anything.

The port, the readiness check and the URL all follow from one question: does anything need to reach this process? Heroku answered it years ago with the Procfile, where only the process type named web receives traffic. On a platform with per-app settings it should be one switch, asked once, that takes all three with it. Asked when the app is created, it is a sentence anyone can answer. Discovered later, it is three settings in three places that have to be undone in the right order.

One case should be refused rather than handled. If an app with a custom domain is switched to a worker, the domain has nothing to route to. Removing it quietly would break a hostname somebody set up in DNS, possibly one the public uses, so the switch should refuse until the domain is removed by hand.

The start command that started the wrong thing

Workers tend to arrive with a second problem. A common shape is one image with two binaries, /api and /worker, and a start command that picks one. Whether that works depends on a Kubernetes detail that is easy to get backwards: a container's command replaces the image's ENTRYPOINT, and its args replace CMD.

Pass the start command as args to an image whose entrypoint is /api, and the container runs /api /worker. The API starts, with an argument it ignores, and the worker never runs. Nothing fails. The app is healthy, and it is the wrong app.

Passing it as command fixes that and breaks buildpack images, whose entrypoint is a launcher that sets up the environment and PATH the buildpacks configured; replacing it means the process starts without them. And running every command under /bin/sh -c, the usual way to make $PORT expand, fails on a distroless image that has no shell. The rule that works reads the image: hand the command to the launcher on a buildpack image, replace the entrypoint on any other, and add a shell only when the command uses one.

How this works in Anyport

Creating an app asks Serves web traffic, on by default. Switched off, the app is a worker and gets no port, no readiness check and no URL; its project card says it is a worker, and in an anyport.yaml it is kind: worker. The same switch is in an existing app's Domains settings, and it refuses while a domain still points at the app.

A start command on a buildpack image runs in a shell behind the buildpack's launcher. On any other image, from a Dockerfile build or a registry, it replaces the entrypoint: a plain command such as /worker runs directly, and one that uses the shell is run in /bin/sh -c. Which kind of image it is gets read from the image itself on every rollout, so a repository that gains a Dockerfile changes with it. The documentation has both rules in a paragraph each.