Skip to content
← All posts

The Anyport Team

A cron job is a second copy of your app, and that is the bug

Most scheduled jobs are an existing service with a different command. Defining them separately is how the nightly report ends up running last month's image with a rotated-out password. What a task should inherit, what it should decide for itself, and why a migration is not a task at all.

Ask where the nightly report runs and the answer, on most clusters, is a CronJob manifest someone wrote a year ago. It names an image tag that was current then. It has its own copy of the environment variables, one of which is a database password that was rotated in the app's Secret in March and not here. It has never failed loudly, because it has been failing quietly, and the report it produces is read by nobody who would notice a missing day.

The manifest was not written carelessly. It was written the way Kubernetes asks for it: a CronJob is a complete pod spec, so the person writing it copied the app's pod spec and changed the command. The copy is the bug. From that moment the job and the app are two things that used to agree.

Inherit, then override

Most scheduled work is an app you already run, with a different entrypoint. The report is the API's codebase invoked as report --since yesterday. The cleanup is the worker's image with a different subcommand. So the primary way to define a task should be to name the app, and inherit from it the image, the variables, the secret references, the resource requests and the registry credentials. When the app deploys a new image, the task runs the new image next time. When the password rotates, the task reads the rotated one.

What remains is what the task genuinely decides for itself. The command, obviously. Variables layered on top of the inherited ones, so the same image can be pointed at a different queue. And often the resources, because a migration or a backfill wants more memory than a request handler and no CPU floor, which is rarely the app's shape. A task defined from a bare image, with no app behind it, should also exist, for work that belongs to no service. It is the second form, not the first.

What happens when the last run is still going

Every scheduler has to answer this and Kubernetes offers three answers: start another alongside, replace the running one, or refuse to start until it finishes. The default that ships with CronJob is to start another. That is the right default for a stateless batch that processes whatever is in a queue, and the wrong one for nearly everything people actually schedule, where two copies of the same job touching the same rows is a data hazard rather than extra throughput. A platform that knows its tasks are mostly reports, cleanups and backfills should default to refuse and let the person who knows better opt into overlap.

Refuse has a consequence that needs its own control. A run that wedges, a connection that never times out, a lock that is never released, now holds the slot, and every later run is skipped without a sound until someone notices the report stopped arriving. So a per-run timeout is not an optional nicety, it is the thing that makes refuse safe. The platform cannot pick the number, because a backfill legitimately runs for hours, but it can make the field impossible to miss and explain why it is there.

Zero retries is an answer

Retrying a failed run is the right instinct for a job that is idempotent and the wrong one for a migration that got halfway. The second attempt runs against a schema the first attempt already altered, and now the failure is worse and harder to read than the original. The default should be zero, stated as a deliberate choice, with the count raised by the person who knows the job can be safely repeated.

Stop it without deleting it

At three in the morning, the correct response to a misbehaving job is to make it stop firing and fix it in daylight. Deleting the CronJob does that and also deletes its history, so the runs that would explain the fault are gone by the time anyone looks. Suspend is the operation that is needed: the schedule stops, the definition and every past run stay where they are, and unsuspending later is one field.

History is not the same as logs

The run that failed overnight has to still be there in the morning: when it ran, what triggered it, how it ended and, when the cluster refused or killed it, the reason the cluster gave. That record is cheap and should be kept, bounded per outcome so a job that runs every minute does not accumulate a year of rows.

Logs are a different matter, and a platform should be honest about it. A finished run's pod is gone, and with it the pod's logs, unless something was shipping them elsewhere while it ran. A task whose output matters later should write it somewhere that keeps it, and the platform should say that plainly rather than imply that a history row is a transcript.

The migration that is not a task

A database migration looks like a task and is not one. It has to run with the code that needs it, once per version, before that version serves traffic, and if it fails the rollout has to stop with the previous version still serving. None of that is scheduling; it is a step in a deploy. So it belongs on the app, as a pre-deploy command that runs with the incoming image.

The tempting design is to let the app reference a task for this. The problem is that a reference can dangle, and a dangling reference here blocks every deploy of the app with no way to express the fix in the file that caused it. A command on the app cannot dangle. If the same migration should also be runnable by hand, it can be a task as well, and the duplication is one line that is obviously the same line.

How this works in Anyport

A task names an app and inherits its image, variables, secrets, resources and registry credentials, or names an image directly. Give it a cron expression and a timezone and it runs on that schedule; leave the schedule off and it is a job you run on demand. The overlap policy defaults to refuse, retries default to zero, and the history limit defaults to three finished runs per outcome; a timeout has no default, for the reason above. A scheduled task can be suspended without losing its history. Every run is kept with its outcome and trigger; a finished run's logs are not, and the documentation says so.

An app can carry a pre-deploy command, which runs once per version with the incoming image before it reaches the app; a non-zero exit stops the rollout. It is a command, not a task reference, so it cannot dangle. In a preview environment the same command is what turns a fresh, empty database into one the branch can run against. The documentation has the fields and the defaults. The decision this post is about is earlier than any of them: when you add scheduled work to a service, is it a second definition of that service, or the same one with a different command.