Running GitHub Actions on AWS Spot: what the discount asks of you
What moving GitHub Actions to AWS Spot taught us about capacity, safe retries, and deciding whether cheaper compute is worth the operational work.
6 min read
Contents
We moved our GitHub Actions workloads from GitHub-hosted runners to self-hosted Spot runners to cut CI costs, and for me the hardest part was reliability: making machines that AWS can reclaim feel dependable to the people waiting on their builds. Most of my effort went into choosing capacity, retrying lost jobs without hiding real failures, and checking each month whether the setup was still worth it.
The deal behind Spot is simple. You get spare EC2 capacity at a discount, on the condition that AWS can take the machine back when it needs it. AWS tries to give two minutes of warning before it stops or terminates one, and it says plainly that the warning is best effort. CI looks like a natural customer for that deal: most jobs are short and repeatable, and nobody is attached to the machine a test suite ran on.
GitHub still runs the show
Self-hosting changes less than the name suggests. GitHub Actions still receives the push, schedules the workflow, and shows the logs. The only thing you replace is the computer that picks up each job.
GitHub Actions
schedules the job
Runner module
matches job labels
EC2 Spot runner
runs one job
Retired
ephemeral
We used the open-source GitHub runners on AWS Terraform module. It listens for queued jobs, matches them to a runner configuration by their labels, and starts EC2 capacity to serve them. We began with a pilot on one repository and grew from there into separate fleets for different kinds of work.
Our runners were configured as ephemeral, handling one job before being retired. That makes replacing a lost runner easier to design for than restoring a machine with a long-lived workspace. It doesn’t make the job itself recoverable; that needs separate handling.
Owning the machines also means owning everything about them: images, capacity, cleanup, and what a workflow running on them is allowed to reach. GitHub’s security guidance is worth reading before the first runner goes live, especially if anyone outside your team can trigger a workflow.
The cheapest instance is rarely the right one
It is tempting to pick one cheap instance type and use it everywhere. That narrows the capacity AWS can choose from. Availability varies across instance types and Availability Zones, so allowing more compatible options gives the fleet more ways to find a machine.
Our approach was to give AWS a list of acceptable instance types and let it balance price against capacity availability. The lowest hourly price means little if the machine disappears halfway through a build and the job has to start over.
The list needed pruning. Some types were excluded because of interruption churn. Others didn’t have enough memory for the builds we ran on them. The configuration also allowed on-demand fallback for a specific insufficient-capacity error at launch. That fallback only covers the moment of launch. It does nothing for a job whose machine is reclaimed while it is running.
I adjusted pool sizes and that price-versus-availability balance repeatedly. Three costs pull against each other: paying for capacity, waiting for capacity, and redoing work after losing it. Push one down and another can rise. That made capacity tuning something I kept revisiting.
Retrying without hiding real failures
Once you accept that machines will occasionally vanish, you need retries. The tricky part is retrying the right things. A job that lost its runner deserves another try. A job with a failing test deserves a human. A retry system that confuses the two teaches the team that red builds don’t mean anything.
The recovery I built was a workflow that runs after the main build fails or is cancelled. It reads the messages GitHub attaches to each failed job and looks for signs of a lost runner: lost communication, a shutdown signal, or a job that timed out waiting for a machine. It deliberately ignores ordinary timeouts. A job that ran and overran its own time limit may be slow for reasons a retry won’t fix.
An earlier version had an interesting bug. A workflow run is a dependency graph. Build and test jobs do the work, and jobs further down, such as gates, deployments, and notifications, wait on them. Losing a build runner could also produce downstream failures with ordinary error messages. Those messages described the consequences of the upstream failure, without identifying the lost runner.
The classifier asked whether every failed job showed signs of runner loss. When downstream jobs had different annotations, that collateral damage vetoed a valid recovery. The fix was to consult the failed build and test jobs and leave out the known downstream ones. If all of those failures match runner-loss signals, the workflow reruns the failed jobs. If runner loss shows up next to some other kind of failure, it notifies a person instead of guessing. And it only acts on the first attempt, so this workflow can request one automatic retry.
Main build fails or is cancelled
recovery workflow starts
- retry already used Leave it for a human
Read the failed build and test jobs
known downstream jobs left out
Do all of them show runner-loss signals?
lost communication, shutdown, or waited too long for a machine
- mixed with other failures Notify a person
- none A real failure: stays red
Rerun the failed jobs
one automatic retry
This is still a heuristic. It recognizes messages we know about, which isn’t the same as knowing the cause, and its list of downstream jobs is tied to the shape of our pipeline. When the pipeline changes, the recovery logic has to change with it. The broader lesson carries over to any system that diagnoses failures in a pipeline: find the job where the trouble started before you let its downstream symptoms vote.
Keeping score every month
The migration didn’t end when the jobs moved. Every month I looked at job failures and at what the runners cost.
Failures need splitting before they say anything useful. Interrupted jobs, runs the retry recovered, and runs that still failed afterward are three different numbers, and only together do they tell you whether a capacity change helped.
| Number | What it tells you |
|---|---|
| Interrupted jobs | How often AWS took a machine back mid-job |
| Runs the retry recovered | How much of that the recovery workflow absorbed |
| Runs still failing after retry | What the retry couldn’t fix, and what needs a closer look |
The cost side has its own trap: compare the minutes jobs actually ran, not how long the machines were up, because an instance can be running without doing any useful work.
My rough savings estimate is around $2,000 a month, based on pricing our current job minutes at GitHub’s hosted-runner rates. It isn’t a before-and-after comparison of invoices or a verified net saving: the calculation still needs reconciliation against attributable AWS costs, GitHub’s included minutes, and per-job rounding. The engineering time behind the tuning, the retry workflow, and the monthly reviews belongs in that calculation too, even without a dollar figure attached.
When Spot CI is worth it
Looking back, I would ask a few questions before starting.
Is there enough paid hosted usage to make the savings matter? If the bill is small, the fleet will probably cost more attention than it saves.
Do your jobs tolerate being restarted? Short, repeatable builds and tests fit Spot well. Long jobs, or jobs with side effects that are hard to repeat, make every interruption more expensive.
Who owns the fleet? Someone has to tune capacity, maintain the retry rules as pipelines change, review the failures, and keep an eye on what workflow code can reach. That work doesn’t stop after the migration.
If the answers line up, Spot can be a good home for CI. The condition is treating a lost machine as a normal event to design around, and continuing to check failures and costs long after the jobs have moved.