Score the job before choosing the price
A batch job belongs on Spot when it can restart safely after an interruption. It also needs time for at least one retry.
Idempotent means a repeat run gives the same safe result. For example, it can reprocess an image with the same output key. It must not charge a card twice.
Checkpointing means saving progress outside the virtual machine. It is like saving a document before a laptop battery dies.
| Job test | Spot-first fit | Commitment or On-Demand fit |
|---|
| Retry tolerance | At least 1 full retry is acceptable | No retry is acceptable |
| Checkpoint interval | Every 5 to 15 minutes for long runs | State cannot be safely saved |
| Deadline slack | Queue wait plus one rerun fits before deadline | A missed window causes direct harm |
| Demand pattern | Bursts or flexible run times | Stable 24/7 baseline |
Pros of interruptible capacity
Spot works well for rendering, genomics, media conversion, large reports, and test builds. It also fits container workloads that read data from durable storage.
Spot can also run Kubernetes worker nodes. The scheduler must replace lost nodes, and the work must spread across several instance types.
Flexible jobs gain the most from Spot capacity.
Cons that change the answer
A job with unrecoverable memory state should not start on Spot. The same is true for a hardware license tied to one machine.
Irreversible database writes also rule out Spot. Choose Spot-first if: your jobs are retry-safe and have durable checkpoints. They must also tolerate a capacity shortage without breaking a customer promise.
Reserved discounts cut bills, not queue risk
Reserved Instances cut eligible Amazon EC2 charges when billing attributes match. They do not attach to a named AWS job.
Reserved Instances also do not guarantee capacity. Think of them as a lower electricity rate, not a reserved seat.
| Purchase model | Capacity promise | Interruption | Best batch use | Commitment |
|---|
| Spot | No | AWS can reclaim capacity | Retryable bursts | None |
| On-Demand | No reserved stock | No Spot eviction | Hard deadlines and fallback | None |
| Reserved Instance | No, by itself | No Spot eviction | Narrow, steady EC2 use | 1 or 3 years |
| Savings Plan | No | No Spot eviction | Stable spend across changing fleets | 1 or 3 years |
Savings Plans often fit batch platforms better when workers change families. This includes c6i, m7i, Graviton, or GPU families.
They also suit moves between US East (Ohio) and US West (Oregon). The discount covers eligible compute spend more broadly than a narrow RI.
Google Cloud Committed Use Discounts teach the same lesson. A discount does not promise that a machine will be available.
Choose Savings Plans or Reserved Instances if: you have a stable baseline for most monthly hours. Choose Savings Plans when families, sizes, architectures, or Regions may change.
Choose a narrow RI only when that EC2 shape is truly stable.
Safe batch capacity flow
Job queue
→
Spot environment
retryable work
→ interruption or shortage →
On-Demand environment
deadline fallback
Savings Plan or RI covers the stable billing floor. Durable storage keeps checkpoints outside both worker pools.
A cost decision should use completed-job cost, not the Spot hourly rate. Assume a steady batch baseline needs 20 vCPUs for 730 monthly hours.
Assume burst processing adds 10,000 vCPU-hours. A Savings Plan or scoped Reserved Instance can cut the predictable baseline cost.
Spot can handle burst work at a lower rate than On-Demand Instances. But retry overhead belongs in the Spot total.
Assume 8% of Spot jobs face interruption. Assume each interrupted job uses 20% extra compute for restart and recovery.
That extra work changes the true Spot cost.
Urgent jobs may also spill to On-Demand during a capacity shortage. Include those hours in the cost model.
Spot remains attractive when its discount exceeds retry, queue-wait, and fallback costs. A posted hourly rate alone cannot make that choice.
For most batch platforms, use Spot for retry-safe bursts and commit only the steady floor. Keep On-Demand capacity for jobs near their deadline. Savings Plans fit changing fleets better than narrow RIs. But neither plan reserves machines. If a deadline cannot move, budget for On-Demand or a supported Capacity Reservation.
Questions & answers
Are Spot Instances worth it for batch jobs?
Yes, Spot is worth it when a job can retry safely. Its deadline must allow queue delay plus at least one full rerun.
Spot poorly fits work that loses value after one missed processing window.
Can AWS Batch use Reserved Instances?
Yes, AWS Batch can consume EC2 usage eligible for an RI discount. You cannot assign an RI directly to a Batch job.
An RI alone does not guarantee capacity. Use a Capacity Reservation when you need a supported capacity guarantee.
How serious are Spot interruptions for throughput?
Interruptions are serious when all workers need one scarce instance type or Availability Zone. Spread capacity across several types and Zones.
Then test the completed-job rate. Do not trust the listed hourly rate alone.
Are Azure Spot VMs the same as AWS Spot?
No, Azure Spot Virtual Machines use different eviction rules and policy choices. Some configurations use capacity-only eviction or price-and-capacity behavior.
Azure Batch needs its own eviction tests, retry rules, and fallback pool. Do not copy AWS settings unchanged.
Choose hybrid capacity for safe savings
As Alan Curtis, I have tested and reviewed hosting solutions for over 12 years. I have seen a nightly pipeline look cheap until Spot shortages hit.
Late jobs then moved to On-Demand workers near the window's end. That erased much of the apparent Spot saving.
A queue-age rule works well. Move a job to On-Demand when remaining deadline slack is below runtime plus one retry.
This protects completion without paying On-Demand rates for normal runs.
Spot pool operating rules
Allow multiple instance families. Cap each job's retry attempts. Alert on queue age, not only instance count.
Store progress in Amazon S3, a database, or another durable service. Do not store it only on a worker's local disk.
For AWS interruption guidance, review the EC2 Spot interruption notice documentation. AWS sends a two-minute notice before it reclaims a Spot worker.
Do not use a Spot-first strategy for strict, non-negotiable completion deadlines. Avoid it for unrecoverable in-memory state, licensing constraints, or hardware constraints. Avoid it when retries cost more than the Spot discount. Also avoid Reserved Instances and Savings Plans with highly sporadic use. They need a dependable baseline. In that edge case, On-Demand can cost more per hour but less overall.
The strongest recommendation is a hybrid model. Commit only the measured baseline with a Savings Plan.
Send retry-safe burst work to Spot. Keep an On-Demand escape path for deadline-critical jobs.
Use RI discounts only for a stable, narrow EC2 footprint. Never treat a billing discount as capacity assurance.
Choose this hybrid design if: your workload has recurring demand and bursts. Most growing batch platforms fit this pattern.
In AWS Batch, create separate managed compute environments for Spot and On-Demand Instances. Do not make fallback a manual choice.
The Spot environment should allow several compatible instance families and sizes. The On-Demand environment should fit deadline-critical work.
Attach both environments to a job queue. Order them so retry-safe jobs try Spot capacity first.
For strict service windows, use a separate high-priority queue. It should target the On-Demand environment first.
You can also route jobs when queue age passes a set threshold. This makes shortage behavior clear.
Flexible work waits or retries on Spot. Jobs near hard deadlines get a predictable non-preemptible path.
A Spot interruption usually gives an EC2 worker a two-minute termination notice. The worker must start shutdown work at once.
The application should stop accepting new work. It should write a durable checkpoint and record the attempt state.
It should then exit so AWS Batch can apply its batch job retry policy. Do not wait for the job to fail.
Keep checkpoints frequent enough that lost work stays affordable. A 10-minute interval can lose almost 10 minutes of progress.
A two-minute interval cuts rework. But it can add storage and coordination overhead.
Idempotent jobs matter because a retry can replay the last checkpointed segment. Store checkpoints and output manifests in S3, DynamoDB, or another durable system.
Never store them only on local instance storage. These rules also apply to Preemptible Instances and other eviction-based worker models.
