At 500 RPS with a 200 ms average duration, your app needs about 100 concurrent runs. That is before retries, spare capacity, or slow downstream services.
When this load runs 24/7, per-invocation billing can raise your monthly compute bill. Cold starts, concurrency caps, and throttling can also hurt p95 and p99 latency.
Serverless can work for predictable high throughput, but rarely as the only compute layer. Reserved containers or VMs often cost less and provide steadier p95 and p99 latency.
The usual answer is a hybrid design. Run baseline traffic on persistent services, then use serverless for bursts, queues, scheduled jobs, and asynchronous work.
Is serverless worth it for steady high throughput?
Serverless can support steady high throughput. Its value depends on whether reduced operational work outweighs higher always-on prices, quotas, and latency risk.
A Service Level Objective, or SLO, is your promised reliability target. Think of it as the delivery time you promise customers.
A useful threshold is sustained compute use above about 40% to 60% each month. This is not a provider rule.
At that point, reserved capacity often beats per-request function billing. Count logs, gateways, network traffic, and warm capacity in the total.
A hybrid architecture often works best for predictable high throughput: persistent compute handles the floor, and serverless handles the ceiling.
That design avoids function rates for constant demand. It still provides fast capacity for sudden spikes.
“Serverless” is not one pricing or performance model. Review each layer on its own.
Function services, such as AWS Lambda, bill mainly for requests and run time. Lambda costs can rise quickly with steady traffic.
API gateways route requests, but may add per-request fees and throttling. Serverless databases and analytics tools reduce administrative work.
Their connection limits, scaling rules, and query fees may become your real bottleneck.
Serverless containers sit closer to containers than to functions. Minimum instances or reserved capacity can protect latency.
VMs may still cost less for an always-on baseline. A hybrid cloud design can mix these models.
Calculate concurrency before you migrate
Size compute by counting requests that overlap during real production load. Use peak RPS, p95 and p99 durations, retries, traffic spikes, and downstream limits.
RPS means requests per second. Concurrency means requests being handled at the same time.
For 500 RPS and a 0.25-second average duration, the base math is 500 × 0.25 = 125 concurrent runs. If p99 reaches one second, 500 RPS may need 500 concurrent runs.
That count excludes uneven arrivals and retries. Both are common during real incidents.
Size from p95 and p99, not averages
Start with this formula: target concurrency = peak RPS × p99 duration × 1.2 to 1.5. The 20% to 50% buffer covers traffic clumps, retries, deploys, and modest growth.
Average duration can hide your worst user experience. P99 means 99 out of every 100 requests finish within that time.
The most frequent error is sizing from average time alone. A slow database call can multiply concurrency demand in minutes.
Check quotas and downstream ceilings
Autoscaling does not mean unlimited capacity. AWS Lambda has regional concurrency quotas, and Amazon API Gateway has throttling limits.
Databases also have connection and write ceilings. They may fail before the function platform fails.
Ask for quota increases before migration day. Then load-test the database, cache, and partner APIs at your planned ceiling.
For a 500 RPS API with one-second p99 latency, plan roughly 600 to 750 concurrent runs before retry volume increases further.
Choose persistent compute for stable demand above 40% to 60% monthly use. Use serverless for bursts when its quota, cold-start, and downstream limits have been tested. This does not apply to low or highly uneven traffic, where per-request billing may still save money. For a steady 24/7 API, price the full stack before moving production traffic.
Compare monthly cost beyond function pricing
Compare total monthly operating cost, not only function execution. Include requests, memory, routing, logs, network traffic, warm capacity, databases, queues, caches, and monitoring.
Also count the engineering time needed to keep the workload within its SLO. Cheap compute can become costly if teams spend hours fixing throttles.
Consider a US East workload of 500 RPS, running 24 hours daily. Assume 512 MB memory and 200 to 250 milliseconds of run time.
That equals about 1.30 billion requests in a 30-day month. It also uses about 130 to 162 million GB-seconds of function compute.
At public list-price ranges, compute and request fees alone can reach $2,400 to $3,000 monthly. This excludes API handling, logs, and data transfer.
The same stable baseline may fit six to ten reserved container tasks or VMs. Those often cost about $700 to $1,700 monthly before database and egress fees.
CPU, memory, region, and commitment term change the result. Price each item using your own traffic data.
| Compute choice | Best traffic pattern | Illustrative monthly compute | p99 control |
|---|
| AWS Lambda-style functions | Sharp peaks, idle periods | $2,400 to $3,000 at the example load | Needs quota and cold-start planning |
| Reserved containers, Cloud Run, ECS | Steady baseline with bursts | $700 to $1,700 before add-ons | Strong with fixed minimum capacity |
| VPS or dedicated server | Stable, well-known demand | $400 to $1,500 before redundancy | Highest host and network control |
Include the hidden line items
Amazon API Gateway, load balancers, egress, NAT Gateway traffic, and log intake can change the result. Tracing, queues, caches, and database reads also add costs.
A $500 function estimate can grow much larger. This happens when each request crosses a paid gateway and writes verbose logs.
A common case is a busy API that logs full request bodies. Log storage and search fees can exceed the function bill.
Use two load models, not one
Model a normal hour and a peak hour separately. For each model, calculate RPS, p95, p99, memory, request count, and data sent out.
Also count database work and warm instances. One monthly average hides both the bill and the risk of peak traffic.
At steady 500 RPS, function costs can exceed reserved compute by hundreds or thousands of dollars each month.
A hybrid path for 500 RPS baseline and 1,500 RPS peaks
500 RPS baseline
Reserved containers
Stable p99 control
1,000 RPS burst
Serverless functions
Short overflow work
Slow work
Queue + workers
Backpressure protects data stores
Protect p99 latency from throttles and cold starts
Protecting p99 means measuring the full request path. Control overload before it reaches databases, caches, partner APIs, or function quotas.
Customer latency includes much more than function code time. A slow service behind the function can hold each request open.
Set a user-facing target before choosing compute. A cached read API may aim for p95 below 200 milliseconds.
It may also aim for p99 below 500 milliseconds. A report generator that queries several systems should send results asynchronously.
That keeps slow work out of the user request. It also reduces the number of concurrent runs.
Measure every hop in the request
Trace DNS lookup, TLS setup, API Gateway routing, and function startup. Trace VPC networking, cache access, database time, and response transfer too.
Distributed tracing links these steps under one request ID. It shows where a delay began.
Most guides say provisioned concurrency solves cold starts. What they miss is that it cannot fix a saturated database.
Stop overload before it spreads
Set rate limits at the edge. Cap function concurrency when a database or partner API has a hard limit.
Then use queue-based buffering for extra work. The queue lets work wait safely instead of causing a retry storm.
A retry storm is like shoppers pushing through one locked store door. More requests make the outage worse.
Queues protect p99 only when workers consume jobs below the database's safe write and connection limits.
This comparison changes for low, highly uneven, or test traffic.
Serverless can cost less there, even with a higher unit price. GPU work, long jobs, persistent connections, live streaming, and strict network control usually favor specialized containers, VMs, or dedicated servers.
Questions & answers
Is serverless cheaper than a VPS at 500 RPS?
Usually not when 500 RPS runs around the clock. At 1.30 billion monthly requests, functions, gateways, logs, and warm capacity can cost hundreds or thousands more.
How much concurrency should a 500 RPS API have?
Plan from p99 duration, not average duration. At 500 RPS and one-second p99 runtime, start near 600 to 750 concurrency with a 20% to 50% buffer.
Can provisioned concurrency eliminate cold starts?
It can reduce cold starts for the environments kept ready. It cannot remove gateway, network, database, or downstream delay, and it adds a recurring fee.
When should I leave serverless for containers?
Move the baseline to containers when use stays near 40% to 60%. Move when costs rise with stable demand or p99 needs paid warm capacity.
Keep functions for burst work and asynchronous jobs. Do this unless their limits create more work than they remove.
Choose hybrid compute for your baseline
Choose persistent containers or VMs for predictable baseline traffic. Use serverless for short, bursty, scheduled, or asynchronous demand.
This split gives stable p99 performance. It also keeps elastic capacity for spikes you cannot forecast.
For a 24/7 high-throughput API, reserve the baseline and pay per request only for the overflow.
A practical deployment split
Run synchronous read and write APIs on reserved containers, Cloud Run minimum instances, managed Kubernetes, or a VM pool. Put them behind load balancing.
Send emails, webhooks, media conversion, scheduled reports, and retryable jobs to a queue. Let serverless workers process those jobs.
This works well when queued work can wait. It does not fit a user action that needs an instant response.
Review the decision each quarter
Recheck the model after major traffic or code changes. A move from 200 milliseconds to 600 milliseconds at p99 triples concurrency demand.
This can happen even when average request time barely changes. Test peak load before renewing contracts or changing architecture.
Your bill and latency should guide the choice. Do not let a platform label make it for you.
