Auto-Scaling Policies: CPU vs Request Rates
Choosing between CPU utilization and request rate as an auto-scaling metric is a critical decision that balances application performance against cloud infrastructure costs. This article explores how web development teams evaluate both approaches, outlining workload characteristics, response times, scaling lag, and practical scenarios where CPU, request rates, or a hybrid approach provide the most reliable scaling behavior.
Understanding the Core Metrics
CPU-based auto-scaling monitors the average processor utilization across instances, triggering new instances when usage crosses a predefined threshold (such as 70%). This metric reflects actual computational stress on the hardware.
Request rate-based auto-scaling monitors the volume of incoming traffic, typically measured in requests per second (RPS) at the load balancer or API gateway. It triggers scaling actions when total throughput per target reaches a predetermined capacity limit.
Key Considerations for Decision-Making
1. Nature of the Workload: Compute-Bound vs. I/O-Bound
The primary factor in choosing a metric is how the application consumes server resources:
- Compute-Bound Applications: Tasks involving video rendering, data encryption, machine learning inference, or heavy mathematical calculations heavily tax the CPU. For these workloads, CPU utilization directly correlates with system health, making it the ideal metric.
- I/O-Bound Applications: Modern asynchronous web APIs, microservices querying external databases, and streaming endpoints spend most of their time waiting for network or disk I/O. In these cases, servers can become overwhelmed with open connections, exhausted socket pools, or thread starvation while CPU utilization remains low. Scaling on request rates prevents outages in these environments.
2. Metric Latency and Traffic Spikiness
Traffic patterns dictate how quickly a scaling event must occur:
- CPU Metric Lag: CPU usage is a lagging indicator. It takes time for incoming requests to increase CPU usage, and metrics are typically aggregated over 1- to 5-minute intervals to smooth out momentary spikes. This delay can lead to dropped requests during sudden traffic bursts.
- Request Rate Proactivity: Request rates reflect traffic at the ingress layer in near real-time. Teams anticipating sudden, predictable surges—such as flash sales, ticket releases, or push notification campaigns—benefit from request-based scaling because it reacts faster to incoming demand.
3. Request Uniformity and Payload Heterogeneity
For request rate scaling to work accurately, requests must have relatively uniform resource requirements. If an application serves both lightweight health checks and heavy, long-running database reports on the same endpoints, request rate becomes an unreliable indicator of load. A low volume of resource-heavy requests could exhaust the server before the RPS threshold is reached. In heterogeneous environments, CPU scaling often provides a safer baseline.
4. Instance Boot and Initialization Time
The time required for an application instance to start, pass health checks, and accept traffic impacts the choice of policy:
- For applications with slow start times (e.g., legacy Java applications or heavy container images), request rate thresholds must be set conservatively to allow new nodes enough lead time to initialize before existing nodes saturate.
- For applications with fast start times (e.g., lightweight Go or Node.js services), CPU scaling with short evaluation periods can respond rapidly without risking capacity shortfalls.
The Hybrid Approach: Multi-Metric Scaling
Many production architectures utilize both metrics simultaneously to compensate for the weaknesses of each:
- Request Rate as the Primary Scale-Out Metric: Configured to rapidly add capacity when traffic volumes surge suddenly.
- CPU Utilization as the Safeguard: Configured to catch background tasks, memory leaks leading to thrashing, or unexpectedly heavy individual requests that RPS metrics fail to represent.
When configuring multi-metric policies, cloud orchestrators typically adopt the policy demanding the highest capacity, ensuring service availability under unpredictable operating conditions.