Pod Rightsizing: Why It Fails to Reduce Kubernetes Costs

If you’re operating Kubernetes in production, you probably know this story: you analyze metrics, tune CPU and memory requests, validate them against historical data, and roll changes out carefully. Pods are no longer wildly overprovisioned. And the cloud bill stays pretty much the same.

If that’s familiar, it’s not necessarily caused by poor execution. It’s more likely a result of how Kubernetes is designed to balance elasticity and safety. Accurate requests are necessary, but they don’t eliminate waste on their own.

This article walks through where that remaining waste comes from, how to recognize it, and what actually removes it from your cluster, step by step, from the pod level up through the node and the invoice.

Step 1: Understand What Accurate Requests Really Optimize

Accurate requests mean a pod’s resource requests closely reflect its typical runtime needs. For CPU, that usually means aligning with sustained usage rather than peaks. For memory, it often means sizing close to the working set to avoid OOMKills.

What accurate requests do well:

• Improve scheduler decisions
• Reduce node overcommit risk
• Stabilize HPA behavior
• Prevent pathological overprovisioning at the pod level

What they do not do:

• Maximize utilization
• Eliminate idle capacity
• Reduce node count automatically
• Guarantee lower cloud costs

The mental shift that matters here: Kubernetes uses requests as reservations, not forecasts. Once a request is set, the scheduler treats that capacity as unavailable to others, regardless of whether it’s actually used.

This distinction sounds subtle but changes everything about how to think about waste. A forecast is a prediction that gets revised as new data comes in. A reservation is a commitment that holds regardless of what actually happens afterward. Kubernetes was designed around the second model on purpose, because the alternative, letting the scheduler guess at real-time usage and place pods more densely than their requests would suggest, trades away the predictability that makes production clusters reliable. That tradeoff is reasonable. It’s just rarely stated explicitly, which is why teams keep expecting rightsizing to behave like a forecasting problem when it’s actually a reservation problem.

Checkpoint: if your pods are no longer throttling or crashing due to resource pressure, and HPA behaves predictably, your requests are likely accurate. That’s the foundation. It’s not the whole picture.

Step 2: The HPA Headroom Buffer

The Horizontal Pod Autoscaler works by comparing observed utilization to a target, and that target is almost never 100%, for good reason.

Consider this configuration:

  resources:

  requests:

    cpu: "1"

hpa:

  metrics:

    - type: Resource

      resource:

        name: cpu

        target:

          type: Utilization

          averageUtilization: 80

This setup intentionally keeps each pod running at around 800 millicores. The remaining 200 millicores aren’t a mistake. They’re a safety margin, because HPA polls metrics on an interval, scaling actions take time to schedule and start, and without headroom, traffic spikes overwhelm existing pods before scaling can react.

Set the target to 100% and you’re betting that load never spikes faster than your scale-out loop can respond. In production, that bet usually loses. Common pitfall: chasing higher utilization by raising HPA targets often produces latency spikes or failed requests before any real savings show up.

Each pod runs below its requested capacity by design. That’s not a configuration mistake to fix; it’s a structural cost of how HPA works when it’s tuned in isolation.

Step 3: Replica-Level Guarantees

Minimum replicas are a reliability mechanism. They ensure capacity is always available, even during quiet periods, and they reserve that capacity whether or not it’s being used.

Example:

• Request per pod: 1 CPU
• HPA target: 80%
• minReplicas: 10

This configuration reserves 10 CPUs at all times. Under steady load, only about 8 CPUs are actively used. The rest is idle but unavailable to other workloads, because Kubernetes doesn’t partially reclaim requests from replicas that are mostly idle. From the scheduler’s perspective, those CPUs are owned.

Checkpoint: if reducing traffic doesn’t reduce reserved capacity, minReplicas is usually why. And minReplicas has a habit of only moving in one direction: it goes up after an incident and almost never comes back down once things stabilize.

The asymmetry is easy to understand once you notice it. Raising minReplicas after an outage is a low-risk decision made under pressure, with a clear justification everyone remembers. Lowering it later is a higher-risk decision made during a calm period, with no urgency behind it and no obvious reward if it goes well. Given that asymmetry, it’s not surprising that minReplicas values across a fleet tend to reflect the worst week each service has ever had, rather than its typical week.

Reversing that ratchet manually means someone has to actively decide to lower a safety number with no immediate upside, which is a hard habit to build into a team’s workflow. Zesty’s Multi-Dimensional Autoscaling (MDA) removes the asymmetry by continuously re-evaluating minReplicas against current traffic patterns instead of leaving the decision to whoever last had a reason to raise it, so the value can drift back down as safely as it drifted up.

Step 4: Buffer Multiplication

By this point, two independent buffers exist: per-pod headroom from HPA targets, and per-workload reservations from replica counts. These buffers don’t cancel each other out. They multiply.

Even if every pod is perfectly sized, each pod carries unused headroom, each workload carries unused replicas, and the scheduler treats all of it as non-negotiable. At scale, this effect becomes dominant: large fleets of well-sized workloads still accumulate substantial idle capacity, simply because safety margins stack on top of each other rather than sharing a single, coordinated buffer.

This is why teams often see rightsizing improve their metrics without improving utilization. The dashboards that track request accuracy will show real progress: the gap between what’s requested and what’s used narrows, throttling incidents drop, HPA behaves predictably. None of those dashboards, though, are built to show the combined size of the stacked buffers sitting on top of that accurate baseline, so the improvement registers everywhere except the one place it was supposed to show up, the invoice.

This is also the point where the fix stops being about any single setting. Requests, HPA targets, and minReplicas are three separate reservations that need to be sized against each other, continuously, not tuned once and left alone. Zesty’s Multi-Dimensional Autoscaling (MDA) treats them as one coordinated system rather than three independent knobs: it continuously analyzes real workload behavior and adjusts CPU and memory requests, HPA targets, and replica counts together, so the buffers stop stacking on top of each other and start sharing a single, right-sized margin instead.

Step 5: How the Scheduler Locks In Unused Capacity

The Kubernetes scheduler places pods based on requests, not actual usage. It has no way of knowing that a pod typically uses half of what it asked for.

Example: a pod requests 4 CPUs with an HPA target utilization of 50%. The scheduler places it on a node with at least 4 free CPUs and blocks those CPUs for other pods. No additional pods land there, even if real usage stays around 2 CPUs.

At scale, this produces node fragmentation: nodes appear full from the scheduler’s perspective, actual usage stays low, and autoscalers add nodes instead of packing existing ones more tightly. Common pitfall: assuming low average utilization means the scheduler can rebalance automatically. It can’t. Rebalancing existing pods across nodes isn’t something the default scheduler does on its own.

This is a frequent source of confusion because cluster-wide utilization dashboards often look fine, or at least not alarming, even while individual nodes are heavily fragmented. A cluster running at an average of 55% CPU utilization can still be full of nodes that are 95% allocated and 20% used, sitting next to other nodes running the reverse pattern. The average hides the fragmentation. Only a per-node view of allocated versus actual usage reveals it.

Fixing the requests that caused the fragmentation doesn’t automatically un-fragment the cluster, because the pods are already placed. Zesty’s Adaptive Pod Placement (APP) is built for exactly this gap: it identifies pods that are blocking consolidation and repositions them so nodes carrying that jagged, low-utilization pattern can actually be reclaimed, rather than waiting for the next natural pod churn to fix it by accident.

Step 6: Why Manual Tuning Chases Its Own Tail

Changing one scaling parameter always affects the others. Increasing requests changes how much load each replica can handle. Changing HPA targets alters replica counts. Adjusting minReplicas shifts baseline capacity.

These parameters form a feedback loop, and tuning one in isolation often destabilizes another. There’s no static “correct” configuration, because workloads evolve, traffic patterns change, and yesterday’s optimal setting becomes today’s waste.

Checkpoint: if you find yourself returning to the same YAML every few weeks, this is the loop you’re experiencing, and it’s not a sign you’re tuning badly. It’s a sign the three parameters need to be adjusted together, continuously, rather than one at a time on a schedule set by whoever last got paged.

The reason this loop is so persistent is that each parameter gets owned, informally, by a different trigger. Requests get changed after a capacity incident. HPA targets get changed after a latency complaint. minReplicas gets changed after an on-call engineer decides they never want to see that alert again. None of these changes are wrong in isolation, but none of them are made with visibility into the other two, so the system as a whole drifts toward being over-provisioned on all three dimensions at once, never under-provisioned on any of them, because nobody is incentivized to be the one who tightens a setting and gets paged for it later.

This is the loop Zesty’s Multi-Dimensional Autoscaling is built to end, not by giving any single owner more visibility into the other two settings, but by removing the need for separate, informally-triggered adjustments altogether. Requests, HPA targets, and replica counts get evaluated together on an ongoing basis, so no single incident, complaint, or on-call decision permanently ratchets one dimension upward while the other two sit untouched.

Step 7: Workload Drift, Not Steady State

Even predictable workloads drift, across daily cycles, weekly business patterns, and seasonal demand shifts. To stay safe during peaks, teams often size for the worst case and accept waste during off-peak hours. That waste doesn’t show up in pod metrics, but it’s very real at the infrastructure level.

Static minimums that protect a cluster at 3 PM burn money at 3 AM without anyone noticing, because the pods themselves look fine on every dashboard that only tracks throttling and crash rates.

Sizing for the worst case is a rational response to one specific fear: what happens if demand spikes and there’s no headroom left to absorb it. Zesty’s FastScaler exists to answer that fear directly. It maintains a pool of hibernated nodes with preloaded container images, ready to scale up roughly 5 times faster than a cold start when a spike actually arrives. That speed is what makes it reasonable to size for typical load instead of the worst case in the first place.

Step 8: Stop Optimizing One Resource at a Time

Most HPA configurations scale on CPU. Many workloads are actually constrained by memory, I/O, or GPU availability instead.

A memory-heavy service may never reach its CPU target while still running out of memory. A GPU workload might have a fixed device count with highly variable CPU usage. Optimizing CPU requests alone in these cases can increase waste elsewhere: high memory requests force larger nodes, and the CPU capacity on those nodes becomes stranded.

Multi-resource coupling means the most restrictive resource determines placement and cost, not the most visible one on a CPU-focused dashboard.

This matters more than it initially seems, because most rightsizing efforts start, understandably, with CPU: it’s the metric HPA scales on by default, and it’s the one most monitoring setups surface first. A team can spend weeks tightening CPU requests across a fleet and see real, measurable improvement on every CPU chart, while memory requests sit untouched at whatever value was copy-pasted in at deployment time. If memory is the resource actually determining node size for a given workload, all that CPU work has no effect on the bill, because the node was never going to shrink on the CPU dimension in the first place.

This is the specific gap the “multi-dimensional” part of Zesty’s Multi-Dimensional Autoscaling is named for: it adjusts CPU and memory requests together, as one system, rather than optimizing whichever resource happens to be easiest to see on a dashboard.

Step 9: Kubernetes Efficiency Isn’t the Same as Cloud Cost Reality

Cloud providers bill for nodes and commitments, not pods. You can reduce pod requests and still run the same number of nodes, hold the same instance commitments, and pay for capacity that Kubernetes can’t consolidate.

Affinity rules, anti-affinity, and the absence of active bin-packing all prevent node scale-down, even when every pod on the cluster is efficiently sized. Checkpoint: if node count doesn’t decrease after rightsizing, cost won’t either. This is the step where a lot of otherwise well-executed rightsizing projects stop paying off without anyone noticing, because the savings were assumed to flow through to the node count automatically, and they don’t.

This gap between Kubernetes-level efficiency and cloud-bill-level savings is often the most frustrating part of a rightsizing initiative, precisely because every internal metric looks like a success. Requests match usage. HPA behaves predictably. minReplicas is tuned to the lowest safe value. And the AWS invoice arrives looking almost exactly like it did before the project started, because none of that pod-level accuracy translated into fewer nodes running underneath it. The infrastructure layer and the workload layer are optimizing for different things, and accuracy at one layer doesn’t automatically produce results at the other.

This is the same consolidation gap from Step 5, showing up again at the billing layer instead of the utilization layer. Accurate requests make a smaller cluster possible; they don’t make it happen if pods are pinned in place by affinity rules or disruption budgets. Zesty’s Adaptive Pod Placement (APP) is what closes that specific gap, freeing up the node count that rightsizing alone made available but couldn’t reach on its own.

Accept the Right Waste as a Design Tradeoff, Then Optimize Above It

Accurate requests are essential. Without them, nothing else works correctly. But they were never meant to be a complete cost solution on their own.

Kubernetes intentionally trades some utilization for reliability, predictable scaling, and operational safety. The resulting waste is structural, not accidental, which is exactly why it doesn’t respond to more careful manual tuning. It responds to system-level thinking that spans pods, workloads, nodes, and cloud billing constructs together.

If accurate requests are your foundation, the real work starts above them, at the layer where buffers stack, placement fragments, and node count stops moving even though every pod on the dashboard looks correctly sized. That’s a different project than the one most teams start with, and it’s the one that actually shows up on the bill.

This is the layer Zesty is built for. Multi-Dimensional Autoscaling (MDA) keeps requests, HPA targets, and replica counts sized against each other continuously, so the buffers described above stop stacking. Adaptive Pod Placement (APP) clears the consolidation blockers that keep node count high even when every pod is accurately sized. And FastScaler removes the reason teams size for the worst case in the first place, absorbing spikes fast enough that a leaner baseline is safe to run. Together, they cover the pod, node, and drift layers that this article walks through, so the work doesn’t stop at accurate requests, it continues automatically above them.

FAQs