When I first started working with Kubernetes, the scheduler felt like a black box. I’d deploy something, and it would just end up… somewhere. Sometimes it worked perfectly; other times, critical workloads would land on overloaded nodes while others sat idle. I’ve seen applications slow to a crawl because they were sharing resources with noisy neighbors or even crash because the node didn’t have enough capacity. It’s frustrating when you’re trying to run a reliable system and feel like you’re not in control.

The scheduler is responsible for determining where every pod in your cluster will run. By default, it does a decent job balancing workloads across available nodes, but it doesn’t know the specific requirements of your application unless you tell it. That’s where tools like node selectors and affinity rules come in. They let you influence the scheduler’s decisions so you can optimize for performance, reliability, and cost-efficiency.

In this article, we’ll dig into how Kubernetes scheduling works, why it’s important to guide the scheduler’s decisions, and how to use node selectors, affinity, and anti-affinity to take control. By the end, you’ll have the tools and knowledge to ensure your applications run on the right nodes, avoiding bottlenecks and ensuring a smooth, efficient deployment.

Node Selector: The Basics

Node selectors are the simplest way to control scheduling. Think of them as a filter: you’re telling Kubernetes, “Only run this pod on nodes that match these labels.” Here’s how it works:

  1. Label Your Nodes: Add labels to your nodes to categorize them. For example: kubectl label node node1 environment=production kubectl label node node2 environment=staging
  2. Use Node Selectors in Your Pod Specs: In your pod’s YAML file, specify a nodeSelector that matches the node labels: apiVersion: v1 kind: Pod metadata: name: example-pod spec: containers: - name: nginx image: nginx nodeSelector: environment: production

Now, this pod will only run on nodes labeled environment=production. It’s a straightforward way to target specific nodes, but it’s also quite rigid. If there are no matching nodes, the pod will stay pending indefinitely.

Using Node Selectors in GitOps Environments

If your cluster is managed using a GitOps approach with tools like ArgoCD, making manual changes to individual objects—like directly editing pod specifications—can lead to conflicts with your infrastructure’s source of truth. GitOps systems continuously reconcile the actual state of your cluster with the desired state stored in a Git repository. Any changes made outside this system might be overwritten, causing unexpected behavior.

To avoid this, always apply node selector configurations in the Git repository that manages your cluster’s state. For example, update the Helm chart, Kustomize overlay, or YAML manifest stored in Git to include the node selector settings. This ensures that the changes persist and align with your cluster’s declarative configuration management approach.

Node Affinity: A More Flexible Approach

Node affinity is like an upgraded version of node selectors. It gives you more control and flexibility by allowing you to define “soft” and “hard” rules. Node affinity uses nodeAffinity in your pod spec and supports two types of rules:

  1. Required During Scheduling, Ignored During Execution: These rules must be satisfied for the pod to be scheduled.
  2. Preferred During Scheduling, Ignored During Execution: These rules act as preferences rather than strict requirements.

Here’s an example:


  apiVersion: v1
kind: Pod
metadata:
  name: example-pod
spec:
  containers:
    - name: nginx
      image: nginx
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: environment
                operator: In
                values:
                  - production
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 1
          preference:
            matchExpressions:
              - key: tier
                operator: In
                values:
                  - backend


In this example:

  • The pod will only be scheduled on nodes with environment=production (hard rule).
  • Among those nodes, it will prefer nodes labeled tier=backend (soft rule).

When to Use Node Selectors vs. Node Affinity

Node selectors are great for simple use cases where you need pods to run on specific nodes. However, if you need more nuanced control—like prioritizing certain nodes while still allowing flexibility—node affinity is the better choice. For example:

  • Use Node Selectors: When you have dedicated nodes for a specific purpose, such as GPU workloads.
  • Use Node Affinity: When you want pods to prefer certain nodes but still run elsewhere if those nodes aren’t available.

Combining Affinity and Anti-Affinity

Sometimes, it’s not just about where pods should go—it’s also about where they shouldn’t go. This is where pod anti-affinity comes in. Pod anti-affinity ensures that certain pods don’t run on the same node, which can be useful for high availability.

Here’s an example of pod anti-affinity:


  apiVersion: v1
kind: Pod
metadata:
  name: example-pod
spec:
  containers:
    - name: nginx
      image: nginx
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
              - key: app
                operator: In
                values:
                  - nginx
          topologyKey: "kubernetes.io/hostname"

This configuration prevents pods with the label app=nginx from being scheduled on the same node. It’s a simple yet effective way to spread workloads across nodes.

Scheduling Strategies: Pod Topology Spread Constraints and Resource Bin Packing

Beyond node selectors and affinity rules, Kubernetes offers built-in strategies to optimize workload distribution across nodes. These strategies focus on balancing resources or consolidating workloads efficiently, providing flexibility in meeting different operational goals.

  1. Pod Topology Spread Constraints: This strategy ensures even distribution of pods across specified topological domains, such as zones or nodes, to enhance fault tolerance. For example, you can configure a spread constraint to prevent all replicas of a service from running on the same node, reducing the risk of a single point of failure. This feature allows you to define rules that specify how pods should be distributed relative to each other within the cluster. Learn more in the official Kubernetes documentation.
  2. Resource Bin Packing: This strategy aims to consolidate workloads onto a minimal number of nodes by optimizing resource usage. For instance, you can prioritize nodes based on their available CPU or memory, scheduling pods in a way that fills resource gaps efficiently. This is particularly valuable in cost-sensitive environments, as it reduces the number of active nodes and thus lowers operational expenses. See details in the resource bin packing documentation.

It’s important to note that while these strategies are available in self-managed or on-premises clusters, managed Kubernetes services like Amazon EKS often rely on the default scheduler behavior without offering direct customization. This makes these features more accessible in environments where you have greater control over the cluster configuration.

For advanced use cases, Kubernetes also supports scheduler extensions. These allow developers to write custom filtering or scoring plugins to tailor the scheduling process to specific requirements. While covering extensions in depth is beyond the scope of this section, they offer a powerful way to address unique scheduling challenges.

Practical Tips for Scheduler Control

  1. Plan Your Labels Carefully: A well-thought-out labeling strategy makes all the difference. Use consistent and meaningful labels across your nodes.
  2. Test Configurations in Staging: Before applying complex affinity rules in production, test them in a staging environment to ensure they behave as expected.
  3. Monitor Scheduling Behavior: Use tools like kubectl describe pod to debug scheduling issues and verify that your rules are working correctly.
  4. Avoid Over-Constraining: Overly strict rules can leave pods unschedulable, so balance constraints with flexibility.
  5. Work with GitOps Systems: If you’re using GitOps tools like ArgoCD, make all scheduler-related configurations—such as node selectors or affinity rules—directly in the Git repository that manages your cluster’s state. This prevents conflicts and ensures that changes persist without being overwritten by automated reconciliation processes.
Get K8s tips straight to your inbox

No spam—just strategies to help you excel at what you do.

Plan your configurations ahead

The Kubernetes scheduler is powerful, but you need to understand how to work with it to harness its full potential. By understanding and leveraging tools like node selectors, affinity, and anti-affinity, you can ensure that your workloads are running in the right place at the right time. Take the time to plan your configurations, and you’ll have a cluster that’s not just functional but optimized for your specific needs.