Types of Ephemeral Storage

Kubernetes provides several types of ephemeral storage, each suited for different use cases:

  1. emptyDir:
    • emptyDir is a temporary directory created when a pod is assigned to a node. The storage lasts as long as the pod is running on the node but is deleted once the pod is terminated.
    • Use Case: emptyDir is often used as scratch space for temporary data, such as logs, cache files, or intermediary computation data. Containers in the same pod can share the emptyDir volume, making it useful for collaborative tasks between containers.
  2. ConfigMap and Secret Volumes:
    • ConfigMap and Secret volumes are technically ephemeral storage types because they provide data temporarily to pods. They are typically used to inject configuration data or sensitive information like passwords and API tokens.
    • Use Case: These volumes are ideal for storing configuration data that needs to be accessible across pods but does not require persistent storage. Data in ConfigMaps and Secrets can be updated without restarting the pod, providing flexibility for dynamic configurations.
  3. Downward API Volume:
    • The Downward API volume provides pod-specific metadata (e.g., pod name, namespace) to containers. This is an ephemeral storage type because it provides data related to the pod that may change or become irrelevant once the pod terminates.
    • Use Case: Use the Downward API volume to make pod information available to applications, enabling them to adjust behavior based on metadata.
  4. container storage (Root Filesystem):
    • Every container in Kubernetes has a root filesystem, which is ephemeral by nature. Data written here is lost once the container or pod is deleted or restarted.
    • Use Case: Temporary application data, such as log files or temporary caches, can be stored in the container’s root filesystem. However, it’s best practice to use dedicated volumes for larger or more critical temporary data to avoid filling up the container’s primary storage.
  5. Ephemeral Volume Types:
    • Kubernetes 1.16 introduced the concept of generic ephemeral volumes, which are dynamically provisioned and attached to a pod. Unlike other ephemeral types, these volumes are created and managed through Persistent Volume Claims (PVCs) but are deleted when the pod is removed.
    • Use Case: Ephemeral volumes provide a way to dynamically provision storage for applications needing temporary storage but with more control over volume characteristics, such as storage class, size, and access modes.

How Ephemeral Storage Works

Ephemeral storage is managed locally on the node where the pod is scheduled. Kubernetes divides the available storage on each node into two main categories:

  1. Root Filesystem (rootfs):
    This is the primary filesystem on the node, where Kubernetes stores the OS files, binaries, logs, and application data. The root filesystem is limited and shared by all pods on the node, so excessive usage by a single pod can affect other workloads.
  2. Quota (emptyDir and other temporary volumes):
    Kubernetes can allocate a portion of a node’s available disk space for ephemeral storage. Administrators can set limits that each pod or container can use by specifying ephemeral-storage requests and limits.

Configuring Ephemeral Storage in Pods

Kubernetes allows you to define limits on a per-container basis using resources.requests and resources.limits fields in the pod specification. These fields control the storage available to each container, helping manage node storage resources and avoid potential conflicts with other containers on the same node.

Example Pod YAML with ephemeral storage limits:


  apiVersion: v1
kind: Pod
metadata:
  name: example-pod
spec:
  containers:
  - name: example-container
    image: nginx
    resources:
      requests:
        ephemeral-storage: "1Gi"
      limits:
        ephemeral-storage: "2Gi"
    volumeMounts:
    - mountPath: /tmp
      name: temp-storage
  volumes:
  - name: temp-storage
    emptyDir: {}

In this example:

  • The container requests 1Gi of ephemeral storage but is limited to 2Gi. If it exceeds the limit, Kubernetes will evict the pod based on its quality of service (QoS) class to free up space.

Use Cases and Best Practice

Ephemeral storage is valuable for tasks where data persistence beyond the pod’s lifetime is unnecessary. Examples include:

  • Data Caching: Applications that process large datasets often use ephemeral storage to cache intermediary results.
  • Log Aggregation: Short-lived containers may write logs, which are then collected by logging agents before the pod is terminated.
  • Temporary Files: Is commonly used for temporary file storage, such as scratch files or transient data generated during processing.

Best Practices:

  1. Set Limits: Define ephemeral-storage limits to prevent pods from consuming excessive node storage, which could impact other pods.
  2. Monitor Node Storage: Use monitoring tools like Prometheus to track usage across nodes and pods. Excessive storage usage by one pod can lead to node pressure and trigger evictions.
  3. Avoid Critical Data in Ephemeral Storage: Ephemeral storage is not meant for data that must survive beyond pod restarts. Use Persistent Volumes for critical data needs.

Challenges

  1. Node Storage Pressure: Since ephemeral storage is shared on the node, overuse by one pod can affect other workloads and lead to performance issues or pod evictions.
  2. Lack of Persistence: Data stored in ephemeral volumes is lost when a pod is terminated or evicted, which limits the use cases for applications that require data persistence.
  3. Monitoring Complexity: Managing ephemeral storage across multiple pods requires careful monitoring to avoid resource conflicts. This is especially important in multi-tenant environments or clusters with dynamic workloads.

Monitoring Usage

Monitoring is essential to ensure nodes aren’t overwhelmed by excessive usage. Kubernetes reports ephemeral storage usage metrics, which can be accessed via tools like Prometheus or Grafana. By setting up alerts on storage usage thresholds, administrators can detect and address high usage before it impacts node performance.

Sample Prometheus Metric:

  • kubelet_volume_stats_used_bytes: Reports used storage for a specific volume in a pod.
  • kube_pod_container_resource_requests_storage_bytes: Tracks storage requests per pod, useful for ensuring requests stay within node limits.

References and Further Reading

  1. Kubernetes Official Documentation on Ephemeral Storage
    Ephemeral Storage in Kubernetes – Kubernetes documentation detailing ephemeral storage and how to manage it with requests and limits.
  2. Using emptyDir Volumes
    emptyDir Volume Documentation – A guide on using emptyDir volumes in Kubernetes for temporary storage within pods.
  3. Ephemeral Volume Types
    Ephemeral Volumes in Kubernetes – Learn about ephemeral volume types, including generic ephemeral volumes introduced in Kubernetes 1.16.
  4. ConfigMap and Secret Volumes
    ConfigMap and Secret Documentation – Overview of ConfigMaps and Secrets, explaining how these types are used to manage configurations and sensitive data in Kubernetes.
  5. Kubernetes Quality of Service (QoS) Classes
    Kubernetes QoS Documentation – Learn how Kubernetes uses QoS classes to manage pod resources, under node pressure conditions.