Kubernetes Image Cleanup Timeout: containerd vs Kubelet Explained

Introduction

If you’ve ever searched for “Kubernetes image cleanup timeout,” you’ve probably run into vague answers suggesting that containerd automatically deletes images after some period of time. That’s not how it works.

The short answer: containerd has no default time-based image cleanup timeout. It does not delete images simply because they’ve been sitting on a node for a certain number of hours or days. Image lifecycle management in Kubernetes is driven by the kubelet, not the container runtime.

This post explains exactly how image garbage collection works in Kubernetes, what containerd actually does, and how to configure the kubelet to match your operational needs.

Key Takeaways

  • containerd does not delete images based on age or a timeout. Its garbage collector only removes unreferenced content (layers, manifests) that no image points to.
  • The kubelet is responsible for image garbage collection in Kubernetes, triggered by disk usage thresholds.
  • Default thresholds: image GC starts at 85% disk usage and stops at 80%.
  • imageMaximumGCAge exists as an optional field but defaults to "0s" (disabled).
  • Images are deleted oldest-unused-first when GC is triggered.

Does containerd Automatically Clean Images After a Timeout?

No. This is one of the most common misconceptions in Kubernetes operations.

containerd does not track how long an image has been sitting unused. Its garbage collector only removes leftover data — such as image layers — after the image itself has been explicitly deleted and no other image shares those same layers. As long as an image exists in containerd’s store, it stays. There is no expiry, no TTL, and no scheduled cleanup based on age.

containerd does support containerd.io/gc.expire labels on leases and certain resources, but these are used by clients managing temporary operations (like in-progress pulls), not for general image lifecycle management.

How Kubernetes Actually Handles Image Cleanup

In a Kubernetes cluster, the kubelet on each node is the component responsible for deciding when to remove container images. It does this through its built-in image garbage collection loop.

The kubelet periodically checks disk usage on the filesystem where images are stored (known as imageFs). When usage crosses a configured threshold, the kubelet instructs the container runtime (containerd, CRI-O, etc.) to remove unused images until disk usage drops below a lower threshold.

The key point: image cleanup is disk-pressure-driven, not time-driven.

Default Kubelet Image GC Behavior

The following are the default kubelet garbage collection settings in Kubernetes:

Parameter Default Value Description
imageGCHighThresholdPercent 85 Image GC is triggered when disk usage reaches 85%
imageGCLowThresholdPercent 80 Image GC stops once disk usage drops to 80%
imageMinimumGCAge 2m Images must be unused for at least 2 minutes before GC can remove them
imageMaximumGCAge 0s (disabled) No maximum age enforced by default; images are not removed based on age alone

Additional GC intervals:

  • Image GC check interval: every 5 minutes
  • Container GC check interval: every 1 minute

When image GC is triggered, the kubelet removes images in order of last usage time, starting with the oldest-unused images first, until disk usage falls below the low threshold.

Comparison: containerd vs Kubelet Image Cleanup

Behavior containerd GC Kubelet Image GC
Trigger mechanism Unreferenced content (no image pointing to it) Disk usage exceeds threshold
Time-based cleanup No (no default timeout) Optional via imageMaximumGCAge (disabled by default)
Disk-aware No Yes (primary trigger)
Deletion order Any unreferenced content Oldest unused images first
Check frequency Event-driven (on content removal) Every 5 minutes
Scope Content blobs and snapshots Full container images
Configurable Via labels and leases (advanced) Via KubeletConfiguration fields

Why Images Disappear from Nodes

If images are vanishing from your nodes and you didn’t explicitly remove them, here are the most common causes:

  • Kubelet disk-pressure GC: Disk usage crossed the 85% threshold, triggering automatic cleanup of unused images. This is the most frequent cause.
  • Node recycling or scaling events: In cloud environments, autoscaler-managed nodes are ephemeral. When a node is terminated and replaced, all local images are lost.
  • Manual cleanup: Someone ran crictl rmi, ctr image rm, or a cleanup CronJob on the node.
  • DaemonSet or system tool: Some clusters run image pruning tools (like Eraser or custom scripts) as DaemonSets.
  • imageMaximumGCAge configured: If your cluster has explicitly set this field, images unused beyond that duration will be removed regardless of disk pressure.

How to Check Kubelet Configuration

To inspect the active kubelet configuration on a node, including image GC settings:

kubectl get --raw "/api/v1/nodes/<NODE_NAME>/proxy/configz" | jq '.kubeletconfig'

Look for these fields in the output:

{
  "imageGCHighThresholdPercent": 85,
  "imageGCLowThresholdPercent": 80,
  "imageMinimumGCAge": "2m0s",
  "imageMaximumGCAge": "0s"
}

If imageMaximumGCAge is "0s", age-based cleanup is disabled and only disk-pressure triggers apply.

Example Kubelet Configuration (YAML)

Here’s a KubeletConfiguration snippet with image GC settings tuned for a production workload:

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
imageGCHighThresholdPercent: 80
imageGCLowThresholdPercent: 70
imageMinimumGCAge: "5m"
imageMaximumGCAge: "168h"   # Remove unused images older than 7 days

In this example, imageMaximumGCAge is set to 168 hours (7 days), meaning any image unused for more than a week will be garbage collected regardless of disk pressure. Adjust this value based on your image pull latency, registry availability, and node disk capacity.

Best Practices

  • Set imageMaximumGCAge in security-sensitive environments. Stale images may contain known vulnerabilities. A 7- to 14-day maximum age ensures old images don’t linger on nodes indefinitely.
  • Monitor disk usage on imageFs. Use node-exporter filesystem metrics (e.g., node_filesystem_avail_bytes) or the kubelet summary API (/stats/summary) to track the image filesystem. Alert before you hit the 85% threshold to avoid surprise GC events.
  • Lower thresholds on nodes with small disks. If your nodes have limited storage, consider setting imageGCHighThresholdPercent to 75 and the low threshold to 65.
  • Don’t rely on containerd for image lifecycle management. containerd’s GC is a low-level mechanism for cleaning orphaned blobs. It is not designed for policy-driven image cleanup.
  • Pre-pull critical images. If certain images must always be available (e.g., for fast failover), use a DaemonSet with imagePullPolicy: IfNotPresent to keep them referenced and protected from GC.
  • Audit cleanup tools. If you run third-party image pruning tools, ensure they don’t conflict with kubelet GC behavior.

Frequently Asked Questions

Does containerd delete images after 24 hours?

No. containerd has no time-based image deletion. The 24-hour expiry you may have seen in documentation refers to lease expiration for temporary operations (like in-progress image pulls), not image cleanup.

What triggers image deletion in Kubernetes?

The kubelet triggers image deletion when disk usage on the image filesystem exceeds imageGCHighThresholdPercent (default: 85%). It removes unused images, oldest first, until usage drops below imageGCLowThresholdPercent (default: 80%).

Can I force images to be deleted after a specific time?

Yes. Set imageMaximumGCAge in your KubeletConfiguration. For example, "72h" will remove any image unused for more than 3 days. This field defaults to "0s" (disabled).

Will images used by running pods be deleted?

No. The kubelet only garbage collects images that are not currently in use by any running container on the node. Additionally, imageMinimumGCAge (default: 2 minutes) ensures that recently used images are not immediately eligible for GC after their last container stops.

How do I prevent a specific image from being garbage collected?

The most reliable Kubernetes-native approach is to keep the image in use by running a lightweight DaemonSet that references it with imagePullPolicy: IfNotPresent. This ensures the image stays referenced on every node and is protected from GC.

Does this behavior differ between containerd and CRI-O?

The kubelet image GC logic is runtime-agnostic. It works the same way whether your nodes use containerd or CRI-O. The kubelet communicates via the CRI (Container Runtime Interface) to list and remove images.

Conclusion

There is no default image cleanup timeout in containerd. It does not remove images based on age, schedule, or elapsed time. Its garbage collector only cleans up unreferenced content blobs.

In Kubernetes, image lifecycle management is handled entirely by the kubelet, driven by disk usage thresholds. By default, cleanup kicks in at 85% disk usage and stops at 80%. If you need time-based cleanup, explicitly configure imageMaximumGCAge in your KubeletConfiguration.

Understanding this distinction prevents misdiagnosis when images disappear from nodes and helps you build reliable, predictable image management strategies for your clusters.

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top