Lazy-Pulling in Containerd v2: Why Pull Time Isn't the Right Metric

Lazy-Pulling in Containerd v2: Why Pull Time Isn’t the Right Metric

by

If you’re running containerized workloads at scale, you’ve likely encountered the cold-start problem: a pod gets scheduled, the container runtime begins pulling a multi-gigabyte image, and your application sits idle while layers download. This latency directly impacts developer velocity, autoscaling responsiveness, and user experience.

Related reading: daemonless image tooling with Skopeo, Crane and regctl.

Containerd’s lazy-pulling feature (also known as on-demand pulling or stargz) promises to solve this by fetching only the metadata initially, then downloading individual file chunks as the container accesses them. The promise is compelling—near-instant container startup regardless of image size. However, the community conversation around this feature has been dominated by a misleading metric: pull time.

In this article, we’ll dismantle the pull time benchmark fallacy and examine the real performance characteristics that matter for production workloads. We’ll explore how lazy-pulling actually works under the hood, what metrics you should be measuring instead, and when this technology makes sense for your infrastructure.

How Lazy-Pulling Actually Works in Containerd

Before we can understand the metrics, we need to understand the mechanism. Traditional container image pulling follows a sequential, all-or-nothing approach:

  1. The container runtime requests an image manifest.
  2. It downloads all layer blobs (compressed tarballs) to local storage.
  3. It decompresses and extracts all layers to create the final root filesystem.
  4. Only then can the container start.

Lazy-pulling fundamentally changes this workflow. Based on the containerd operations documentation and community implementations, the process looks more like this:

  1. The runtime fetches the image manifest and a specialized index file that maps file paths to byte ranges within compressed layers.
  2. It prepares a virtual filesystem that appears to contain the complete image.
  3. When a process inside the container attempts to read a file, the filesystem driver intercepts the request.
  4. It consults the index, fetches only the specific byte ranges needed for that file from the registry, decompresses them on-the-fly, and serves the data to the application.
  5. Fetched chunks are cached locally to avoid redundant network requests.

This approach is conceptually similar to how modern game engines stream assets or how HTTP range requests enable video streaming. The key insight is that most containers only need a small subset of their total image content during initial startup.

The Pull Time Fallacy: What Traditional Benchmarks Get Wrong

When lazy-pulling first gained attention, the most common benchmark comparison was simple: “How long does it take to pull a 1GB image versus lazy-pull the same image?” This metric is not just incomplete—it’s actively misleading for several reasons.

1. Pull Time Measures the Wrong Phase

With traditional pulling, the pull command blocks until all layers are downloaded and extracted. With lazy-pulling, the pull command returns almost immediately after fetching metadata. Comparing these two durations is like comparing the time to download a movie trailer versus the time to buffer the entire film—they’re measuring fundamentally different operations.

The meaningful comparison isn’t pull time versus lazy-pull time. It’s total time to application readiness. This includes:

  • Metadata fetch time
  • Container initialization time
  • Application bootstrap time (where file accesses actually happen)
  • Any network latency incurred during on-demand fetching

2. It Ignores Workload Access Patterns

As detailed analysis has shown, the performance impact of lazy-pulling depends entirely on your container’s filesystem access patterns during startup. Consider two extreme examples:

  • Node.js application: Starts quickly, requires few files (node binary, package.json, a handful of .js files). Lazy-pulling provides near-instant startup.
  • Java application with large classpath: Scans hundreds of JAR files during initialization, triggering many small reads across the filesystem. This could result in hundreds of HTTP range requests, adding significant latency.

A single “pull time” metric cannot capture this complexity. The real question is: How does the filesystem access pattern during application startup map to the image’s layer structure?

3. It Misses the Caching Benefit

Traditional pulling has a clear caching model: once a layer is pulled, subsequent containers using that layer start instantly (assuming no registry changes). Lazy-pulling introduces a more nuanced caching model where individual file chunks are cached after first access.

This means the performance of lazy-pulling improves for frequently accessed files across container instances, but benchmarking typically focuses on cold starts only.

The Metrics That Actually Matter for Production

To properly evaluate lazy-pulling for your workloads, you need to measure these key performance indicators:

1. Time-to-First-Byte (TTFB) for Application Requests

This is the most critical metric for user-facing services. How long from kubectl apply or horizontal pod autoscaler decision until your application can serve its first request? This metric encompasses:

  • Pod scheduling
  • Image provisioning (metadata fetch for lazy-pulling)
  • Container runtime initialization
  • Application startup (including any on-demand file fetching)
  • Health check passes

For autoscaling scenarios, TTFB directly impacts your ability to handle traffic spikes.

2. Application Startup Latency Breakdown

Instead of measuring “pull time,” instrument your application to report key milestones:

# Example instrumentation points for a web application
START_TIME=$(date +%s.%N)
# ... application code ...
CONTAINER_READY=$(date +%s.%N)  # Runtime reports container running
IMPORTS_LOADED=$(date +%s.%N)   # After requiring dependencies
DATABASE_CONNECTED=$(date +%s.%N) # After DB connection established
HTTP_LISTENING=$(date +%s.%N)   # Server listening on port

By comparing these timestamps between traditional and lazy-pulled containers, you can identify exactly where latency is introduced or saved.

3. Network Request Patterns During Startup

Monitor the filesystem layer to understand what’s being fetched:

  • How many HTTP range requests occur during startup?
  • What’s the total bandwidth transferred versus the full image size?
  • What’s the request pattern (sequential vs random access)?
  • How effective is the local chunk cache?

This data helps you optimize image construction. If your application sequentially reads every file in /usr/lib during startup, you might want to ensure those files are contiguous in the image layer to minimize range requests.

4. Registry Load and Scaling Characteristics

Lazy-pulling changes the load pattern on your container registry. Instead of occasional large downloads, you get frequent small range requests. You need to measure:

  • Registry requests per second during scaling events
  • Cache hit ratios at registry CDN level
  • Impact on registry authentication systems (more requests = more token validation)

Implementation Considerations and Gotchas

Based on the Kubernetes images documentation and containerd’s implementation, here are practical considerations for adopting lazy-pulling:

Compatibility and Requirements

Lazy-pulling requires specific configuration and compatible images:

  • Runtime support: Containerd with appropriate snapshotter (e.g., stargz, overlaybd)
  • Image format: Images must be built with lazy-pulling support (e.g., using ctr-remote or nerdctl with stargz compression)
  • Registry support: The registry must support HTTP range requests (most do, but verify)
  • Kubernetes version: No specific version requirement, but newer kubelets have better integration

When Lazy-Pulling Excels

  • Large base images with small applications: e.g., a 2GB ubuntu:latest base running a 50MB Go binary
  • Ephemeral workloads: CI/CD runners, batch jobs, function-as-a-service platforms
  • Memory-constrained environments: Lazy-pulling avoids decompressing entire layers to disk
  • Autoscaling scenarios: When you need to scale from zero quickly

When Traditional Pulling May Be Better

  • Applications with random access patterns: Databases that read many files during startup
  • Network-limited environments: If each range request has high latency, the overhead adds up
  • Air-gapped or registry-constrained deployments: Predictable bandwidth usage may be preferable
  • Images with poor locality: Files accessed together aren’t stored together in the image

Practical Recommendations for Platform Teams

1. Profile Before You Optimize

Don’t implement lazy-pulling globally based on synthetic benchmarks. Start with profiling:

  1. Identify your most frequently scaled or restarted workloads
  2. Measure their actual file access patterns during startup (using strace or eBPF tools)
  3. Build test images with lazy-pulling support
  4. A/B test against traditional images in staging
  5. Measure the real metrics: TTFB, resource usage, registry load

2. Optimize Image Construction

Lazy-pulling performance depends heavily on image layout. Follow these guidelines:

  • Place files accessed during startup close together in the layer
  • Keep startup dependencies in as few layers as possible
  • Consider separating runtime dependencies from build dependencies
  • Use multi-stage builds to minimize final image size

3. Implement Gradual Rollout

When deploying lazy-pulling in production:

  1. Start with non-critical, ephemeral workloads
  2. Implement comprehensive monitoring for both performance and errors
  3. Set up alerts for abnormal request patterns to your registry
  4. Have a rollback plan (traditional images still available)

4. Monitor the Right Things

Beyond application metrics, monitor infrastructure impacts:

  • Registry request rate and latency percentiles
  • Node filesystem cache efficiency
  • Network bandwidth patterns (many small requests vs few large ones)
  • Container startup failure rates

The Future of Container Image Distribution

Lazy-pulling represents a fundamental shift from treating container images as monolithic blobs to treating them as structured, queryable datasets. This evolution will likely continue with:

  • Intelligent prefetching: Systems that learn access patterns and prefetch likely-needed chunks
  • Content-defined chunking: More efficient than fixed-range requests
  • Peer-to-peer distribution: Nodes sharing fetched chunks locally
  • Integration with GitOps: Knowing which image versions will be needed based on Git changes

The key takeaway is that we’re moving beyond simple “pull time” optimization toward holistic container lifecycle optimization. The metric that matters isn’t how fast we can download bytes, but how quickly we can deliver working applications to users.

Frequently Asked Questions

Conclusion

Lazy-pulling in containerd v2 is a powerful technology that can dramatically improve container startup performance, but only if measured and implemented correctly. The traditional “pull time” benchmark is a relic of an earlier approach to container images and should be replaced with application-centric metrics like time-to-first-byte and startup latency breakdowns.

For platform engineers and SREs, the path forward is clear: profile your actual workloads, understand their filesystem access patterns, test lazy-pulling with realistic metrics, and implement gradually with proper monitoring. When applied to the right workloads with optimized images, lazy-pulling can transform your scaling responsiveness and resource efficiency.

Remember: the goal isn’t faster image pulls—it’s faster applications. Measure what matters.