Worker OOM when exporting cache after builds with different compression types for images with lots of layers

[Fricounet]

Contributing guidelines and issue reporting guide

• I've read the contributing guidelines and wholeheartedly agree. I've also read the issue reporting guide.

Well-formed report checklist

• I have found a bug that the documentation does not mention anything about my problem
• I have found a bug that there are no open or closed issues that are related to my problem
• I have provided version/information about my environment and done my best to provide a reproducer

Description of bug

Bug description

The bug triggers when trying to export the build cache for an image that has a lot of layers (30 will surely trigger the bug for instance). If the same image was previously built on the host with a different compression type, the cache export step will cause the buildkit worker to OOM after a few seconds.

I think this bug was introduced in #2405. This PR, and in particular the getAvailableBlobs function tries to build the connection graph between all layers in a build and all the available compressions already present in the worker.
My understanding of the function is that:

• recursive function that unstacks the layer parent chain until it reaches the last layer in the chain
• then, going from the root blob down to every child blobs, it finds each compression types available for the blob
• for each compression and for each child blob, we create a new remote which contains all possible parents

This behavior causes the complexity to explode as we end up generating all possible combinations of compression types in the remotes. The memory complexity of the operation is c^#layers with c the number of compression types for the image. If c=2 (gzip and zstd) and we have an image with 30 layers, if we only considered that each remote only contains an integer, even though the object is way larger, the total memory needed would be 2^30*8bytes = 8.5TB (way higher in practice as each remote wheighs more than an integer)

Additionally, this situation was worsened by this PR #2603. It introduced a walkBlob function to find the different compression for a blob. walkBlob is called with a function that adds a descriptor to a list each time the function is called. However, the function is called more than once per blob (it’s called twice of the first blob) which means the final list has duplicated elements. Which means that the function complexity is now (c+1)^#layers

Fixing walkBlob is easy enough but I'm not sure how to fix the root issue: if we need to keep the full combination matrix of blobsXcompression, I don't see how we can have a reasonable amount of memory used. But maybe we don't need the full matrix?

Reproduction

Any build using an image with enough layers will trigger the bug. For instance:

FROM mcr.microsoft.com/devcontainers/universal:2.13.0-linux
RUN echo "test"

Optional, start a registry if you don't have one available:

docker run -d -p 5000:5000 --name registry registry

Trigger a zsh build with force compression:

docker buildx build --platform=linux/amd64 -f Dockerfile --output type=image,compression=zstd,force-compression=true,oci-mediatypes=true .

Trigger a gzip build without force compression and export the build cache:

docker buildx build --platform=linux/amd64 -f Dockerfile --cache-to=type=registry,ref=localhost:5000/repro/cache,mode=max .

note: this works in this case because the initial image mcr.microsoft.com/devcontainers/universal:2.13.0-linux
is using gzip layers. If it was using zstd layer, the first build would have to force-compress to gzip instead.

The second build should fail because the buildkit worker will be OOM killed.

Version information

(buildkit version is coming from my fork, which dates back from May, the additional commit just adds some binaries to the final image, unrelated to the bug)

buildctl --version && buildkitd version
buildctl github.com/moby/buildkit 0cb7f6c1b.m 0cb7f6c1bb7a68a0eb42b941b22634c1661f9d20.m
INFO[2025-07-16T09:47:59Z] auto snapshotter: using overlayfs
WARN[2025-07-16T09:47:59Z] using host network as the default
INFO[2025-07-16T09:47:59Z] found worker "msafv6ydxpayr0pa0qwcr2i0r", labels=map[org.mobyproject.buildkit.worker.executor:oci org.mobyproject.buildkit.worker.hostname:baptiste-box org.mobyproject.buildkit.worker.network:host org.mobyproject.buildkit.worker.oci.process-mode:sandbox org.mobyproject.buildkit.worker.selinux.enabled:false org.mobyproject.buildkit.worker.snapshotter:overlayfs], platforms=[linux/amd64 linux/amd64/v2 linux/amd64/v3 linux/386]
WARN[2025-07-16T09:47:59Z] skipping containerd worker, as "/run/containerd/containerd.sock" does not exist
INFO[2025-07-16T09:47:59Z] found 1 workers, default="msafv6ydxpayr0pa0qwcr2i0r"
WARN[2025-07-16T09:47:59Z] currently, only the default worker can be used.
INFO[2025-07-16T09:47:59Z] running server on /run/buildkit/buildkitd.sock

docker buildx version && docker buildx inspect debug
github.com/docker/buildx v0.14.0 171fcbeb69d67c90ba7f44f41a9e418f6a6ec1da
Name:          debug
Driver:        docker-container
Last Activity: 2025-07-16 09:31:30 +0000 UTC

Nodes:
Name:                  debug0
Endpoint:              unix:///var/run/docker.sock
Driver Options:        image="moby/buildkit:local" memory="5GiB" network="host"
Status:                running
BuildKit daemon flags: --allow-insecure-entitlement=network.host
BuildKit version:      0cb7f6c1b.m
Platforms:             linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/386
Labels:
 org.mobyproject.buildkit.worker.executor:         oci
 org.mobyproject.buildkit.worker.hostname:         baptiste-box
 org.mobyproject.buildkit.worker.network:          host
 org.mobyproject.buildkit.worker.oci.process-mode: sandbox
 org.mobyproject.buildkit.worker.selinux.enabled:  false
 org.mobyproject.buildkit.worker.snapshotter:      overlayfs
GC Policy rule#0:
 All:           false
 Filters:       type==source.local,type==exec.cachemount,type==source.git.checkout
 Keep Duration: 48h0m0s
GC Policy rule#1:
 All:           false
 Keep Duration: 1440h0m0s
 Keep Bytes:    9.313GiB
GC Policy rule#2:
 All:        false
 Keep Bytes: 9.313GiB
GC Policy rule#3:
 All:        true
 Keep Bytes: 9.313GiB

[tonistiigi]

@ktock Can you take a look?

[ktock]

However, the function is called more than once per blob (it’s called twice of the first blob) which means the final list has duplicated elements. Which means that the function complexity is now (c+1)^#layers

Thanks for pointing this out, @Fricounet. Yes, deduplicating the return value from GetRemotes looks a necessary fix. I can work on this fix (unless you already have ongoing work around this)

[Fricounet]

Hey @ktock I can submit a fix for the small duplication part today.
Though that leaves the main issue that we are making #compression^#layers allocations which clearly will not scale well with the number of layer. Any idea on how we can fix this part?

[ktock]

@Fricounet Thanks for the PR.

Though that leaves the main issue that we are making #compression^#layers allocations which clearly will not scale well with the number of layer. Any idea on how we can fix this part?

I think we can add a filter in ExportTo for selecting the return values of LoadRemotes. For now only the specified compression might be enough (?) but in the future we can add richer filtering like #2347. WDYT @tonistiigi ?