allow exporting cache layers in parallel to the remote registry

[compumike]

Description

exporting cache to registry: Concurrency for writing layer:

In cache/remotecache/export.go the build cache layers are exported to the remote registry cache sequentially, one layer at a time:

buildkit/cache/remotecache/export.go

Lines 199 to 210 in 955c2b2

	for _, l := range config.Layers {
	dgstPair, ok := descs[l.Blob]
	if !ok {
	return nil, errors.Errorf("missing blob %s", l.Blob)
	}
	layerDone := progress.OneOff(ctx, fmt.Sprintf("writing layer %s", l.Blob))
	if err := contentutil.Copy(ctx, ce.ingester, dgstPair.Provider, dgstPair.Descriptor, ce.ref, logs.LoggerFromContext(ctx)); err != nil {
	return nil, layerDone(errors.Wrap(err, "error writing layer blob"))
	}
	layerDone(nil)
	cache.AddCacheBlob(dgstPair.Descriptor)
	}

It would be great if it were possible to export these layers in parallel, similar to dockerd's max-concurrent-uploads option, which defaults to 5.

My use case is that using --cache-to type=registry,ref=${image}:build-cache,mode=max with a remote registry, I've observed that each writing layer sha256:... round trip takes a minimum of 0.3 seconds or more per layer (and sometimes a few seconds), even when that layer blob is already present remotely. The net effect is something like 30 seconds of my build time is just spent in exporting cache to registry to verify that each blob is already present, even when no actual new data is being uploaded! Allowing some concurrency here would make remote registry caching much more performant.

---

Use --cache-from data to filter --cache-to blob list

An alternative, complementary performance enhancement might be to use the manifest retrieved in the --cache-from step, if the same type=registry,ref=... is used (which it usually will be!), to filter out layers which we know the remote registry server already has.