feat: wcow: add support for bind and cache mounts #5708

profnandaa · 2025-02-06T10:36:15Z

Currently, mounts are not supported for WCOW builds, see #5678. This commit introduces support for bind and cache mounts (through the dockerfile frontend). The remaining two require a little more work and consultation with the platform teams for enlightment.

WIP Checklist:

Support for bind mounts
Support for cache mounts
add frontend/dockerfile integration tests
add client integration tests (not all, others waiting complete llb.AddMount support)
add documentation

Addresses part of #5678
Fixes #5603

Demo

Prep the context directory:

mkdir mounts-demo
cd mounts-demo
mkdir temp
echo "hello: root" > root.txt
echo "hello: inner" > .\temp\inner.txt

Dockerfile:

FROM mcr.microsoft.com/windows/nanoserver:ltsc2022 AS base
# this is needed for the access of the mounts
USER ContainerAdministrator

# cache mount example
RUN --mount=type=cache,target=/mycache echo "hello" > \mycache\foo
RUN --mount=type=cache,target=/mycache dir \mycache\foo

# bind mount examples
FROM base
# bind root of the context dir
RUN --mount=type=bind,target=/out type out\root.txt
# bind an inner directory (/temp) in the context
RUN --mount=type=bind,src=/temp,dst=/out type out\inner.txt

FROM wintools/nanoserver AS tools
RUN mkdir \bin
RUN copy \Windows\System32\whoami.exe \bin\whoami.exe


FROM base
# bind mount from another stage
RUN --mount=type=bind,from=tools,src=/bin,dst=/bin \bin\whoami.exe

Build command:

buildctl build --frontend dockerfile.v0 `
--local context=. --local dockerfile=. `
--output type=image,name=docker.io/profnandaa/mount-test,push=false `
--progress plain --no-cache

Build log:

#8 [base 2/3] RUN --mount=type=cache,target=/mycache echo "hello" > mycachefoo
#8 ...

#9 [tools 2/3] RUN mkdir bin
#9 DONE 2.8s

#8 [base 2/3] RUN --mount=type=cache,target=/mycache echo "hello" > mycachefoo
#8 DONE 2.8s

#10 [base 3/3] RUN --mount=type=cache,target=/mycache dir mycachefoo
#10 ...

#11 [tools 3/3] RUN copy WindowsSystem32whoami.exe binwhoami.exe
#11 8.133         1 file(s) copied.
#11 ...

#10 [base 3/3] RUN --mount=type=cache,target=/mycache dir mycachefoo
#10 8.217  Volume in drive C has no label.
#10 8.218  Volume Serial Number is 1E6D-2BB8
#10 8.218
#10 8.218  Directory of C:\mycache
#10 8.218
#10 8.222 02/06/2025  02:52 PM                10 foo
#10 8.222                1 File(s)             10 bytes
#10 8.222                0 Dir(s)  389,156,036,608 bytes free
#10 ...

#11 [tools 3/3] RUN copy WindowsSystem32whoami.exe binwhoami.exe
#11 DONE 8.6s

#10 [base 3/3] RUN --mount=type=cache,target=/mycache dir mycachefoo
#10 DONE 8.6s

#12 [stage-3 1/1] RUN --mount=type=bind,from=tools,src=/bin,dst=/bin binwhoami.exe
#12 4.449 user manager\containeradministrator
#12 DONE 4.9s

#13 exporting to image
#13 exporting layers

billywr

the fix LGTM

danielnilsson9 · 2025-02-07T12:18:20Z

I have done some testing on this with cache mounts and it works! But I'm a bit confused by the mount target path.
I'm running the binaries from this build: https://github.com/moby/buildkit/actions/runs/13193504481

It doesn't seem to be possible to specify an absolute path, specifying a full windows path "C:/Users/ContainerAdministrator/.conan2/p" does not work:

RUN --mount=type=cache,id=conan-cache-v1,sharing=locked,target=C:/Users/ContainerAdministrator/.conan2/p `
failed to create shim task: hcs::CreateComputeSystem si4wd93a8a3gf69wkaqhu86uz: The parameter is incorrect.: unknown

If I specify "/Users/ContainerAdministrator/.conan2/p" is seems to be relative to the current set WORKDIR and not the root of the C:/ disk which I think is inconsistent with the linux implementation.

profnandaa · 2025-02-07T15:02:53Z

Thanks for testing, let me take a look at this on Monday. --- // sent from a tiny device while on the move. forgive the tie pose.

…

On Fri, Feb 7, 2025, 15:18 Daniel Nilsson ***@***.***> wrote: I have done some testing on this with cache mounts and it works! But I'm a bit confused by the mount target path. I'm running the binaries from this build: https://github.com/moby/buildkit/actions/runs/13193504481 It doesn't seem to be possible to specify an absolute path, specifying a full windows path "C:/mycache" does not work: RUN --mount=type=cache,id=conan-cache-v1,sharing=locked,target=C:/Users/ContainerAdministrator/.conan2/p ` failed to create shim task: hcs::CreateComputeSystem si4wd93a8a3gf69wkaqhu86uz: The parameter is incorrect.: unknown If I specify "/Users/ContainerAdministrator/.conan2/p" is seems to be relative to the current set WORKDIR and not the root of the C:/ disk which I think is inconsistent with the linux implementation. — Reply to this email directly, view it on GitHub <#5708 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAB7ZELBM76SSE6V7EIIZRD2OSQCFAVCNFSM6AAAAABWTGIERSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNBSG43DQNBYGM> . You are receiving this because you authored the thread.Message ID: ***@***.***>

danielnilsson9 · 2025-02-07T15:09:01Z

Thanks :) Here is a small reproduce:

# escape=`

FROM mcr.microsoft.com/windows/servercore:ltsc2022

SHELL ["C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]

USER ContainerAdministrator

WORKDIR /build

# Does not work, cache directory mounted in C:/build/cache instead of C:/cache
RUN --mount=type=cache,id=cache-mount-v3,target=/cache `
    Get-ChildItem -Path "C:\\" ; `
    Get-ChildItem -Path "C:\\build"

# Does not work, no directory mounted, not in C:/cache and not in C:/build/cache
RUN --mount=type=cache,id=cache-mount-v3,target=C:/cache `
    Get-ChildItem -Path "C:\\" ; `
    Get-ChildItem -Path "C:\\build"

Thanks for testing, let me take a look at this on Monday. --- // sent from a tiny device while on the move. forgive the tie pose.
…
On Fri, Feb 7, 2025, 15:18 Daniel Nilsson @.> wrote: I have done some testing on this with cache mounts and it works! But I'm a bit confused by the mount target path. I'm running the binaries from this build: https://github.com/moby/buildkit/actions/runs/13193504481 It doesn't seem to be possible to specify an absolute path, specifying a full windows path "C:/mycache" does not work: RUN --mount=type=cache,id=conan-cache-v1,sharing=locked,target=C:/Users/ContainerAdministrator/.conan2/p ` failed to create shim task: hcs::CreateComputeSystem si4wd93a8a3gf69wkaqhu86uz: The parameter is incorrect.: unknown If I specify "/Users/ContainerAdministrator/.conan2/p" is seems to be relative to the current set WORKDIR and not the root of the C:/ disk which I think is inconsistent with the linux implementation. — Reply to this email directly, view it on GitHub <#5708 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAB7ZELBM76SSE6V7EIIZRD2OSQCFAVCNFSM6AAAAABWTGIERSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNBSG43DQNBYGM . You are receiving this because you authored the thread.Message ID: @.>

profnandaa · 2025-02-11T04:26:53Z

@danielnilsson9 -- you are right, this is not as it is on Linux, eg:

FROM alpine
WORKDIR /build
RUN --mount=type=cache,target=/cache ls / && echo -- && ls ./

build log:

#7 [stage-0 3/3] RUN --mount=type=cache,target=/cache ls / && echo -- && ls ./ 
#7 0.216 bin
#7 0.216 build 
#7 0.216 cache <--- 
#7 0.216 dev
#7 0.216 etc
#7 0.216 home

Ok, will fix that. Thanks for the catch!

profnandaa · 2025-02-11T16:16:25Z

@danielnilsson9 -- fixed, please take a look.

For a consistent experience, we'll add to the documentation that for mount src and dst options, users supply unix paths instead.

danielnilsson9 · 2025-02-11T16:49:07Z

@profnandaa Nice! Can confirm it is working as expected now.

slonopotamus · 2025-02-11T19:08:37Z

users supply unix paths instead.

That's a kinda strange rule. Can't I have multiple drives in Windows container and want to mount cache to other letter than C:\?

profnandaa · 2025-02-12T05:57:34Z

users supply unix paths instead.

That's a kinda strange rule. Can't I have multiple drives in Windows container and want to mount cache to other letter than C:\?

I see... makes sense. I had noticed for instance when target=C:/something vs target=/something, they are treated differently for the cache coz of:

buildkit/frontend/dockerfile/dockerfile2llb/convert_runmount.go

Line 116 in 6702365

mount.CacheID = path.Clean(mount.Target)

I didn't know which other more surprises might be there.

However, for other drive locations, that should still work okay. Let me run some tests.

profnandaa · 2025-02-13T06:00:13Z

No issues mounting from a different drive than C:

FROM mcr.microsoft.com/windows/nanoserver:ltsc2022
USER ContainerAdministrator

RUN --mount=type=bind,target=/test type \test\example.txt

> ls -Name E:\sample\
example.txt

> buildctl build --frontend dockerfile.v0 --local context=E:\sample --local dockerfile=. `
--output type=image,name=docker.io/profnandaa/mount-test,push=false `
--progress plain --no-cache

...
#6 [stage-0 2/2] RUN --mount=type=bind,target=/test type testexample.txt
#6 1.387 mounted!!! from E:\
#6 DONE 1.7s

profnandaa · 2025-02-14T05:43:52Z

NOTE: with this diff, now supports backslashes on windows paths for mount options. However, you have to change the default escape token from \ to `, e.g.

#escape=`

FROM mcr.microsoft.com/windows/nanoserver:ltsc2022
USER ContainerAdministrator

RUN --mount=type=cache,target=C:\mycache echo "hello" > \mycache\foo
# then for options on multilines you have to go with ` instead of \
RUN --mount=type=cache,`
target=C:\mycache dir \mycache\foo

profnandaa · 2025-02-14T10:58:47Z

I'll carry out the rest of the llb.AddMount support as a separate PR. See #5678 (comment)

This should be ready for review. /cc. @tonistiigi @crazy-max

billywr

LGTM!

profnandaa · 2025-02-19T09:13:10Z

NOTE: taken it back to draft to cover some cases in llb.AddMount that are related to this, will update.

profnandaa · 2025-02-20T10:39:51Z

With the latest update, you can now set / as source from an image. Initially, this would raise some access denied error like:

error: failed to solve: 
failed to compute cache key: failed to calculate checksum of ref p11yciubfsq2s4670o7j7htlv::shece2fgarkdis8ki7jsuzhvd: 
failed to open /WcSandboxState/Hives/DefaultUser_Delta: 
open C:\Users\annandaa\AppData\Local\Temp\buildkit-mount577988685\WcSandboxState\Hives\DefaultUser_Delta: 
Access is denied.

Now, such a dockerfile can build:

# escape=`

FROM mcr.microsoft.com/windows/nanoserver:ltsc2022
USER ContainerAdministrator

RUN --mount=type=bind,from=docker.io/wintools/nanoserver,`
src=/,dst=/bin C:\bin\Windows\System32\whoami.exe

tonistiigi · 2025-02-20T17:36:27Z

cache/contenthash/checksum.go

@@ -862,6 +862,10 @@ func (cc *cacheContext) scanChecksum(ctx context.Context, m *mount, p string, fo
 }

 func (cc *cacheContext) checksum(ctx context.Context, root *iradix.Node[*CacheRecord], txn *iradix.Txn[*CacheRecord], m *mount, k []byte, followTrailing bool) (*CacheRecord, bool, error) {
+	// only applies for Windows, it's a no-op on non-Windows.
+	enableProcessPrivileges()


shouldn't this be in a more top-level function. This one gets called recursively I think.

Let me crosscheck.

You right, it's recursive here. I've taken it up to a more top-level function now.

tonistiigi · 2025-02-20T17:39:07Z

executor/oci/spec_windows.go

@@ -119,3 +119,9 @@ func generateCDIOpts(_ *cdidevices.Manager, devices []*pb.CDIDevice) ([]oci.Spec
 	// https://github.com/cncf-tags/container-device-interface/issues/28
 	return nil, errors.New("no support for CDI on Windows")
 }
+
+func getMountType(_ string) string {


Isn't this true only for bind? In that case should be normalizeBindType() with an input parameter comparison.

Not just for bind, all of them are referred to as windows-layer by this point. They all fail with:

failed to create shim task: invalid OCI spec - Type 'windows-layer' not supported: unknown

However, I think I can rename this to normalizeMountType instead?

tonistiigi · 2025-02-20T17:40:34Z

frontend/dockerfile/dockerfile2llb/convert_runmount.go

@@ -113,12 +114,12 @@ func dispatchRunMounts(d *dispatchState, c *instructions.RunCommand, sources []*
 				sharing = llb.CacheMountLocked
 			}
 			if mount.CacheID == "" {
-				mount.CacheID = path.Clean(mount.Target)
+				mount.CacheID = filepath.Clean(mount.Target)


filepath in here doesn't look right as this code can run in a different OS than daemon or the mount.Target

This was my attempt to have the Windows path resolve to the same default cache ID e.g.

C:/temp -> C:\\temp

though this quickly falls apart with cases like /temp -> \\temp.

Best guarantee is just for the user to set their own IDs if they are using different path schemes. Will add that as part of the documentation note.

Let me revert back to path.Clean.

Currently, mounts are not supported for WCOW builds, see moby#5678. This commit introduces support for bind and cache mounts. The remaining two require a little more work and consultation with the platform teams for enlightment. WIP Checklist: - [x] Support for bind mounts - [x] Support for cache mounts - [x] add frontend/dockerfile integration tests - [x] add client integration tests (not all, `llb.AddMount` not complete) Fixes moby#5603 Signed-off-by: Anthony Nandaa <[email protected]>

After completing support for `llb.AddMount` in moby#5708 -- this made possible to add some more `client` integration tests that were making `llb.AddMount` calls. Most of the tests below are related to this: - [x] testCacheExportCacheKeyLoop - [x] testOCILayoutSource - [x] testBuildMultiMount - [x] testTarExporterSymlink - [x] testCacheExportIgnoreError - [x] testBasicCacheImportExport - [x] testUncompressedLocalCacheImportExport - [x] testImageManifestRegistryCacheImportExport - [x] testBasicLocalCacheImportExport - [x] testBasicInlineCacheImportExport - [x] testRegistryEmptyCacheExport - [x] testCachedMounts - [x] testDuplicateCacheMount - [x] testRunCacheWithMounts - [x] testMountWithNoSource - [x] testProxyEnv - [x] testRelativeMountpoint - [x] testValidateDigestOrigin - [x] testExportAnnotations - [x] testExportAnnotationsMediaTypes - [x] testAttestationDefaultSubject - [x] testAttestationBundle - [x] testLLBMountPerformance Skipped tests (due to registry setup issue, to be addressed separately): - testUncompressedRegistryCacheImportExport - testBasicRegistryCacheImportExport - testMultipleRegistryCacheImportExport Related to moby#5678 Signed-off-by: Anthony Nandaa <[email protected]>

github-actions bot added area/dockerfile area/testing area/windows area/frontend area/executor labels Feb 6, 2025

profnandaa force-pushed the wcow_mount_support branch from 8bf0b5a to bc2decc Compare February 6, 2025 15:44

billywr reviewed Feb 6, 2025

View reviewed changes

profnandaa force-pushed the wcow_mount_support branch from bc2decc to ec20b95 Compare February 7, 2025 04:58

profnandaa changed the title ~~wip: wcow: hack complete for bind and cache mounts support~~ feat: wcow: add support for bind and cache mounts Feb 7, 2025

This was referenced Feb 7, 2025

Tracking Windows buildkit required features adamrehn/ue4-docker#365

Open

WCOW: RUN with bind/cache mounts #5603

Closed

profnandaa force-pushed the wcow_mount_support branch from ec20b95 to 9c74564 Compare February 7, 2025 13:11

profnandaa force-pushed the wcow_mount_support branch from 9c74564 to 261ee88 Compare February 10, 2025 05:18

profnandaa force-pushed the wcow_mount_support branch from 261ee88 to b2c2b01 Compare February 11, 2025 16:10

github-actions bot added area/client area/util labels Feb 11, 2025

profnandaa force-pushed the wcow_mount_support branch from b2c2b01 to 00b9867 Compare February 11, 2025 16:18

profnandaa force-pushed the wcow_mount_support branch 2 times, most recently from 42f8214 to ba74219 Compare February 12, 2025 07:04

profnandaa force-pushed the wcow_mount_support branch 2 times, most recently from 6fef8df to 59b5b09 Compare February 14, 2025 05:37

profnandaa mentioned this pull request Feb 14, 2025

WCOW: need support for llb.AddMount and RUN --mount for WCOW #5678

Open

profnandaa force-pushed the wcow_mount_support branch from 59b5b09 to 39e3ff8 Compare February 14, 2025 10:57

profnandaa marked this pull request as ready for review February 14, 2025 10:59

billywr reviewed Feb 16, 2025

View reviewed changes

billywr approved these changes Feb 16, 2025

View reviewed changes

profnandaa mentioned this pull request Feb 18, 2025

[POC] wcow: option to run builds in hyperv isolation #5753

Draft

4 tasks

profnandaa marked this pull request as draft February 19, 2025 08:50

profnandaa force-pushed the wcow_mount_support branch from 39e3ff8 to 54924cc Compare February 20, 2025 09:39

github-actions bot added the area/storage label Feb 20, 2025

profnandaa force-pushed the wcow_mount_support branch from 54924cc to b94b60c Compare February 20, 2025 12:57

profnandaa marked this pull request as ready for review February 20, 2025 13:12

profnandaa force-pushed the wcow_mount_support branch 3 times, most recently from 26dfae5 to 7ba4fb1 Compare February 20, 2025 13:37

tonistiigi reviewed Feb 20, 2025

View reviewed changes

profnandaa force-pushed the wcow_mount_support branch from 7ba4fb1 to 877d8a7 Compare February 20, 2025 18:19

tonistiigi approved these changes Feb 20, 2025

View reviewed changes

tonistiigi added the status/merge label Feb 20, 2025

tonistiigi merged commit 26b1f2b into moby:master Feb 20, 2025
104 checks passed

profnandaa deleted the wcow_mount_support branch February 24, 2025 17:40

profnandaa mentioned this pull request Feb 26, 2025

tests: client: windows: add various llb.AddMount related tests #5782

Closed

23 tasks

profnandaa mentioned this pull request Apr 4, 2025

tests: client: windows: add various llb.AddMount related tests #5888

Merged

23 tasks

feat: wcow: add support for bind and cache mounts #5708

feat: wcow: add support for bind and cache mounts #5708

Uh oh!

Conversation

profnandaa commented Feb 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Demo

Uh oh!

billywr left a comment

Choose a reason for hiding this comment

Uh oh!

danielnilsson9 commented Feb 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

profnandaa commented Feb 7, 2025 via email

Uh oh!

danielnilsson9 commented Feb 7, 2025

Uh oh!

profnandaa commented Feb 11, 2025

Uh oh!

profnandaa commented Feb 11, 2025

Uh oh!

danielnilsson9 commented Feb 11, 2025

Uh oh!

slonopotamus commented Feb 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

profnandaa commented Feb 12, 2025

Uh oh!

profnandaa commented Feb 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

profnandaa commented Feb 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

profnandaa commented Feb 14, 2025

Uh oh!

billywr left a comment

Choose a reason for hiding this comment

Uh oh!

profnandaa commented Feb 19, 2025

Uh oh!

profnandaa commented Feb 20, 2025

Uh oh!

tonistiigi Feb 20, 2025

Choose a reason for hiding this comment

Uh oh!

profnandaa Feb 20, 2025

Choose a reason for hiding this comment

Uh oh!

profnandaa Feb 20, 2025

Choose a reason for hiding this comment

Uh oh!

tonistiigi Feb 20, 2025

Choose a reason for hiding this comment

Uh oh!

profnandaa Feb 20, 2025

Choose a reason for hiding this comment

Uh oh!

tonistiigi Feb 20, 2025

Choose a reason for hiding this comment

Uh oh!

profnandaa Feb 20, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

profnandaa commented Feb 6, 2025 •

edited

Loading

danielnilsson9 commented Feb 7, 2025 •

edited

Loading

slonopotamus commented Feb 11, 2025 •

edited

Loading

profnandaa commented Feb 13, 2025 •

edited

Loading

profnandaa commented Feb 14, 2025 •

edited

Loading