[BUG] Build secret files not mounted in /run/secrets when using docker compose build with profiles

[kmfarley11]

Description

If I set up a Dockerfile for build secrets, docker build --secret ... works just fine per the official docs.

If I set up a docker compose project per the official docs (this one too) secrets do still seem to work just fine.

However, if I add a profile to my service that has build secrets, and try to specifically build the service in isolation, I run into issues!
Further, if i do a direct up --build on the command behind the profile, that still works though...

Note: this is highly related to other issues such as #9822 & #13019 (sorry for the confusion on that latter one). But it describes a different issue with profiles and build vs. up behavior on the latest docker engine.

Steps To Reproduce

1. Create a simple local test file: echo "SECRET TEST" > test-secret.txt
2. Create a Dockerfile with this content

#Dockerfile
FROM debian:stable-slim
RUN --mount=type=secret,id=test-secret ls -la /run/secrets/; cp /run/secrets/test-secret /tmp

CMD ["cat", "/tmp/test-secret"]

3. Create a docker-compose file with this content:

#docker-compose.yaml
secrets:
    test-secret:
        file: test-secret.txt

services:
    secret-build-test:
        profiles: ["test"]
        build:
            context: .
            dockerfile: Dockerfile
            secrets:
                - test-secret

4. Verify that a docker compose build with profiles activated works as expected

docker builder prune -fa
docker compose --progress=plain --profile test build secret-build-test

should output something like the following:

...
#6 [stage-0 2/2] RUN --mount=type=secret,id=test-secret ls -la /run/secrets/; cp /run/secrets/test-secret /tmp
#6 0.156 total 12
#6 0.156 drwxr-xr-x 2 root root 4096 Jul  4 15:42 .
#6 0.156 drwxr-xr-x 1 root root 4096 Jul  4 15:42 ..
#6 0.156 -r-------- 1 root root   12 Jul  4 15:42 test-secret
#6 DONE 0.2s
...

5. Demonstrate that a direct docker compose build fails!

docker builder prune -fa
docker compose build secret-build-test --progress=plain --no-cache

This outputs the following for me

...
#6 [stage-0 2/2] RUN --mount=type=secret,id=test-secret ls -la /run/secrets/; cp /run/secrets/test-secret /tmp
#6 0.149 ls: cannot access '/run/secrets/': No such file or directory
#6 0.150 cp: cannot stat '/run/secrets/test-secret': No such file or directory
#6 ERROR: process "/bin/sh -c ls -la /run/secrets/; cp /run/secrets/test-secret /tmp" did not complete successfully: exit code: 1
------
 > [stage-0 2/2] RUN --mount=type=secret,id=test-secret ls -la /run/secrets/; cp /run/secrets/test-secret /tmp:
0.149 ls: cannot access '/run/secrets/': No such file or directory
0.150 cp: cannot stat '/run/secrets/test-secret': No such file or directory
------
...

6. Show that a plain up command still works though...

docker builder prune -fa
docker compose up --build secret-build-test

This outputs the following for me:

...
 => [stage-0 2/2] RUN --mount=type=secret,id=test-secret ls -la /run/secrets/; cp /run/secrets/test-secret /tmp                                            0.2s
 => exporting to image                                                                                                                                     0.0s
 => => exporting layers                                                                                                                                    0.0s
 => => writing image sha256:fe3ec2d88d0be58cdfc31774c851e50c985d05b7f86d36b2bf24da0d9e511ea7                                                               0.0s
 => => naming to docker.io/library/compose-secrets-issue-secret-build-test                                                                                 0.0s
 => resolving provenance for metadata file                                                                                                                 0.0s
[+] Running 2/2
 ✔ secret-build-test                                    Built                                                                                              0.0s
 ✔ Container compose-secrets-issue-secret-build-test-1  Recreated                                                                                          0.1s
Attaching to secret-build-test-1
secret-build-test-1  | SECRET TEST
secret-build-test-1 exited with code 0

Compose Version

Docker Compose version v2.37.1-desktop.1

Docker Environment

Client:
 Version:    28.2.2
 Context:    default
 Debug Mode: false
 Plugins:
  ai: Docker AI Agent - Ask Gordon (Docker Inc.)
    Version:  v1.4.0
    Path:     /usr/local/lib/docker/cli-plugins/docker-ai
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.24.0-desktop.2
    Path:     /usr/local/lib/docker/cli-plugins/docker-buildx
  cloud: Docker Cloud (Docker Inc.)
    Version:  v0.3.9
    Path:     /usr/local/lib/docker/cli-plugins/docker-cloud
  compose: Docker Compose (Docker Inc.)
    Version:  v2.37.1-desktop.1
    Path:     /usr/local/lib/docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.41
    Path:     /usr/local/lib/docker/cli-plugins/docker-debug
  desktop: Docker Desktop commands (Docker Inc.)
    Version:  v0.1.9
    Path:     /usr/local/lib/docker/cli-plugins/docker-desktop
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.29
    Path:     /usr/local/lib/docker/cli-plugins/docker-extension
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.4.0
    Path:     /usr/local/lib/docker/cli-plugins/docker-init
  mcp: Docker MCP Plugin (Docker Inc.)
    Version:  dev
    Path:     /usr/local/lib/docker/cli-plugins/docker-mcp
  model: Docker Model Runner (Docker Inc.)
    Version:  v0.1.24
    Path:     /usr/local/lib/docker/cli-plugins/docker-model
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /usr/local/lib/docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.18.1
    Path:     /home/kevin/.docker/cli-plugins/docker-scout
WARNING: Plugin "/usr/local/lib/docker/cli-plugins/docker-dev" is not valid: failed to fetch metadata: fork/exec /usr/local/lib/docker/cli-plugins/docker-dev: no such file or directory
WARNING: Plugin "/usr/local/lib/docker/cli-plugins/docker-feedback" is not valid: failed to fetch metadata: fork/exec /usr/local/lib/docker/cli-plugins/docker-feedback: no such file or directory
WARNING: Plugin "/usr/local/lib/docker/cli-plugins/docker-scan" is not valid: failed to fetch metadata: fork/exec /usr/local/lib/docker/cli-plugins/docker-scan: no such file or directory

Server:
 Containers: 1
  Running: 0
  Paused: 0
  Stopped: 1
 Images: 6
 Server Version: 28.2.2
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Discovered Devices:
  cdi: docker.com/gpu=webgpu
 Swarm: inactive
 Runtimes: nvidia runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 05044ec0a9a75232cad458027ca83437aae3f4da
 runc version: v1.2.5-0-g59923ef
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.6.87.2-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 20
 Total Memory: 31.18GiB
 Name: docker-desktop
 ID: adac6913-bc44-4345-b8d0-ba432c46a5ce
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=unix:///var/run/docker-cli.sock
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  ::1/128
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: DOCKER_INSECURE_NO_IPTABLES_RAW is set

Anything else?

As a general note: in the past, I've typically been able to expect that direct service callouts to build, up etc. will work in isolation regardless of profiles. But it seems there's been some tumult in profile behavior recently. Is there a desire / trend from compose developers to make profiled services only work when profiles are provided? Or can I still reliably count on isolated callouts to work regardless?

[ndeloof]

tested with latest codebase (v2.38.1-10-gc626befee) and can't reproduce
v2.37.3 introduced build-related fixes that my cover this scenario
would you have a chance to give latest release a try ?

[kmfarley11]

tested with latest codebase (v2.38.1-10-gc626befee) and can't reproduce v2.37.3 introduced build-related fixes that my cover this scenario would you have a chance to give latest release a try ?

Just updated, now docker compose version yields

Docker Compose version v2.38.1-desktop.1

With that, I'm still seeing the same results from steps 4, 5, & 6 per the issue description:
this is ok -> docker builder prune -fa; docker compose --profile test build secret-build-test
this fails -> docker builder prune -fa; docker compose build secret-build-test --progress=plain --no-cache
this is ok -> docker builder prune -fa; docker compose up --build secret-build-test

[ndeloof]

My bad, ran the wrong command. Issue is 100% valid and reproducible