fix: wcow: fix race condition in localmounter

[profnandaa]

WIP: still ongoing with investigation to determine
exactly which process is accessing the file, will
update on the issue #5807 thread for the records.

Fix the race condition with maximum 2 retries.
From several tests, 1 retry seems to be
enough, even without backoff. Added a simple
linear backoff for each retry starting at 30 ms.

fixes #5807

---

Here is a repro run + the mitigation - repro_run_8min.webm. See around t 1:55, on the 25th run. Hit the issue 1 in 100 runs, mostly it's been 2-3 in 100 runs.

Here's the repro script I was using:

# .\repro.ps1
param(
	[Parameter(Position = 0)]
	[int]$Count = 50,
	[Parameter(Position = 1)]
	[int]$Pause = 3
)

$dockerfile = @"
FROM mcr.microsoft.com/windows/nanoserver:ltsc2022 AS build
RUN mkdir out\sub && mklink /D sub out\sub && mklink /D sub2 out\sub && echo data> sub\foo 

FROM mcr.microsoft.com/windows/nanoserver:ltsc2022
COPY --from=build /sub/foo .
COPY --from=build /sub2/foo bar
"@

if (-not (Test-Path -Path ".\Dockerfile")) {
	Set-Content -Path .\Dockerfile -Value $dockerfile
}

$cmd = "buildctl build --frontend dockerfile.v0 --local context=. --local dockerfile=. --output type=image,name=docker.io/profnandaa/repro-5807,push=false --progress plain --no-cache"

for ($i = 1; $i -le $Count; $i++) { 
	Write-Host "`n=== Run $i`n"
	iex $cmd
	sleep $Pause
}

To run, save the script in repro.ps1, run buildkitd with breakpoint set at snapshot/localmounter_windows.go:78 , then:

mkdir auto
cp repro.ps1 auto
cd auto
.\repro.ps1 100 0

[profnandaa]

Just some more stats:

• build-log-1000-repro.txt - 11 hits in 1000 runs (before fixing)
• build-log-1000-fixed.txt

type .\build-log-1000-repro.txt | sls "error: failed to solve:" | measure
Count             : 11

type .\build-log-1000-fixed.txt | sls "error: failed to solve:" | measure
Count             : 0