Skip to content

[11.0-stable] Update github workflows: perform login on docker, self-hosted runners and linuxkit update #5043

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 3 commits into
base: 11.0-stable
Choose a base branch
from
Draft

[11.0-stable] Update github workflows: perform login on docker, self-hosted runners and linuxkit update #5043

wants to merge 3 commits into from

Conversation

rene
Copy link
Contributor

@rene rene commented Jul 2, 2025 •
edited
Loading

Description

This PR brings several new features and changes from current workflows to 11.0-stable branch:

  • Provide Docker login action on the build workflows.
  • Switch to Zededa's self hosted runners
  • Update linuxkit to version 1.7.0
  • Port Makefile changes to support linuxkit 1.7.0

How to test and validate this PR

Run GH workflows.

Changelog notes

None.

Checklist

  • I've provided a proper description
  • I've added the proper documentation
  • I've tested my PR on amd64 device
  • I've tested my PR on arm64 device
  • I've written the test verification instructions
  • I've set the proper labels to this PR
  • I've checked the boxes above, or I've provided a good reason why I didn't
    check them.

@rene rene requested a review from eriknordmark as a code owner July 2, 2025 12:44
@rene rene marked this pull request as draft July 2, 2025 12:47
@rene rene force-pushed the add-dockerlogin-11.0 branch 3 times, most recently from 37c449c to bffc2ee Compare July 2, 2025 13:52
@OhmSpectator
Copy link
Member

A new LinuxKit should implement part of it... I prefer to get 1.7.0 working (or 1.8.0?). Previous fixes were more workarounds.

@rene rene force-pushed the add-dockerlogin-11.0 branch 4 times, most recently from 76c99c9 to 516f3be Compare July 2, 2025 18:51
@rene rene marked this pull request as ready for review July 2, 2025 19:15
@rene
Copy link
Contributor Author

rene commented Jul 2, 2025 •
edited
Loading

Tested on my repo: https://github.com/rene/eve/actions/runs/16033358147?pr=75

rucoder and others added 3 commits July 3, 2025 12:20
@rucoder @rene
Port Makefile changes to improve/fix linuxkit build
967a4d1 
This commit ports (adapted) the following commits:

Upgrade Linuxkit to 1.7.0

This version has many important fixes
1. Import of Docker credentials
2. --builder-config can pass a buildkitd.toml to a builder container

So we do not need an ensure-builder target anymore. It was very useful
for debugging Docker login / mirror related issues but I hope it is gone
for good.

Signed-off-by: Mikhail Malyshev <[email protected]>

Create linuxkit-builder manually to support parallel build

Linuxkit tries to create docker builder container if it doesn't exist.
It works fine for -j 1 but fails for parallel build. LK tries to create or update the
builder concurrently and fails. To avoid a race we create builder manually

- the builder name is hard-coded and is linuxkit-builder
- override context name for `linuxkit pkg build` so it doesn't try
to create a builder by itself
- ensure-builder target checks if the container exists and of required version
- docker prune operation cannot be executed concurrently, use flock to handle that
this is especially important for CI build with PRUNE=1 set

Signed-off-by: Mikhail Malyshev <[email protected]>

Fix 'unauthenticated pull rate limit reached' problem

- action/docker-login stores credential in ~/.docker/config.json. In theory this file should be mounted
into linuxkit-builder container but I did not know about it. Dut to that fact buildkit was running unauthenticated
- upgrade to linuxkit 1.6.5 that has fixes for a race in builder creation and cache access
- mount /etc/buildkit/buildkitd.toml that MAY define a mirror to speedup the build. LK doens't do it
other wise we could remove the whole block for manual creation of buildkit runner

Signed-off-by: Mikhail Malyshev <[email protected]>

Makefile: Bump linuxkit to version 1.5.3

Bump linuxkit to version 1.5.3 in order to get a new feature that supports
--tag in build.yml. This version also introduces some changes that
requires some adjusts in the Makefile:

1. Introduces different verbose levels (this commit sets to level 2 to keep
   the same amount of information provided by the current version)
2. Remove --arch parameter from cache command (in favor of a --platform
   docker style parameter)

Signed-off-by: Renê de Souza Pinto <[email protected]>

Signed-off-by: Renê de Souza Pinto <[email protected]>
@rene
github/workflows: Add docker login to build workflow
69a41d6 
Provide docker login for build.yml workflow.

Signed-off-by: Renê de Souza Pinto <[email protected]>
@rene
github/workflows: Switch to self hosted runners
6c0b2c9 
Switch to Zededa's self hosted runners on build and publish workflows.

Signed-off-by: Renê de Souza Pinto <[email protected]>
@rene rene force-pushed the add-dockerlogin-11.0 branch from 516f3be to 6c0b2c9 Compare July 3, 2025 10:21
OhmSpectator
OhmSpectator reviewed Jul 3, 2025
View reviewed changes
Copy link
Member

@OhmSpectator OhmSpectator left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

Error: Username and password required

.github/workflows/build.yml
@@ -18,11 +18,11 @@ jobs:
fail-fast: false
matrix:
include:
- os: buildjet-4vcpu-ubuntu-2204-arm
- os: zededa-ubuntu-2204
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it was the reason why I had zero problems building and publishing for 11:
We did not hit any 429 error on that branch because we use virtual runners there. Our runners all share the same IP.
FYI, @rene, @rucoder, @uncleDecart

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On the other hand, I see that Eden tests are starting to experience the same issue, and they run on virtual runners. And they have nothing with parallel build as well. I really think it's some new changes on Dockerhub. The policy officially changed on April 1st, but we are now starting to see the consequences.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://www.docker.com/blog/revisiting-docker-hub-policies-prioritizing-developer-experience/
Do we have updated information? In this link it says:

We did not enforce the Docker Hub rate limit changes previously scheduled for April 1, 2025. The current limits—100 pulls per 6 hours for unauthenticated users and 200 pulls per 6 hours for Docker Personal users—will remain in place as we continue to evaluate developer feedback and identify appropriate long-term limits.

We will announce any future enforcement at least 6 months in advance.

But they might have changed their minds... or they are doing some tests...

Btw, it also says:

Unlimited Pull rates for Paid Users (As Announced Earlier) – Starting April 1, 2025, all paid Docker subscribers will have unlimited image pulls (with fair use limits) to ensure a seamless experience.

For us the pull rate limits shouldn't be a problem at all by doing login....

@rene
Copy link
Contributor Author

rene commented Jul 3, 2025

Last test on my repo: https://github.com/rene/eve/actions/runs/16047887243?pr=75

@OhmSpectator
Copy link
Member

Just a note: The issue with accessing secrets in the workflow file arises because it uses the pull_request trigger. In this scenario, the workflow cannot access secrets. To enable access to these secrets, the trigger should be changed to pull_request_target.

@rene rene changed the title [11.0-stable] Perform login on docker [11.0-stable] Update github workflows: perform login on docker, self-hosted runners and linuxkit update Jul 3, 2025
@OhmSpectator OhmSpectator added the side-quest A worthy adventure, but not essential for victory. Tackle when the main quest is safe! label Jul 7, 2025
@europaul
Copy link
Contributor

Is this PR stalled because we still see 429's on the zededa runners? Or can we proceed?

@rene rene marked this pull request as draft August 12, 2025 09:32
@rene
Copy link
Contributor Author

rene commented Aug 12, 2025

Is this PR stalled because we still see 429's on the zededa runners? Or can we proceed?

We've got more updates on the workflows @europaul , I will need to rebase this PR, so just moved to Draft for now....

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@OhmSpectator OhmSpectator OhmSpectator left review comments

@eriknordmark eriknordmark Awaiting requested review from eriknordmark eriknordmark is a code owner

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
side-quest A worthy adventure, but not essential for victory. Tackle when the main quest is safe!
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@rene @OhmSpectator @europaul @rucoder