fix(ci): treat an unreadable cache size as unstable when settling - #7274
Merged
Merged
Conversation
The settle loop initialised prev='' and, after 4795d04 made total() return empty on failure, two consecutive failed `buildctl du` reads compared equal and tripped the stability counter. The loop then exited after ~2s instead of its 120s bound — exactly when du is failing and the prune is most likely still deleting, which is the case the wait exists to cover. Handing back early there risks the builder post-step SIGKILLing buildkitd and skipping the sticky disk commit. An empty reading can only mean du failed. buildctl prints its `Total:` line unconditionally (cmd/buildctl/diskusage.go), so an empty cache still reports `Total: 0B` and settles normally. Guarding on a non-empty reading therefore costs nothing in the healthy paths: verified a steady value and an empty cache both still exit after 3 iterations, while a persistently failing du now waits out all 60. Reported by cubic on #7273.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
Greptile SummaryThis PR corrects the Docker build cache settle loop so failed, empty
Confidence Score: 5/5The PR appears safe to merge because the changed condition correctly distinguishes unreadable cache-size results from valid stable values. The settle loop now counts only matching non-empty readings as stable, while failed reads remain bounded by the existing 60-iteration limit and valid zero-cache output continues to settle normally.
|
| Filename | Overview |
|---|---|
| .github/actions/docker-build/action.yml | Adds a non-empty guard to the cache-size stability comparison, correctly preventing repeated failed reads from triggering early settlement. |
Reviews (1): Last reviewed commit: "fix(ci): treat an unreadable cache size ..." | Re-trigger Greptile
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
cubic flagged this on #7273 and it's correct. Verified rather than taken on trust.
The settle loop initialises
prev='', and since4795d0426fmadetotal()return empty on failure, two consecutive failedbuildctl dureads compare equal and trip the stability counter:So the loop gives up after ~2s instead of its 120s bound — precisely when
duis failing and the prune is most likely still deleting, which is the exact case the wait exists to cover. Handing back early there risks the builder post-step SIGKILLing buildkitd and skipping the sticky disk commit.Why guarding on non-empty is the right fix, with no downside
An empty reading can only mean
dufailed.buildctlprints itsTotal:line unconditionally —cmd/buildctl/diskusage.go:161, outside any branch:So a genuinely empty cache still reports
Total: 0Band settles normally. There's no risk of adding 120s to builds with a cold cache — which was the obvious objection to this fix.Confirmed across all three paths:
dubehaviourTotal: 0B)Severity
Lower than P1 in practice: since #7252 the step can't fail a build, so the consequence is a possibly-skipped disk commit — one run's cache lost, not a red build. But the logic is plainly wrong and the fix is one condition, so it's worth taking before promotion.
Type of Change
Testing
Reproduced the early exit and verified the fix under the exact shell flags GitHub uses (
bash -e -o pipefail), across failing / steady / empty-cache readings.shellcheckclean,actionlintunchanged from baseline (13 pre-existing),bun run lintand all 39check:auditspass.Checklist