fix(amber): declare cloudpickle in LICENSE-binary-python - #8293
Conversation
joblib 1.6.0 stopped vendoring cloudpickle -- joblib/externals/cloudpickle
is gone from the wheel and the metadata now carries
`Requires-Dist: cloudpickle>=3.0` -- so pip installs cloudpickle as its own
distribution. amber/operator-requirements.txt reaches joblib transitively
through scikit-learn==1.7.2, which makes cloudpickle newly visible to
pip-licenses and unclaimed by amber/LICENSE-binary-python.
That fails the `Check installed Python packages against per-module
LICENSE-binary files` step of `build / pyamber (ubuntu-latest, 3.12)` on
every open PR, regardless of what the PR touches:
NEW Python packages not claimed by LICENSE-binary:
+ cloudpickle==3.1.2 (license: BSD License)
cloudpickle 3.1.2 is BSD-3-Clause (its METADATA says
`License: BSD-3-Clause`; PyPI's classifier only reports the generic
"BSD License"), which is ASF Category A. Its wheel ships no NOTICE and
licenses/LICENSE-BSD-3-Clause.txt already carries the license text, so the
one manifest bullet is the whole fix.
The transitive version drift the same step reports is left alone: it is
informational under --ignore-transitive-version, and the nightly
exact-match check on main owns refreshing it.
Backport auto-label reportThis
|
Automated Reviewer SuggestionsBased on the
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #8293 +/- ##
============================================
- Coverage 93.93% 93.90% -0.04%
Complexity 4762 4762
============================================
Files 1191 1191
Lines 48308 49051 +743
Branches 5377 5377
============================================
+ Hits 45380 46060 +680
- Misses 1494 1557 +63
Partials 1434 1434
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
| config | throughput | MB/s | latency | max Δ latest / 7d | |
|---|---|---|---|---|---|
| 🔴 | bs=10 sw=10 sl=64 | 560 | 0.342 | 17,115/27,086/27,086 us | 🔴 +5.6% / 🔴 +73.9% |
| 🟢 | bs=100 sw=10 sl=64 | 1,150 | 0.702 | 85,112/118,599/118,599 us | 🟢 -11.7% / 🟢 -15.5% |
| ⚪ | bs=1000 sw=10 sl=64 | 1,318 | 0.804 | 755,442/825,214/825,214 us | ⚪ within ±5% / 🟢 +26.5% |
Baseline details
Latest main 50321e4 from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---|---|---|---|---|
| bs=10 sw=10 sl=64 | throughput | 560 tuples/sec | 570 tuples/sec | 787.98 tuples/sec | -1.8% | -28.9% |
| bs=10 sw=10 sl=64 | MB/s | 0.342 MB/s | 0.348 MB/s | 0.481 MB/s | -1.7% | -28.9% |
| bs=10 sw=10 sl=64 | p50 | 17,115 us | 16,834 us | 12,593 us | +1.7% | +35.9% |
| bs=10 sw=10 sl=64 | p95 | 27,086 us | 25,649 us | 15,579 us | +5.6% | +73.9% |
| bs=10 sw=10 sl=64 | p99 | 27,086 us | 25,649 us | 18,786 us | +5.6% | +44.2% |
| bs=100 sw=10 sl=64 | throughput | 1,150 tuples/sec | 1,125 tuples/sec | 1,008 tuples/sec | +2.2% | +14.1% |
| bs=100 sw=10 sl=64 | MB/s | 0.702 MB/s | 0.687 MB/s | 0.615 MB/s | +2.2% | +14.1% |
| bs=100 sw=10 sl=64 | p50 | 85,112 us | 83,841 us | 100,701 us | +1.5% | -15.5% |
| bs=100 sw=10 sl=64 | p95 | 118,599 us | 134,321 us | 107,244 us | -11.7% | +10.6% |
| bs=100 sw=10 sl=64 | p99 | 118,599 us | 134,321 us | 116,122 us | -11.7% | +2.1% |
| bs=1000 sw=10 sl=64 | throughput | 1,318 tuples/sec | 1,341 tuples/sec | 1,042 tuples/sec | -1.7% | +26.5% |
| bs=1000 sw=10 sl=64 | MB/s | 0.804 MB/s | 0.819 MB/s | 0.636 MB/s | -1.8% | +26.4% |
| bs=1000 sw=10 sl=64 | p50 | 755,442 us | 741,999 us | 981,959 us | +1.8% | -23.1% |
| bs=1000 sw=10 sl=64 | p95 | 825,214 us | 800,914 us | 1,023,080 us | +3.0% | -19.3% |
| bs=1000 sw=10 sl=64 | p99 | 825,214 us | 800,914 us | 1,051,697 us | +3.0% | -21.5% |
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,357.21,200,128000,560,0.342,17115.16,27085.98,27085.98
1,100,10,64,20,1739.42,2000,1280000,1150,0.702,85111.64,118598.84,118598.84
2,1000,10,64,20,15174.97,20000,12800000,1318,0.804,755442.05,825214.29,825214.29
|
CI note for reviewers — the one red check on this PR is not caused by this diff. The fix itself is verified on a real runner. and The red check is The backport check runs It surfaced here only because Also worth ignoring: a Happy to drop the Update — the runs finished, and they isolate the failure cleanly. That gives a controlled A/B on this exact commit: the same job name, |
|
Two things: this PR's own CI is not verifying it, and here is the verification it is missing. Why its CI proves nothingEvery stack skipped on the latest run. The precheck log says why: The
So the manual-label workaround does not hold: it survives until the next The verification, run locally
Pre-fix manifest ( Post-fix manifest (this PR) — same input, same command: That 110 is the same count the description quotes from a passing run, which is a useful cross-check that the synthesized package set matches the real installed one. Reproducible from the repo root: git show origin/main:amber/LICENSE-binary-python > /tmp/A.txt
git show HEAD:amber/LICENSE-binary-python > /tmp/B.txt
# build /tmp/pip-licenses.csv = every "- name==ver" bullet under a "Python packages:"
# heading in /tmp/A.txt, plus cloudpickle,3.1.2,"BSD License"
./bin/licensing/check_binary_deps.py --ignore-transitive-version python /tmp/pip-licenses.csv --license-binary /tmp/A.txt # exit 1
./bin/licensing/check_binary_deps.py --ignore-transitive-version python /tmp/pip-licenses.csv --license-binary /tmp/B.txt # exit 0I have not re-added the |
|
Backport to |
…e#8375) ### What changes were proposed in this PR? `typing-extensions` is a direct dependency, and the version it resolves to moved to 4.16.0 while `amber/LICENSE-binary-python` still claimed 4.14.1. Direct-dependency drift hard-fails the license check rather than being reported as informational, so `build / pyamber (ubuntu-latest, 3.12)` fails its license step on every pull request and on `main` — the last four `Required Checks` runs on `main` are red — and every later step in that job is skipped. Re-running does not help: the resolver picks up 4.16.0 each time. The version is the only thing that changed. PyPI reports `license_expression: PSF-2.0` for both 4.14.1 and 4.16.0, so the entry stays in the Python Software Foundation License section it is already listed under. The transitive entries printed alongside it in the same output are advisory; the nightly exact-match check on `main` refreshes those. ### Any related issues, documentation, discussions? Closes apache#8374. Same failure mode as apache#8293, with a different package. apache#7285 is the weekly aggregate tracker `license-binary-checker.yml` re-uses; it does not name the offending dependency. ### How was this PR tested? Verified before editing rather than after: the failing run is [33710669719](https://github.com/apache/texera/actions/runs/33710669719), whose license step prints `~ typing-extensions: LICENSE-binary=4.14.1 bundled=4.16.0` under `DRIFT (direct)`. PyPI's metadata for both versions reports `license_expression: PSF-2.0`, and the entry is already in the Python Software Foundation License section, so no section move is needed. The check itself runs in CI on this PR — a green `build / pyamber (ubuntu-latest, 3.12)` is the confirmation that the manifest now matches what gets installed. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (claude-opus-5)
What changes were proposed in this PR?
One bullet in
amber/LICENSE-binary-python, in the BSD 3-Clause section:Why it is a bug.
build / pyamber (ubuntu-latest, 3.12)has been redrepo-wide since ~09:00-13:00 UTC on 2026-08-31, on every open PR regardless of
what the PR touches, because joblib 1.6.0 stopped vendoring cloudpickle:
joblib/externals/cloudpickle/in the wheelRequires-Distcloudpickle>=3.0pip-licensesamber/operator-requirements.txtreaches joblib transitively, so pip nowinstalls cloudpickle as its own distribution and the manifest no longer
describes what the image bundles:
Note that the code was always in the distribution -- it shipped inside the
joblib wheel. What changed is that it became a separate distribution, and so
became visible to the license scanner.
License clearance. cloudpickle 3.1.2's wheel METADATA says
License: BSD-3-Clause(PyPI's classifier reports only the genericBSD License, which is what the CI line quotes), and the bundledLICENSEisthe canonical three-clause text. That is ASF Category A. The wheel ships no
NOTICE, andlicenses/LICENSE-BSD-3-Clause.txtalready carries the text thatthe section references, so no
NOTICE-binary/NOTICE-binary-pythonchange isneeded and the one bullet is the entire fix.
Deliberately out of scope. The same step prints a long
DRIFT (transitive, informational)list (charset-normalizer,click,joblib,regex,scipy, ...). PR runs pass--ignore-transitive-version, sonone of it is fatal, and the nightly exact-match check on
mainowns refreshingthose pins -- bulk-bumping 22 moving targets here would just add churn. That is
why
joblibstays at1.5.3in the manifest even though1.6.0is whatintroduced the coupling.
This unblocks every open PR's
build / pyamber (ubuntu-latest, 3.12)job.Any related issues, documentation, discussions?
Closes #8292
Related: #8294
How was this PR tested?
Reproduced the CI check locally, byte-for-byte, without a full
pip install.CI's own report is a complete description of the diff between the manifest and
reality (1
+, 0 stale, 0 direct drift, 22 transitive drifts), so the installedset was rebuilt as "every claimed bullet at its bundled version, plus
cloudpickle==3.1.2" and fed to the checker exactly as
build.ymlinvokes it:+ cloudpickle==3.1.2 (license: BSD License)-- identical to run 33394216939OK: 110 Python packages match LICENSE-binary.The
--ignore-transitive-version-less (nightly) mode still exits 1 on the 22transitive drifts both before and after this change, so the behavior there is
unchanged -- that is the nightly refresh's job, not this PR's.
Upstream cause confirmed straight from the published wheels:
The failure was verified as repo-wide rather than PR-specific on two unrelated
branches --
chore/remove-superseded-sql-updates(run 33394216939, 12:55 UTC)and
ci/8084-backport-manager-approval-gate(run 33392828233, 12:39 UTC) -- bothfailing on the same
+ cloudpickle==3.1.2line, against green license steps at08:39 UTC (#8282) and 08:49 UTC (#8286).
One wrinkle worth flagging for the reviewer:
buildwas skipped on thisPR's first runs.
amber/LICENSE-binary-pythonmatches no glob in.github/labeler.yml, so no stack label was applied,precheckselected nostacks, and the aggregate gate accepts
build: skippedas passing -- i.e. achange to this manifest is not normally gated by the check that validates it.
The
pyamberlabel was applied by hand here so the license check actually runsagainst the fix. That gap is filed separately as #8294 and left out of this
diff to keep the hotfix a one-liner.
Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 5)