Pin 202609 pipeline toolchain (duckdb/pandas/pyarrow/rdflib) + isolated venv

[rdhyee]

Summary

• 🤖 rbotyee (Claude, operated by @rdhyee) — Raymond's intent: close the toolchain-pin gap a clean-checkout reproducibility test found on 2026-08-31 (decision 0 of the isamples_202609 publish plan), before assembling a publish set. What I did: exact-pinned pandas/pyarrow/rdflib in scripts/requirements.txt alongside the existing duckdb==1.4.4 pin, and added scripts/setup_pipeline_venv.sh so those pins install into an isolated scripts/.venv (via uv) instead of a shared interpreter.
• Why this was needed: only duckdb was pinned, and nothing ever installed from scripts/requirements.txt into an isolated env — scripts ran against whatever was in the shared pyenv myenv (used by many other projects). Between 2026-08-28 and 2026-08-31, an unrelated myenv upgrade moved duckdb 1.4.4 → 1.5.5, which changed one step's output bytes and hard-crashed another. Full writeup: provenance/isamples_202609/step9_clean_rerun_2026-08-31.md (isamples-suite provenance repo, not in this PR).
• Mirrors the hermetic pattern the pqg repo's step 1 already uses (its own uv.lock) — applied here as a pinned requirements.txt + uv-managed venv since these are one-shot scripts, not an installable package.

Test plan

• bash scripts/setup_pipeline_venv.sh creates scripts/.venv and reports duckdb 1.4.4 pandas 2.3.3 pyarrow 21.0.0 rdflib 6.3.2 — matches the 2026-08-28 reference build's own manifest.json environment fields exactly.
• Full clean-checkout chain rerun (steps 2–7) against this pinned venv, byte-compared to the 2026-08-28 reference — in progress, will report results in the provenance repo (not part of this PR).
• Codex review

🤖 Generated with Claude Code

https://claude.ai/code/session_01NdwkgvUJhJ3G9cJNj9oUuZ

[rdhyee]

🤖 rbotyee (Claude, operated by @rdhyee) — Raymond's intent: have Codex review this PR before merge, per the repo's standard multi-AI review loop. What I did: ran a Codex review; findings and my response below.

Codex review (first pass)

High — Makefile's PY ?= python default meant make all/make test still ran against the ambient interpreter even after scripts/setup_pipeline_venv.sh created the pinned venv — the exact failure mode this PR set out to close, just reachable through a different door.

Medium — the PR's language overstated hermeticity: pandas/pyarrow/rdflib's transitive dependencies, the Python interpreter version, and uv itself are still resolver-chosen, so this is narrower than a full lockfile (unlike pqg's uv.lock-based step 1).

Low — a few wording issues: "major-version bump" for a 1.4→1.5 release-line change; "the pin drifted" when the file's pin didn't move (the ambient install drifted because nothing had ever installed from the pin); setup_pipeline_venv.sh printed "ready" before its own verification step, so an import-time failure could look like success; and DATA_PROVENANCE.md's new bullet sat next to a pre-existing "still unresolved" note about an unrelated (data-lineage) reproducibility gap, reading as a contradiction.

Response (pushed in 710adf7)

• Makefile's PY now defaults to scripts/.venv/bin/python (override with PY=python etc. — CI already installs into its own ephemeral runner via plain pip, so it isn't exposed to the ambient-drift risk this PR is about); warns clearly if the venv is missing. Verified make test (62/62) against the pinned venv.
• Narrowed the "exact pins"/"hermetic" language in scripts/requirements.txt and setup_pipeline_venv.sh to what's actually true: the 4 output-critical direct dependencies are pinned; transitive deps/Python/uv are not, and that's flagged as a known gap rather than implied to be closed.
• Fixed the three wording issues and the print-order bug; added a sentence in DATA_PROVENANCE.md distinguishing this (now-closed) toolchain-pin gap from the separate, still-open data-lineage gap described immediately below it.

Full Codex transcript available on request (kept in session notes, not posted here to keep this comment scannable). Second pass in progress.

[rdhyee]

🤖 rbotyee (Claude, operated by @rdhyee) — Raymond's intent: same as above — Codex sign-off before merge. What I did: fixed the 2 remaining low-severity nits from Codex's second pass (f8c05af) and got its final verdict.

Codex second pass — verdict

Verdict: all four original findings are substantively resolved.

Two low-severity nits it caught, both fixed in f8c05af:

• Makefile's missing-venv check used wildcard, which doesn't search $PATH — so make ... PY=python produced a false "not found" warning. Now only checks when PY looks like a path (contains /).
• DATA_PROVENANCE.md credited make with "installing" packages; only scripts/setup_pipeline_venv.sh installs — make just uses the resulting venv. Reworded.

Codex's own verification: make test (62/62) passing, bash -n and git diff --check clean, PR CI green.

Claude + Codex: LGTM. Not merged — Raymond's call.

https://claude.ai/code/session_01NdwkgvUJhJ3G9cJNj9oUuZ