Open dataset HDF5 files in read-only mode when loading - #1847
Open
MaxGhenis wants to merge 1 commit into
Open
Conversation
pandas opens HDFStore in append mode by default, so loading a dataset from a read-only file (a Hugging Face cache snapshot, a read-only mount, or a chmod 444 copy) failed with PermissionError before any data was read. Pass mode="r" at both read sites in UKSingleYearDataset and UKMultiYearDataset; the two save() sites keep write access. Add regression tests that load synthetic single- and multi-year datasets from read-only files without network access, plus a microsimulation test that loads a read-only copy of enhanced_frs_2023_24_tiny.h5 and computes household_net_income. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
UKSingleYearDatasetandUKMultiYearDatasetopen dataset files withpd.HDFStore(file_path). pandas' default mode is append ("a"), which requires write access, so loading any dataset that lives in a read-only location fails before a single row is read:This bites Hugging Face cache snapshots (
~/.cache/huggingface/hub/models--policyengine--policyengine-uk-data-private/snapshots/<rev>/enhanced_frs_2023_24.h5), datasets on read-only mounts, and any copy without the write bit. Found on 2026-09-03 while reproducing PolicyEngine/uk-trade-shock-study, where two separate sessions had to patch their venv copy of the loader to proceed.Reproduction on
main:Fix
Pass
mode="r"to the twoHDFStorecalls that only read (the__init__loaders of both dataset classes). The twosave()calls keep their default write mode. Loading never writes, so this changes nothing for writable files.Tests
policyengine_uk/tests/test_read_only_dataset.py:chmod 444, reload, and assert the contents and the file's SHA-256 are unchanged. Both fail onmainand pass with this change. They skip if the environment ignores permission bits (root).microsimulation-marked test: downloadenhanced_frs_2023_24_tiny.h5@1.57.2, copy it read-only, buildMicrosimulation(dataset=path), and computehousehold_net_incomefor 2025. Passes locally (1,000 households, all finite, file hash unchanged). Skips withoutHUGGING_FACE_TOKEN, like the other microsimulation tests.make formatclean.Changelog
changelog.d/hdfstore-read-only.fixed.md(patch bump).🤖 Generated with Claude Code