Skip to content

Add an optional native parser backend with pure-Python fallback - #694

Open
lowmiaq-gmail wants to merge 3 commits into
theskumar:mainfrom
lowmiaq-gmail:codex/native-backend-adapter-20260827
Open

Add an optional native parser backend with pure-Python fallback#694
lowmiaq-gmail wants to merge 3 commits into
theskumar:mainfrom
lowmiaq-gmail:codex/native-backend-adapter-20260827

Conversation

@lowmiaq-gmail

Copy link
Copy Markdown

Problem

Some parse-heavy consumers of python-dotenv can spend measurable time in the parser, while the default package must remain pure Python and portable.

Proposal

Add a small optional adapter at the parser boundary:

  • pip install python-dotenv remains the current pure-Python path.
  • pip install "python-dotenv[native]" opts into fast-dotenv-rs-backend on CPython.
  • PyPy and environments without the backend continue using the existing parser.
  • The adapter returns the existing upstream Binding and Original objects.
  • Backend contract failures are visible; they are not silently converted into a false native pass.

The backend-only distribution does not install dotenv or a console script, so it can coexist with this package. The Rust implementation and wheels are maintained separately in fast-dotenv-rs.

Compatibility

The patch is based on upstream main at a00cb2eed0704cd6d2071b2004c37e95ccc86ee5 and only changes the parser boundary plus the optional extra. Existing Binding, Original, parse_stream, CLI, and fallback behavior remain the public contract.

Local validation:

  • Upstream suite: 263 passed, 1 skipped in an isolated CPython 3.14 environment; one unrelated macOS printenv --version fixture differs from Linux behavior.
  • Native adapter tests pass.
  • Differential corpus: 1,088 records, zero mismatches across Python/native paths.
  • Property-style corpus: 500 valid + 500 malformed inputs, zero mismatches.
  • Backend CI: 10/10 jobs passed across Linux x86_64/arm64, macOS arm64/Intel, and Windows x86_64 on CPython 3.10 and 3.14.

Performance evidence

In the real-consumer benchmark, 21/30 p50 rows improved, 8 were neutral, and 1 small cold-start CLI row regressed. Representative absolute savings were approximately 0.13 ms for a small dotenv_values workload, 0.75 ms for medium input, and 4.94 ms for large input. The adapter is intentionally optional because small cold-start workloads are not universally faster.

Maintenance and rollback

  • No Rust compiler is required for the default install.
  • The native dependency is restricted to CPython; PyPy remains pure Python.
  • No compatible backend means the existing parser remains active.
  • The backend wheel matrix and release workflow are maintained in the separate backend repository.

This is the concrete implementation for #693; the backend-only package release is being completed independently so that the upstream package does not need to own a Rust build pipeline.

@lowmiaq-gmail

Copy link
Copy Markdown
Author

Update: the backend-only distribution is now publicly available as fast-dotenv-rs-backend 0.1.1.

The release workflow built and audited 5 platform wheels plus an sdist; the publish job succeeded, and fresh public PyPI install/coexistence/native-call checks pass locally. Evidence run: https://github.com/lowmiaq-gmail/fast-dotenv-rs/actions/runs/33084678870. The PR remains default-off: without the optional backend, python-dotenv stays on the existing pure-Python parser.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant