Release netCDF memory-map pages after each frame - #5463
Conversation
NCDFReader._read_frame now drops the pages of scipy's memory map once the frame has been copied into the Timestep. Nothing else unmaps them, so without this the resident memory of the process grows towards the size of the whole trajectory file while it is read. The map is read only, so dropping is safe: pages are faulted back in if the data are read again. MADV_DONTNEED does not exist on Windows, where the call is skipped and the reader is unchanged.
Documentation build overview
5 files changed ·
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## develop #5463 +/- ##
========================================
Coverage 93.87% 93.87%
========================================
Files 182 182
Lines 22522 22526 +4
Branches 3206 3207 +1
========================================
+ Hits 21143 21147 +4
Misses 917 917
Partials 462 462 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
| The pages of the memory map are released after every frame, so that | ||
| memory use no longer grows towards the size of the trajectory file | ||
| while it is read. Requires ``MADV_DONTNEED``, which is not available | ||
| on Windows. |
There was a problem hiding this comment.
FWIW, I don't think it is necessary to add a versionchanged directive for a bug fix--mostly just appropriate for user-facing behavior changes, otherwise we may have way too many such directives.
There was a problem hiding this comment.
Okay makes sense, removed it.
| SimpleNamespace(madvise=advice.append), | ||
| ) | ||
| universe.trajectory[1] | ||
| assert advice == [mmap.MADV_DONTNEED] |
There was a problem hiding this comment.
The test is a bit convoluted with monkeypatch--could we avoid that?
Also, am I understanding correctly that on this branch something like https://github.com/pythonprofilers/memory_profiler should show a constant level of physical memory consumption, while it should increase linearly when reading in/iterating over a trajecotry of the appropriate type on the develop branch?
We could probably capture that with https://asv.readthedocs.io/en/stable/writing_benchmarks.html#peak-memory as well, assuming you're saying that physical memory is being used excessively on develop in these scenarios.
There was a problem hiding this comment.
Yea I agree, I removed the test.
There was a problem hiding this comment.
But yes I ran memory profilers locally on 4GB sized trajectory.
Reading frame by frame with develop leads to memory usage climbing steadily and reaching peak memory of 4GB since nothing unmaps the scipys memory map.
With the fix in place the memory didnt climb at all and behaved as expected when reading frame by frame.
Review feedback: a versionchanged directive is not needed for a bug fix with no user-facing behaviour change, and the test only asserted that madvise is called with a particular flag, which is an implementation detail. The values read from the trajectory are unchanged and already covered by the existing NCDF reader tests.
3 participants
Description
NCDFReader._read_frame now drops the pages of scipy's memory map once the frame has been copied into the Timestep. Nothing else unmaps them, so without this the resident memory of the process grows towards the size of the whole trajectory file while it is read, as documented in discussion here. (ping @orbeckst )
MADV_DONTNEED does not exist on Windows, so the call is guarded by hasattr(mmap, "MADV_DONTNEED") and the reader is unchanged there.
Changes made in this Pull Request:
NCDFReader._read_framenow drops the pages of scipy's memory map once theframe has been read into the
Timestep.LLM / AI generated code disclosure
LLMs or other AI-powered tools (beyond simple IDE use cases) were used in this contribution: yes test written by AI
PR Checklist
package/CHANGELOGfile updated?package/AUTHORS?Developers Certificate of Origin
I certify that I can submit this code contribution as described in the Developer Certificate of Origin, under the MDAnalysis LICENSE.