chunk() takes a path. A caller that has work to do before chunking – filtering rows, adding columns – therefore writes its result to disk only so chunk() can read it back and write the same bytes again as chunks. Internally chunk() consumes lines through readline.createInterface and uses the path for basename() and an error message.
In geonames-rdf that is lde/download.ts: it streams allCountries.txt out of the zip, adds two columns per row, writes geonames.tsv (~1.7 GB), calls chunk() on it and removes it; the same for alternate-names.tsv (~0.7 GB). About 2.4 GB written and read again per weekly run, and peak disk holds a table and its chunks at once. The download step already went from 228 s to 95 s by cutting passes over the table; this is the last one left.
Suggestion
Accept a line source next to the path:
chunk(input: string | AsyncIterable<string> | Readable, options: ChunkOptions & { name?: string })name is required when the input is not a path, since that is where the chunk names came from. The loop body does not change. removeChunksOf() keys on name as it does now.
Found while porting geonames-rdf (netwerk-digitaal-erfgoed/geonames-rdf#50), see #782 and #793.
chunk()takes a path. A caller that has work to do before chunking – filtering rows, adding columns – therefore writes its result to disk only sochunk()can read it back and write the same bytes again as chunks. Internallychunk()consumes lines throughreadline.createInterfaceand uses the path forbasename()and an error message.In geonames-rdf that is
lde/download.ts: it streamsallCountries.txtout of the zip, adds two columns per row, writesgeonames.tsv(~1.7 GB), callschunk()on it and removes it; the same foralternate-names.tsv(~0.7 GB). About 2.4 GB written and read again per weekly run, and peak disk holds a table and its chunks at once. The download step already went from 228 s to 95 s by cutting passes over the table; this is the last one left.Suggestion
Accept a line source next to the path:
nameis required when the input is not a path, since that is where the chunk names came from. The loop body does not change.removeChunksOf()keys onnameas it does now.Found while porting geonames-rdf (netwerk-digitaal-erfgoed/geonames-rdf#50), see #782 and #793.