grist

grist streams zstandard-compressed Reddit dumps, filters their NDJSON records, and writes matching records without loading a dump into memory. Inputs are processed in argument order and matching output remains in source order.

Build

Go 1.26 or newer is required.

make test
make build
./dist/grist --help

The build is a stripped, static binary (CGO_ENABLED=0). You can also install it with go install github.com/johndfowler/grist@latest.

Usage

grist filter RC_2025-01.zst RS_2025-01.zst \
  --subreddit golang \
  --author some_user \
  --after 2025-01-01 \
  --before 2025-02-01 \
  --format csv \
  --output matches.csv

Each input must contain zstd-compressed, newline-delimited JSON objects such as the Arctic Shift Reddit comment and submission dumps. Use - as the only input to read compressed data from stdin:

cat RC_2025-01.zst | grist filter - --subreddit golang

Filters and options

• --subreddit: case-insensitive subreddit match
• --author: case-insensitive author match
• --after: inclusive lower timestamp bound
• --before: exclusive upper timestamp bound
• --format, -f: jsonl (default) or csv
• --output, -o: output path (stdout by default)
• --workers, -w: concurrent filter workers (logical CPU count by default)
• --no-progress: disable progress display

All active filters combine with AND. Dates accept RFC 3339 or YYYY-MM-DD; date-only values mean midnight UTC. Records missing a field required by an active filter don't match. created_utc may be a JSON number or numeric string containing Unix seconds.

JSONL output preserves each matching source object and appends one newline. CSV uses this fixed schema:

id,created_utc,subreddit,author,permalink,title,body,url,score

Missing CSV fields are blank and standard CSV quoting handles commas, quotes, and embedded newlines. Interactive terminals show compressed-byte progress for files or record progress for stdin. Progress and diagnostics go to stderr and never contaminate stdout.