vrk jsonl

About

Converts between JSON arrays and JSONL (one record per line). Splits arrays for pipeline processing, or collects JSONL lines back into an array when an API expects one. Uses a streaming decoder, so it handles arrays larger than available memory.

The problem

You have a 3GB JSON array from a data export and need to process records one per line. jq -c ‘.[]’ loads the entire array into memory and gets OOM-killed. Going the other direction - collecting JSONL back into an array for an API that expects one - requires careful bracket and comma handling that breaks on empty input.

Before and after

Before

# split: jq loads entire array into memory, OOM on large files
cat data.json | jq -c '.[]'
# collect: manual bracket/comma handling
echo '['; cat records.jsonl | sed 's/$/,/' | sed '$ s/,$//' ; echo ']'

After

cat data.json | vrk jsonl
cat records.jsonl | vrk jsonl --collect

Example

cat data.json | vrk jsonl

Exit codes

Flags