mastros
Back to Blog

JSONL vs JSON: Which to Use for APIs, Logs, and ML

July 31, 2026·
JSONL vs JSON: Which to Use for APIs, Logs, and ML

Use JSON for single documents, API responses, and config files. Use JSONL for logs, streaming datasets, ML training data, and any pipeline where you're appending records continuously. That's the whole decision for most engineering tasks.

A few specifics worth locking in before you read further:

  • Format standard: JSON is governed by RFC 8259, which defines a single root value per document. JSONL (also called JSON Lines or NDJSON) has no RFC equivalent but is the de-facto standard for ML training data at OpenAI, Hugging Face, and BigQuery streaming inserts.
  • Append complexity: Appending to a JSON array is O(n) because you must rewrite the file. Appending to JSONL is O(1) — you write one new line.
  • Streaming: JSONL can be parsed line by line without loading the full file. JSON requires a full parse before any record is accessible.
  • Corruption resilience: A single syntax error in JSON can invalidate the entire file. In JSONL, a bad line can be skipped; the rest of the file stays readable.

Table of Contents

What is JSON and when should you use it?

JSON (JavaScript Object Notation) is a text format that represents a single value — an object, array, string, number, boolean, or null — per document. RFC 8259 is the governing standard. Its strength is hierarchical structure: objects can nest arrays, which can contain other objects, making it natural for representing complex, cross-referenced data in one self-contained payload.

A compact example:

{
  "user": {
    "id": 1042,
    "name": "Priya Nair",
    "roles": ["admin", "editor"],
    "preferences": {
      "theme": "dark",
      "notifications": true
    }
  }
}

That nested shape is exactly what makes JSON the right call for REST API responses, OpenAPI specs, and config files. The whole document is one coherent unit, and any consumer can traverse it as a tree.

Common JSON use cases:

  • REST API responses (single resource or paginated envelope)
  • Application config files (package.json, tsconfig.json, .eslintrc)
  • Small data exchange payloads between services
  • Browser-side state and localStorage payloads
  • OpenAPI / Swagger specs and schema definitions

The key constraint: JSON is a document format. It assumes the reader will consume the whole thing at once.

What is JSONL (JSON Lines / NDJSON)?

JSONL — also called JSON Lines or NDJSON (Newline Delimited JSON) — stores one independent JSON value per line, with each line terminated by . There is no outer array, no commas between records, and no shared root object. The file as a whole is not valid JSON; each individual line is.

Two records as JSONL:

{"id": 1, "event": "page_view", "url": "/pricing", "ts": 1718000001}
{"id": 2, "event": "click", "url": "/signup", "ts": 1718000045}

That's it. No brackets wrapping the pair. A tool expecting a JSON array will fail on this file — which is the single most common mistake developers make when first working with the format.

A few validity notes worth keeping straight:

  • Each line must be a complete, self-contained JSON value.
  • Internal newlines inside a record break the format. Keep records compact.
  • File extensions .jsonl and .ndjson are effectively interchangeable across tooling, though .jsonl dominates in ML ecosystems and .ndjson is more common in logging pipelines.
  • Blank lines are technically allowed by some implementations but are best avoided for compatibility.

How do JSONL and JSON actually differ?

The table below maps the dimensions engineers care about most.

Engineer analyzing JSON vs JSONL comparison tables

Dimension JSON JSONL
Top-level structure Single root value (object or array) Sequence of independent values, one per line
Streaming / incremental parse No — full parse required Yes — parse line by line
Append performance O(n) — full file rewrite O(1) — write a new line
Corruption resilience One error invalidates the file Bad line can be skipped
Human readability Pretty-printable, easy to inspect Compact by design; harder to scan visually
Tooling JSON parsers in every language, jq Unix line tools, jq, ndjson-tools, pandas
Typical uses APIs, configs, small exports Logs, ML datasets, ETL staging, scraping
Conversion complexity Low for small files; memory-heavy at scale Same — streaming approach needed at scale

Streaming and parsing. JSON requires loading the full document into memory before your code can touch a single field. JSONL lets you open a file, read line one, process it, discard it, and move to line two — constant memory regardless of file size. For a 2GB log file, that difference is the gap between a working pipeline and an OOM crash.

Append behavior. If you're writing a JSON array and your process crashes mid-write, you have a broken file. With JSONL, every successfully written line is already a valid, recoverable record. This makes JSONL the natural choice for any process that writes incrementally — a scraper, a log shipper, or a browser extension exporting records one at a time.

Readability tradeoff. JSON wins for human inspection. You can open a pretty-printed JSON file and immediately understand its shape. JSONL is compact by design, and a file with a million records is not something you browse manually. Use JSON when a developer needs to read the output; use JSONL when a machine does.

When should you pick JSON vs. JSONL?

The format decision usually comes down to destination and write pattern, not personal preference.

Pick JSON when:

  • You're returning a single resource from a REST endpoint.
  • You're writing a config file that a developer will read and edit.
  • Your payload is small enough to fit comfortably in memory.
  • The consumer is a browser, a mobile app, or any client that expects a single document.
  • You need nested, cross-referenced structure (e.g., a user object with embedded address and order history).

Pick JSONL when:

  • You're shipping log events to a log aggregator (Elasticsearch, Loki, Datadog).
  • You're building an ML training dataset for fine-tuning. OpenAI's fine-tuning API requires JSONL, with each line containing a {"messages": [...]} object. Hugging Face datasets follow the same pattern.
  • You're running a web scraper that produces output continuously — JSONL lets you append each scraped record without touching previous output.
  • You're staging data for a warehouse. BigQuery streaming inserts expect JSONL; so does most ETL tooling that feeds into Spark or dbt.
  • Your pipeline needs fault tolerance: if a write fails, you lose one line, not the whole file.

A practical rule: if the data is a thing (a user, a config, an API response), use JSON. If the data is a stream of things (events, records, training examples), use JSONL.

Performance, appendability, and what happens when things break

Append complexity is where the json vs jsonl difference becomes a real operational concern. To add a record to a JSON array, you have to read the file, parse it, push the new item, serialize the whole structure, and write it back. At 10MB that's fine. At 500MB it's slow. At 5GB it's a problem you don't want in a hot path.

Hands typing JSONL file content on laptop

JSONL sidesteps this entirely. Open the file in append mode, write one line, close. That's O(1) regardless of how many records already exist.

Memory follows the same pattern. JSON parsers — whether Python's json.load(), Node's JSON.parse, or jq — load the full document into a parse tree. JSONL parsers iterate line by line, so memory stays constant even for multi-gigabyte files. For browser-based exporters, this is especially relevant: keeping a large object graph in browser memory risks crashing the tab on long exports.

Pro Tip: Write JSONL lines atomically — serialize the full record to a string first, then write the string plus in a single call. A partial line write (from a crash mid-record) produces a corrupt line that will fail JSON parsing. Atomic writes mean the worst case is a missing last line, not a broken one.

Corruption resilience follows directly from structure. A missing closing brace in a JSON file makes the entire document unparseable. In JSONL, a malformed line is one bad record. You can log it, skip it, and keep processing. For production pipelines ingesting third-party data, that resilience is worth a lot.

Reading and writing JSON and JSONL in Python, Node.js, and the CLI

Python

import json

# Read JSON
with open("data.json") as f:
    obj = json.load(f)

# Write JSON
with open("out.json", "w") as f:
    json.dump(obj, f)

# Read JSONL (streaming — constant memory)
with open("data.jsonl") as f:
    for line in f:
        record = json.loads(line.strip())
        process(record)

# Write JSONL
with open("out.jsonl", "a") as f:
    f.write(json.dumps(record) + "
")

The jsonlines library on PyPI wraps this pattern cleanly and handles edge cases like blank lines and BOM characters.

Node.js

// Read JSON
const obj = JSON.parse(fs.readFileSync("data.json", "utf8"));

// Read JSONL with readline (streaming)
const rl = readline.createInterface({ input: fs.createReadStream("data.jsonl") });
rl.on("line", (line) => {
  const record = JSON.parse(line);
  process(record);
});

The readline interface handles backpressure automatically. Don't use fs.readFileSync on large JSONL files — you'll load the whole thing into memory and lose the streaming benefit.

CLI tooling

Task Tool Command pattern
Pretty-print JSON jq jq . data.json
Extract field from JSON jq jq '.user.name' data.json
Count JSONL records Unix wc -l data.jsonl
Filter JSONL lines jq + flags jq -c 'select(.event=="click")' data.jsonl
Extract field from JSONL jq jq -r '.url' data.jsonl
Grep JSONL by value grep grep '"event":"click"' data.jsonl

CLI and Unix tools like grep, wc -l, and awk are natural fits for JSONL because the format is line-oriented. jq works on both formats but is typically used for full-document JSON operations. For JSONL, pass the -c flag to keep output compact (one result per line).

When LLM-generated JSON comes back malformed — extra text, trailing commas, unescaped characters — a repair tool like datatool.dev can fix broken JSON before it hits your pipeline.

Converting between JSON and JSONL

From JSONL to a JSON array

  1. Open the JSONL file and iterate line by line.
  2. Parse each line with json.loads().
  3. Collect records into a list.
  4. Serialize the list with json.dump().
import json

with open("data.jsonl") as infile, open("out.json", "w") as outfile:
    records = [json.loads(line) for line in infile if line.strip()]
    json.dump(records, outfile)

Memory warning: this loads all records into memory at once. For large files, use a streaming approach or ClickHouse local mode, which can run SQL against files without a full server install.

From a JSON array to JSONL

  1. Confirm the source is an array at the root level, not a single object.
  2. Iterate over each element and serialize it as a single compact line.
  3. Never just strip the outer brackets — that leaves trailing commas and multi-line records that break streaming parsers.
import json

with open("data.json") as infile, open("out.jsonl", "w") as outfile:
    records = json.load(infile)
    for record in records:
        outfile.write(json.dumps(record) + "
")

jq one-liner (JSON array → JSONL):

jq -c '.[]' data.json > out.jsonl

jq one-liner (JSONL → JSON array):

jq -s '.' data.jsonl > out.json

Common pitfalls

  • Pretty-printed JSON → JSONL: you cannot convert a pretty-printed JSON array to valid JSONL by removing brackets. Every item must be emitted as a single self-contained JSON value on one line.
  • Non-atomic conversion: if your conversion script crashes mid-write, the output file is partial. Write to a temp file and rename atomically.
  • Trailing commas: some JSON-like configs (JSONC, JSON5) allow trailing commas. Standard json.loads() will reject them. Strip them before converting.

Best practices for production use

A few rules that save debugging time later:

  • One record per line, no internal newlines. A pretty-printed record inside a JSONL file breaks every streaming parser that splits on .
  • Terminate every line with . Some tools expect a trailing newline on the last line; include it by default.
  • Never rename a pretty-printed file to .jsonl. The extension signals format; the content must match.
  • Compress for storage. JSONL compresses well with gzip or zstd because repeated field names across lines create high redundancy. A raw 1GB JSONL file often compresses to under 100MB.
  • Validate per line in production. Use JSON Schema validation on each record as it's written, not after the fact. The jsonschema library in Python and ajv in Node.js both support per-record validation.
  • Document your schema. JSONL has no envelope to carry schema metadata. Keep a companion schema file (JSON Schema or a README) alongside any JSONL dataset you hand off.
  • Watch for NDJSON vs. JSON array mismatches. Consumers like Elasticsearch and BigQuery expect NDJSON. Consumers like some REST clients expect a JSON array. Confirm before you ship.

For downstream analytics, a common production pattern is to ingest to JSONL as a landing zone, then compact to Parquet for columnar query performance. JSONL gives you fault-tolerant, appendable staging; Parquet gives you fast aggregations. They're complementary, not competing.

How Mastros handles JSON and JSONL exports from the browser

Browser-based exporters face a constraint that server-side pipelines don't: browser memory is limited, and a tab that runs out of it closes. This is where the JSONL vs JSON difference becomes a practical product decision, not just a theoretical one.

Mastros exports data from Telegram Web, WhatsApp Web, and LinkedIn directly in your browser, with no data sent to an external server. For small exports — a few hundred group members or a short chat thread — JSON works fine. The full dataset fits in memory, and the output is easy to inspect manually or import into a spreadsheet.

For large batch exports — thousands of chat messages, a full Telegram group history, or a bulk media export — JSONL is the right call. Mastros writes records as lines to the output file incrementally, so browser memory stays low regardless of export size. If the export is interrupted, every line already written is a valid, recoverable record. You don't lose the whole file.

Mastros supports CSV, JSON, and JSONL exports depending on the extension and use case. For teams feeding exports into ML pipelines, analytics warehouses, or ETL workflows, JSONL is the format to pick. For recruiters or sales teams importing into a CRM, CSV or JSON usually fits better.

Key Takeaways

Use JSON for single documents and APIs; use JSONL for streams, logs, and ML datasets where append performance and streaming matter.

Point Details
Format by destination JSON for APIs and configs; JSONL for logs, ML training data, and ETL staging.
Append complexity Appending to JSONL is O(1); appending to a JSON array requires a full file rewrite (O(n)).
Streaming and memory JSONL enables constant-memory streaming; JSON requires loading the full document before parsing.
Corruption resilience A bad line in JSONL can be skipped; one syntax error in JSON invalidates the entire file.
Mastros exports Mastros supports CSV, JSON, and JSONL; choose JSONL for large batch exports and ML pipelines, JSON for small manual-inspection exports.

The format choice most teams get wrong

Most teams default to JSON because it's familiar, and that's fine for APIs and configs. The mistake is carrying that default into pipelines where data grows continuously. A JSON array that starts at 10MB and grows to 2GB is a rewrite-on-every-append problem that compounds quietly until someone's pipeline starts timing out.

The real decision point isn't "which format do I know?" It's "how does this data grow, and who reads it?" If a human reads it once and it stays small, JSON is perfectly good. If a machine appends to it, streams it, or ingests it into a warehouse, JSONL is the right starting point. From there, the common path is JSONL as a landing zone, then compaction to Parquet for analytics, as the jsonlkit.com comparison describes. That two-step pattern handles both fault tolerance and query performance without asking you to choose between them.

One practical test before you commit: take a 100MB sample of your expected data, try appending 10,000 records to it in both formats, and measure wall time and peak memory. The numbers usually make the decision obvious. Then validate that your target consumer — OpenAI fine-tuning, BigQuery, Elasticsearch, your CRM — actually expects the format you've chosen. Mismatched expectations at the consumer end are the most common source of "it works locally but fails in production" bugs with these formats.

Mastros exports data to JSON and JSONL directly from your browser

If you're pulling data from Telegram, WhatsApp, or LinkedIn and need it in a format your pipeline can actually use, Mastros gives you CSV, JSON, and JSONL exports without routing your data through an external server. Everything runs locally in your Chrome browser.

Mastros

The Telegram extension exports group members, chat messages, recent contacts, mutual groups, and bulk media, with no API credentials to register. The WhatsApp extension works in read-only mode from WhatsApp Web, no API access required. The LinkedIn extension exports people, companies, jobs, and Sales Navigator leads.

For large exports destined for ML training or analytics ingestion, pick JSONL. For quick manual inspection or CRM imports, pick JSON or CSV. Check the Mastros product page for current export limits, plan details, and supported formats, or go straight to the LinkedIn scraper if that's your use case.

Authoritative references and further reading

  • JSON Lines specification — jsonlines.org: the canonical reference for the JSONL format, including the one-record-per-line rule and streaming rationale.
  • RFC 8259 — IETF JSON standard: the governing standard for JSON syntax and encoding.
  • Line-delimited JSON — Wikipedia: background on NDJSON, streaming JSON protocols, and historical context.
  • JSONL vs JSON comparison — jsonlkit.com: side-by-side technical breakdown including the JSONL landing zone → Parquet pipeline pattern.
  • JSONL format guide — jsonic.io: practical notes on .jsonl vs .ndjson extension compatibility.
  • Convert JSONL to JSON — ClickHouse engineering: recommended approach for large-file conversion without loading everything into memory.
  • jq manual — jqlang.github.io: full reference for jq, the standard CLI tool for JSON and JSONL processing.
  • jsonlines Python package — PyPI: Python library for reading and writing JSONL with edge-case handling.
  • JSONL vs JSON key differences — jsonltools.com: notes on Unix tooling compatibility and CLI workflows for JSONL.
  • OpenAI fine-tuning data format: OpenAI's documentation on the JSONL format required for fine-tuning datasets.
  • Mastros browser exporters: product details for Telegram, WhatsApp, and LinkedIn exports in CSV, JSON, and JSONL.

Recommended

More from the blog