A data format that puts JSON on a token diet without going binary

What it does

TONL is a TypeScript-first serialization format that encodes structured data more compactly than JSON while keeping it plain text. It ships with a document API supporting JSONPath-like queries, mutations, aggregations, and streaming for multi-gigabyte files. The project also includes a CLI tool with interactive dashboards and a schema language for validation.

The interesting bit

Rather than resorting to binary blobs that save bytes but destroy debuggability, TONL uses a tabular, schema-hinted syntax—imagine CSV’s more sophisticated cousin for nested objects—to cut token counts. It also bundles an adaptive optimizer that automatically picks between delta encoding, bit packing, and dictionary compression depending on the data shape.

Key highlights

• Claims 36% smaller byte size and 45% fewer GPT-5 tokens than JSON on typical e-commerce data, with round-trip safe conversion.
• Query engine supports filtering, wildcards, recursive traversal, and aggregation (groupBy, stats), backed by hash and B-tree indexes.
• Zero runtime dependencies, browser-ready at ~29.5 KB gzipped, and ships with an interactive CLI featuring neon and matrix themes.
• Includes advanced optimization modules: dictionary encoding, delta encoding, run-length encoding, and bit packing for booleans.
• Schema definition via TSL (TONL Schema Language) with 13 constraint types and auto-generated TypeScript types.

Caveats

• The README cites both “698 tests” and “2,300+ tests” in different sections, so the exact test count is unclear.
• The performance table labels the 1600× query speed improvement as a “Target: <1ms,” suggesting that figure may be aspirational rather than measured.

Verdict

Developers who frequently hit LLM token limits or need to stream large structured datasets with lightweight tooling should evaluate this. If you do not need human-readable debugging or JSONPath-style queries over compact data, the format’s extra complexity may not pay off.