An Elasticsearch-shaped escape hatch from token bloat

What it does

XERJ is a search engine that auto-indexes folders of code, docs, logs, and PDFs so AI agents can query them through an Elasticsearch-compatible API instead of reading entire files into their context window. It infers schemas on the fly, parses source files into AST-aware records via tree-sitter, and exposes full-text, vector, and hybrid search over the same wire protocol Elasticsearch uses.

The interesting bit

The project treats context-window exhaustion as a storage problem: rather than grepping or loading 5.2 million tokens of WordPress PHP into an LLM, an agent can query a pre-built index and get answers for a few kilobytes per question. It also openly documents its four benchmark losses—read p99 under heavy concurrent write load—instead of burying them.

Key highlights

• Elasticsearch API compatibility: 1,360 of 1,363 conformance tests pass, so existing clients, Kibana, and dashboards connect unchanged.
• Zero-configuration autoindexing: drop a folder on it and it sniffs file types, spins up per-dataset indexes, and handles everything from CSV to SQLite to DOCX.
• One static Rust binary, no JVM or dependencies, prebuilt for Linux, macOS, and Windows on x86-64 and arm64.
• Head-to-head benchmarks against Elasticsearch 8.13.4 show 55 wins, 26 ties, and 4 losses, with 1.72x faster ingest and 1.61x smaller on-disk footprint.
• Vector and hybrid search are built in, so you do not need a separate vector database.

Caveats

• The four benchmark losses are all in read p99 latency while a high-rate writer runs; the team publishes the finding rather than hiding it, but it is a real rough edge.
• Compatibility is with the Elasticsearch API, not its internals: if you rely on deep Elasticsearch plugin behavior, this is not a drop-in reimplementation.
• Three of 1,363 conformance tests still fail; the gaps are listed in the repo.

Verdict

Worth a look if you are feeding large codebases or document dumps to agents and want to keep your vector search, full-text queries, and existing Elasticsearch tooling under one small binary. Skip it if you need a battle-tested, years-old distributed cluster or deep ES plugin compatibility.