Production logs without the sanitization scrub

What it does

Loghub hosts 16 raw system log datasets—HDFS, Windows, Linux, Apache, Android, supercomputer traces, and others—totaling hundreds of millions of lines. The maintainers gathered production logs from prior studies and their own lab systems, deliberately leaving them unsanitized and unanonymized wherever possible. It is offered free for academic and research use, with downloads hosted on Zenodo.

The interesting bit

Most public log datasets are scrubbed before release; Loghub keeps the original formatting and noise intact, which is exactly what makes them useful for stress-testing parsers and anomaly detectors. A handful of datasets even carry ground-truth labels for supervised experiments.

Key highlights

• 16 datasets across distributed systems, supercomputers, operating systems, mobile, server, and standalone software
• Raw sizes range from ~2 MiB (Proxifier) to nearly 30 GiB (Thunderbird supercomputer logs)
• Some datasets include ground-truth labels (HDFS_v1, BGL, Thunderbird, OpenStack, Hadoop, HDFS_v3); most are unlabeled
• Downloaded by over 450 organizations from industry and academia
• Maintained by the LogPAI group, with companion papers at ISSRE 2023 and ISSTA 2024

Caveats

• Explicitly licensed for research or academic work only; commercial use is not addressed
• Several datasets list “N.A.” for time span, and Hadoop’s labeling status is unclear without checking an open issue
• The repo is strictly a catalog and download index—no parsing code or tools are included

Verdict

Grab this if you are building or benchmarking log parsers, anomaly detectors, or AIOps pipelines and need realistic, messy data. Skip it if you are looking for a log analysis framework or production-ready tools.