Training LLMs to forget the present

What it does

TimeCapsuleLLM is a research project that curates period-specific text corpora—currently 90 GB of London writing from 1800 to 1875—and trains small language models from scratch on nothing else. The goal is to isolate the linguistic patterns, knowledge, and cultural assumptions of an era without contamination from modern concepts. Versions range from early nanoGPT prototypes to a 90-billion-token Llama-style model trained through 182 K steps.

The interesting bit

The project treats temporal bias as a feature, not a bug: by deliberately walling off training data, the model hallucinates in historically consistent ways. In one v1 prompt about 1834 London, the output referenced a real protest and Lord Palmerston—a genuine historical connection drawn from a dataset that never mentions smartphones or social media.

Key highlights

• Built on successive architectures: nanoGPT (v0/v0.5), Microsoft Phi-1.5 (v1), and llamaforcausallm (v2).
• v2 trained on 136,344 documents totaling 90 GB raw / 112 GB tokenized.
• Custom tokenizer trained on the cleaned historical corpus to match period vocabulary.
• Published dataset and bias reports (pronoun, geographic, temporal) available on Hugging Face.
• Now expanding beyond London under academic supervision at Muhlenberg College and in collaboration with University College London.

Caveats

• Outputs remain highly hallucinatory; facts are often invented even when the tone feels era-appropriate.
• Earlier versions suffer from OCR noise (“Digitized by Google”) and tokenization glitches that insert spurious spaces between characters.
• The repository is primarily data curation and tokenizer preparation; it defers the full training pipeline to upstream projects like nanoGPT.

Verdict

Historians, linguists, and ML researchers studying cultural bias should take a look. Anyone needing a reliable, factually accurate model should steer clear—this is a deliberate anachronism engine, not a trustworthy encyclopedia.