This repository provides a complete encoder-decoder transformer with linear-time attention, based on the Linformer paper. It implements the projected attention mechanism that compresses the key and value matrices to achieve linear complexity while maintaining model performance. The implementation includes a language model class for easy experimentation with very long sequences (1000+ tokens) on standard hardware.