The package provides tools for fine-tuning pretrained GPT-2 (124M-774M) and GPT Neo (125M-350M) models on custom datasets, or training from scratch. It leverages pytorch-lightning for distributed training across multiple GPUs with optimized memory usage and fast generation. The project also supports model upload/download from the HuggingFace model hub and includes caching, dataset merging, and cross-training capabilities.