This repository contains the code and pre-trained model for the paper that introduced the generative pre-training approach for transformer language models. The implementation supports fine-tuning on downstream NLP tasks, with the ROCStories cloze test as the demonstrated benchmark. It provides both pre-trained weights and training code for fine-tuning on custom datasets.