The repository replicates BERT’s pre-training strategy and experiments with replacing the Transformer backbone with TextCNN. It implements masked language model pre-training and fine-tuning for various NLP tasks such as text classification, document classification, and question answering. The work demonstrates that the pre-train and fine-tuning paradigm is model-independent and can boost performance significantly even with moderate-sized datasets.