SciBERT is a pretrained contextualized embeddings model based on BERT architecture, trained on a large corpus of scientific papers from Semantic Scholar. It is designed to improve performance on downstream NLP tasks in the scientific domain such as named entity recognition, relation extraction, and citation intent classification.