UP-DETR introduces random query patch detection as a pretext task to pre-train transformers for object detection without human annotations. The approach inherits the DETR architecture with ResNet-50 backbone and transformer encoder-decoder, achieving 43.1 AP on COCO after fine-tuning. Pre-training can initialize CNN weights from self-supervised methods like SwAV while freezing them during transformer pre-training.