Finally, BERT-style pretraining that actually works on ResNets

What it does

SparK is the official PyTorch implementation of an ICLR 2023 Spotlight paper that pretrains standard CNNs—ResNets, ConvNeXts, and theoretically any convnet—using BERT-style masked autoencoding. It provides pretraining and finetuning scripts, plus a suite of pretrained weights that drop into standard model definitions.

The interesting bit

The authors pitch this as the first time masked modeling has worked on vanilla convolutional networks without turning them into transformers. The README even includes a notebook diagnosing the “mask pattern vanishing” issue that supposedly stymied earlier attempts—dense convolutions were apparently washing out the holes.

Key highlights

• Claims to outperform pretrained Swin-Transformers on ImageNet-1K after finetuning, with smaller pretrained models sometimes beating larger unpretrained ones.
• Ships with pretrained weights for ResNet50/101/152/200 and ConvNeXt-S/B/L, available both as bare backbones and with a UNet-style decoder attached.
• Includes Colab notebooks for visualizing reconstructions and the sparse convolution masking behavior.
• Codebase is intentionally minimal, relying on standard timm ResNet definitions and the official ConvNeXt implementation.

Caveats

• Weights are hosted on Google Drive rather than Hugging Face or timm; both integrations are still unchecked on the todo list.
• Reproduction is pinned to PyTorch 1.10 and timm 0.5.4, which may not play nicely with current environments.

Verdict

A solid stopgap if you need self-supervised pretraining for existing CNN pipelines without rewriting your stack around Vision Transformers. If you have already moved to ViTs and MAE, there is less urgency.