A CNN that stole the ViT playbook and beat it on an iPhone

What it does

RepViT is a family of lightweight CNNs built by incrementally upgrading MobileNetV3 with structural insights borrowed from efficient Vision Transformers, while stubbornly remaining purely convolutional. The same repository also ships RepViT-SAM, which replaces the heavyweight transformer encoder in the Segment Anything Model with a RepViT backbone to chase real-time segmentation on mobile hardware. Both lines of work treat on-device latency as a first-class metric, benchmarking via Core ML on an iPhone 12.

The interesting bit

The project’s central conceit is that lightweight ViTs win on mobile not because of self-attention, but because of smarter architectural choices—RepViT simply copies those choices into a CNN and, according to the authors, ends up faster than the ViTs it imitates. RepViT-SAM then demonstrates the payoff: swapping this CNN into SAM purportedly delivers zero-shot transfer performance comparable to MobileSAM with a fraction of the inference time.

Key highlights

• RepViT-M1.0 achieves about 80% ImageNet top-1 accuracy at roughly 1 ms latency on an iPhone 12; the largest variant, M2.3, reaches 83.7% at 2.3 ms.
• RepViT-SAM claims nearly 10× faster inference than MobileSAM by dropping self-attention entirely in favor of a RepViT encoder.
• All models are integrated into timm and come with Core ML exports for direct mobile benchmarking.
• The codebase includes downstream pipelines for object detection, instance segmentation, and semantic segmentation built on MMDetection and MMSegmentation.
• Grounded-SAM has adopted the encoder as Grounded-RepViT-SAM.

Caveats

• The README links to detection and segmentation guides but keeps most implementation details behind separate readmes, so surface-level documentation is thin.
• Latency figures are tied to iPhone 12 and Core ML Tools; translating those numbers to other mobile chipsets or Android runtimes is left as an exercise for the reader.

Verdict

Worth a look if you need pretrained mobile vision backbones backed by actual on-device latency tables rather than theoretical FLOP counts. Look elsewhere if you want architectural novelty around attention mechanisms—this project explicitly avoids them.