EoMT is a minimalist image segmentation model that converts a standard Vision Transformer into a unified architecture for image and video segmentation tasks. The model encodes both image patches and segmentation queries as tokens within the plain ViT, eliminating the need for adapters or task-specific decoders. Leveraging large-scale pretrained ViTs, EoMT achieves accuracy comparable to state-of-the-art methods while being significantly faster, demonstrating up to 4× speedup with ViT-L over more complex approaches.