This project presents DAM4SAM, a visual object tracking system that augments the Segment Anything Model 2 (SAM2) with a distractor-aware memory mechanism. The approach addresses a key weakness of memory-based trackers by better handling distracting objects in the scene. The method achieves state-of-the-art performance on six benchmarks including VOT/S challenges without additional training. The authors also introduce the DiDi dataset for studying distractor handling in tracking.