DeepSIM learns to map between a primitive representation of an image (such as edges or segmentation) and the image itself, trained from a single training pair with extensive augmentation including thin-plate-spline transforms. The model enables image manipulation by editing the primitive representation and reconstructing the corresponding image through a conditional GAN architecture, supporting applications like flipping eyebrows, changing object shapes, and converting car types.