Rotary Transformer implements RoPE, a relative position encoding method where context embeddings (query and key) are multiplied by rotation matrices dependent on absolute position. The key property is that the inner product of context embeddings depends only on relative position, making it suitable for linear attentions. The implementation modifies the self-attention layer with minimal code changes using bert4keras.