Differentiable physics meets real quadrotors in flight

What it does This repository contains the training code for a Nature Machine Intelligence paper on vision-based agile quadrotor flight. It learns control policies by backpropagating through a differentiable physics simulator instead of treating dynamics as an opaque forward-only environment. The codebase handles both solo high-speed maneuvers and multi-agent swarm coordination.

The interesting bit Instead of wrapping a black-box simulator around the policy, the physics engine itself is a layer in the gradient computation graph. The network therefore learns how its motor commands propagate through time, and the authors claim this is the first pipeline of its kind to actually fly on a real robot.

Key highlights

• End-to-end vision policies trained inside a CUDA-accelerated differentiable physics engine
• Supports both single-agent aggressive flight and multi-agent swarm tasks
• Evaluation depends on a separate simulator release hosted on the GitHub release page
• Authors describe this as the first real-world quadrotor trained via differentiable physics

Caveats

• The evaluation simulator and validation code are not included in the repository itself
• README offers minimal architectural detail; exactly how the vision backbone couples to the physics gradients is unclear
• Requires compiling custom CUDA operations and is tested only against PyTorch 2.2.2 and CUDA 11.8

Verdict A solid reference if you are studying sim-to-real transfer or differentiable simulation in robotics. If you need a production flight stack, this remains a research artifact.