Copycat driving with transformers in the CARLA simulator

What it does

TransFuser is an end-to-end driving agent for the CARLA simulator that learns by imitating a privileged autopilot. It consumes RGB images, LiDAR point clouds, depth maps, semantic segmentation, and top-down maps, then fuses them with a transformer to output driving actions. The repository includes the full training pipeline, a 210GB dataset generated across eight towns, and pretrained checkpoints.

The interesting bit

Rather than relying on geometric projections or hand-tuned fusion rules, it uses a transformer to let sensor modalities interact directly. The authors also open-sourced the privileged autopilot itself, so you can generate fresh expert trajectories or audit exactly what the student is trying to copy.

Key highlights

• Imitation-learning setup trained on a 210GB multi-modal dataset spanning RGB, depth, semantics, LiDAR, bounding boxes, and ego measurements from eight CARLA towns
• Evaluation support for the Longest6 benchmark and official CARLA Leaderboard submission via Docker
• Pretrained agent weights available, though the README mentions “all 4 methods” without naming them
• Multi-GPU training support and a results parser that maps infractions back onto town maps

Caveats

• Tightly coupled to CARLA 0.9.10.1, with no mention of newer simulator versions
• Evaluating a model trained on a single GPU requires manually removing a line in submission_agent.py
• Architectural details are scarce; you will likely need the papers to understand the fusion mechanism beyond the high-level description

Verdict

Grab this if you are researching sensor-fusion architectures or need a strong CARLA baseline to beat. Skip it if you want a production-ready stack or a simulator-agnostic framework.