Training a neural network to drive by repeatedly crashing in CARLA

What it does

This is an end-to-end autonomous driving experiment built on the CARLA simulator (version 0.9.8). A Proximal Policy Optimization agent learns to steer through two predefined towns by consuming low-dimensional latent representations produced by a frozen Variational Autoencoder, rather than raw camera pixels. The result is a pretrained agent that can navigate Town 02 and Town 07 without real-world collision costs.

The interesting bit

Instead of feeding raw images directly into the policy network, the project first trains a VAE on 12,000 semantically segmented frames to compress 160×80 observations into a latent space, then freezes those weights while the PPO agent trains. It is a tidy separation of perception and control that keeps the reinforcement learning problem from drowning in high-dimensional visual noise.

Key highlights

• Built from scratch in PyTorch with a custom gym-inspired CARLA environment wrapper
• Ships with pretrained PPO agents for both Town 02 and Town 07
• Uses a frozen VAE to compress semantically segmented camera images into a latent state representation
• Includes Tensorboard logging and checkpointing for training runs
• Also contains a work-in-progress Duelling DQN implementation for discrete control

Caveats

• The discrete Duelling DQN driver is explicitly marked as a work in progress
• Tied to CARLA 0.9.8, an older release that only officially supports Windows and Linux
• The VAE requires collecting thousands of simulator frames before the driving agent can train

Verdict

Worth a look if you are studying the mechanics of end-to-end autonomous driving or need a minimal, from-scratch PyTorch PPO implementation tied to a physics simulator. Skip it if you want a mature self-driving stack or support for modern CARLA versions.