Autonomous Racing using Reinforcement Learning
A car that learned to drive by crashing into walls thousands of times and slowly figuring out how not to. No lines of steering code, just trial, error, and a reward signal, first in 2D, then scaled up into a real Unity go-kart.
This project builds a driving policy using reinforcement learning instead of writing any explicit steering logic, in two stages: first a 2D agent trained in Gymnasium's CarRacing-v3 environment, then a 3D agent trained inside a Unity go-kart simulation with real physics. Both are trained with PPO (Proximal Policy Optimization), learning purely from a reward for staying on track and a penalty for crashing, no hand-coded rules for when to turn or brake.
It started small and 2D. Using Gymnasium's CarRacing-v3 environment, the car begins as a blank slate that only sees a top-down camera view of the track and has no idea what steering or gas even means. Early on it just spins in circles and drives off the road within seconds. Using an algorithm called PPO (Proximal Policy Optimization), it tries thousands of small variations of its own driving, keeps whatever recently scored higher, and slowly tightens up its behavior over roughly a million and a half timesteps until it's taking corners cleanly and holding the track. That's phase one, worked all the way through: reward, watch, retrain, repeat.
From there the idea scaled up into 3D using Unity and a toolkit called ML-Agents: a flat top-down image swapped for an actual go-kart racing around a modeled track, with real physics, lap checkpoints, and five raycast "whiskers" standing in for the car's eyes instead of pixels. Four karts trained in parallel on identical copies of the same track, all pooling their experience back into one shared PPO brain, which pushed through a full million training steps in a single run instead of four separate ones. The trained brain was exported as a standalone Windows build, so the kart drives itself with no Unity Editor or Python process running behind it.
How it works
Phase one trains in Gymnasium's CarRacing-v3, a 2D top-down racing simulator, using a CNN policy (Stable-Baselines3's CnnPolicy) that reads a stack of 4 grayscale frames and outputs continuous steering, gas, and brake values. Frame-stacking exists because a single image can't encode motion, stacking 4 in a row gives the network enough short-term context to infer how fast and in what direction the car is already moving.
The agent is trained with PPO, an algorithm that nudges the driving policy toward whatever recently earned it a higher reward, while a clipping mechanism (clip range 0.2) stops any single update from swinging its behavior too wildly and destabilizing what it already learned. Reward comes from progress along the track and staying on the road, with a penalty for going off it, so over roughly 1.5M timesteps the mean episode reward climbs from around -50 to consistently above 800.
Phase two moves into Unity with ML-Agents, on a Karting Microgame template scene reworked for training. A custom C# script, KartAgent.cs, implements the ML-Agents API: it feeds the policy a vector observation (5 raycast distance sensors plus the kart's speed) instead of pixels, hands out reward at each track checkpoint the kart clears in order, and penalizes it for going off-track or driving backward through a checkpoint.
Two more scripts were modified to make the stock Karting Microgame trainable rather than just playable: ArcadeKart.cs (the driving physics controller) was patched to handle input refreshing and destroyed-object references safely across thousands of automatic episode resets, and GameFlowManager.cs was changed to skip the game's pre-race 3-2-1 countdown and scene-loading transitions during training, since those exist for a human player and just waste wall-clock time when an agent is resetting hundreds of times an hour.
Four karts train at once on parallel copies of the track, sharing one PPO policy so their experience gets pooled into the same gradient update, which is what let a 1M-step run stand in for what would otherwise be four separate single-agent runs. The best checkpoint is exported straight to ONNX (KartAgent.onnx), and Unity's ML-Agents runtime can run that ONNX file for inference entirely on its own, no Python or PyTorch needed once training is done, which is also what makes the standalone build possible.
Technical Breakdown
The project has two acts: a 2D Gymnasium car learning to drive from scratch, then the same PPO idea scaled up into a real Unity go-kart. These are the actual training curves and in-engine captures from both runs, not staged screenshots.
Phase 1, checkpoint 0032: driving blind
An early PPO checkpoint on CarRacing-v3. At this point the policy has barely started training, so it drifts off the asphalt almost immediately and has no real notion of "steer toward the road."
Phase 1, checkpoint 0142: learning to correct
A mid-training checkpoint. The faint arcs behind the car are its own recent trajectory, and it's now visibly overcorrecting through a turn rather than driving straight off, evidence the reward signal for staying on-track is starting to shape its steering.
Phase 1, checkpoint 0273: holding the line
A late checkpoint, tracking straight down the center of the road into an upcoming corner. This is the same policy architecture as the first image, just a lot more PPO updates later, going from spinning out on straightaways to holding a clean racing line.
Gymnasium: mean episode reward
Mean reward per episode over 1.5M training steps. It climbs from around -50 to roughly 800 by step 600K, dips during a stretch where PPO's exploration bonus temporarily pushes it to try riskier lines, then recovers and peaks above 800 again by the end of training, more or less what a healthy (if slightly noisy) PPO run is supposed to look like.
Gymnasium: entropy loss
Entropy measures how random the policy's action choices still are. It rises early on as PPO's entropy bonus deliberately encourages exploring different driving lines, then falls steadily for the rest of training as the car commits more and more confidently to specific steering decisions instead of guessing.
Unity: four karts training in parallel
The actual Unity Editor scene mid-training: four karts (Kart_AI through Kart_AI (3)) running identical copies of the same oval track at once, each firing its own fan of raycasts (the white lines) out to sense the road edge. Every kart's experience feeds into one shared PPO policy, which is what let a single 1M-step run do the work of four separate ones.
Unity: training console log
Raw ml-agents-learn output from the final stretch of the 1M-step run. Mean reward holds in the 180-200 range with the standard deviation across the 4 karts settling down over time (from the high-20s down to the mid-teens near step 1,000,000), a sign the four karts were converging on similar driving behavior rather than each learning something different.
Unity: reward across saved checkpoints
Reward at the three ONNX checkpoints exported during the run (250K, 500K, and 750K steps). It isn't a clean monotonic climb, it peaks around the 500K-step checkpoint and dips slightly by 750K, a reminder that PPO training isn't guaranteed to improve step over step, which is exactly why checkpointing and picking the best snapshot afterward matters.
Unity: where the training time actually went
A profile of the 1M-step run's wall-clock time. Stepping the Unity environment itself (env_step, physics + rendering four karts) ate roughly 3,000 of the run's ~3,400 seconds, versus about 411 seconds spent actually advancing the PPO trainer (the gradient updates). The bottleneck was Unity simulating the world, not the neural network learning from it.