Traffic Light Control Using Reinforcement Learning Daniel Goldberg Andrew Elstein Final Presentation
Feb 23, 2016
Traffic Light Control Using Reinforcement Learning
Daniel GoldbergAndrew Elstein
Final Presentation
The Problem• Traffic congestion is estimated to cost Americans
$121 billion in lost productivity fuel, and other costs.• Traffic Lights are imperfect and contribute to this• Usually statically controlled
• A better method of controlling them can reduce waiting times significantly
Approach
• Implement a “Reinforcement Learning” (RL) algorithm to control traffic lights
• Create a simulation of traffic to tweak and test traffic light optimizations
Implementation• If minor adjustments
were made to the algorithm, it could operate within existing infrastructure • Optimally, a camera
system and would be added
Simulation
Insert picture of visualization
Simulation Structure• To build the simulation we created the follow Data
Structures:
• CarsPosition, Destination, Velocity, Map, Color
• Roads• Lanes
• Individual Cells• Intersection location matrix
• Intersections• Position, Traffic Lights
• In total, the simulation is coded in MATLAB with 3100 lines of code
Cars Struct
Simulation Dynamics• Cars are spawned randomly
• They follow an randomly generated path to destination
• Cars follow normal traffic rules
• Road Cells are discretized to easily simulate traffic, only one car can exist in each road cell. Cars move ahead one or two cells in each time-step, depending on the car's max velocity and whether there is an open spot.
Demo
Reinforcement Learning• Weiring - Multi-Agent Reinforcement Learning for
Traffic Light Control
• It introduced an objective function to minimize or maximize a goal value
tl = traffic lightp = current position d = destinationL = light decision = discounting constant ‘ = next
Reinforcement Learning Theory
• Coordinating a system of lights to respond to current conditions can reap exceptional benefit
• The theory cleverly merges probability, game theory and machine learning to efficiently control traffic
• In our case, the expected value of each of a light’s possible states are calculated
• With this value function a game is played to maximize it, in turn minimizing waiting time
Results
Wrote a script to compare the smart algorithm to static On-Off-On-Off lights.
Our algorithm reduced average waiting time—and thus traveling time—for a system with any number of cars
Travelling time for our implementation was reduced by an average of 10%. There was a 15% reduction for sparse traffic systems from a static control, but only a 3% decrease for heavy congestion.
Results cont.
Extensions• Fairness-weighted objective:
•
• ω = weighting constant• t = current time
• ti = time of arrival for car i
• • If F(t) > 1, cars on road 1 get to go• If F(t) < 1, cars on road 2 get to go
Further Extensions• Car Path optimization and rerouting
• Model expansion to traverse an entire city
• Inter-traffic-light communication
• Retesting with increased computational resources for modeling accuracy and robustness
RL In the News• Samah El-Tantawy, 2012 PhD recipient from the
University of Toronto, won the 2013 IEEE best dissertation award for her research in RL.
• Her RL traffic model showed reduced rush-hour travel times by as much as 26 percent and is working on monetizing her research with small MARLIN-ATSC (Multi-agent Reinforcement Learning for Integrated Network of Adaptive Traffic Signal Controllers) computers.
Challenges• Difficult to understand data structures and
how they would interact• Object Oriented Approach vs. MATLAB’s index-based
structures• Understand how cars would interact with each
other• Understanding RL algorithm• Adapting our model to use RL algorithm• Limited computational resources