Reinforcement Learning for Railway Scheduling Overcoming Data Sparseness through Simulations Dr. Erik Nygren Research and Innovation Lab Swiss Federal Railway
Reinforcement Learning
for Railway Scheduling
Overcoming Data Sparseness through Simulations
Dr. Erik Nygren Research and Innovation Lab Swiss Federal Railway
© SBB • Solution Center Infrastructure • Research & Innovation • October 2017 2
Swiss Railway Network. A Complex Dynamical System.
Influencing Factors Facts
1
10,000
1,210,000
Weather People
Infrastructure Events
12,997
Most dense network 33,000
210,000 t
1t
31,266
3,230 km
KM
Energy
© SBB • Solution Center Infrastructure • Research & Innovation • October 2017 3
Train Dispatching and Scheduling. Challenges in the Worlds Densest Train Network.
RCS
Train runs
Production
Timetable
Evolution of Dispatching. Towards Full Automation.
Today
Future
Past
© SBB • Solution Center Infrastructure • Research & Innovation • October 2017 4
© SBB • Solution Center Infrastructure • Research & Innovation • October 2017 5
Automated Train Dispatching. Current Challenges.
Big Data
Big Data: Not enough relevant information
Automated
Dispatching
Learning
Measure-
ments Action
© SBB • Solution Center Infrastructure • Research & Innovation • October 2017 6
Reinforcement Learning for Railway Dispatching. Overcoming Data Sparseness through Simulations.
WIP
Measure- ments Action
Validation
Data generation
Learning
Learning
Action
Artificial Data
Big Data
High Performance Simulation
© SBB • Solution Center Infrastructure • Research & Innovation • October 2017 7
High Performance Simulations. Unleashing the Power of Parallel Computing.
DGX-1 High Performance Simulations
Time speedup Scenario variations Influencing factor analysis
© SBB • Solution Center Infrastructure • Research & Innovation • October 2017 8
Preliminary Results. Visualization of Simulation Results.
2h realtime
500x
5000x Simulation speed
Visualization speed
© SBB • Solution Center Infrastructure • Research & Innovation • October 2017 9
Reinforcement Learning. Playing the Dispatcher Game.
Action
Reward
DGX-1 High Performance Simulations
Artificial Data
DGX-1 Automated Dispatcher
© SBB • Solution Center Infrastructure • Research & Innovation • October 2017 10
Machine Learning on Artificial Data. Generating, Evaluating and Optimizing Train Dispatching.
Automated Dispatcher
Reinforcement Learning
Tree Search
Evolutionary Strategies
Building Blocks Variable Topologies
1
2
3
Mixed Integer Linear Programming
Genetic Algorithm
© SBB • Solution Center Infrastructure • Research & Innovation • October 2017 11
Current State... And Future Expected Reward.
DGX-1 High Performance Simulations
DGX-1 Automated Dispatcher
Fully Automated Process
Train runs
Production
Timetable
Take Home.
Big Data Big Information
© SBB • Solution Center Infrastructure • Research & Innovation • October 2017 13
Take Home.
AI
Model
Big Data Big Information
Dr. Erik Nygren
AI Researcher
Research Team
© SBB • Solution Center Infrastructure • Research & Innovation • October 2017 14
Reward Function. How to Reward an Artificial Dispatcher.
Reward