Developing aDeveloping aSelf-Driving CarSelf-Driving Car
for the 2007 DARPAfor the 2007 DARPAUrban ChallengeUrban Challenge
Seth TellerCS and AI Laboratory
EECS DepartmentMIT
Joint work with: Matt Antone, David Barrett,Mitch Berger, Bryt Bradley, Ryan Buckley, StefanCampbell, Alexander Epstein, Gaston Fiore, LukeFletcher, Emilio Frazzoli, Jonathan How, AlbertHuang, Troy Jones, Sertac Karaman, Olivier Koch,Yoshi Kuwata, Victoria Landgraf, John Leonard,Keoni Maheloni, David Moore, Katy Moyer, EdwinOlson, Andrew Patrikalakis, Steve Peters, StephenProulx, Nicholas Roy, Chris Sanders, Justin Teo,Robert Truax, Matthew Walter, Jonathan Williams
12/13/2007
Grand ChallengeGrand Challenge
• Military interest in autonomous land vehicles– Congressional mandate (H.R. 4205/P.L. 106-398):
“one third of operational ground combatvehicles to be unmanned by 2015”
• DGC 1: March 2004– 142 miles in 10 hours, $1M prize– 106 entering teams; no finishers– Dense GPS corridor– One vehicle at a time!– No moving obstacles (static world)
• Whenever two robots came close,one was manually paused
• DGC 2: October 2005– 132 miles in 10 hours, $2M prize– One vehicle at a time– 195 teams, 5 finishers
12/13/2007
Urban Challenge (2007)Urban Challenge (2007)
• Novel elements:– Urban road network– Moving traffic– No course inspection– 60 miles in 6 hours– $3.5M prize pool– 89 entering teams
• Program goals– Safe (no collisions)– Capable (turns, stops, intersection,
passing, merging, parking, following)– Robust (blocked roads, erratic drivers,
sparse waypoints, GPS degradation and outages)
Source: DARPA Urban ChallengeParticipants Briefing, May 2006
12/13/2007
Program ScopeProgram Scope
• In scope:– Following -- Emergency stops– Intersections -- Timely left turns across traffic– Passing, Merging -- Potholes, construction sites– Parking, U-turns -- Blockages, replanning
• Out of scope:– Pedestrians– High speed (> 30 mph)– Traffic signals, signage– Difficult off-road terrain– Highly inclement weather
12/13/2007
Source: DARPA Participants Briefing, May 2006
RNDF Versus Actual ConditionsRNDF Versus Actual Conditions
12/13/2007
Fundamental QuestionsFundamental Questions
• Autonomous driving includes four key problems:– Where is the road?
• Identify drivable road surface at fine grain
– Where are the static obstacles?• Hard: curbs, potholes, signposts, buildings
• Soft: lane markings, shoulders, vegetation
– Where are the other vehicles?• Where might they move in the near future?
– How should the vehicle behave?• Codify (non-algorithmic) rules of safe, legal, “human-like” driving
• Solve all of the above, with available (uncertain)sensor data, in real time (without killing anyone).
12/13/2007
Related WorkRelated Work
• Fully Autonomous Driving Systems– Limited domain (highways, traffic-free roads)
– Require human to stage control handoff, monitoroperation, and take over in emergency situations
– Munich’s VaMoRs (1985-2004), VAMP (1993-2004);CMU’s NAVLAB (1985); Penn (Southall & Taylor 2001)
• Assistive Driving Technologies– Limited duty cycle (cruising, emergencies, staged
parking) and actuation (e.g. none, or brakes only)
– Require human handoff and resumption of control
– Automakers’ ABS, cruise control, self-parking systems
– Lane departure warnings (Mobileye, Iteris, ANU)
12/13/2007
Assessment and StrategyAssessment and Strategy
• Human-level urban driving not achievablewith existing technology and methods in 2006– Key issues: uncertainty; computational resources; safety
– Example: if vehicle is unsure of where the road is,identifying the appropriate traffic behavior is very difficult
• Strategy– Technical footprint for success covers many disciplines interdisciplinary approach integrating EECS/AA/ME
– Spiral design approach figure out how to solve theproblem while designing the system at the same time
12/13/2007
Team Members & RolesTeam Members & Roles
• MIT faculty, postdocs, students, staff– Operating software, sensor & computer
selection and configuration (~8 full timegraduate student programmers)
– Project Management• Draper Laboratory IR&D
– System Engineering, Vehicle Integration& Test Support, Logistics Support
• Olin College of Engineering– Vehicle Engineering (Mech. & Elec.)– System Testing support
• Other Team Sponsors– Quanta Computer, BAE Systems, Ford,
Land Rover, MIT SOE, CSAIL, EECS,A/A, MIT IS&T, MIT Lincoln Laboratory,…
10
12/13/2007
Planned TimelinePlanned Timeline
• Bring up rapid prototype vehicle(Ford Escape) summer/fall ‘06– Gain experience with sensors,
dynamics, coding, configuration
• Bring up competition vehicle(LandRover LR3) spring ‘07– Develop mature algorithms,
tune for qualifiers and final race
Site Visit(6/20/07)
Semi-final(10/26-31)
Final Event(11/3)
ParticipantsConference (5/20)
ProgramAnnounced (5/1)
Site Visit(10/27)
Track A Announced
2006 2007
(10/2)
Ford Escape LR3
12/13/2007
Design StrategyDesign Strategy
• Sensor-rich, CPU-intensive architecture– Intensive use of live and logged visualization
• Redundancies:– Sensor type and spatial coverage
– Closed-loop planning
– Computation failover at process level
– Firmware-mediated vehicle control
• Failsafe behaviors– If no progress, relax perceived constraints
12/13/2007
Our ApproachOur Approach
Velodyne HDL
PushbroomSick LIDAR (5)
ACC RADAR (15)
SkirtSick LIDAR (7)
Cameras (6)
13
12/13/2007
VelodyneVelodyne Lidar Lidar
• 64 lasers, 360° HFOV• Spins at 15 Hz• Vertical FOV
-24° +2°• Redundant (albeit
relatively noisy) lidar
12/13/2007
Automotive RadarsAutomotive Radars
• 15 Delphi automotive radars• Doppler range, bearing, closing
speed of 20 objects @ 10Hz• Narrow beam width• Good far-field car detectors
12/13/2007
Sample Radar DataSample Radar DataRaw range, bearing, range rate data, false-colored by radar ID
12/13/2007
Video CamerasVideo Cameras
• 5 Firewire Cameras– Point Grey Firefly MV
• 720x480 8bpp Bayerpattern @ 22.8 fps
• ~40 MB/s (2.5 GB/min)Lots of data!
• Purpose: Detection ofpainted lane markings
Rear view Narrow forward view
Forward left Forward center Forward right
12/13/2007
Lane EstimationLane Estimation
Road paintdetectors
CurbDetectors
Lane centerline estimator
Lane tracker
RNDF
12/13/2007
System ArchitectureSystem Architecture
• Perception– Vehicle surroundings
– Vehicle location w.r.t.surroundings and RNDF
• Planning & Control– Codified driving rules
– How to reach the goal
• AEVIT Vehicle Conversion (EMC) control unit– Continuous signal (steering, gas/brake)– Discrete signal (turn signals, gear shift)
Perception
NavigatorMDF
Goal
Trajectory
Steer, gas/brake
Vehicle states
Drivable surface, lanemarkings, Obstacles;
Traffic vehicle
Local mapDrivableSurface,Hazards
SituationalPlanner
VehicleController
Vehicle
Vehicle StateEstimator
SensorsSensorsSensorsSensorsSensorsSensorsSensorsRNDF
12/13/2007
NavigatorNavigator
• Mission planner
• Sets high level goals– Carrot for the motion planner
• Replan around blocked roads
• Knob on constraints in drivability map– Perception algorithms are not perfect
– If car stuck and isn’t making progress, startignoring perception (invoke failsafe levels)
Navigator
DrivabilityMap
MotionPlanner
Controller
12/13/2007
RRT-based PlanningRRT-based Planning
• Sample the input to the controller
– Dynamically feasible path– Closed-loop stability
• Every trajectory ends with a stop– Continuously replan, extend trajectories
so that terminal segments are not used.
• Various types of hard constraints– Obstacle, lane markings, stop lines,
etc.– Navigator dynamically revises
constraints
ControllerVehicleModel
Obstacleinfeasible
Road infeasible
Car
Goal
Divider infeasible
12/13/2007
Land Rover LR3Land Rover LR3
• Linux blade cluster with two fast interconnection networks– 10 blades each with 2.33GHz quad-core processor 40 cores– Approximately 80 driving-related processes steady-state
• Many sensors– Applanix IMU/GPS– Hi-res odometry– 12 SICK Lidars– Velodyne (~64 Lidars)– 15 automotive radars– 5 video cameras
• Roof-mounted AC• Power consumption
~5500W total• Internal gas generator
12/13/2007
Autonomous Driving Test SiteAutonomous Driving Test Site
• South Weymouth Naval Air Station– About 40 min. from MIT off 3S
– Usually $10K/day; free to uswhen no paying customer
• Large tarmac area– Can create arbitrary (flat) road networks
– Environmentally sensitive:• Obstacles: traffic cones
• Lane markings: flour
• Traffic: team members’ cars
12/13/2007
DARPA Site Visit (June 2006)DARPA Site Visit (June 2006)
• Weymouth RNDF– Lanes known very accurately
• Demonstrate “basic” naviga-tion and traffic capabilities– Stay in lane
– Pass a stopped car
– Intersection precedence
– Make 3pt turn at stub road
– Speed limit: 15 mph
• Rain: phantom obstacles
• Results– 3 missions completed
– 1 mission success on retry
12/13/2007
Closed-Loop TrackingClosed-Loop Tracking
Playback speed: 2x
One-way two-lane roadRNDF-interpolated estimate goes through trees and bushes!
12/13/2007
NQE Area BNQE Area B
Playback speed: 3.3x
One of only 2 teams to complete Area B on the first attempt.
12/13/2007
Area AArea A
• Advanced traffic capabilities– Merging into traffic
– Left turn across oncoming traffic
– Excessive delay (> 10 sec.) prohibited
• ~10 traffic vehicles moving at 10mph.
• 1st trial – 7 laps in 24 min• 2nd trial – 10 laps in 12 min
12/13/2007
Area CArea C
• Objectives– Intersection precedence (turn-taking)
– Blocked check points (replanning, 3-point turns)
Route blockage
12/13/2007
Urban Challenge Event Urban Challenge Event (11/03/07)(11/03/07)
• Only 11 teams selected,due to safety concerns
• 50 human-driven traffic vehicles
• 3 MDFs – total of ~60 miles
• Focus very different from spec, NQE– Allowed pre-inspection
– Much simpler setup• No passing
• No road blockage
• No sparse points
• Empty parking lot
– No GPS degradation• Most RF turned off
– Other robot cars Highly stochastic 1 mile
12/13/2007
Final CompetitionFinal Competition
• Media Coverage– Discovery, German TV,
Local News– Live WebCast from DARPA– Discovery special will
feature team (in production)• Early Attrition
– 5 Teams eliminated fromrace within first hour
– Oshkosh drove toward building,Carolo drove into MIT …
• Summary– Drove 57 miles in 5hrs 35min
run time (10.2 mph average)– 2 Collisions (Carolo, Cornell)
39
12/13/2007
Degraded PerformanceDegraded Performance
• Phantom curbs on the dirt segment– Had never tested on steeply-sloped dirt roads
– Failsafe timer kicked in disregard curbs
12/13/2007
Accident with Accident with CarOLOCarOLO
• Accident had several contributory causes:– CarOLO drove into us damaged, removed from competition
– Hard to detect slowly moving objects, without false positives
– Could have embedded a better evasive maneuver
The first bot-on-bot caraccident in
history!
12/13/2007
Accident with CornellAccident with Cornell
• Cornell– Stopped, then reversed
in the intersection
– Started moving as we passed
DARPA: “no fault” incident;both teams continued
The second bot-on-bot car accident in history!
• MIT– Tried passing near intersection
– Returned to lane too quickly
12/13/2007
Traffic JamTraffic Jam
• Each car waiting for another car to move
• Excess delay = 10sec traffic jam
12/13/2007
High-Speed SectionHigh-Speed Section
• MDF speed limit: 30mph– Braking distance = 36m (with 2.5m/s2 deceleration)
– Standoff distance = 10m
– Requires reliable detection range: 50m
Capped @25mphby our software
12/13/2007
ResultsResults
• 6 teams finished; 5 othersremoved from competitionby DARPA officials– Excess delay– Collisions
• Many race-day firsts for us:– More than 20miles in single day– Steep dirt (unpaved) segment– Mile-long, wide lanes @25mph– Interaction with other robots
• We drove safely– No processes died– Our chase driver: “your vehicle
was always safe, in my opinion”
CMU Stanford Virginia Tech
12/13/2007
Failure ModesFailure Modes
• Perception limitations– Hallucinated curbs (at detection size threshold)– Vulnerability to shadows, sun blinding– Sensitivity to vehicle pitch– Inability to track slow-moving vehicles (< 3mph)
• Control/planning limitations– Occasionally failed to achieve target orientation– Caused over-correction, unsafe maneuvers
• Failsafe strategy– Unclear when to relax or observe constraints– Example: U-turn at roadblock, or drive around?
12/13/2007
AchievementsAchievements
• Respectable rookieshowing– First time in DGC for
everyone on the team
• Fourth place overall– One of only 6 teams (of
89 initially entering) tocomplete UCE course
• Completed NQEwithout manualannotation of RNDF
12/13/2007
Lessons LearnedLessons Learned
• About competing effectively
• About DARPA’s expectations
• About the autonomous driving task
12/13/2007
AcknowledgmentsAcknowledgments
• THANK YOU to MIT IS&T:– Jerry Grochow
– Theresa Regan
– Mitch Berger