Developing a Self-Driving Car for the 2007 DARPA …ist.mit.edu/sites/default/files/migration/about/reports/darpa.pdfDeveloping a Self-Driving Car for the 2007 DARPA Urban Challenge

Developing aDeveloping aSelf-Driving CarSelf-Driving Car

for the 2007 DARPAfor the 2007 DARPAUrban ChallengeUrban Challenge

Seth TellerCS and AI Laboratory

EECS DepartmentMIT

Joint work with: Matt Antone, David Barrett,Mitch Berger, Bryt Bradley, Ryan Buckley, StefanCampbell, Alexander Epstein, Gaston Fiore, LukeFletcher, Emilio Frazzoli, Jonathan How, AlbertHuang, Troy Jones, Sertac Karaman, Olivier Koch,Yoshi Kuwata, Victoria Landgraf, John Leonard,Keoni Maheloni, David Moore, Katy Moyer, EdwinOlson, Andrew Patrikalakis, Steve Peters, StephenProulx, Nicholas Roy, Chris Sanders, Justin Teo,Robert Truax, Matthew Walter, Jonathan Williams

12/13/2007

Grand ChallengeGrand Challenge

• Military interest in autonomous land vehicles– Congressional mandate (H.R. 4205/P.L. 106-398):

“one third of operational ground combatvehicles to be unmanned by 2015”

• DGC 1: March 2004– 142 miles in 10 hours, $1M prize– 106 entering teams; no finishers– Dense GPS corridor– One vehicle at a time!– No moving obstacles (static world)

• Whenever two robots came close,one was manually paused

• DGC 2: October 2005– 132 miles in 10 hours, $2M prize– One vehicle at a time– 195 teams, 5 finishers

12/13/2007

Urban Challenge (2007)Urban Challenge (2007)

• Novel elements:– Urban road network– Moving traffic– No course inspection– 60 miles in 6 hours– $3.5M prize pool– 89 entering teams

• Program goals– Safe (no collisions)– Capable (turns, stops, intersection,

passing, merging, parking, following)– Robust (blocked roads, erratic drivers,

sparse waypoints, GPS degradation and outages)

Source: DARPA Urban ChallengeParticipants Briefing, May 2006

12/13/2007

Program ScopeProgram Scope

• In scope:– Following -- Emergency stops– Intersections -- Timely left turns across traffic– Passing, Merging -- Potholes, construction sites– Parking, U-turns -- Blockages, replanning

• Out of scope:– Pedestrians– High speed (> 30 mph)– Traffic signals, signage– Difficult off-road terrain– Highly inclement weather

12/13/2007

Source: DARPA Participants Briefing, May 2006

RNDF Versus Actual ConditionsRNDF Versus Actual Conditions

12/13/2007

Fundamental QuestionsFundamental Questions

• Autonomous driving includes four key problems:– Where is the road?

• Identify drivable road surface at fine grain

– Where are the static obstacles?• Hard: curbs, potholes, signposts, buildings

• Soft: lane markings, shoulders, vegetation

– Where are the other vehicles?• Where might they move in the near future?

– How should the vehicle behave?• Codify (non-algorithmic) rules of safe, legal, “human-like” driving

• Solve all of the above, with available (uncertain)sensor data, in real time (without killing anyone).

12/13/2007

Related WorkRelated Work

• Fully Autonomous Driving Systems– Limited domain (highways, traffic-free roads)

– Require human to stage control handoff, monitoroperation, and take over in emergency situations

– Munich’s VaMoRs (1985-2004), VAMP (1993-2004);CMU’s NAVLAB (1985); Penn (Southall & Taylor 2001)

• Assistive Driving Technologies– Limited duty cycle (cruising, emergencies, staged

parking) and actuation (e.g. none, or brakes only)

– Require human handoff and resumption of control

– Automakers’ ABS, cruise control, self-parking systems

– Lane departure warnings (Mobileye, Iteris, ANU)

12/13/2007

Assessment and StrategyAssessment and Strategy

• Human-level urban driving not achievablewith existing technology and methods in 2006– Key issues: uncertainty; computational resources; safety

– Example: if vehicle is unsure of where the road is,identifying the appropriate traffic behavior is very difficult

• Strategy– Technical footprint for success covers many disciplines interdisciplinary approach integrating EECS/AA/ME

– Spiral design approach figure out how to solve theproblem while designing the system at the same time

12/13/2007

Team FormationTeam Formation

9

12/13/2007

Team Members & RolesTeam Members & Roles

• MIT faculty, postdocs, students, staff– Operating software, sensor & computer

selection and configuration (~8 full timegraduate student programmers)

– Project Management• Draper Laboratory IR&D

– System Engineering, Vehicle Integration& Test Support, Logistics Support

• Olin College of Engineering– Vehicle Engineering (Mech. & Elec.)– System Testing support

• Other Team Sponsors– Quanta Computer, BAE Systems, Ford,

Land Rover, MIT SOE, CSAIL, EECS,A/A, MIT IS&T, MIT Lincoln Laboratory,…

10

12/13/2007

Planned TimelinePlanned Timeline

• Bring up rapid prototype vehicle(Ford Escape) summer/fall ‘06– Gain experience with sensors,

dynamics, coding, configuration

• Bring up competition vehicle(LandRover LR3) spring ‘07– Develop mature algorithms,

tune for qualifiers and final race

Site Visit(6/20/07)

Semi-final(10/26-31)

Final Event(11/3)

ParticipantsConference (5/20)

ProgramAnnounced (5/1)

Site Visit(10/27)

Track A Announced

2006 2007

(10/2)

Ford Escape LR3

12/13/2007

Design StrategyDesign Strategy

• Sensor-rich, CPU-intensive architecture– Intensive use of live and logged visualization

• Redundancies:– Sensor type and spatial coverage

– Closed-loop planning

– Computation failover at process level

– Firmware-mediated vehicle control

• Failsafe behaviors– If no progress, relax perceived constraints

12/13/2007

Our ApproachOur Approach

Velodyne HDL

PushbroomSick LIDAR (5)

ACC RADAR (15)

SkirtSick LIDAR (7)

Cameras (6)

13

12/13/2007

SICK SICK LidarsLidarsFalse colored by height

12/13/2007

VelodyneVelodyne Lidar Lidar

• 64 lasers, 360° HFOV• Spins at 15 Hz• Vertical FOV

-24° +2°• Redundant (albeit

relatively noisy) lidar

12/13/2007

Sample Sample VelodyneVelodyne Data DataFalse colored by height

12/13/2007

Detection of Static ObstaclesDetection of Static Obstacles

12/13/2007

Automotive RadarsAutomotive Radars

• 15 Delphi automotive radars• Doppler range, bearing, closing

speed of 20 objects @ 10Hz• Narrow beam width• Good far-field car detectors

12/13/2007

Sample Radar DataSample Radar DataRaw range, bearing, range rate data, false-colored by radar ID

12/13/2007

Vehicle Detection and TrackingVehicle Detection and Tracking

12/13/2007

Tracking in Real SituationsTracking in Real Situations

12/13/2007

Video CamerasVideo Cameras

• 5 Firewire Cameras– Point Grey Firefly MV

• 720x480 8bpp Bayerpattern @ 22.8 fps

• ~40 MB/s (2.5 GB/min)Lots of data!

• Purpose: Detection ofpainted lane markings

Rear view Narrow forward view

Forward left Forward center Forward right

12/13/2007

Lane EstimationLane Estimation

Road paintdetectors

CurbDetectors

Lane centerline estimator

Lane tracker

RNDF

12/13/2007

System ArchitectureSystem Architecture

• Perception– Vehicle surroundings

– Vehicle location w.r.t.surroundings and RNDF

• Planning & Control– Codified driving rules

– How to reach the goal

• AEVIT Vehicle Conversion (EMC) control unit– Continuous signal (steering, gas/brake)– Discrete signal (turn signals, gear shift)

Perception

NavigatorMDF

Goal

Trajectory

Steer, gas/brake

Vehicle states

Drivable surface, lanemarkings, Obstacles;

Traffic vehicle

Local mapDrivableSurface,Hazards

SituationalPlanner

VehicleController

Vehicle

Vehicle StateEstimator

SensorsSensorsSensorsSensorsSensorsSensorsSensorsRNDF

12/13/2007

NavigatorNavigator

• Mission planner

• Sets high level goals– Carrot for the motion planner

• Replan around blocked roads

• Knob on constraints in drivability map– Perception algorithms are not perfect

– If car stuck and isn’t making progress, startignoring perception (invoke failsafe levels)

Navigator

DrivabilityMap

MotionPlanner

Controller

12/13/2007

RRT-based PlanningRRT-based Planning

• Sample the input to the controller

– Dynamically feasible path– Closed-loop stability

• Every trajectory ends with a stop– Continuously replan, extend trajectories

so that terminal segments are not used.

• Various types of hard constraints– Obstacle, lane markings, stop lines,

etc.– Navigator dynamically revises

constraints

ControllerVehicleModel

Obstacleinfeasible

Road infeasible

Car

Goal

Divider infeasible

12/13/2007

Land Rover LR3Land Rover LR3

• Linux blade cluster with two fast interconnection networks– 10 blades each with 2.33GHz quad-core processor 40 cores– Approximately 80 driving-related processes steady-state

• Many sensors– Applanix IMU/GPS– Hi-res odometry– 12 SICK Lidars– Velodyne (~64 Lidars)– 15 automotive radars– 5 video cameras

• Roof-mounted AC• Power consumption

~5500W total• Internal gas generator

12/13/2007

LR3 Cargo CompartmentLR3 Cargo Compartment

12/13/2007

Autonomous Driving Test SiteAutonomous Driving Test Site

• South Weymouth Naval Air Station– About 40 min. from MIT off 3S

– Usually $10K/day; free to uswhen no paying customer

• Large tarmac area– Can create arbitrary (flat) road networks

– Environmentally sensitive:• Obstacles: traffic cones

• Lane markings: flour

• Traffic: team members’ cars

12/13/2007

DARPA Site Visit (June 2006)DARPA Site Visit (June 2006)

• Weymouth RNDF– Lanes known very accurately

• Demonstrate “basic” naviga-tion and traffic capabilities– Stay in lane

– Pass a stopped car

– Intersection precedence

– Make 3pt turn at stub road

– Speed limit: 15 mph

• Rain: phantom obstacles

• Results– 3 missions completed

– 1 mission success on retry

12/13/2007

Closed-Loop TrackingClosed-Loop Tracking

Playback speed: 2x

One-way two-lane roadRNDF-interpolated estimate goes through trees and bushes!

12/13/2007

NQE and CompetitorsNQE and Competitors

• photos

12/13/2007

Area BArea B

• Our very first NQE run.Sparse Waypoints

Parking

Gate

Gauntlet

Start chute

12/13/2007

NQE Area BNQE Area B

Playback speed: 3.3x

One of only 2 teams to complete Area B on the first attempt.

12/13/2007

Area B Parking TestArea B Parking Test

• Parking: RNDF target position was blocked

12/13/2007

Area AArea A

• Advanced traffic capabilities– Merging into traffic

– Left turn across oncoming traffic

– Excessive delay (> 10 sec.) prohibited

• ~10 traffic vehicles moving at 10mph.

• 1st trial – 7 laps in 24 min• 2nd trial – 10 laps in 12 min

12/13/2007

Area CArea C

• Objectives– Intersection precedence (turn-taking)

– Blocked check points (replanning, 3-point turns)

Route blockage

12/13/2007

Urban Challenge Event Urban Challenge Event (11/03/07)(11/03/07)

• Only 11 teams selected,due to safety concerns

• 50 human-driven traffic vehicles

• 3 MDFs – total of ~60 miles

• Focus very different from spec, NQE– Allowed pre-inspection

– Much simpler setup• No passing

• No road blockage

• No sparse points

• Empty parking lot

– No GPS degradation• Most RF turned off

– Other robot cars Highly stochastic 1 mile

12/13/2007

Final CompetitionFinal Competition

• Media Coverage– Discovery, German TV,

Local News– Live WebCast from DARPA– Discovery special will

feature team (in production)• Early Attrition

– 5 Teams eliminated fromrace within first hour

– Oshkosh drove toward building,Carolo drove into MIT …

• Summary– Drove 57 miles in 5hrs 35min

run time (10.2 mph average)– 2 Collisions (Carolo, Cornell)

39

12/13/2007

12/13/2007

CruisingCruising

• MIT starting order: 6th

12/13/2007

Degraded PerformanceDegraded Performance

• Phantom curbs on the dirt segment– Had never tested on steeply-sloped dirt roads

– Failsafe timer kicked in disregard curbs

12/13/2007

Accident with Accident with CarOLOCarOLO

• Accident had several contributory causes:– CarOLO drove into us damaged, removed from competition

– Hard to detect slowly moving objects, without false positives

– Could have embedded a better evasive maneuver

The first bot-on-bot caraccident in

history!

12/13/2007

Accident with CornellAccident with Cornell

• Cornell– Stopped, then reversed

in the intersection

– Started moving as we passed

DARPA: “no fault” incident;both teams continued

The second bot-on-bot car accident in history!

• MIT– Tried passing near intersection

– Returned to lane too quickly

12/13/2007

Traffic JamTraffic Jam

• Each car waiting for another car to move

• Excess delay = 10sec traffic jam

12/13/2007

High-Speed SectionHigh-Speed Section

• MDF speed limit: 30mph– Braking distance = 36m (with 2.5m/s2 deceleration)

– Standoff distance = 10m

– Requires reliable detection range: 50m

Capped @25mphby our software

12/13/2007

ResultsResults

• 6 teams finished; 5 othersremoved from competitionby DARPA officials– Excess delay– Collisions

• Many race-day firsts for us:– More than 20miles in single day– Steep dirt (unpaved) segment– Mile-long, wide lanes @25mph– Interaction with other robots

• We drove safely– No processes died– Our chase driver: “your vehicle

was always safe, in my opinion”

CMU Stanford Virginia Tech

12/13/2007

Failure ModesFailure Modes

• Perception limitations– Hallucinated curbs (at detection size threshold)– Vulnerability to shadows, sun blinding– Sensitivity to vehicle pitch– Inability to track slow-moving vehicles (< 3mph)

• Control/planning limitations– Occasionally failed to achieve target orientation– Caused over-correction, unsafe maneuvers

• Failsafe strategy– Unclear when to relax or observe constraints– Example: U-turn at roadblock, or drive around?

12/13/2007

Teamwork!Teamwork!

12/13/2007

AchievementsAchievements

• Respectable rookieshowing– First time in DGC for

everyone on the team

• Fourth place overall– One of only 6 teams (of

89 initially entering) tocomplete UCE course

• Completed NQEwithout manualannotation of RNDF

12/13/2007

Lessons LearnedLessons Learned

• About competing effectively

• About DARPA’s expectations

• About the autonomous driving task

12/13/2007

AcknowledgmentsAcknowledgments

• THANK YOU to MIT IS&T:– Jerry Grochow

– Theresa Regan

– Mitch Berger

12/13/2007

Questions?Questions?