3D Vision Guided Robotic Charging Station for Electric and Plug-In Hybrid Vehicles

3D Vision Guided Robotic Charging Station

for Electric and Plug-in Hybrid Vehicles

J. Mišeikis, M. Rüther, B. Walzel, M. Hirz, H. Brunner

Electric Vehicles are becoming great!

… but EV owners still have plenty to

complain about!

Electric Vehicles are becoming great!

… but EV owners still have plenty to

complain about!

Electric Vehicles are becoming great!

Charging

Image source: http://www.electriccarpledge.com/electric-vehicle-resources/electric-car-plug-types/

Image source: http://www.quick220.com/page/cp-product-solutions-2.html

Image source: https://twitter.com/dunneautomotive/status/780996480417034241

Image source: https://twitter.com/dunneautomotive/status/780996480417034241

Image source: https://twitter.com/dunneautomotive/status/780996480417034241

But the charging cable becomes very thick and heavy!

Let’s allow the robot to

do the dirty work!

Use Case

The car is parked in the designated parking spot

and the robot handles the whole charging

process

Related Work

Our Concept: Robotic EV / PHEV Charging Station

Our System Setup

Algorithm

Detection of the

charging port

3D Pose Estimation

Robot Motion

Planning

3-Step Plug-In

Procedure

Charging!

Algorithm

Detection of the

charging port

3D Pose

Estimation

Robot Motion

Planning

3-Step Plug-In

ProcedureCharging!

Step-by-step

Shape-Based Template Matching: Templates

Detection of the

charging port

3D Pose

Estimation

Robot Motion

Planning

3-Step Plug-In

ProcedureCharging!

Shape-Based Template Matching: Evaluation

Detection of the

charging port

3D Pose

Estimation

Robot Motion

Planning

3-Step Plug-In

ProcedureCharging!

Shape-Based Template Matching: Process

Detection of the

charging port

3D Pose

Estimation

Robot Motion

Planning

3-Step Plug-In

ProcedureCharging!

Left Cam Right Cam

Shape-Based Template Matching: Process

Detection of the

charging port

3D Pose

Estimation

Robot Motion

Planning

3-Step Plug-In

ProcedureCharging!

Left Cam Right Cam

Shape-Based Template Matching: Process

Detection of the

charging port

3D Pose

Estimation

Robot Motion

Planning

3-Step Plug-In

ProcedureCharging!

Left Cam Right Cam

Left Cam Right Cam

3D Pose Estimation: Stereo Triangulation

Detection of the

charging port

3D Pose

Estimation

Robot Motion

Planning

3-Step Plug-In

ProcedureCharging!

3D Pose Estimation: Stereo Triangulation

Detection of the

charging port

3D Pose

Estimation

Robot Motion

Planning

3-Step Plug-In

ProcedureCharging!

Point 1 (X, Y)

Point 2 (X, Y)

Point 3 (X, Y)

Point 1 (X, Y)

Point 2 (X, Y)

Point 3 (X, Y)

Left Cam Right Cam

3D Pose Estimation: Stereo Triangulation

Detection of the

charging port

3D Pose

Estimation

Robot Motion

Planning

3-Step Plug-In

ProcedureCharging!

We use stereo triangulation to calculate depth

Left Cam Right Cam

3D Pose Estimation: Stereo Triangulation

Detection of the

charging port

3D Pose

Estimation

Robot Motion

Planning

3-Step Plug-In

ProcedureCharging!

Real-World Coordinates

Point 1 (X, Y, Z)

Point 2 (X, Y, Z)

Point 3 (X, Y, Z)

Left Cam Right Cam

3D Pose Estimation: Find the Relative Orientation

Detection of the

charging port

3D Pose

Estimation

Robot Motion

Planning

3-Step Plug-In

ProcedureCharging!

Roll

Pitch

Yaw

3D Pose Estimation: Find the Relative Orientation

Detection of the

charging port

3D Pose

Estimation

Robot Motion

Planning

3-Step Plug-In

ProcedureCharging!

Roll

Pitch

Yaw

We can think about this as 3D plane fitting on selected points

3D Pose Estimation: Find the Relative Orientation

Detection of the

charging port

3D Pose

Estimation

Robot Motion

Planning

3-Step Plug-In

ProcedureCharging!

Roll

Pitch

Yaw

We can think about this as 3D plane fitting on selected points

3D Pose Estimation

Detection of the

charging port

3D Pose

Estimation

Robot Motion

Planning

3-Step Plug-In

ProcedureCharging!

Charging Port Position in

Real-World Coordinates

(X, Y, Z, roll, pitch, yaw)

Left Cam Right Cam

Marker-less Eye-to-Hand Calibration

Cam to Plug

Transform

End-Effector to

Plug

Transform

Marker-less Eye-to-Hand Calibration

3D Pose Estimation

Detection of the

charging port

3D Pose

Estimation

Robot Motion

Planning

3-Step Plug-In

ProcedureCharging!

Now we have the following information:

● Distance to the port from the vision sensor in X, Y and Z

● Orientation of the port in relation to the vision sensor

● Location of the vision sensor in relation to the robot

● Location of the charging plug in relation to the robot

3D Pose Estimation

Detection of the

charging port

3D Pose

Estimation

Robot Motion

Planning

3-Step Plug-In

ProcedureCharging!

Let’s move the robot!

Now we have the following information:

● Distance to the port from the vision sensor in X, Y and Z

● Orientation of the port in relation to the vision sensor

● Location of the vision sensor in relation to the robot

● Location of the charging plug in relation to the robot

Robot Motion Planning: RRT-Connect

Detection of the

charging port

3D Pose

Estimation

Robot Motion

Planning

3-Step Plug-In

ProcedureCharging!

Plug-In Procedure: 3 Steps

Detection of the

charging port

3D Pose

Estimation

Robot Motion

Planning

3-Step Plug-In

ProcedureCharging!

Charging! And Unplugging Afterwards

Detection of the

charging port

3D Pose

Estimation

Robot Motion

Planning

3-Step Plug-In

ProcedureCharging!

Simply track-back the same trajectory!

Results

Exp Charging Port at 10° angle Charging Port at 30° angle

1 Success Success: Misalignment

2 Success: Misalignment Failed: Missed rotation

3 Success Success

4 Failed: Missed rotation Success: Misalignment

5 Success: Misalignment Success: Misalignment

Attempts: 10

Successful Charging: 8

Small alignment issues: 5

Failed connection: 2

Due to initial calibration issues

Future Work

● Test on a real vehicle

● Automatically open charging port lid

● Add more safety features

● Automatic communication between

the charger and the vehicle

● Incorporate automatic parking

Thank You!

Any Questions?