ALFRED THOMPSON MICROSOFT ACADEMIC TEAM HTTP://BIT.LY/ALFREDTH Kinect for FRC 2012.

ALFRED THOMPSONMICROSOFT ACADEMIC TEAM

HTTP://BIT.LY/ALFREDTH

Kinect for FRC 2012

Outline

2

IntroductionKinect HardwareKinect for WindowsSkeleton, depth, image and audio dataKinect for FRC 2012Default FRC gesturesProgramming your own gesturesAdditional Resources

Kinect Hardware

3

Multi-Array Mic Motorized Tilt

IR Projector

RGB Camera

IR Camera

Kinect and FRC 2012

The Kinect has multiple powerful sensors for depth, skeleton, video and audio input

The Kinect is likely intended for robot control via the drivers station for the 2012 FRC game

4

Kinect for Windows Architecture5

Skeleton Data

Kinect Audio

Four microphone array with hardware-based audio processing– Multichannel echo cancellation (MEC)– Sound position tracking– Other digital signal processing (noise

suppression and reduction)

7

Kinect Sensor Data8

FIRST Kinect Architecture

Driver Station Robot

FRC Driver Station

Kinect for Windows SDKKinect for Windows SDK

WPI Kinect Server

cRIO

WPI Lib (Kinect

Enabled)

KinectKinect

Joystick Position and Skeleton points

9

Kinect for FRC 2012

10

Driver Station PCKinect SDK

FRC Kinect Server (C#)

Driver Station (Kinect Diags)

Dashboard(Skeletal Display)

Robot Gesture Code (LabVIEW, C/C++ or Java)

Scenario 1 Scenario 2

Raw Skeletal Data, Diagnostic Data,

Default Gesture Data (Kinect Sticks),Optional Data

Default Gesture Data (Kinect Sticks)

Raw Skeletal Data, Diagnostic Data,

Optional Data

Programming

Teams using C# can change/modify the WPI Kinect Server on the Driver Station

Teams using LabView, Java or C++ can process joystick AND skeleton point data on the cRIO robot controller

11

Default FRC Gestures

12

90

-90

0

RIGHT

Right KinectStick2 Y valuesMap to Wrist/Shoulder Angle

Left KinectStick1 Y valuesMap to Wrist/Shoulder Angle

Y= 1.00

Y= -1.00

Y= 0.00

Y= 0.50

Y=- 0.50

Fig 1 Human Player facing Kinect Y joystick value / angle interpretation

90

-90

0

Y= 1.00

Y= -1.00

Y= 0.00

Y= 0.50

Y=- 0.50

LEFT

70

Y= -1.00

70

Y= -1.00

70

Y= 1.00

35

-35

35

35

-70

Y= 1.0090

-90

0

-45

45NAN / Zero

90

-90

0

-45

45

RIGHT LEFT

NAN / Zero

Facing Kinect

Fig 2 Wrist/hand to Elbow angle definition of Zero

Buttons:1) Head to the Left2) Head to the Right3) Right Leg Out

4) Left Leg Out5) Right Leg Forward6) Right Leg Back

7) Left Leg Forward8) Left Leg Back9) Kinect Control “Enabled”

Control Considerations

Robot control is achieved via skeletal position

Gestures/poses relate to skeletal position and inform which control is desired

Consider your gestures carefully!

Response times will be different with Kinect compared to traditional joysticks

13

Using the FRC Gestures

14

Labview

KinectStick leftArm(1);KinectStick rightArm(2);

leftArm.getY()leftArm.getRawButton(#)

C++

KinectStick leftArm;KinectStick rightArm;…leftArm = new KinectStick(1);rightArm = new KinectStick(2);…leftArm.getY()leftArm.getRawButton(#)

Java

Gesture Programming

15

Teams may modify the Kinect Server code (C#) using Visual C# Express. Allows access to depth/RGB data Easier to display feedback on dashboard Keeps cRIO processing burden low Gesture examples available from general Kinect

development community Teams may use the raw skeleton data sent to the robot to

process their own gestures in their chosen language (LabVIEW, C/C++ or Java)

Language familiarity

Getting Help

1. FRC Forums

2. Public Kinect for Windows Forums

3. Escalate to Microsoft via official FIRST contacts

16

Demo and Questions

Go Teams, go!

17

Additional Resources

18

Official Kinect for Windows Website FRC Beta Kinect Forum

Kinect SDK Quickstart Video Tutorials Coding4Fun Kinect Toolkit Coding4Fun Kinect Projects Visual C# Express Kinect for Windows SDK Beta Forums

Getting Started with Microsoft Kinect for FRC document – released at Kickoff

Kinect Server Code Walkthough – released at Kickoff