1 Artificial Intelligence in Games: Week 9 Direct Learning Steve Rabin [email protected] TA: Chi-Hao Kuo ([email protected])

1

Artificial Intelligence in Games:Week 9

Direct Learning

Steve [email protected]

www.aiwisdom.com/digipen

TA: Chi-Hao Kuo ([email protected])

2

Questions1. Describe indirect learning.2. Describe direct learning.3. In player modeling, UsesRocketLauncher is a ______.4. UsesRocketLauncher can have a value in what range?5. In player modeling, these things we’re tracking could describe a player’s ______, ______, or _______.6. Using a Bi-Gram, find the probabilities for possible next actions in the following sequence:

kick, punch, kick, sideswipe, punch, kick, kick, kick, punch, punch, kick, kick, punch, kick, …

Impressionism

3

Impressionism

4

Impressionism

5

Post-Impressionism

6

Post-Impressionism

7

Post-Impressionism

8

Post-Impressionism

9

Meet the Roulins!

10

Augustine Roulin

11

Augustine Roulin

12

Augustine Roulin

13

Augustine Roulin & Baby

14

Baby Marcelle Roulin

15

Baby Marcelle Roulin

16

Postman Joseph Roulin

17


18


19


20

Camille Roulin

21

Camille Roulin

22

Armand Roulin

23

Armand Roulin

24

Meet the Roulins!

25

Meet the Roulins!

26

Project 1 Winners10% Extra Credit on Project 1 Tues

3D Formations - Yong Koon Wilson Phoo Flight of the Bumblebee - Hui Wen Leong Vacuum Strike - Kokleong Woon

Thurs Clicking Game - Teong Chye Chuah Snake Game - Zhi Hao Othniel Choo Space Invaders - Paulo Gutierres

Boghossian Neto

27

28

Research Project Work on this! Talk to me if you need any guidence

Demos should illustrate the research Show the result of the algorithm Show how the algorithm works

Papers Professional, yet friendly Indistinguishable from a Game Programming

Gems or AI Game Programming Wisdom article

29

30

Supervised Learning Provide the algorithm a data set

We are supervising the algorithm It learns the association between the inputs and

outputs

$

Square Footage

31

Supervised Learning Continuous output prediction = Regression

$

Square Footage

32

Supervised Learning Discrete output prediction = Classification

Malignant

Tumor Size

Benign

33

Supervised Learning 2 Input Classification Example (age and tumor size) Infinite inputs? (support vector machines)

Tumor Size

Age

Supervised Learning Algorithms Analytical learning Artificial neural network Backpropagation Boosting (meta-algorithm) Bayesian statistics Case-based reasoning Decision tree learning Inductive logic programming Gaussian process regression Gene Expression Programming Kernel estimators Learning Automata Minimum message

length (decision trees, decision graphs, etc.)

Naive bayes classifier Nearest Neighbor Algorithm

Probably approximately correct learning (PAC) learning

Ripple down rules, a knowledge acquisition methodology

Symbolic machine learning algorithms

Subsymbolic machine learning algorithms

Support vector machines Random Forests Ensembles of Classifiers Ordinal classification Data Pre-processing Handling imbalanced datasets Statistical relational learning Proaftn, a multicriteria

classification algorithm

34

Supervised Learning Algorithms Analytical learning Artificial neural network Backpropagation Boosting (meta-algorithm) Bayesian statistics Case-based reasoning Decision tree learning Inductive logic programming Gaussian process regression Gene Expression Programming Kernel estimators Learning Automata Minimum message

length (decision trees, decision graphs, etc.)

Naive bayes classifier Nearest Neighbor Algorithm

Probably approximately correct learning (PAC) learning

Ripple down rules, a knowledge acquisition methodology

Symbolic machine learning algorithms

Subsymbolic machine learning algorithms

Support vector machines Random Forests Ensembles of Classifiers Ordinal classification Data Pre-processing Handling imbalanced datasets Statistical relational learning Proaftn, a multicriteria

classification algorithm

35

36

Unsupervised Learning Find interesting things about the data

Such as clusters

http://scikit-learn.org/stable/tutorial/machine_learning_map/ 37

http://www.lauradhamilton.com/machine-learning-algorithm-cheat-sheet

38

Algorithm Pros Cons Good At

Linear regression

- Very fast (runs in constant time)- Easy to understand the model- Less prone to overfitting

- Unable to model complex relationships-Unable to capture nonlinear relationships without first transforming the inputs

- The first look at a dataset- Numerical data with lots of features

Decision trees

- Fast- Robust to noise and missing values- Accurate

- Complex trees are hard to interpret- Duplication within the same sub-tree is possible

- Star classification- Medical diagnosis- Credit risk analysis




39


Neural networks

- Extremely powerful- Can model even very complex relationships- No need to understand the underlying data– Almost works by “magic”

- Prone to overfitting- Long training time- Requires significant computing power for large datasets- Model is essentially unreadable

- Images- Video- “Human-intelligence” type tasks like driving or flying- Robotics

Support Vector Machines

- Can model complex, nonlinear relationships- Robust to noise (because they maximize margins)

- Need to select a good kernel function- Model parameters are difficult to interpret- Sometimes numerical stability problems- Requires significant memory and processing power

- Classifying proteins- Text classification- Image classification- Handwriting recognition




40


K-Nearest Neighbors

- Simple- Powerful- No training involved (“lazy”)- Naturally handles multiclass classification and regression

- Expensive and slow to predict new instances- Must define a meaningful distance function- Performs poorly on high-dimensionality datasets

- Low-dimensional datasets- Computer security: intrusion detection- Fault detection in semiconducter manufacturing- Video content retrieval- Gene expression- Protein-protein interaction



41

Quick and Dirty Intro to Neural Networks

42

Neural Networks What are they?

Based on biological neurons in the brain NNs are complex non-linear functions

that relate one or more input variables to an output variable

NNs are a series of identical non-linear processing elements (analogous to neurons) connected together in a network by weights (analogous to synapses).

43

Neural Networks

w 1

w 2

w 3

w n

i1

i2

i3

in

Inp uts W eights A d d er A c tivatio nF unc tio n

O utp ut

O

i1.w1

i2.w2

i3.w3

in .wn

C e ll B o dy

D e ndr i te s

Axo n

Synaps e

B utto n

44

Neural Networks

S pe e d H e a lthM e le e R a nge d

A tta c k F le e D e fe nd

Input L aye r

O utput L aye r

H idde n L aye r

W e ightsInput-H idde n

W e ightsH idde n-O utput

45

Neural Networks Must be trained to produce a particular

function Show it examples of inputs and outputs Weights adjusted to minimize error between

output and desired output (Backpropagation) Can require hundreds or thousands of

examples Computationally intensive For games, training must be done offline

Not guaranteed to learn the right thing Requires too many examples

Blazingly fast once it is trained and frozen

46

Neural Networks What are neural networks really good at?

Face and fingerprint recognition Handwriting recognition Gesture recognition

NNs excel at taking noisy data and processing it to produce a match

Game success stories Colin McRae Rally 2.0: Steering racing cars Forza Motorsport "Drivatar" Creatures: Control and learning

Bottom line: Statistically, game developers are not using NNs

47

Perceptrons Perceptron network is simpler version of

a NN Single-layer neural network Easier to train than an NN or decision tree Each perceptron outputs a “yes” or “no”

Either it gets stimulated enough to trigger, or it does not

Can learn Boolean decisions such as attack or don’t attack

Drawback: Can only learn simple (linearly separable) functions

48

Perceptrons:Black & White Example Three inputs used to determine if

Creature was hungry hunger, tasty food, unhappiness

If Creature ate and received positive or negative feedback, then weights were adjusted, thus facilitating learning

49

Perceptrons:Black & White Example Inputs

Hunger Tasty food Unhappiness

W e ig h t

W e ig h t

W e ig h t

I n pu t

I n pu t

I n pu t

Th re s h o ld D e s ire I n te n s ity*

*

*

50

Perceptrons:Black & White Example

Desire Sources Actual Intended(Feedback)

Hunger Tastiness Unhappiness

Value Weight Value Weight Value Weight

0.8 0.500 0.8 0.500 0 0.500 0.8 1.000

0.8 0.564 0.2 0.564 0 0.500 0.564 1.000

0.1 0.599 0.6 0.573 0 0.500 0.404 0.200

0 0.597 0 0.560 1 0.5 0.5 0.002

0.3 0.597 0.4 0.560 1 0.450 0.853 0.400

- 0.58341

- 0.542 - 0.4047 - -

51

Decision Tree Relates inputs from game world to an

output representing something you want to predict

Series of rules arranged in a tree structure Example: Will bot survive engagement

with player? Is bot’s health low?

Yes: will not survive No: check next question

Is bot’s ammunition low? Yes: will not survive No: will survive

Important for learning Algorithms exist to create decision trees in near

real-time (ID3)

Yes NoHealth

Low?

Yes NoAmmo

Low?

Die

Die Live

52

Decision Tree Learning:Black & White Example What objects will satisfy a Creature’s

desire to eat?

Scenario Creature eats something Creature gets either positive or negative

feedback Player can stroke or slap Creature Certain objects are tastier than others

Decision tree is created that reflects feedback and past experience

Decision tree influences future decisions to eat a particular object

53

Decision Tree Learning:Black & White Example

What Creature Ate Feedback (how tasty)

A big rock -1.0

A small rock -0.5

A small rock -0.4

A tree -0.2

A cow +0.6

An im ate o r I n an im ate?

+ 0 .6 - 0 .5

An im ate I n an im ate

54

Decision Tree Learning:Minimizing Entropy

E n tr o p y

R atio o f R ew ar d s to T o ta l F eed b ac k0 1

1

Pick decisions that minimize entropy Random outcomes result in entropy

of 1

55

Decision Tree Learning:Anger Decision Tree

What Creature Attacked

Feedback from Player

Friendly, weak, Celtic -1.0

Enemy, weak, Celtic +0.4

Friendly, strong, Norse -1.0

Enemy, strong, Norse -0.2

Friendly, weak, Greek -1.0

Enemy, medium, Greek +0.2

Enemy, strong, Greek -0.4

Enemy, medium, Aztec 0.0

Friendly, weak, Aztec -1.0

56


Alleg ian c e?

- 1 .0 D ef en s e?

Alleg ian c e : F r ien d ly Alleg ian c e : E n em y

+ 0 .1+ 0 .4 - 0 .3

D ef en s e : N o n e D ef en s e : M ed iu m D ef en s e : M ax im u m

57













58













Metaheuristic and Stochastic Optimization Genetic Algorithms Simulated Annealing Quantum Annealing Probability Collectives Reactive Search Optimization Cross-Entropy Method Random Search Stochastic Tunneling Parallel Tempering Stochastic Hill Climbing Swarm Algorithms Tabu Search Extreme Optimization

59

Evolutionary Algorithms Genetic Algorithms Genetic Programming Evolutionary Programming Gene Expression Programming Evolution Strategy Memetic Algorithm Differential Evolution Neuroevolution Learning classifier system

60

61

Genetic Algorithms

62

Genetic Algorithms Based on evolutionary principals

Collection of genes forms a chromosome All combinations of genes defines the search space A chromosome defines a point in the search space

Technique Create a population of chromosomes Select the best chromosomes Mate best chromosomes to create a new generation

Crossover Mutation Elitism

GAs outperform many other techniques when search space contains many optima

63

Genetic Algorithms:Crossover

64

Genetic Algorithms Extremely interesting and fascinating

technology What is it used for?

Optimization technique to find a solution Most useful when the search space is poorly

understood Unfortunately, not very useful for game

development Too slow for real-time Not guaranteed to find an optimal solution Can be used during development to discover the best

combination of settings If game changes slightly during tuning, GA result is likely

invalid

65

Genetic Algorithms Demo Genome encoding:

Steering | Throttle | Steering | Throttle | …

66

Genetic Algorithms http://boxcar2d.com/

http://boxcar2d.com/

67

Movie Time!

68

Gesture Recognition What kind of algorithm can do

this?

Generalizing from examples? How?

Comparing against examples? How?

69

Gesture Recognition:Steps Record gesture

Must know when gesture starts and stops! Holding a button or passing a threshold

Process Gesture Position independence Size independence Irregularities Redundant data Feature extraction

Apply matching algorithm

70

Gesture Recognition:Processing

processing

71

Gesture Recognition:Processing Normalize size Target length Target segments

72

Gesture Recognition:Nearest Neighbor

Example 1 Example 2 Example 3

Player Move

Which example best matches the player's???

73

Gesture Recognition:Nearest Neighborerror = 0° error = 90° error = 90°


74



75



76


Player Move


0+45+45=90 90+45+0=135 90+0+0=90

77


Player Move


0+452+452=4050 902+452+0=10125 902+0+0=8100

Square the individual error,

then sum!!!

78

Gesture Recognition Compare the vectors and perform

nearest neighbor

79

Gesture Recognition

Demo!

Microsoft’sOriginal Kinect $149.99 at retail

Sold 10M+ so far $56 worth of parts

$17 for PrimeSense motion detector

$500M ad campaign during Christmas Ad spending was $100 per Kinect sold

80

Xbox One Kinect 3x fidelity IR sensor (no visible light) 60% wider field of vision

Closest 3ft vs 6ft Detect heart rate, facial expression Position and orientation of 25

joints Weight put on each limb

81

Xbox One Kinect

82

83

“Microsoft exec caught in privacy snafu, says Kinect might tailor ads to you”

From Engadget.com:Microsoft's Dennis Durkin voiced an interesting idea at an investment summit last week – the idea that the company's Kinect camera might pass data to advertisers about the way you look, play and speak. "We can cater what content gets presented to you based on who you are," he told investors, suggesting that the Kinect offered business opportunities that weren't possible "in a controller-based world."

84


“And over time that will help us be more targeted about what content choices we present, what advertising we present, how we get better feedback. And data about how many people are in a room when an advertisement is shown, how many people are in a room when a game is being played, how are those people engaged with the game? How are they engaged with a sporting event? Are they standing up? Are they excited? Are they wearing Seahawks jerseys?”

– Microsoft's Dennis Durkin

85


From Engadget.com:…but moreover it's explicitly against the privacy policy Microsoft presents Kinect users. "Third party partners use aggregated data to deliver Kinect experiences (games or applications), to understand how customers use their Kinect experiences, and to improve performance or even to help plan new experiences," the Kinect Privacy and Online Safety FAQ reads, but also "They are not permitted to use the information for marketing purposes such as selling you games or services, or for personalizing advertising."

86

Kinect:Depth Camera

87

Kinect:Depth Camera

http://www.youtube.com/watch?v=nvvQJxgykcU&feature=player_embedded

88

Kinect:Depth Camera

http://www.youtube.com/watch?v=7QrnwoO1-8A

89

Kinect:Your Bones

90

Kinect:Skeleton Patent“To determine whether a target or object in the scene corresponds to a human target, each of the targets may be flood filled and compared to a pattern of a human body model. Each target or object that matches the human body model may then be scanned to generate a skeletal model associated therewith. The skeletal model may then be provided to the computing environment such that the computing environment may track the skeletal model, render an avatar associated with the skeletal model, and may determine which controls to perform in an application executing on the computing environment based on, for example, gestures of the user that have been recognized from the skeletal model. A gesture recognizer engine, the architecture of which is described more fully below, is used to determine when a particular gesture has been made by the user.”

91

Kinect:Skeleton from a Point Cloud Recorded hundreds of people

moving in front of depth camera Manually hand labeled a skeleton on

top of every frame Recorded similar mocap data to

movements Machine learning to match noisy

data to a skeleton (with probabilities)

92

Kinect: Skeleton

93

Kinect Step 1:(from Popular Mechanics, Jan 2010)

As you stand in front of the camera, it judges the distance to different points on your body. In the image on the far left, the dots show what it sees, a so-called "point cloud" representing a 3-D surface; a skeleton drawn there is simply a rudimentary guess. (The image on the top shows the image perceived by the color camera.)

94


Then the brain guesses which parts of your body are which. It does this based on all of its experience with body poses. Depending on how similar your pose is to things it's seen before, Natal can be more or less confident of its guesses. In the color-coded person (bottom center), the darkness, lightness, and size of different squares represent how certain Natal is that it knows what body-part that area belongs to.

95

Kinect Step 2 (continued):(from Popular Mechanics, Jan 2010)

(For example, the three large red squares indicate that it’s highly probable that those parts are “left shoulder,” “left elbow” and “left knee"; as the pixels become smaller and muddier in color, such as the grayish pixels around the hands, that’s an indication that Natal is hedging its bets and isn’t very sure of its identity.)

96


Then, based on the probabilities assigned to different areas, Natal comes up with all possible skeletons that could fit with those body parts. (This step isn't shown in the image, but it looks similar to the stick-figure drawn on the left, except there are dozens of possible skeletons overlaid on each other.) It ultimately settles on the most probable one. Its reasoning here is partly based on its experience, and partly on more formal kinematics models that programmers added in.

97


Once Natal has determined it has enough certainty about enough body parts to pick the most probable skeletal structure, it outputs that shape to a simplified 3D avatar [image at right]. That’s the final skeleton that will be skinned with clothes, hair, and other features and shown in the game.

98


Once Natal has determined it has enough certainty about enough body parts to pick the most probable skeletal structure, it outputs that shape to a simplified 3D avatar [image at right]. That’s the final skeleton that will be skinned with clothes, hair, and other features and shown in the game.

99

Kinect Gesture Recognition? Microsoft doesn’t distribute any code

to recognize poses or gestures Maybe because it’s a patent minefield…

The Kinect SDK just outputs raw skeleton poses Up to individual developers to figure out

how to use raw skeleton data

100

1 Artificial Intelligence in Games: Week 9 Direct Learning Steve Rabin [email protected] TA: Chi-Hao Kuo ([email protected])

Documents

square footage slide

indirect learning

camille roulin

armand roulin

postman joseph roulin

augustine roulin baby

baby marcelle roulin

direct learning steve