Classification and Regression I Krzysztof Dembczy´ nski Intelligent Decision Support Systems Laboratory (IDSS) Pozna´ n University of Technology, Poland Software Development Technologies Master studies, second semester Academic year 2018/19 (winter course) 1 / 43
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Classification and Regression I
Krzysztof Dembczynski
Intelligent Decision Support Systems Laboratory (IDSS)Poznan University of Technology, Poland
Software Development TechnologiesMaster studies, second semester
Academic year 2018/19 (winter course)
1 / 43
Review of the previous lectures
• Mining of massive datasets.
2 / 43
Outline
1 Motivation
2 Statistical Learning Theory
3 Learning Paradigms and Principles
4 Summary
3 / 43
Outline
1 Motivation
2 Statistical Learning Theory
3 Learning Paradigms and Principles
4 Summary
4 / 43
We live in the era of Big Data and Machine Learning.
5 / 43
Search engines: website ranking and personalization
6 / 43
Recommender systems: movie, book, product recommendations
7 / 43
Online shopping/auctions
8 / 43
Autonomous vehicles
9 / 43
Spam filtering
10 / 43
A plenty of machine learning competitions
11 / 43
Machine learning is everywhere. . .
search engines recommender systems online advertising translation
autonomous cars face recognition image recognition voice recognition
fraud detection healthcare medical research bioinformatics
neuroscience climate science astronomy physics12 / 43
Machine learning
• What is machine learning?
I Machine learning is the science of getting computers to act withoutbeing explicitly programmed. (Andrew Ng)
• Supervised learningI Learn a computer to predict an unknown response/value of a decision
attribute for an object described by several features.
• Two main problems:I Classification: Prediction of categorical response,I Regression: Prediction of continuous response.
• Examples:I Spam filtering,I Handwriting recognition,I Text classification,I Stock prices,I etc.
13 / 43
Machine learning
• What is machine learning?I Machine learning is the science of getting computers to act without
being explicitly programmed. (Andrew Ng)
• Supervised learningI Learn a computer to predict an unknown response/value of a decision
attribute for an object described by several features.
• Two main problems:I Classification: Prediction of categorical response,I Regression: Prediction of continuous response.
• Examples:I Spam filtering,I Handwriting recognition,I Text classification,I Stock prices,I etc.
13 / 43
Machine learning
• What is machine learning?I Machine learning is the science of getting computers to act without
being explicitly programmed. (Andrew Ng)
• Supervised learningI Learn a computer to predict an unknown response/value of a decision
attribute for an object described by several features.
• Two main problems:I Classification: Prediction of categorical response,I Regression: Prediction of continuous response.
• Examples:I Spam filtering,I Handwriting recognition,I Text classification,I Stock prices,I etc.
13 / 43
Machine learning
• What is machine learning?I Machine learning is the science of getting computers to act without
being explicitly programmed. (Andrew Ng)
• Supervised learningI Learn a computer to predict an unknown response/value of a decision
attribute for an object described by several features.
• Two main problems:I Classification: Prediction of categorical response,I Regression: Prediction of continuous response.
• Examples:I Spam filtering,I Handwriting recognition,I Text classification,I Stock prices,I etc.
13 / 43
Machine learning
• What is machine learning?I Machine learning is the science of getting computers to act without
being explicitly programmed. (Andrew Ng)
• Supervised learningI Learn a computer to predict an unknown response/value of a decision
attribute for an object described by several features.
• Two main problems:I Classification: Prediction of categorical response,I Regression: Prediction of continuous response.
• Examples:I Spam filtering,I Handwriting recognition,I Text classification,I Stock prices,I etc.
13 / 43
Machine learning
• We know relatively much about solving simple learning problems suchas binary classification:
I Advanced theory,I Implemented fast algorithms,I Almost a mature technology.
• The main challenges are:
I Feature engineering,I Supervision of examples,I New applications,I Complex problems,I Large-scale problems.
14 / 43
Machine learning
• We know relatively much about solving simple learning problems suchas binary classification:
I Advanced theory,
I Implemented fast algorithms,I Almost a mature technology.
• The main challenges are:
I Feature engineering,I Supervision of examples,I New applications,I Complex problems,I Large-scale problems.
14 / 43
Machine learning
• We know relatively much about solving simple learning problems suchas binary classification:
I Advanced theory,I Implemented fast algorithms,
I Almost a mature technology.
• The main challenges are:
I Feature engineering,I Supervision of examples,I New applications,I Complex problems,I Large-scale problems.
14 / 43
Machine learning
• We know relatively much about solving simple learning problems suchas binary classification:
I Advanced theory,I Implemented fast algorithms,I Almost a mature technology.
• The main challenges are:
I Feature engineering,I Supervision of examples,I New applications,I Complex problems,I Large-scale problems.
14 / 43
Machine learning
• We know relatively much about solving simple learning problems suchas binary classification:
I Advanced theory,I Implemented fast algorithms,I Almost a mature technology.
• The main challenges are:
I Feature engineering,I Supervision of examples,I New applications,I Complex problems,I Large-scale problems.
14 / 43
Machine learning
• We know relatively much about solving simple learning problems suchas binary classification:
I Advanced theory,I Implemented fast algorithms,I Almost a mature technology.
• The main challenges are:I Feature engineering,
I Supervision of examples,I New applications,I Complex problems,I Large-scale problems.
14 / 43
Machine learning
• We know relatively much about solving simple learning problems suchas binary classification:
I Advanced theory,I Implemented fast algorithms,I Almost a mature technology.
• The main challenges are:I Feature engineering,I Supervision of examples,
I New applications,I Complex problems,I Large-scale problems.
14 / 43
Machine learning
• We know relatively much about solving simple learning problems suchas binary classification:
I Advanced theory,I Implemented fast algorithms,I Almost a mature technology.
• The main challenges are:I Feature engineering,I Supervision of examples,I New applications,
I Complex problems,I Large-scale problems.
14 / 43
Machine learning
• We know relatively much about solving simple learning problems suchas binary classification:
I Advanced theory,I Implemented fast algorithms,I Almost a mature technology.
• The main challenges are:I Feature engineering,I Supervision of examples,I New applications,I Complex problems,
I Large-scale problems.
14 / 43
Machine learning
• We know relatively much about solving simple learning problems suchas binary classification:
I Advanced theory,I Implemented fast algorithms,I Almost a mature technology.
• The main challenges are:I Feature engineering,I Supervision of examples,I New applications,I Complex problems,I Large-scale problems.
14 / 43
Software
• Weka (http://www.cs.waikato.ac.nz/ml/weka/)
• R-project (http://www.r-project.org/),
• Octave (https://www.gnu.org/software/octave/),
• Julia (http://julialang.org/),
• Scikit-learn (http://scikit-learn.org/stable/)
• Matlab (http://www.mathworks.com/products/matlab/)
• H20 (http://0xdata.com/)
• GraphLab (http://dato.com/)
• MLLib (https://spark.apache.org/mllib/)
• Mahout (http://mahout.apache.org/)
• . . .
15 / 43
Outline
1 Motivation
2 Statistical Learning Theory
3 Learning Paradigms and Principles
4 Summary
16 / 43
Supervised learning
Training data{xi, yi}n1
Learning algorithm
Model f(x)
Predictionby using h(x)
Test example xPredicted outcome
y = h(x)
True outcome y
Evaluation
Risk
Evaluation Estimated risk
≈
Loss `(y, y)
17 / 43
Supervised learning
Training data{xi, yi}n1
Learning algorithm
Model f(x)
Predictionby using h(x)
Test example xPredicted outcome
y = h(x)
True outcome y
Evaluation
Risk
Evaluation Estimated risk
≈
Loss `(y, y)
17 / 43
Supervised learning
Training data{xi, yi}n1
Learning algorithm
Model f(x)
Predictionby using h(x)
Test example xPredicted outcome
y = h(x)
True outcome y
Evaluation
Risk
Evaluation Estimated risk
≈
Loss `(y, y)
17 / 43
Supervised learning
Training data{xi, yi}n1
Learning algorithm
Model f(x)
Predictionby using h(x)
Test example xPredicted outcome
y = h(x)
True outcome y
Evaluation
Risk
Evaluation Estimated risk
≈
Loss `(y, y)
17 / 43
Supervised learning
Training data{xi, yi}n1
Learning algorithm
Model f(x)
Predictionby using h(x)
Test example xPredicted outcome
y = h(x)
True outcome y
Evaluation
Risk
Evaluation Estimated risk
≈
Loss `(y, y)
17 / 43
Supervised learning
Training data{xi, yi}n1
Learning algorithm
Model f(x)
Predictionby using h(x)
Test example xPredicted outcome
y = h(x)
True outcome y
Evaluation
Risk
Evaluation Estimated risk
≈
Loss `(y, y)
17 / 43
Supervised learning
Training data{xi, yi}n1
Learning algorithm
Model f(x)
Predictionby using h(x)
Test example xPredicted outcome
y = h(x)
True outcome y
Evaluation
Risk
Evaluation Estimated risk
≈
Loss `(y, y)
17 / 43
Supervised learning
Training data{xi, yi}n1
Learning algorithm
Model f(x)
Predictionby using h(x)
Test example xPredicted outcome
y = h(x)
True outcome y
Evaluation
Risk
Evaluation Estimated risk
≈
Loss `(y, y)
17 / 43
Supervised learning
Training data{xi, yi}n1
Learning algorithm
Model f(x)
Predictionby using h(x)
Test example xPredicted outcome
y = h(x)
True outcome y
Evaluation
Risk
Evaluation Estimated risk
≈
Loss `(y, y)
17 / 43
Supervised learning
Training data{xi, yi}n1
Learning algorithm
Model f(x)
Predictionby using h(x)
Test example xPredicted outcome
y = h(x)
True outcome y
Evaluation
Risk
Evaluation Estimated risk
≈
Loss `(y, y)
17 / 43
Supervised learning
Training data{xi, yi}n1
Learning algorithm
Model f(x)
Predictionby using h(x)
Test example xPredicted outcome
y = h(x)
True outcome y
Evaluation
Risk
Evaluation Estimated risk
≈
Loss `(y, y)
17 / 43
Supervised learning
Training data{xi, yi}n1
Learning algorithm
Model f(x)
Predictionby using h(x)
Test example xPredicted outcome
y = h(x)
True outcome y
Evaluation
Risk
Evaluation Estimated risk
≈
Loss `(y, y)
17 / 43
Supervised learning
Training data{xi, yi}n1
Learning algorithm
Model f(x)
Predictionby using h(x)
Test example xPredicted outcome
y = h(x)
True outcome y
Evaluation
Risk
Evaluation Estimated risk
≈
Loss `(y, y)
17 / 43
Supervised learning
Training data{xi, yi}n1
Learning algorithm
Model f(x)
Predictionby using h(x)
Test example xPredicted outcome
y = h(x)
True outcome y
Evaluation
Risk
Evaluation Estimated risk
≈
Loss `(y, y)
17 / 43
Statistical learning framework
• Input x ∈ X drawn from a distribution P (x).
I usually a feature vector, X ⊆ Rd.
• Outcome y ∈ Y drawn from a distribution P (y |x).I target of our prediction: class label, real value, label vector, etc.,I alternative view: example (x, y) drawn from P (x, y).
• Prediction y = h(x) by means of prediction function h : X → Y.
I h returns prediction y = h(x) for every input x.
• Loss of our prediction: `(y, y).
I ` : Y × Y → R+ is a problem-specific loss function.
• Goal: find a prediction function with small loss.
18 / 43
Statistical learning framework
• Input x ∈ X drawn from a distribution P (x).
I usually a feature vector, X ⊆ Rd.
• Outcome y ∈ Y drawn from a distribution P (y |x).I target of our prediction: class label, real value, label vector, etc.,I alternative view: example (x, y) drawn from P (x, y).
• Prediction y = h(x) by means of prediction function h : X → Y.
I h returns prediction y = h(x) for every input x.
• Loss of our prediction: `(y, y).
I ` : Y × Y → R+ is a problem-specific loss function.
• Goal: find a prediction function with small loss.
18 / 43
Statistical learning framework
• Input x ∈ X drawn from a distribution P (x).
I usually a feature vector, X ⊆ Rd.
• Outcome y ∈ Y drawn from a distribution P (y |x).I target of our prediction: class label, real value, label vector, etc.,I alternative view: example (x, y) drawn from P (x, y).
• Prediction y = h(x) by means of prediction function h : X → Y.
I h returns prediction y = h(x) for every input x.
• Loss of our prediction: `(y, y).
I ` : Y × Y → R+ is a problem-specific loss function.
• Goal: find a prediction function with small loss.
18 / 43
Statistical learning framework
• Input x ∈ X drawn from a distribution P (x).
I usually a feature vector, X ⊆ Rd.
• Outcome y ∈ Y drawn from a distribution P (y |x).I target of our prediction: class label, real value, label vector, etc.,I alternative view: example (x, y) drawn from P (x, y).
• Prediction y = h(x) by means of prediction function h : X → Y.
I h returns prediction y = h(x) for every input x.
• Loss of our prediction: `(y, y).
I ` : Y × Y → R+ is a problem-specific loss function.
• Goal: find a prediction function with small loss.
18 / 43
Statistical learning framework
• Input x ∈ X drawn from a distribution P (x).
I usually a feature vector, X ⊆ Rd.
• Outcome y ∈ Y drawn from a distribution P (y |x).I target of our prediction: class label, real value, label vector, etc.,I alternative view: example (x, y) drawn from P (x, y).
• Prediction y = h(x) by means of prediction function h : X → Y.
I h returns prediction y = h(x) for every input x.
• Loss of our prediction: `(y, y).
I ` : Y × Y → R+ is a problem-specific loss function.
• Goal: find a prediction function with small loss.
18 / 43
Statistical learning framework
• Input x ∈ X drawn from a distribution P (x).
I usually a feature vector, X ⊆ Rd.
• Outcome y ∈ Y drawn from a distribution P (y |x).I target of our prediction: class label, real value, label vector, etc.,I alternative view: example (x, y) drawn from P (x, y).
• Prediction y = h(x) by means of prediction function h : X → Y.
I h returns prediction y = h(x) for every input x.
• Loss of our prediction: `(y, y).
I ` : Y × Y → R+ is a problem-specific loss function.
• Goal: find a prediction function with small loss.
18 / 43
Risk
• Goal: minimize the expected loss over all examples (risk):
L`(h) = E(x,y)∼P [`(y, h(x))] .
• The optimal prediction function over all possible functions:
h∗ = argminh
L(h),
(so called Bayes prediction function).
• The smallest achievable risk (Bayes risk):
L∗` = L`(h∗).
19 / 43
Risk
• Goal: minimize the expected loss over all examples (risk):
L`(h) = E(x,y)∼P [`(y, h(x))] .
• The optimal prediction function over all possible functions:
h∗ = argminh
L(h),
(so called Bayes prediction function).
• The smallest achievable risk (Bayes risk):
L∗` = L`(h∗).
19 / 43
Risk
• Goal: minimize the expected loss over all examples (risk):
L`(h) = E(x,y)∼P [`(y, h(x))] .
• The optimal prediction function over all possible functions:
h∗ = argminh
L(h),
(so called Bayes prediction function).
• The smallest achievable risk (Bayes risk):
L∗` = L`(h∗).
19 / 43
Decomposition of risk
L`(h) = E(x,y) [`(y, h(x))]
=
∫X×Y
`(y, h(x))P (x, y)dxdy
=
∫X
(∫Y`(y, h(x))P (y |x)dy
)P (x)dx
= E x [L`(h |x)] .
• L`(h |x) is the conditional risk of y = h(x) at x.
• Bayes prediction minimizes the conditional risk for every x:
h∗(x) = argminh
L`(h |x).
20 / 43
Decomposition of risk
L`(h) = E(x,y) [`(y, h(x))]
=
∫X×Y
`(y, h(x))P (x, y)dxdy
=
∫X
(∫Y`(y, h(x))P (y |x)dy
)P (x)dx
= E x [L`(h |x)] .
• L`(h |x) is the conditional risk of y = h(x) at x.
• Bayes prediction minimizes the conditional risk for every x:
h∗(x) = argminh
L`(h |x).
20 / 43
Decomposition of risk
L`(h) = E(x,y) [`(y, h(x))]
=
∫X×Y
`(y, h(x))P (x, y)dxdy
=
∫X
(∫Y`(y, h(x))P (y |x)dy
)P (x)dx
= E x [L`(h |x)] .
• L`(h |x) is the conditional risk of y = h(x) at x.
• Bayes prediction minimizes the conditional risk for every x:
h∗(x) = argminh
L`(h |x).
20 / 43
Decomposition of risk
L`(h) = E(x,y) [`(y, h(x))]
=
∫X×Y
`(y, h(x))P (x, y)dxdy
=
∫X
(∫Y`(y, h(x))P (y |x)dy
)P (x)dx
= E x [L`(h |x)] .
• L`(h |x) is the conditional risk of y = h(x) at x.
• Bayes prediction minimizes the conditional risk for every x:
h∗(x) = argminh
L`(h |x).
20 / 43
Decomposition of risk
L`(h) = E(x,y) [`(y, h(x))]
=
∫X×Y
`(y, h(x))P (x, y)dxdy
=
∫X
(∫Y`(y, h(x))P (y |x)dy
)P (x)dx
= E x [L`(h |x)] .
• L`(h |x) is the conditional risk of y = h(x) at x.
• Bayes prediction minimizes the conditional risk for every x:
h∗(x) = argminh
L`(h |x).
20 / 43
Decomposition of risk
L`(h) = E(x,y) [`(y, h(x))]
=
∫X×Y
`(y, h(x))P (x, y)dxdy
=
∫X
(∫Y`(y, h(x))P (y |x)dy
)P (x)dx
= E x [L`(h |x)] .
• L`(h |x) is the conditional risk of y = h(x) at x.
• Bayes prediction minimizes the conditional risk for every x:
h∗(x) = argminh
L`(h |x).
20 / 43
Making optimal decisions
Example
• Pack of cards: 7 diamonds (red), 5 hearts (red), 5 spades (black), 2clubs (black).
• Decision = bet (four choices).
• If you win you get 100$, if you loose you must give 50$.
• What is the loss and optimal decision?
• Suppose we know the card is black. What is the optimal decisionnow?
• What are the input variables?
21 / 43
Making optimal decisions
Example
• Pack of cards: 7 diamonds (red), 5 hearts (red), 5 spades (black), 2clubs (black).
• Decision = bet (four choices).
• If you win you get 100$, if you loose you must give 50$.
• What is the loss and optimal decision?
• Suppose we know the card is black. What is the optimal decisionnow?
• What are the input variables?
21 / 43
Making optimal decisions
Example
• Pack of cards: 7 diamonds (red), 5 hearts (red), 5 spades (black), 2clubs (black).
• Decision = bet (four choices).
• If you win you get 100$, if you loose you must give 50$.
• What is the loss and optimal decision?
• Suppose we know the card is black. What is the optimal decisionnow?
• What are the input variables?
21 / 43
Making optimal decisions
Example
• Pack of cards: 7 diamonds (red), 5 hearts (red), 5 spades (black), 2clubs (black).
• Decision = bet (four choices).
• If you win you get 100$, if you loose you must give 50$.
• What is the loss and optimal decision?
• Suppose we know the card is black. What is the optimal decisionnow?
• What are the input variables?
21 / 43
Making optimal decisions
Example
• Pack of cards: 7 diamonds (red), 5 hearts (red), 5 spades (black), 2clubs (black).
• Decision = bet (four choices).
• If you win you get 100$, if you loose you must give 50$.
• What is the loss and optimal decision?
• Suppose we know the card is black. What is the optimal decisionnow?
• What are the input variables?
21 / 43
Making optimal decisions
Example
• Pack of cards: 7 diamonds (red), 5 hearts (red), 5 spades (black), 2clubs (black).
• Decision = bet (four choices).
• If you win you get 100$, if you loose you must give 50$.
• What is the loss and optimal decision?
• Suppose we know the card is black. What is the optimal decisionnow?
• What are the input variables?
21 / 43
Making optimal decisions
Example
• Pack of cards: 7 diamonds (red), 5 hearts (red), 5 spades (black), 2clubs (black).
• Bet the color:
I if the true color is red and you are correct you win 50, otherwise youloose 100,
I if the true color is black and you are correct you win 200, otherwise youloose 100.
• What is the loss and optimal decision now?
22 / 43
Making optimal decisions
Example
• Pack of cards: 7 diamonds (red), 5 hearts (red), 5 spades (black), 2clubs (black).
• Bet the color:
I if the true color is red and you are correct you win 50, otherwise youloose 100,
I if the true color is black and you are correct you win 200, otherwise youloose 100.
• What is the loss and optimal decision now?
22 / 43
Making optimal decisions
Example
• Pack of cards: 7 diamonds (red), 5 hearts (red), 5 spades (black), 2clubs (black).
• Bet the color:I if the true color is red and you are correct you win 50, otherwise you
loose 100,
I if the true color is black and you are correct you win 200, otherwise youloose 100.
• What is the loss and optimal decision now?
22 / 43
Making optimal decisions
Example
• Pack of cards: 7 diamonds (red), 5 hearts (red), 5 spades (black), 2clubs (black).
• Bet the color:I if the true color is red and you are correct you win 50, otherwise you
loose 100,I if the true color is black and you are correct you win 200, otherwise you
loose 100.
• What is the loss and optimal decision now?
22 / 43
Making optimal decisions
Example
• Pack of cards: 7 diamonds (red), 5 hearts (red), 5 spades (black), 2clubs (black).
• Bet the color:I if the true color is red and you are correct you win 50, otherwise you
loose 100,I if the true color is black and you are correct you win 200, otherwise you
loose 100.
• What is the loss and optimal decision now?
22 / 43
Regression
• Prediction of a real-valued outcome y ∈ R.
• Find a prediction function h(x) that accurately predicts value of y.
• The most common loss function used is squared error loss:
`se(y, y) = (y − y)2,
where y = h(x).
23 / 43
Regression
• The conditional risk for squared error loss is :
Lse(h |x) = Ey|x[(y − y)2
]= Ey|x
[(y − µ(x) + µ(x)− y)2
]= Ey|x
[(y − µ(x))2 + 2 (y − µ(x))︸ ︷︷ ︸
=0 under expectation
(µ(x)− y) + (µ(x)− y)2]
= Ey|x[(y − µ(x))2
]︸ ︷︷ ︸independent of y
+(µ(x)− y)2.
µ(x) = Ey |x[y]
• Hence, h∗(x) = µ(x), the conditional expectation of y at x, and:
Lse(h∗ |x) = Ey|x
[(y − µ(x))2
]= Var(y|x).
24 / 43
Regression
• The conditional risk for squared error loss is :
Lse(h |x) = Ey|x[(y − y)2
]
= Ey|x[(y − µ(x) + µ(x)− y)2
]= Ey|x
[(y − µ(x))2 + 2 (y − µ(x))︸ ︷︷ ︸
=0 under expectation
(µ(x)− y) + (µ(x)− y)2]
= Ey|x[(y − µ(x))2
]︸ ︷︷ ︸independent of y
+(µ(x)− y)2.
µ(x) = Ey |x[y]
• Hence, h∗(x) = µ(x), the conditional expectation of y at x, and:
Lse(h∗ |x) = Ey|x
[(y − µ(x))2
]= Var(y|x).
24 / 43
Regression
• The conditional risk for squared error loss is :
Lse(h |x) = Ey|x[(y − y)2
]= Ey|x
[(y − µ(x) + µ(x)− y)2
]
= Ey|x[(y − µ(x))2 + 2 (y − µ(x))︸ ︷︷ ︸
=0 under expectation
(µ(x)− y) + (µ(x)− y)2]
= Ey|x[(y − µ(x))2
]︸ ︷︷ ︸independent of y
+(µ(x)− y)2.
µ(x) = Ey |x[y]
• Hence, h∗(x) = µ(x), the conditional expectation of y at x, and:
Lse(h∗ |x) = Ey|x
[(y − µ(x))2
]= Var(y|x).
24 / 43
Regression
• The conditional risk for squared error loss is :
Lse(h |x) = Ey|x[(y − y)2
]= Ey|x
[(y − µ(x) + µ(x)− y)2
]= Ey|x
[(y − µ(x))2 + 2 (y − µ(x))︸ ︷︷ ︸
=0 under expectation
(µ(x)− y) + (µ(x)− y)2]
= Ey|x[(y − µ(x))2
]︸ ︷︷ ︸independent of y
+(µ(x)− y)2.
µ(x) = Ey |x[y]
• Hence, h∗(x) = µ(x), the conditional expectation of y at x, and:
Lse(h∗ |x) = Ey|x
[(y − µ(x))2
]= Var(y|x).
24 / 43
Regression
• The conditional risk for squared error loss is :
Lse(h |x) = Ey|x[(y − y)2
]= Ey|x
[(y − µ(x) + µ(x)− y)2
]= Ey|x
[(y − µ(x))2 + 2 (y − µ(x))︸ ︷︷ ︸
=0 under expectation
(µ(x)− y) + (µ(x)− y)2]
= Ey|x[(y − µ(x))2
]︸ ︷︷ ︸independent of y
+(µ(x)− y)2.
µ(x) = Ey |x[y]
• Hence, h∗(x) = µ(x), the conditional expectation of y at x, and:
Lse(h∗ |x) = Ey|x
[(y − µ(x))2
]= Var(y|x).
24 / 43
Regression
• The conditional risk for squared error loss is :
Lse(h |x) = Ey|x[(y − y)2
]= Ey|x
[(y − µ(x) + µ(x)− y)2
]= Ey|x
[(y − µ(x))2 + 2 (y − µ(x))︸ ︷︷ ︸
=0 under expectation
(µ(x)− y) + (µ(x)− y)2]
= Ey|x[(y − µ(x))2
]︸ ︷︷ ︸independent of y
+(µ(x)− y)2.
µ(x) = Ey |x[y]
• Hence, h∗(x) = µ(x), the conditional expectation of y at x, and:
Lse(h∗ |x) = Ey|x
[(y − µ(x))2
]= Var(y|x).
24 / 43
Regression
• Another loss commonly used in regression is the absolute error:
`ae(y, y) = |y − y|.
• The Bayes classifier for the absolute-error loss is:
h∗(x) = argminh
Lae(h |x) =
median(y|x) ,
i.e., median of the conditional distribution of y given x.
25 / 43
Regression
• Another loss commonly used in regression is the absolute error:
`ae(y, y) = |y − y|.
• The Bayes classifier for the absolute-error loss is:
h∗(x) = argminh
Lae(h |x) = median(y|x) ,
i.e., median of the conditional distribution of y given x.
25 / 43
Regression
−3 −2 −1 0 1 2 3
01
23
4
y −− f((x))
L((y
,, f((
x))))
Squared−error lossAbsolute−error loss
Figure: Loss functions for regression task26 / 43
Binary Classification
• Prediction of a binary outcome y ∈ {−1, 1} (alternatively y ∈ {0, 1}).
• Find a prediction function h(x) that accurately predicts value of y.
• The most common loss function used is 0/1 loss:
`0/1(y, y) = Jy 6= yK ={
0, if y = y ,1, otherwise .
27 / 43
Binary Classification
• Define η(x) = P (y = 1|x).
• The conditional 0/1 risk at x is:
L0/1(h|x) = η(x)Jh(x) 6= 1K + (1− η(x))Jh(x) 6= −1K.
• The Bayes classifier:
h∗(x) ={
1 if η(x) > 1− η(x)−1 if η(x) < 1− η(x) = sgn (η(x)− 1/2) ,
and the Bayes conditional risk:
L`(h∗ |x) = min{η(x), 1− η(x)}.
28 / 43
Binary Classification
• Define η(x) = P (y = 1|x).• The conditional 0/1 risk at x is:
L0/1(h|x) = η(x)Jh(x) 6= 1K + (1− η(x))Jh(x) 6= −1K.
• The Bayes classifier:
h∗(x) ={
1 if η(x) > 1− η(x)−1 if η(x) < 1− η(x) = sgn (η(x)− 1/2) ,
and the Bayes conditional risk:
L`(h∗ |x) = min{η(x), 1− η(x)}.
28 / 43
Binary Classification
• Define η(x) = P (y = 1|x).• The conditional 0/1 risk at x is:
L0/1(h|x) = η(x)Jh(x) 6= 1K + (1− η(x))Jh(x) 6= −1K.
• The Bayes classifier:
h∗(x) ={
1 if η(x) > 1− η(x)−1 if η(x) < 1− η(x) = sgn (η(x)− 1/2) ,
and the Bayes conditional risk:
L`(h∗ |x) = min{η(x), 1− η(x)}.
28 / 43
Binary Classification
• Define η(x) = P (y = 1|x).• The conditional 0/1 risk at x is:
L0/1(h|x) = η(x)Jh(x) 6= 1K + (1− η(x))Jh(x) 6= −1K.
• The Bayes classifier:
h∗(x) ={
1 if η(x) > 1− η(x)−1 if η(x) < 1− η(x) = sgn (η(x)− 1/2) ,
and the Bayes conditional risk:
L`(h∗ |x) = min{η(x), 1− η(x)}.
28 / 43
Binary Classification
• Define η(x) = P (y = 1|x).• The conditional 0/1 risk at x is:
L0/1(h|x) = η(x)Jh(x) 6= 1K + (1− η(x))Jh(x) 6= −1K.
• The Bayes classifier:
h∗(x) ={
1 if η(x) > 1− η(x)−1 if η(x) < 1− η(x) = sgn (η(x)− 1/2) ,
and the Bayes conditional risk:
L`(h∗ |x) = min{η(x), 1− η(x)}.
28 / 43
Binary Classification
• Define η(x) = P (y = 1|x).• The conditional 0/1 risk at x is:
L0/1(h|x) = η(x)Jh(x) 6= 1K + (1− η(x))Jh(x) 6= −1K.
• The Bayes classifier:
h∗(x) ={
1 if η(x) > 1− η(x)−1 if η(x) < 1− η(x) = sgn (η(x)− 1/2) ,
and the Bayes conditional risk:
L`(h∗ |x) = min{η(x), 1− η(x)}.
28 / 43
Multi-class classification
• Domain of outcome variable y is a set of labels Y = {1, . . . ,K}.• Goal: find a prediction function h(x) that for any object x predicts
accurately the actual value of y.
• Loss function: the most common is 0/1 loss:
`0/1(y, y) =
{0, if y = y ,1, otherwise .
29 / 43
Multi-class classification
• The conditional risk of the 0/1 loss is:
L0/1(h |x) = Ey|x`0/1(y, h(x))
=∑k∈Y
P (y = k|x)`0/1(k, h(x))
• Therefore, the Bayes classifier is given by:
h∗(x) = argminh
L0/1(h |x)
= argmaxk
P (y = k|x) ,
the class with the largest conditional probability P (y|x).• The Bayes conditional risk:
L`(h∗ |x) = min{1− P (y = k|x) : k ∈ Y}.
30 / 43
Multi-class classification
• The conditional risk of the 0/1 loss is:
L0/1(h |x) = Ey|x`0/1(y, h(x))
=∑k∈Y
P (y = k|x)`0/1(k, h(x))
• Therefore, the Bayes classifier is given by:
h∗(x) = argminh
L0/1(h |x)
= argmaxk
P (y = k|x) ,
the class with the largest conditional probability P (y|x).• The Bayes conditional risk:
L`(h∗ |x) = min{1− P (y = k|x) : k ∈ Y}.
30 / 43
Multi-class classification
• The conditional risk of the 0/1 loss is:
L0/1(h |x) = Ey|x`0/1(y, h(x))
=∑k∈Y
P (y = k|x)`0/1(k, h(x))
• Therefore, the Bayes classifier is given by:
h∗(x) = argminh
L0/1(h |x)
= argmaxk
P (y = k|x) ,
the class with the largest conditional probability P (y|x).• The Bayes conditional risk:
L`(h∗ |x) = min{1− P (y = k|x) : k ∈ Y}.
30 / 43
Multi-class classification
• The conditional risk of the 0/1 loss is:
L0/1(h |x) = Ey|x`0/1(y, h(x))
=∑k∈Y
P (y = k|x)`0/1(k, h(x))
• Therefore, the Bayes classifier is given by:
h∗(x) = argminh
L0/1(h |x)
= argmaxk
P (y = k|x) ,
the class with the largest conditional probability P (y|x).
• The Bayes conditional risk:
L`(h∗ |x) = min{1− P (y = k|x) : k ∈ Y}.
30 / 43
Multi-class classification
• The conditional risk of the 0/1 loss is:
L0/1(h |x) = Ey|x`0/1(y, h(x))
=∑k∈Y
P (y = k|x)`0/1(k, h(x))
• Therefore, the Bayes classifier is given by:
h∗(x) = argminh
L0/1(h |x)
= argmaxk
P (y = k|x) ,
the class with the largest conditional probability P (y|x).• The Bayes conditional risk:
L`(h∗ |x) = min{1− P (y = k|x) : k ∈ Y}.
30 / 43
Multi-class classification
• The conditional risk of the 0/1 loss is:
L0/1(h |x) = Ey|x`0/1(y, h(x))
=∑k∈Y
P (y = k|x)`0/1(k, h(x))
• Therefore, the Bayes classifier is given by:
h∗(x) = argminh
L0/1(h |x)
= argmaxk
P (y = k|x) ,
the class with the largest conditional probability P (y|x).• The Bayes conditional risk:
L`(h∗ |x) = min{1− P (y = k|x) : k ∈ Y}.
30 / 43
Deterministic learning framework
• Input x ∈ X drawn from a distribution P (x).
• Outcome y ∈ Y.
• Unknown target function h∗ : X → Y, such that y = h∗(x).
• Goal: discover h∗ by observing examples of (x, y).
• This is a special case of the statistical framework:I What is P (y|x)?
• P (y|x) is a degenerate distribution for every x.
I Bayes prediction function?
• h∗
I Risk of h∗? (assuming `(y, y) = 0 whenever y = y)
• h∗ has zero risk.
I Unrealistic scenario in real life.
31 / 43
Deterministic learning framework
• Input x ∈ X drawn from a distribution P (x).
• Outcome y ∈ Y.
• Unknown target function h∗ : X → Y, such that y = h∗(x).
• Goal: discover h∗ by observing examples of (x, y).
• This is a special case of the statistical framework:I What is P (y|x)?
• P (y|x) is a degenerate distribution for every x.
I Bayes prediction function?
• h∗
I Risk of h∗? (assuming `(y, y) = 0 whenever y = y)
• h∗ has zero risk.
I Unrealistic scenario in real life.
31 / 43
Deterministic learning framework
• Input x ∈ X drawn from a distribution P (x).
• Outcome y ∈ Y.
• Unknown target function h∗ : X → Y, such that y = h∗(x).
• Goal: discover h∗ by observing examples of (x, y).
• This is a special case of the statistical framework:I What is P (y|x)?
• P (y|x) is a degenerate distribution for every x.
I Bayes prediction function?
• h∗
I Risk of h∗? (assuming `(y, y) = 0 whenever y = y)
• h∗ has zero risk.
I Unrealistic scenario in real life.
31 / 43
Deterministic learning framework
• Input x ∈ X drawn from a distribution P (x).
• Outcome y ∈ Y.
• Unknown target function h∗ : X → Y, such that y = h∗(x).
• Goal: discover h∗ by observing examples of (x, y).
• This is a special case of the statistical framework:I What is P (y|x)?
• P (y|x) is a degenerate distribution for every x.
I Bayes prediction function?• h∗
I Risk of h∗? (assuming `(y, y) = 0 whenever y = y)
• h∗ has zero risk.
I Unrealistic scenario in real life.
31 / 43
Deterministic learning framework
• Input x ∈ X drawn from a distribution P (x).
• Outcome y ∈ Y.
• Unknown target function h∗ : X → Y, such that y = h∗(x).
• Goal: discover h∗ by observing examples of (x, y).
• This is a special case of the statistical framework:I What is P (y|x)?
• P (y|x) is a degenerate distribution for every x.
I Bayes prediction function?• h∗
I Risk of h∗? (assuming `(y, y) = 0 whenever y = y)• h∗ has zero risk.
I Unrealistic scenario in real life.
31 / 43
Deterministic learning framework
• Input x ∈ X drawn from a distribution P (x).
• Outcome y ∈ Y.
• Unknown target function h∗ : X → Y, such that y = h∗(x).
• Goal: discover h∗ by observing examples of (x, y).
• This is a special case of the statistical framework:I What is P (y|x)?
• P (y|x) is a degenerate distribution for every x.
I Bayes prediction function?• h∗
I Risk of h∗? (assuming `(y, y) = 0 whenever y = y)• h∗ has zero risk.
I Unrealistic scenario in real life.
31 / 43
Outline
1 Motivation
2 Statistical Learning Theory
3 Learning Paradigms and Principles
4 Summary
32 / 43
Learning
• Distribution P (x, y) is unknown unknown.
• Therefore, Bayes classifier h∗ is also unknown.
• Instead, we have access to n independent and identically distributed(i.i.d) training examples (sample):
{(x1, y1), (x2, y2), . . . , (xn, yn)}.
• Learning: use training data to find a good approximation of h∗.
33 / 43
Spam filtering
• Problem: Predict whether a given email is spam or not.• An object to be classified: an email.• There are two possible responses (classes): spam, not spam.
34 / 43
Spam filtering
Example
• Representation of an email through (meaningful) features:
I length of subjectI length of email body,I use of colors,I domain,I words in subject,I words in body.
length of length of use ofsubject body colors domain gold price USD . . . machine learning spam?
7 240 1 live.fr 1 1 1 . . . 0 0 12 150 0 poznan.pl 0 0 0 . . . 1 1 02 250 0 tibco.com 0 1 1 . . . 1 1 0
4 120 1 r-project.org 0 1 0 . . . 0 0 ?
35 / 43
Spam filtering
Example
• Representation of an email through (meaningful) features:I length of subjectI length of email body,I use of colors,I domain,I words in subject,I words in body.
length of length of use ofsubject body colors domain gold price USD . . . machine learning spam?
7 240 1 live.fr 1 1 1 . . . 0 0 12 150 0 poznan.pl 0 0 0 . . . 1 1 02 250 0 tibco.com 0 1 1 . . . 1 1 0
4 120 1 r-project.org 0 1 0 . . . 0 0 ?
35 / 43
Training/Test Data in Computer Format
Example (ARFF format for training/test data)
@relation weather
@attribute outlook {sunny , overcast , rainy}
@attribute temperature real
@attribute humidity real
@attribute windy {true , false}
@attribute play {yes , no}
@data
sunny ,85,85,false ,no
sunny ,80,90,true ,no
overcast ,83,86,false ,yes
rainy ,70,96,false ,yes
rainy ,68,80,false ,yes
rainy ,65,70,true ,no
overcast ,64,65,true ,yes
sunny ,72,95,false ,no
sunny ,69,70,false ,yes
rainy ,75,80,false ,yes
sunny ,75,70,true ,yes
overcast ,72,90,true ,yes
overcast ,81,75,true ,yes
rainy ,71,91,true ,no
36 / 43
Learning
• Four types of datasets:I training data: past emails,I validation data: a portion of past email used for tuning learning
algorithmsI test data: a portion of past emails used for estimating the risk,I new incoming data to be classified: new incoming emails.
37 / 43
Different learning paradigms
• Generative learningI Follow a data generating processI Learn a model of the joint distribution P (x, y) and then use the Bayes
theorem to obtain P (y |x).I Make the final prediction by computing the optimal decision based onP (y |x) with respect to a given `(y, y).
• Discriminative learningI Approximate h∗(x) which is a direct map from x to y orI Model the conditional probability P (y |x) directly, andI Make the final prediction by computing the optimal decision based onP (y |x) with respect to a given `(y, y).
• Two phases of the learning models: learning and prediction(inference).
38 / 43
Different learning paradigms
• Generative learning
I Follow a data generating processI Learn a model of the joint distribution P (x, y) and then use the Bayes
theorem to obtain P (y |x).I Make the final prediction by computing the optimal decision based onP (y |x) with respect to a given `(y, y).
• Discriminative learningI Approximate h∗(x) which is a direct map from x to y orI Model the conditional probability P (y |x) directly, andI Make the final prediction by computing the optimal decision based onP (y |x) with respect to a given `(y, y).
• Two phases of the learning models: learning and prediction(inference).
38 / 43
Different learning paradigms
• Generative learningI Follow a data generating processI Learn a model of the joint distribution P (x, y) and then use the Bayes
theorem to obtain P (y |x).I Make the final prediction by computing the optimal decision based onP (y |x) with respect to a given `(y, y).
• Discriminative learningI Approximate h∗(x) which is a direct map from x to y orI Model the conditional probability P (y |x) directly, andI Make the final prediction by computing the optimal decision based onP (y |x) with respect to a given `(y, y).
• Two phases of the learning models: learning and prediction(inference).
38 / 43
Different learning paradigms
• Generative learningI Follow a data generating processI Learn a model of the joint distribution P (x, y) and then use the Bayes
theorem to obtain P (y |x).I Make the final prediction by computing the optimal decision based onP (y |x) with respect to a given `(y, y).
• Discriminative learning
I Approximate h∗(x) which is a direct map from x to y orI Model the conditional probability P (y |x) directly, andI Make the final prediction by computing the optimal decision based onP (y |x) with respect to a given `(y, y).
• Two phases of the learning models: learning and prediction(inference).
38 / 43
Different learning paradigms
• Generative learningI Follow a data generating processI Learn a model of the joint distribution P (x, y) and then use the Bayes
theorem to obtain P (y |x).I Make the final prediction by computing the optimal decision based onP (y |x) with respect to a given `(y, y).
• Discriminative learningI Approximate h∗(x) which is a direct map from x to y orI Model the conditional probability P (y |x) directly, andI Make the final prediction by computing the optimal decision based onP (y |x) with respect to a given `(y, y).
• Two phases of the learning models: learning and prediction(inference).
38 / 43
Different learning paradigms
• Generative learningI Follow a data generating processI Learn a model of the joint distribution P (x, y) and then use the Bayes
theorem to obtain P (y |x).I Make the final prediction by computing the optimal decision based onP (y |x) with respect to a given `(y, y).
• Discriminative learningI Approximate h∗(x) which is a direct map from x to y orI Model the conditional probability P (y |x) directly, andI Make the final prediction by computing the optimal decision based onP (y |x) with respect to a given `(y, y).
• Two phases of the learning models: learning and prediction(inference).
38 / 43
Different learning paradigms
• Various principles on how to learn:I Empirical risk minimization,I Maximum likelihood principle,I Bayes approach,I Minimum description length,I . . .
39 / 43
Empirical Risk Minimization (ERM)
• Choose a prediction function h which minimizes the loss on thetraining data within some restricted class of functions H.
h = argminh∈H
1
n
n∑i=1
`(yi, h(xi)).
• The average loss on the training data is called empirical risk L`(h).
• H can be: linear functions, polynomials, trees of a given depth, rules,linear combinations of trees, etc.1
Elements of Statistical Learning (2nd Ed.) c©Hastie, Tibshirani & Friedman 2009 Chap 2
Linear Regression of 0/1 Response
.. . . . . .. . . . . . . . . . . .. . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . .. . . . . . . .. .
oo
ooo
o
o
o
o
o
o
o
o
oo
o
o o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
oo
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
oo
o
o
o
o
o
o
o
o
o
oo o
oo
oo
o
oo
o
o
o
oo
o
o
o
o
o
o
o
o
o
o
o
o
oo
o
o
o
oo
o
o
o
o
o
oo
o
o
o
o
o
o
o
oo
o
o
o
o
o
o
o
o ooo
o
o
ooo o
o
o
o
o
o
o
o
oo
o
o
oo
ooo
o
o
ooo
o
o
o
o
o
o
o
oo
o
o
o
o
o
o
oo
ooo
o
o
o
o
o
o
oo
oo
oo
o
o
o
o
o
o
o
o
o
o
o
FIGURE 2.1. A classification example in two di-mensions. The classes are coded as a binary variable(BLUE = 0, ORANGE = 1), and then fit by linear re-gression. The line is the decision boundary defined by
xT β = 0.5. The orange shaded region denotes that partof input space classified as ORANGE, while the blue regionis classified as BLUE.
Elements of Statistical Learning (2nd Ed.) c©Hastie, Tibshirani & Friedman 2009 Chap 2
1-Nearest Neighbor Classifier
.. .. .. . . . . .. . . . . . .. . . . . . . . .. . . . . . . . .. . . . . . . . . . .. . . . . . . . . . . . . .. . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . .. . . . . . . . . . . .. . . . . . . . . . .. . . . . . . . . . .. . . . . . . . . . . .. . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . .. . . . . . . . . . .. . . . . . . . . .. . . . . . . . . .. . . . . . . . . .. . . . . . . . . .. . . . . . . . .
oo
ooo
o
o
o
o
o
o
o
o
oo
o
o o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
oo
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
oo
o
o
o
o
o
o
o
o
o
oo o
oo
oo
o
oo
o
o
o
oo
o
o
o
o
o
o
o
o
o
o
o
o
oo
o
o
o
oo
o
o
o
o
o
oo
o
o
o
o
o
o
o
oo
o
o
o
o
o
o
o
o ooo
o
o
ooo o
o
o
o
o
o
o
o
oo
o
o
oo
ooo
o
o
ooo
o
o
o
o
o
o
o
oo
o
o
o
o
o
o
oo
ooo
o
o
o
o
o
o
oo
oo
oo
o
o
o
o
o
o
o
o
o
o
o
FIGURE 2.3. The same classification example in twodimensions as in Figure 2.1. The classes are coded asa binary variable (BLUE = 0, ORANGE = 1), and thenpredicted by 1-nearest-neighbor classification.
Elements of Statistical Learning (2nd Ed.) c©Hastie, Tibshirani & Friedman 2009 Chap 9
|
t1
t2
t3
t4
R1
R1
R2
R2
R3
R3
R4
R4
R5
R5
X1
X1X1
X2
X2
X2
X1 ≤ t1
X2 ≤ t2X1 ≤ t3
X2 ≤ t4
FIGURE 9.2. Partitions and CART. Top right panelshows a partition of a two-dimensional feature space byrecursive binary splitting, as used in CART, applied tosome fake data. Top left panel shows a general partitionthat cannot be obtained from recursive binary splitting.Bottom left panel shows the tree corresponding to thepartition in the top right panel, and a perspective plotof the prediction surface appears in the bottom rightpanel.
1
T. Hastie, R. Tibshirani, and J. Friedman. Elements of Statistical Learning: Second Edition.Springer, 2009
40 / 43
Empirical Risk Minimization (ERM)
• Choose a prediction function h which minimizes the loss on thetraining data within some restricted class of functions H.
h = argminh∈H
1
n
n∑i=1
`(yi, h(xi)).
• The average loss on the training data is called empirical risk L`(h).
• H can be: linear functions, polynomials, trees of a given depth, rules,linear combinations of trees, etc.1
Elements of Statistical Learning (2nd Ed.) c©Hastie, Tibshirani & Friedman 2009 Chap 2
Linear Regression of 0/1 Response
.. . . . . .. . . . . . . . . . . .. . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . .. . . . . . . .. .
oo
ooo
o
o
o
o
o
o
o
o
oo
o
o o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
oo
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
oo
o
o
o
o
o
o
o
o
o
oo o
oo
oo
o
oo
o
o
o
oo
o
o
o
o
o
o
o
o
o
o
o
o
oo
o
o
o
oo
o
o
o
o
o
oo
o
o
o
o
o
o
o
oo
o
o
o
o
o
o
o
o ooo
o
o
ooo o
o
o
o
o
o
o
o
oo
o
o
oo
ooo
o
o
ooo
o
o
o
o
o
o
o
oo
o
o
o
o
o
o
oo
ooo
o
o
o
o
o
o
oo
oo
oo
o
o
o
o
o
o
o
o
o
o
o
FIGURE 2.1. A classification example in two di-mensions. The classes are coded as a binary variable(BLUE = 0, ORANGE = 1), and then fit by linear re-gression. The line is the decision boundary defined by
xT β = 0.5. The orange shaded region denotes that partof input space classified as ORANGE, while the blue regionis classified as BLUE.
Elements of Statistical Learning (2nd Ed.) c©Hastie, Tibshirani & Friedman 2009 Chap 2
1-Nearest Neighbor Classifier
.. .. .. . . . . .. . . . . . .. . . . . . . . .. . . . . . . . .. . . . . . . . . . .. . . . . . . . . . . . . .. . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . .. . . . . . . . . . . .. . . . . . . . . . .. . . . . . . . . . .. . . . . . . . . . . .. . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . .. . . . . . . . . . .. . . . . . . . . .. . . . . . . . . .. . . . . . . . . .. . . . . . . . . .. . . . . . . . .
oo
ooo
o
o
o
o
o
o
o
o
oo
o
o o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
oo
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
oo
o
o
o
o
o
o
o
o
o
oo o
oo
oo
o
oo
o
o
o
oo
o
o
o
o
o
o
o
o
o
o
o
o
oo
o
o
o
oo
o
o
o
o
o
oo
o
o
o
o
o
o
o
oo
o
o
o
o
o
o
o
o ooo
o
o
ooo o
o
o
o
o
o
o
o
oo
o
o
oo
ooo
o
o
ooo
o
o
o
o
o
o
o
oo
o
o
o
o
o
o
oo
ooo
o
o
o
o
o
o
oo
oo
oo
o
o
o
o
o
o
o
o
o
o
o
FIGURE 2.3. The same classification example in twodimensions as in Figure 2.1. The classes are coded asa binary variable (BLUE = 0, ORANGE = 1), and thenpredicted by 1-nearest-neighbor classification.
Elements of Statistical Learning (2nd Ed.) c©Hastie, Tibshirani & Friedman 2009 Chap 9
|
t1
t2
t3
t4
R1
R1
R2
R2
R3
R3
R4
R4
R5
R5
X1
X1X1
X2
X2
X2
X1 ≤ t1
X2 ≤ t2X1 ≤ t3
X2 ≤ t4
FIGURE 9.2. Partitions and CART. Top right panelshows a partition of a two-dimensional feature space byrecursive binary splitting, as used in CART, applied tosome fake data. Top left panel shows a general partitionthat cannot be obtained from recursive binary splitting.Bottom left panel shows the tree corresponding to thepartition in the top right panel, and a perspective plotof the prediction surface appears in the bottom rightpanel.
1 T. Hastie, R. Tibshirani, and J. Friedman. Elements of Statistical Learning: Second Edition.Springer, 2009
40 / 43
Outline
1 Motivation
2 Statistical Learning Theory
3 Learning Paradigms and Principles
4 Summary
41 / 43
Summary
• What is machine learning?
• Supervised learning: statistical decision/learning theory, lossfunctions, risk.
• Learning paradigms and principles.
42 / 43
Bibliography
• T. Hastie, R. Tibshirani, and J. Friedman. Elements of Statistical Learning: SecondEdition.
Springer, 2009http://www-stat.stanford.edu/~tibs/ElemStatLearn/
• Christopher M. Bishop. Pattern Recognition and Machine Learning.
Springer-Verlag, 2006
• David Barber. Bayesian Reasoning and Machine Learning.
Cambridge University Press, 2012http://www.cs.ucl.ac.uk/staff/d.barber/brml/
• Yaser S. Abu-Mostafa, Malik Magdon-Ismail, and Hsuan-Tien Lin. Learning FromData.
AMLBook, 2012
43 / 43
Related Documents
