CS 59000 Statistical Machine learningLecture 15
Yuan (Alan) QiPurdue CS
Oct. 21 2008
Outline
• Review of Gaussian Processes (GPs)• From linear regression to GP • GP for regression
• Learning hyperparameters• Automatic Relevance Determination• GP for classification
Gaussian Processes
How kernels arise naturally in a Bayesian setting?
Instead of assigning a prior on parameters w, we assign a prior on function value y.Infinite space in theory
Finite space in practice (finite number of training set and test set)
Linear Regression Revisited
Let
We have
From Prior on Parameter to Prior on Function
The prior on function value:
Stochastic Process
A stochastic process is specified by giving the joint distribution for any finite set of values in a consistent manner (Loosely speaking, it means that a marginalized joint distribution is the same as the joint distribution that is defined in the subspace.)
Gaussian Processes
The joint distribution of any variables is a multivariable Gaussian distribution.
Without any prior knowledge, we often set mean to be 0. Then the GP is specified by the covariance :
Impact of Kernel FunctionCovariance matrix : kernel function
Application economics & finance
Gaussian Process for Regression
Likelihood:
Prior:
Marginal distribution:
Samples of Data Points
Predictive Distribution
is a Gaussian distribution with mean and variance:
Predictive Mean
is the nth component ofWe see the same form as kernel ridge
regression and kernel PCA.
GP Regression
Discussion: the difference between GP regression and Bayesian regression with Gaussian basis functions?
Computational Complexity
GP prediction for a new data point:
GP: O(N3) where N is number of data pointsBasis function model: O(M3) where M is the
dimension of the feature expansionWhen N is large: computationally expensive.Sparsification: make prediction based on only a few
data points (essentially make N small)
Learning Hyperparameters
Empirical Bayes Methods
Automatic Relevance Determination
Consider two-dimensional problems:
Maximizing the marginal likelihood will make certain small, reducing its relevance to prediction.
Example
t = sin(2 π x1)
x2 = x1 +n
x3 = e
Gaussian Processes for Classification
Likelihood:
GP Prior:
Covariance function:
Sample from GP Prior
Predictive Distribution
No analytical solution.Approximate this integration:
Laplace’s methodVariational BayesExpectation propagation
Laplace’s method for GP Classification (1)
Laplace’s method for GP Classification (2)
Taylor expansion:
Laplace’s method for GP Classification (3)
Newton-Raphson update:
Laplace’s method for GP Classification (4)
Gaussian approximation:
Laplace’s method for GP Classification (4)
Question: How to get the mean and the variance above?
Predictive Distribution
Example
