PARAMETER ESTIMATION FOR SEMIPARAMETRIC MODELS WITH CMARS AND ITS APPLICATIONS Fatma YERLIKAYA-ÖZKURT Institute of Applied Mathematics, METU, Ankara,Turkey Gerhard-Wilhelm WEBER Institute of Applied Mathematics, METU, Ankara,Turkey Faculty of Economics, Business and Law, University of Siegen, Germany Center for Research on Optimization and Control, University of Aveiro, Portugal Universiti Teknologi Malaysia, Skudai, Malaysia Pakize TAYLAN Department of Mathematics, Dicle University , Diyarbakır , Turkey 5th International Summer School Achievements and Applications of Contemporary Informatics, Mathematics and Physics National University of Technology of the Ukraine Kiev, Ukraine, August 3-15, 2010
30
Embed
Parameter Estimation for Semiparametric Models with CMARS and Its Applications
AACIMP 2010 Summer School lecture by Gerhard Wilhelm Weber. "Applied Mathematics" stream. "Modern Operational Research and Its Mathematical Methods with a Focus on Financial Mathematics" course. Part 11. More info at http://summerschool.ssa.org.ua
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
PARAMETER ESTIMATION FOR SEMIPARAMETRIC MODELS
WITH CMARS AND ITS APPLICATIONS
Fatma YERLIKAYA-ÖZKURT
Institute of Applied Mathematics, METU, Ankara,Turkey
Gerhard-Wilhelm WEBER
Institute of Applied Mathematics, METU, Ankara,Turkey
Faculty of Economics, Business and Law, University of Siegen, Germany
Center for Research on Optimization and Control, University of Aveiro, Portugal
Universiti Teknologi Malaysia, Skudai, Malaysia
Pakize TAYLAN
Department of Mathematics, Dicle University, Diyarbakır, Turkey
5th International Summer School
Achievements and Applications of Contemporary Informatics,
which can be solved using generalized singular value decomposition (GSVD).
2 2
2 2min ,
preproc
preproc preprocy X L
y preproc
The Least-Squares Estimation with Tikhonov Regularization
• After getting the regression coefficients, we subtract the linear least- square model (without intercept)
from corresponding responses:
• Doing this at the input data, the resulting values ( ) are our new responses.
• Then, based on these new data, we find find the knots for nonparametric part with MARS.
• Again consider model :
and is a smooth function which we try to estimate by CMARS
which is an alternative technique to the well-known data mining tool multivariate adaptive
regression splines (MARS).
1
ˆ.m
j j
j
y X y
y
( ) T
i i i iH x t
The Least-Squares Estimation with Tikhonov Regularization
CMARS Method
• What is MARS?
• Multivariate adaptive regression splines (MARS) is developed in 1991 by Jerome Friedman.
• MARS builds flexible models by introducing piecewise linear regressions.
• MARS uses expansions in piecewise linear basis functions of the form
.
• Set of basis functions:
.
Basic elements in the regression with MARS.
[ ] : max 0,q q ( , ) = [ ( )] , -c x x+ ( , ) = [ ( )] ,c x x
1, 2, ,: ( ) , ( ) , ,..., , 1,2,...,|j j j j N jX X x x x j p
x
y
+( , )=[ ( )]c x x ( , )=[ ( )]-c x x
CMARS Method
• Thus, we can represent by a linear combination which is successively built up by basis
functions and the intercept , such that
.
Here, are basis functions from or products of two or more such functions,
interaction basis functions are created by multiplying an existing basis function with a truncated linear
function involving new variable. are the unknown coefficients for the mth basis function
or for the constant 1
• Provided the observations represented by the data :
.
0
1
( ) ( )M
T
i i i m m i
m
H
x t
i t
0
( 1,2,..., )m m M
m ( 1,2,..., )m M
( 0).m
( 1, 2, ..., )i i Nt
1
( ) : [ ( )]m
m m mj j j
K
m
j
s t
t
CMARS Method
The MARS algorithm for estimating the model function consists of two algorithms:
I. Forward stepwise algorithm:
• Search for the basis functions.
• At each step, the split that minimized some “lack of fit” criterion from all the possible splits on each basis function is chosen.
• The process stops when a user-specified value is reached. At the end of this process we have a large expression in Y .
• This model typically overfits the data; so a backward deletion procedure is applied.
II. Backward stepwise algorithm:
• Prevents from over-fitting by decreasing the complexity of the model without degrading the fit to the data.
• Proposition: We do not employ the backward stepwise algorithm to estimate the function.
At its place, as an alternative we propose to use penalty terms in addition to the least-squares estimation in order to control the lack of fit from the viewpoint of the complexity of the estimation.
maxM
The Penalized Residual Sum of Squares for GLM with MARS
• Let we consider equation
• ,
where and . Let us use the penalized residual sum
of squares with basis functions having been accumulated in the forward stepwise algorithm.
For the GLM model with MARS, PRSS has the following form:
, .
( ) T
i i i iH x t
1, ,...,T
M
maxM
max
1 2
22 2
2
,
1 1 1, ( )( , )
( ) ,MN
T m m
i i i m m r s m
i m r sr s V m
PRSS D d
x t t t
T
i i i t t t
1 2
, ( ) : ( )m m m m
r s m m r sD t t t t
1 2
1 2
1 2 1 2
( ) : | 1, 2,...,
: ( , ,..., )
( , )
: , , 0,1
Km
m
j m
m T
m m m
V m j K
t t t
t =
where
The Penalized Residual Sum of Squares for GLM with MARS
• Our optimization problem bases on the tradeoff between both accuracy, i.e., a small sum of error squares, and not too high a complexity.
• This tradeoff is established through the penalty parameters .
• Let use in approximate PRSS by discretizing the high-dimensional integration:
Then, PRSS becomes
.
m
1
, ,1
ˆm
m mm j m jj j
Km
i l lj
t t t
, , ,
ˆ , ,..., ,m m mm j m j m jj j j
m
i l l lt t t
t
1,2,...,( ) 0,1,2,..., 1
Kmj
mj KN
max
,M
max
max
1 1
1 11, ( ),..., ( ), ( ),..., ( )T
MM M
i i M i M i M i
d t t t t
max
1 2
2
1
( 1) 22
2
,
1 1 1, ( )( , )
( )
ˆ ˆ( ) .
Km
NT
i i i
i
M Nm m
m m r s m i i
m i r sr s V m
PRSS
D
x d
t t
max1 2 1 2: ( , ,..., , , ,..., )MM M M T
i i i i i i i
d t t t t t t
Tikhonov Regularization for GLM with MARS
• For a short representation, we can rewrite the approximate relation as
• We can write PRSS as
where is a block matrix constructed by -matrix and
matrix , is a vector constructed and vectors.
• Then, we deal with the linear systems equations of , approximately.
12
1 2
22
,
1, ( )( , )
ˆ ˆ( ) .m m
im r s m i i
r sr s V m
L D
t t
max ( 1)2
2 2
21 1
( ) ,
KmM N
m im m
m i
PRSS L
X d
1( ) ( ),..., ( )T
N d d d
max ( 1)2
* * 2 2
21 1
,
KmM N
m im m
m i
PRSS L
X
* = ( )X X d ( )N p Xmax( ( 1))N Μ
( )d * = , T
* * X
Tikhonov Regularization for GLM with MARS
• We approach our problem PRSS as a Tikhonov regularization problem by using the same penalty
parameter for each derivative:
Here, high dimentional matrix , where is an matrix with entries being first
or second derivatives of .
• We can easily note that our Tikhonov regularization problem has multiple objective functions through
a linear combination of and . We select the solution such that it minimizes both
first objective function and second objective in the sense of a compromise
(tradeoff) solution.
2( : )m
max( 1)M p
2 2* * * *
2 2.PRSS X L
* *L = R L
*R
2* *
2 X
2* *
2X
2* *
2 X
2* *
2X
An Alternative Solution for Tikhonov Regularization Problem with CQP
• We can solve Tikhonov regularization problem for MARS by continuous optimization techniques, especially, conic quadratic programming.
• We formulate PRSS as a CQP problem:
max
* *
1 1 1 1 1(1,0 ) , ( , ) , (0 , ), , (1,0,...,0) , 0,T T T T T
M p Nz q c u D X d p
, *
* *
2
* *
2
min ,
subject to ,
.
zz
z
M
X
L
2, ( 1,2,..., ). ( )min T
i ii iT q i k
xc Qx Pp Cx D x dIn general : subject to
Max max max
*
2 1 2 1 2 2 2( , ), , and . 0 0 0M M M p q MD L d p
*I
• We first reformulate as a Primal Problem:
with ice-cream (or second order or Lorentz) cones:
*I
max
maxmax
max
max
, *
*
*
1
*11
*
1
21
min ,
such that ,1 0
0,
0
, ,
0
0
0
0
z
N
T
M p
MM
T
M p
MN
z
z
z
M
L L
X
L
1 1 2 2 2
1 2 1 1 1 2( , ,..., ) | ... ( 1).N T N
N N+ NL x x x x x x x N
x R
An Alternative Solution for Tikhonov Regularization Problem with CQP
• The corresponding Dual Problem is
max
max
maxmax max
2max
1 1 2
1
1 2* *11 1
1
1 2
max ( ,0) ,
0 11such that ,
, .
0
00
00 0
M
T T
M
TTMN
T TM pM p M p
N
M
L L
X L
An Alternative Solution for Tikhonov Regularization Problem with CQP
Solution Methods
• CQPs belong to the well-structured convex problems.
• Interior Point algorithms:
– We use the structure of problem.
– Yield better complexity bounds.
– Exhibit much better practical performance.
Application
• GLPMs with CMARS and the parameter estimation for them have been presented and investigated in detail. Now, a numerical example for this study will be given.
• Two data sets are used in our applications:
• Concrete Compressive Strength Data Set
• Concrete Slump Test Data Set
• Data sets are obtained from UCI Machine Learning Repository (http://archive.ics.uci.edu/ml/).
• Salford Systems is used for MARS application, for CMARS a code is written by using MATLAB
and in order to solve the CQP problem in CMARS, MOSEK software is preferred.
For Tikhonov Regularization Regularization Toolbox in MATLAB is used.
• All test data sets are also compared according to the performance measures such as Root Mean Square Error (RMSE), Correlation Coefficient (r), R2, Adjusted R2.
•
• To compare the performances of Tikhonov regularization, CMARS and GPLM models,
let us look at the performance measure values for both data sets.
WORSE BETTER
Evaluation of the models based on performance values:
• CMARS performs better than Tikhonov regularization with respect to all the measures for both data sets.
• On the other hand, GLM with CMARS (GPLM) performs better than both Tikhonov regularization and CMARS with respect to all the measures for both data sets.
Application
Outlook
• Important new class of GPLs: having written the Tikhonov regularization task for GLM using
MARS as a CQP problem, we will call it CGLMARS:
, ,
( , ) GPLM ( ) ( )LM MARS
= +
TE Y X T G X T
X T X T
e.g.,
References
[1] Aster, A., Borchers, B., and Thurber, C., Parameter Estimation and Inverse Problems, Academic
Press, 2004.
[2] Craven, P., and Wahba, G., Smoothing noisy data with spline functions, Numer. Math. 31, Linear
Models, (1979), 377-403.
[3] De Boor, C., Practical Guide to Splines, Springer Verlag, 2001.