Generaliza)ons of support vector machines Elinor Velasquez Bioinforma)cs Department University of California, Santa Cruz
Generaliza)ons,of,support,vector,machines,
Elinor,Velasquez,,Bioinforma)cs,Department,
University,of,California,,Santa,Cruz,
Outline,
• The,problem,that,we,are,trying,to,solve,• An,example,
• Support,vector,machines,defined,
• Rela)on,to,convolu)onal,neural,nets,• SVMs,generalized,to,Gelfand,pairs,
• Some,new,geometrical,generaliza)ons,for,learning,new,features,and,for,classifica)on,
The,problem,we,are,trying,to,solve,
• Given,data,from,a,cancer,pa)ent,,classify,the,subtype,of,the,cancer.,
The,problem,we,are,trying,to,solve,
• What,is,the,data?,Suppose,we,have,gene,expression,intensity,data.,
• The,gene,expression,intensity,increases,when,the,gene,is,“on”,(upPregulated),and,decreases,when,the,gene,is,“off”,(downPregulated).,
Microarray,datachip,(Ohio,State,CompBio,Lab),
Example,
• Suppose,there,are,only,two,types,of,lung,cancer,,named,AC,and,SCC.,
• Data,=,{(xi,,yi)}i=1,,…,,N,with,xi,a,MPdimensional,vector,of,gene,expression,intensity,values,(M,genes),
• yi,=,Dirac,delta,func)on,(1,if,AC,,0,if,SCC),for,,,,,,,i,=,1,,…,,N,pa)ent,samples.,
We,want,to,classify,the,data,samples,
• How,to,do,this?,• Use,supervised,learning,method:,Support,vector,machines,
• Supervised,learning,means,to,predict/classify,
• Unsupervised,learning,is,used,to,uncover,structure,in,the,data,,
A,support,vector,machine,(SVM),is,a,type,of,graphical,model,(Vapnik,,1995),
x1,
x2,
xM,
+1,
Output,=,f(wTx,+,b),,,,,,,,,,,,,,,=,ypredicted(
Loss,func)on,=,max,(0,,1,–,yf(x)),
Input,
Supervised,learning,method,
Generalize,SVMs,to,convolu)onal,neural,nets,
• Why?,,• Convolu)onal,neural,nets,are,performing,beeer,than,SVMs,for,a,variety,of,data,types.,
Convolu)onal,neural,net,is,a,graphical,learning,model,(LeCun,,2006),
x1,
x2,
xN,
+1,
h1,
h1,
h1,
h2,
h2,
h2,
Input,Output,
Supervised,learning,method,
We,want,our,data,to,be,invariant,under,rigid,mo)ons,and,small,deforma)ons,
• Mallat,(2010),solved,this,problem,for,data,learned,in,convolu)onal,neural,nets:,He,used,a,scaeering,operator,,S,,on,f,ε,L2(RM),which,was,composed,of,a,wavelet,transform,,convolu)on,operator,and,modulus,operator.,,
• But,data,lies,on,a,more,complicated,manifold.,
Construc)ng,data,invariant,to,rigid,mo)ons,and,small,deforma)ons,
• Let,G,be,a,locally,compact,group,and,K,a,compact,subgroup,(Gelfand,pair),
• G,approximates,a,complicated,manifold.,• Example:,Let,G,=,group,of,inver)ble,matrices,,K,=,orthogonal,group,
• Suppose,we,can,map,the,data,to,G,(True,for,Lie,groups),• Then,,build,a,convolu)onal,neural,net,for,the,data,on,G/K,,an,“invariant”,space.,
• Will,extend,Mallat’s,work,(work,in,progress),
An,ideal,data,space,
• The,data,actually,lives,on,a,manifold,X.,• Map,the,data,to,X/G,,with,X,a,manifold,and,G,,its,group,of,invariances,,using,geometric,invariant,theory,tools.,
• Build,a,convolu)onal,neural,network,on,this,data,space,,X/G.,,
• (Work,in,progress),
Can,we,improve,upon,convolu)onal,neural,nets?,
x1,
x2,
xN,
h1,
h1,
h1,
h2,
h2,
h2,
h3,
A,cylindrical,neural,net,is,called,a,Recurrent,Neural,Net,(Graves,1980s),
We,can,make,a,convolu)onal,recurrent,neural,net,for,classifica)on,
x1,
x2,
xN,
h1,
h1,
h1,
h2,
h2,
h2,
h3, Aeach,a,recursive,neural,net,to,the,recurrent,neural,net:,
h1,
h2,
Cylinder,(Recurrent,neural,net),
Triangle,(Recursive,neural,net),
Output,
We,can,pinch,the,cylinder,on,each,end,to,make,a,convolu)onal,neural,net,
sphere,minus,the,south,pole,
h3,
h2,
h1,
x1,
h3,
h2,
h1,
x2,
h1, h1,
h2, h2,
h3,
Output,
We,can,construct,a,convolu)onal,neural,net,on,a,smooth,manifold,of,
genus,G,
h9,
h2,
h3,
h4,
h5,
h6,
h7,h8,
h1,
Output,
h1,
h2,
h3,
h4,
h5,h9,
h6,
h7,
h8,
Summary,
• Convolu)onal,neural,nets,are,learning,models,which,classify,data.,
• Data,actually,lives,on,a,complicated,manifold.,
• We,can,remove,data,invariances,before,learning,to,remove,redundancy.,
• The,future,holds,many,possibili)es,for,learning,models.,
Thank,you!,