Top Banner

Click here to load reader

32

NIPS2016論文紹介 Riemannian SVRG fast stochastic optimization on riemannian manifolds

Feb 21, 2017

ReportDownload

Technology

  • NIPS 2016

    Riemannian SVRG: Fast Stochastic Optimization on Riemannian Manifolds

    Takami Sato

    2017/02/03 NIPS2016 1

    Authors: Hongyi Zhang, Sashank J. Reddi and Suvrit Sra

  • SVRGRSVRG

    GDSGD

    (1/2)(1/)

    Gradient Dominated

    Gradient Dominated

    PCARiemman centroid RSVRG

    2017/02/03 NIPS2016 2

  • 2017/02/03 NIPS2016 3

    http://www.slideshare.net/Suurist/hiroyuki-sato-62489428

    http://suuri.st/portfolio/%E4%BD%90%E8%97%A4%E5%AF%9B%E4%B9%8B

    http://www.slideshare.net/Suurist/hiroyuki-sato-62489428http://www.slideshare.net/Suurist/hiroyuki-sato-62489428http://www.slideshare.net/Suurist/hiroyuki-sato-62489428http://www.slideshare.net/Suurist/hiroyuki-sato-62489428http://www.slideshare.net/Suurist/hiroyuki-sato-62489428http://suuri.st/portfolio/%E4%BD%90%E8%97%A4%E5%AF%9B%E4%B9%8Bhttp://suuri.st/portfolio/%E4%BD%90%E8%97%A4%E5%AF%9B%E4%B9%8B

  • 2017/02/03 NIPS2016 4

    n-1

  • 2017/02/03 NIPS2016 5

    Riemannian manifold

    MM

  • 2017/02/03 NIPS2016 6

    Riemannian manifold

    MM

    tangent space

  • 2017/02/03 NIPS2016 7

    RPG

  • SGD

    sub-linear

    Variance Reduction

    2017/02/03 NIPS2016 8

    SVRG

  • SVRG

    2017/02/03 NIPS2016 9

    (smooth)strongly convex

  • SVRG

    2017/02/03 NIPS2016 10

    SAG

    SVRG

    SAGA SAGSVRG

  • +1 = 1

    =1

    =

    =

    1 ( )

    2017/02/03 NIPS2016 11

    +1 = 1

    =1

    +1

    ()

    +1 = 1

    =1

    + ()

    SAG Stochastic Average Gradient

    SVRG Stochastic Variance Reduction Gradient

    SAGA

    SVRG

  • Variance Reduction

    MCMC

    2017/02/03 NIPS2016 12

    = + ()

    () = ) + 1 ( = 1 X

    () = 2( ) + 2(, )

    < 1 SAG

    SVRG: X , =

  • 2017/02/03 NIPS2016 13

    sublinear convergence

    11 O(log1

    )

    linear convergence

    k1/k O(1

    )

    superlinear convergence

    2Second-order convergence

    12 O(loglog1

    )

  • n

    2017/02/03 NIPS2016 14

    log (1

    )

    Nesterov

    log (1

    )

    SVRG + log (1

    )

  • SVRG

    2017/02/03 NIPS2016 15

    NIPS2016 http://www.di.ens.fr/~fbach/fbach_tutorial_vr_nips_2016.pdf

    O()

    O(n)O(1)

    + log (1

    )

    http://www.di.ens.fr/~fbach/fbach_tutorial_vr_nips_2016.pdfhttp://www.di.ens.fr/~fbach/fbach_tutorial_vr_nips_2016.pdfhttp://www.di.ens.fr/~fbach/fbach_tutorial_vr_nips_2016.pdf

  • SVRG

    SVRG[Johnson+ 2013]

    2017/02/03 NIPS2016 16

  • SVRG

    2017/02/03 NIPS2016 17

    SVRG

  • SVRG

    2017/02/03 NIPS2016 18

    SVRG

  • 2017/02/03 NIPS2016 19

    1

    2

    Parallel Transport

  • 2017/02/03 NIPS2016 20

    g-convex()

    -strongly g-convex()

    L-g-smooth()

    -gradient dominated ()

  • 2017/02/03 NIPS2016 21

    /2

  • SVRG 2 RSVRG

    2017/02/03 NIPS2016 22

  • 2017/02/03 NIPS2016 23

  • 2017/02/03 NIPS2016 24

  • 2017/02/03 NIPS2016 25

    -strongly g-convexgradient dominated

    1

  • 2017/02/03 NIPS2016 26

  • Gradient Dominated

    2017/02/03 NIPS2016 27

  • 2017/02/03 NIPS2016 28

  • 1/

    2017/02/03 NIPS2016 29

    1/ 1/2

    y2epoch

    epochepoch

  • Riemman centroid

    2017/02/03 NIPS2016 30

    Riemman centroid

  • Riemman centroid

    2017/02/03 NIPS2016 31

    Riemman centroid

    100100

    RSVRG

  • SVRG

    Vector transport

    2017/02/03 NIPS2016 32

Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.