LSMR: An iterative algorithm for least-squares problems David Fong Michael Saunders Institute for Computational and Mathematical Engineering (iCME) Stanford University Copper Mountain Conference on Iterative Methods Copper Mountain, Colorado Apr 5–9, 2010 David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 1/31
54
Embed
LSMR: An iterative algorithm for least-squares problemsLSMR: An iterative algorithm for least-squares problems David Fong Michael Saunders Institute for Computational and Mathematical
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 16/31
log10(‖E2‖) for LSQR and LSMR
0 200 400 600 800 1000 1200 1400 1600 1800−7
−6
−5
−4
−3
−2
−1
0
iteration count
log(
E2)
Name:lp pilot ja, Dim:2267x940, nnz:14977, id=88
E2 LSQRE2 LSMR
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 17/31
Backward errors for LSQR
0 200 400 600 800 1000 1200 1400 1600 1800−8
−7
−6
−5
−4
−3
−2
−1
0
1
iteration count
log(
Bac
kwar
d E
rror
)
Name:lp cre a, Dim:7248x3516, nnz:18168, id=93
E1 LSQRE2 LSQROptimal LSQR
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 18/31
Backward errors for LSMR
0 200 400 600 800 1000 1200 1400 1600 1800−8
−7
−6
−5
−4
−3
−2
−1
0
1
iteration count
log(
Bac
kwar
d E
rror
)
Name:lp cre a, Dim:7248x3516, nnz:18168, id=93
E1 LSMRE2 LSMROptimal LSMR
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 19/31
Optimal backward errors for LSQR and LSMR
0 100 200 300 400 500 600−7
−6
−5
−4
−3
−2
−1
0
iteration count
log(
||ATr|
|/||r
||)
Name:lp pilot, Dim:4860x1441, nnz:44375, id=107
Optimal LSQROptimal LSMR
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 20/31
Space-time trade-offs
LSMR is well-suited for limited memory computations.
What if we have• more memory• Av expensive
Can we speed things up?
Some ideas:• Reorthogonalization• Restarting• Local reorthogonalization
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 21/31
Space-time trade-offs
LSMR is well-suited for limited memory computations.
What if we have• more memory• Av expensive
Can we speed things up?
Some ideas:• Reorthogonalization• Restarting• Local reorthogonalization
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 21/31
Reorthogonalization
Golub-Kahan processInfinite precision Finite precisionUk, Vk orthonormal Lose orthogonalityAt most min(m,n) iterations Could take 10n or more
Apply modified Gram-Schmidt to uk+1 and/or vk+1:
u← u− (uTj u)uj j = k, k−1, k−2, . . .
(similarly for v)
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 22/31
Reorthogonalization
Golub-Kahan processInfinite precision Finite precisionUk, Vk orthonormal Lose orthogonalityAt most min(m,n) iterations Could take 10n or more
Apply modified Gram-Schmidt to uk+1 and/or vk+1:
u← u− (uTj u)uj j = k, k−1, k−2, . . .
(similarly for v)
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 22/31
Effects of reorthogonalization on various problems
0 10 20 30 40 50 60 70 80 90−8
−7
−6
−5
−4
−3
−2
−1
0
1
iteration count
log(
E2)
Name:lp ship12l, Dim:5533x1151, nnz:16276, id=91
NoOrthoOrthoUOrthoVOrthoUV
0 100 200 300 400 500 600 700 800 900−7
−6
−5
−4
−3
−2
−1
0
1
iteration count
log(
E2)
Name:lp scfxm2, Dim:1200x660, nnz:5469, id=62
NoOrthoOrthoUOrthoVOrthoUV
0 20 40 60 80 100 120 140 160 180−6
−5
−4
−3
−2
−1
0
1
iteration count
log(
E2)
Name:lp agg2, Dim:758x516, nnz:4740, id=58
NoOrthoOrthoUOrthoVOrthoUV
0 2000 4000 6000 8000 10000 12000−7
−6
−5
−4
−3
−2
−1
0
iteration count
log(
E2)
Name:lpi gran, Dim:2525x2658, nnz:20111, id=94
NoOrthoOrthoUOrthoVOrthoUV
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 23/31
Orthogonality of Uk
100
200
300
400
500
600
5
6
Original
100
200
300
400
500
600
5
6
7
8
910
11
1213
141516
MGS on U
100
200
300
400
500
600
5
6
7
8
910
11
1213
141516
MGS on V
100
200
300
400
500
600
5
6
7
8
910
11
1213
141516
MGS on U ,V
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 24/31
Orthogonality of Vk
100
200
300
400
500
600
5
6
Original
100
200
300
400
500
600
5
6
7
8
910
11
1213
141516
MGS on U
100
200
300
400
500
600
5
6
7
8
910
11
1213
141516
MGS on V
100
200
300
400
500
600
5
6
7
8
910
11
1213
141516
MGS on U ,V
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 25/31
What we learnt so far
• Reorthogonalizing Vk (only) is sufficient• Reorthogonalizing Uk (only) is nearly as good• xk converges the same for all options
What can be improved• May still use too much memory• Need more flexibility for space-time trade-off
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 26/31
Reorthogonalization with Restarting
Restarting LSMR
rk = b−Axk min ‖A∆x− rk‖
Restarting leads to stagnation
0 500 1000 1500 2000 2500 3000 3500 4000 4500−7
−6
−5
−4
−3
−2
−1
0
iteration count
Bac
kwar
d E
rror
Name:lp maros, Dim:1966x846, nnz:10137, id=81
NoOrthoRestart5Restart10Restart50NoRestart
0 2000 4000 6000 8000 10000 12000 14000 16000−7
−6
−5
−4
−3
−2
−1
0
1
iteration count
Bac
kwar
d E
rror
Name:lp cre c, Dim:6411x3068, nnz:15977, id=90
NoOrthoRestart5Restart10Restart50NoRestart
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 27/31
Reorthogonalization with Restarting
Restarting LSMR
rk = b−Axk min ‖A∆x− rk‖
Restarting leads to stagnation
0 500 1000 1500 2000 2500 3000 3500 4000 4500−7
−6
−5
−4
−3
−2
−1
0
iteration count
Bac
kwar
d E
rror
Name:lp maros, Dim:1966x846, nnz:10137, id=81
NoOrthoRestart5Restart10Restart50NoRestart
0 2000 4000 6000 8000 10000 12000 14000 16000−7
−6
−5
−4
−3
−2
−1
0
1
iteration count
Bac
kwar
d E
rror
Name:lp cre c, Dim:6411x3068, nnz:15977, id=90
NoOrthoRestart5Restart10Restart50NoRestart
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 27/31
Local reorthogonalization• Reorthogonalize wrto only the last l vectors• Partial speed-up• Less memory• Depends on efficiency of Av and ATu
Speed up with local reorthogonalization
0 50 100 150 200 250 300 350 400 450−7
−6.5
−6
−5.5
−5
−4.5
−4
−3.5
−3
−2.5
−2
iteration count
Bac
kwar
d E
rror
Name:lp fit1p, Dim:1677x627, nnz:9868, id=80
NoOrthoLocal5Local10Local50FullOrtho
0 200 400 600 800 1000 1200 1400−6
−5
−4
−3
−2
−1
0
1
2
iteration count
Bac
kwar
d E
rror
Name:lp bnl2, Dim:4486x2324, nnz:14996, id=89
NoOrthoLocal5Local10Local50FullOrtho
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 28/31
Local reorthogonalization• Reorthogonalize wrto only the last l vectors• Partial speed-up• Less memory• Depends on efficiency of Av and ATu
Speed up with local reorthogonalization
0 50 100 150 200 250 300 350 400 450−7
−6.5
−6
−5.5
−5
−4.5
−4
−3.5
−3
−2.5
−2
iteration count
Bac
kwar
d E
rror
Name:lp fit1p, Dim:1677x627, nnz:9868, id=80
NoOrthoLocal5Local10Local50FullOrtho
0 200 400 600 800 1000 1200 1400−6
−5
−4
−3
−2
−1
0
1
2
iteration count
Bac
kwar
d E
rror
Name:lp bnl2, Dim:4486x2324, nnz:14996, id=89
NoOrthoLocal5Local10Local50FullOrtho
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 28/31
Conclusions
Advantages of LSMR• All the good properties of LSQR• Monotone convergence of ‖ATrk‖• ‖rk‖ still monotonic• Cheap near-optimal backward error estimate⇒ reliable stopping rule
• Backward error almost surely monotonic
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 29/31
Acknowledgement
• Michael Saunders• Chris Paige• Stanford Graduate Fellowship
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 30/31
Questions
LSMR in MATLAB andslides for this talk are
downloadable athttp://zi.ma/lsmr
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 31/31