DSP-CIS Chapter-13: Least Mean Squares (LMS) Algorithm Marc Moonen Dept. E.E./ESAT-STADIUS, KU Leuven marc.moonen@esat.kuleuven.be

DSP-CIS

Chapter-13: Least Mean Squares (LMS) Algorithm

Marc MoonenDept. E.E./ESAT-STADIUS, KU Leuven

marc.moonen@esat.kuleuven.be

www.esat.kuleuven.be/stadius/

DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 2

Part-III : Optimal & Adaptive Filters

: Optimal & Adaptive Filters - Intro• General Set-Up

• Applications

• Optimal (Wiener) Filters

– : Least Squares & Recursive Least Squares Estimation• Least Squares Estimation

• Recursive Least Squares (RLS) Estimation

• Square-Root Algorithms

– : Least Means Squares (LMS) Algorithm• LMS/NLMS : Stochastic Gradient Algorithms

• LMS analysis

• LMS Family

– : Fast Recursive Least Squares Algorithms

: Kalman Filtering

Chapter-11

Chapter-12

Chapter-13

Chapter-14

Chapter-15

DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 3

Least Mean Squares (LMS) Algorithm

DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 4

Least Mean Squares (LMS) Algorithm

DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 5

Least Mean Squares (LMS) Algorithm

(Widrow 1965 !!)

DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 6

Least Mean Squares (LMS) Algorithm

Bernard Widrow

https://www.youtube.com/watch?v=hc2Zj55j1zU

DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 7

Least Mean Squares (LMS) Algorithm

DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 8

Least Mean Squares (LMS) Algorithm

DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 9

Least Mean Squares (LMS) Algorithm

large λ_max implies

a small stepsize

DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 10

Least Mean Squares (LMS) Algorithm

DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 11

Least Mean Squares (LMS) Algorithm

error vector projected onto eigenvectors initial error vector projected onto eigenvectors

(=projection on i-th eigenvector)

small λ_i implies slow convergence λ_min <<λ_max (hence small μ) implies *very* slow convergence

DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 12

Least Mean Squares (LMS) Algorithm

DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 13

Least Mean Squares (LMS) Algorithm

DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 14

Least Mean Squares (LMS) Algorithm

DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 15

Least Mean Squares (LMS) Algorithm

DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 16

Least Mean Squares (LMS) Algorithm

DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 17

Least Mean Squares (LMS) Algorithm

DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 18

Least Mean Squares (LMS) Algorithm

DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 19

Least Mean Squares (LMS) AlgorithmS

kip

th

is s

lide

DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 20

Least Mean Squares (LMS) AlgorithmS

kip

th

is s

lide

DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 21

Least Mean Squares (LMS) AlgorithmS

kip

th

is s

lide

DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 22

Least Mean Squares (LMS) AlgorithmS

kip

th

is s

lide

DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 23

Least Mean Squares (LMS) AlgorithmS

kip

th

is s

lide

DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 24

Least Mean Squares (LMS) AlgorithmS

kip

th

is s

lide

DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 25

Least Mean Squares (LMS) AlgorithmS

kip

th

is s

lide

DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 26

Least Mean Squares (LMS) AlgorithmS

kip

th

is s

lide

DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 27

Least Mean Squares (LMS) AlgorithmS

kip

th

is s

lide

DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 28

Least Mean Squares (LMS) AlgorithmS

kip

th

is s

lide

DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 29

Least Mean Squares (LMS) Algorithm

DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 30

Least Mean Squares (LMS) Algorithm

DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 31

Least Mean Squares (LMS) Algorithm

DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 32

Least Mean Squares (LMS) Algorithm

DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 33

Least Mean Squares (LMS) Algorithm

DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 34

Least Mean Squares (LMS) Algorithm