1 A Fast-Nonegativity-Constrained Least Squares Algorithm R. Bro, S. D. Jong J. Chemometrics,11,393-401, 1997 By : Maryam Khoshkam.

11

A Fast-Nonegativity-A Fast-Nonegativity-Constrained Least Squares Constrained Least Squares

Algorithm Algorithm

R. Bro, S. D. JongR. Bro, S. D. Jong

J. Chemometrics,11,393-401, 1997J. Chemometrics,11,393-401, 1997

By : Maryam Khoshkam

22

IntroductionIntroduction

Algorithm of classical non-negative Algorithm of classical non-negative least square (nnls) least square (nnls)

A numerical exampleA numerical example

Fast nonegative least square (Fnnls)Fast nonegative least square (Fnnls)

Results and discussionResults and discussion

33

introductionEstimation of models subject to non-negativity constraints is of practical importance in chemistry.

The time required for estimating true least squares non-negativity constrained model is typicaly many times longer than for estimating unconstrained model.

Approximations procedures

Unconstrained estimation+ Setting negative value to zero

What is the problem of force to zero algorithm????

44

1. There is no guarantee whatsoever for the quality of the model.

2. When included in multiway algorithm, it can cause the algorithm to diverge. (specially, in noisy data or difficult models)

Non-negativity constrained linear least square

55

NNLS

2

)1()()1()( MMLLf dZxd

NNLS will be stated using the following nomenclature:

0md For all m

Lawson, C.L. and R.J. Hanson, Solving Least-Squares Problems, Prentice-Hall, Chapter 23, p. 161, 1974.

lsqnonneg command in MATLAB is based on this algorithm

)1()()1( MMLL dZx

66

Some important aspects of the algorithm

•This is an active set algorithm.

Thus the nnls first find the true passive and active sets, then perform the least square on corresponding columns of Z.

HOW…????

First of all, it suppose that all elements of d, are in active set.

R={1,2,….,M}

P={}

If the all elements of d are active, estimate that how

the form of initial d….?????

77

•Initial solution vector d is feasible and set equal to a M×1 zero vector

•Using the vector w

•Then it removes the passive sets elements of d, one by one, and STEP by STEP

HOW…????

)( )1()()1()()1( MMLLT

MLM dZxZw

88

2

)1()()1()( MMLLf dZxd )(2)(' xZdZd Tf

w=ZT ( x – Z d )

)('2

1mm dfw

One necessary condition for optimally of a solution , is that the derivatives with respect to the parameters of the passive set be zero. WHY ??

How is the f ’ (dm) if m is an active set ????

99

Rmw

Pmw

m

m

0

0

00)('min)( mm wdffmif dP

00)('0)( mm wdffmif dR

Thus at optimal solution, we expect that:

When we are being at optimal condition, wm >0. wht is the meaning of positive value for wm?

1010

positive wm shows that by increasing dm to a more positive value, the change in residual is negative (f ', which is the slope is negative).

It means that the residual becomes less, and more close to zero, when dm goes toward positive.

1111

Algorithm nnls 1. P=Ø2. R={1,2,…,M}3. d=04. w=ZT(x-Zd)

loop A

No

Opt. sol. Of d

yes

tolwR nRn

)(max&

)(maxarg nRn

wm

P={m} & R=R-{m}

SP=[(ZP)TZP]-

1(ZP)TxNo

0min Ps

d=s

w=ZT(x-Zd)

yesLoop C all sp>0

Loop BBuild sp

1212

A simple numerical example

718980

7272

467487

527173

)34(Z

20

68

67

49

)14(x

loop A

7526

10353

15902

0

0

0

}3,2,1{

{};

w

d

R

p

Max(wn)>1e-15 & R≠ØRn

Loop B

1313

Loop B

6495.0

};3,2{

};1{

ps

R

p

0min Ps

No

1555.8-

1913.5-

0

0

0

6495.0

w

sd

sMax(wn)>1e-15 & R≠Ø

RnOpt. sol. Of d

1414

Graphical representation of nonlinear least square algorithm:

H3A ↔ H+ + H2A- pka1=2.6

H2A- ↔ H+ + HA2- pka2=4.0

HA2- ↔ H+ + A3- pka3=6.3

0 2 4 6 8 10 120

0.002

0.004

0.006

0.008

0.01

0.012

0.014

0.016

0.018

0.02Concentration Profiles

pH

Con

cent

ratio

n

300 350 400 450 500 550 600 6500

0.2

0.4

0.6

0.8

1

1.2

1.4spectral Profiles for H2A

wavelength

abso

rbtio

n

D=X+.Z (ILS)

Z D

0 50 100 150 200 250 300 350 400-0.1

-0.05

0

0.05

0.1

0.15

300 350 400 450 500 550 600 6500

0.002

0.004

0.006

0.008

0.01

0.012

0.014

0.016

0.018

0.02

X D̂

1515

X(10x100) = Z(10x4)

D(4x100)

x10x1=X(:,1)

Step1

d(4x1)

0

0

0

0

)3 d

)()4 )14( ZdxZw T

1 1.5 2 2.5 3 3.5 40

1

2

3

4

5

6

7

8

9x 10

-14

w

1 1.5 2 2.5 3 3.5 40

0.2

0.4

0.6

0.8

1

1.2

1.4x 10

-13

0.005 x D(:,1)

1) P=[];

2) R={1,2,3,4}

1616

0 50 100 150 200 250 300 350 4000

0.2

0.4

0.6

0.8

1

1.2

1.4

0 50 100 150 200 250 300 350 4000

0.02

0.04

0.06

0.08

0.1

0.12

0.14

Force to zeronnls

Variation of w during nonnegative least square

Rmw

Pmw

m

m

0

0

1717

300 350 400 450 500 550 600 650-6

-4

-2

0

2

4

6

8

10

12

14x 10

-13

Residual matrix

300 350 400 450 500 550 600 650-0.5

0

0.5

1

1.5

2

2.5

3x 10

-13

w

w=ZT(x-Zd)

=

w(4×1) r(10x1)Z (10x4)T

rw ii :),(Z

wi in each λ, is the contribution of ith species in in residual vector in each λ.

1818

Why it is necessary to modify theNNLS algorithm?

1919

PARAFAC-ALS

EBCAX )'()( JKI

I

J

K

2

,,)'(min BCAX

CBA

Least square

BCZ )( FJK

2020

1)()()()( )'(ˆ

ZZXZAEZAX JKIFJKT

FIJKI

If the size of X is 10x200x5 in a 3 component system, the size of Z is:

500x3 And so on….

Computation of Z, can be computationally costly for large arrayes

Excessive memory is required to calculate X(IxJK), X(JxIK) and X(K,IJ)

2121

1)'(ˆ ZZXZA

KK

K

BDXBDX

BDBDXXXZ K1

...

]'...'][...[

11

'1

)'(*)'()()'(' BBCCBCBCZ Z

)'(*)'(ˆ1

BBCCBDXA

K

kkk

And in similar way for estimation of B and C

2222

Nnls can not used for this simplified version of PARAFAc, Why??

2323

In estimating A, and using non-negativity constraint on it,

iFiFJKi eaZx )1()(

A(IxF) X(IxJK)

Z

=

SLOW!!!!!!!A modification…

2424

Fast non-negativity least square

1. Accept the cross product (xZ and Z’Z) instead of the row data

w=ZT ( x – Z d ) w=ZTx – (ZTZ)d

SP=[(ZP)TZP]-1(ZP)Tx SP=[(ZTZ)P]-1(ZTx)

2. Set the passive and active variables, before enter to loop B

d is not the zero vector in this case

2525Thanks

2626

Ex.

d=Z\x

Force to zero

RMS=103

718980

7272

467487

527173

Z

20

68

67

49

x

068.2917.0123.1 d

0917.0123.1dnnlsq

00650.0d

RMS=20

2727

068.2917.0123.1 d

3th element of d

In unconstrained solution of d, the mth negative element is active set

00650.0d

In constrained solution of d, the mth zero element is active set

2th and 3th elements of d

What happen if the true active set is known?

Thus it is not possible to realize the true active set form unconstraint least square solution.

2828

For example, if we know that 2th and 3th elements of d are true active sets,

0

01d

d

Then perform the simple unconstrained least square with the 1st column of Z. (columns corresponding to passive sets. selectivity constraint !!!!

NOTE!!!!! 1)

2) f(d) minimized by columns of Z corresponding to passive sets. Thus active sets which for them, dm=0, is not take a part

in minimizing the residuals

R

P

md

md

m

m

if0

if0

2929

Loop C

nn

n

n sd

dP

min

)( dsdd

Update R and P

SP=[(ZP)TZP]-1(ZP)Tx

1)

2)

3)

4)

SR=05)