IEEE UKRI Talk, De Montfort Univ., April 2, 04 1 Open Issues in Constrained Blind Source Separation Jonathon Chambers Cardiff Professorial Research Fellow.

IEEE UKRI Talk, De Montfort Univ., April 2, 04

1

Open Issues in Constrained Blind Source Separation

Jonathon ChambersCardiff Professorial Research Fellow

Cardiff School of EngineeringCardiff University, Wales, U.K.

E-mail: [email protected]


2

Summary of Talk

• Acknowledgement

• Historical background & motivation

• BSS with matrix constraints

• Penalty functions in FD-BSS

• Exploiting periodicity in BSS

• Future application-driven challenges


3

Acknowledgements Jonathon Chambers wishes to express his sincere thanks

for the support of Professor Andrzej Cichocki, Riken Brain Science Institute, Japan

The invitation from the organising committee of the workshop to give this talk.

His co-researchers: Drs Saeid Sanei, Maria Jafari and Wenwu Wang.


4

LMS Algorithm

B. Widrow, and M.E. Hoff, Jr.,

“Adaptive switching circuits,” IRE Wescon Conv. Rec., pt. 4, pp. 96-104, 1960.

LMS Update

t)(xe(t) t)(w 1)t(w


5

Historical Background

• The field of conventional adaptive signal processing has been greatly enhanced by the exploitation of constrained optimisation

• Constraints on the error, and/or structure or some norm of the weights via, for example, Lagrange multipliers and/or Karush-Khun-Tucker conditions


6

Historical BackgroundCertain key papers:

• O.L. Frost, III, “An algorithm for linearly constrained adaptive array processing,” Proc. IEEE, Vol. 60(8), pp. 926-925, 1972

• R.P. Gitlin et al. “The tap-leakage algorithm: an algorithm for the stable operation of a digitally implemented fractionally spaced equalizer,” Bell Sys. Tech. Journal, Vol. 61(8), pp. 1817-1839, 1982.

• D.T.M. Slock, “Convergence behavior of the LMS and Normalised LMS Algorithms,” IEEE Trans. Signal Processing, Vol. 41(9), pp. 2811-2825, 1993.


7

Historical Background Cont.• S.C. Douglas, “A family of normalized LMS algorithms,”

IEEE Signal Processing Letters, Vol. 1(3), pp. 49-51, 1994.

• S.C. Douglas, and M. Rupp, “A posteriori updates for adaptive filters,” Asilomar Conference on Signals, Systems and Computers, Vol. 2, pp 1641-1645, 1997.

• T. Gänsler, et al., “A robust proportionate affine projection algorithm for network echo cancellation,” Proc. ICASSP 2000, Vol. 2, pp. 793-796, 2000.

• O. Vainia, “Polynomial constrained LMS adaptive algorithm for measurement signal processing,” Proc. IECON 2002, Vol. 2, pp. 1479-1482, 2002.


8

Motivation

“In many applications of Independent Component Analysis (ICA) and Blind Source Separation (BSS) estimated source signals and the mixing or separating matrices have some special structure or some constraints are imposed for the matrices…”, Cichocki and Georgiev, 2003


9

H

s1

sN

W

x1

xM

Y1

YN

Unknown

Known

Independent?

Adapt

Mixing Process

Unmixing Process

Fundamental Model for Instantaneous Blind Source

Separation


10

Certain BSS Books

• Andrzej Cichocki and Shun-Ichi Amari, Adaptive Blind Signal and Image Processing, Wiley, 2002

• Simon Haykin Unsupervised Adaptive Filtering, Vols. I and II, Wiley, 2000

• Aapo Hyvärinen, Juha Karhunen and Erkki Oja, Independent Component Analysis, Wiley, 2001

• Te-Won Lee, Independent component analysis: theory and applications, Kluwer, 1998


11

BSS References

• A. Mansour and M. Kawamoto, “ICA Papers Classified According to their Applications and Performances”, IEICE Trans. Fundamentals, Vol. E86-A, No. 3, March 2003, pp. 620-633.

• In 2002, 800 different papers have been published, these are downloadable at http://ali.mansour.free/REF.htm

http://ali.mansour.free/REF.htm

http://ali.mansour.free/REF.htm


12

With a symmetric mixing matrix [C&G,2003]:-

Txxx

Txx

Tx

-xx

-xx

-1opt

VV t)}(xt)(xE{ R

where

VV R H W 21

21

BSS With Matrix Constraints


13

Stable Frobenius norm of the separating matrix

Theorem [C&G 2003]: The learning rule

W(t)t)I(t)(yF(t) dt

dW(t)N

where β > 0 is a scaling factor and

γ(t) = trace(WT(t)F(y(t))W(t)) > 0,

stabilizes the Frobenius norm of W(t) such that

1T2

Ft)W(t)(W traceW(t)

BSS With Matrix Consts. Cont.


14

Consequence: The modified NG descent learning

algorithm, with a forgetting factor, described as

t)W(t)(t)W(t)(WW

J(W)(t)-

dt

dW(t) T

with γ(t) = -trace(WT(t)[J(W)/ W]WT(t)W(t)) > 0

has a W(t) with bounded Frobenius norm throughout

the learning process.



15

Prof. Amari’s “Leaky” NG Algorithm becomes

W(t)t)(yt))(yf(I)(

W(t)(t)(t)-1 1)W(tT

N

t

where 0 << (1-βγ(t)η(t)) < 1 is the leakage factor



16

Introducing a semi-orthogonality constraint so that it is possible to extract an arbitrary group of sources, say e, 1 e N.

Assuming pre-whitened data

Qx x and I }xxE{R N

T

XX

and the mixing matrix A = QH, the demixing

matrix We should satisfy WeA = [Ie,0N-e]



17

A natural gradient algorithm to find We becomes:-

t)(Wy(t)][y(t)ft)(xf[y(t)](t)

- t)( W )1t(W

eTT

ee

With initial conditions which satisfy

eTee I )0(W)0(W



18

)(1 ts

)(2 ts

)(2 tx

)(1 txM icrophone1

M ic rophone2

S peaker1

S peaker2

Real Convolutive Mixing Env. – Cocktail Party Problem


19

)(

)(

)(

)(

tx

tx

ts

ts

tHtH

tHtH

MNMNM

N

11

1

111

)()(

)()(

ConvolutionsHx *

N

j

P

pjij ptsphtx

1

1

0

)()()(

Compact form:

Expansion form:

Convolutive BSS – Model


20

Taxonomy of Existing Sols. To Convolutive BSS

• Performing blind separation in the time domain by extending the existing instantaneous methods to conv. case

• Transforming the convolutive BSS problem into multiple instantaneous (complex) problems in the frequency domain

• Decomposing the system into smaller problems using, for example, a subband approach

• Hybrid frequency and time domain approaches


21

)(

)(

)(

)(

NMNM

N

M s

s

HH

HH

x

x

1

1

1111

)()(

)()(

sHx *DFT

Convolutive BSS problem

Multiple complex-valued instantaneous BSS problems

Transform Convolutive BSS into the Frequency Domain


22

In the frequency domain:-

Mathematical Formulation

NTN1

NTN1

C t),(Xt),...,,(X t),X(

and

C t),(St),...,,(S t),S(

where

t),V( t),)S(H( t),X(


23

De-mixing Operation

independ.

mutually become t),(Yt),...,,(Y

such that determined )in W( Parameters

N1

TN1

NN

t)],(Yt),...,,([Y t),Y( and

C) W(where

t),X() W( t),Y(


24

Constrained Optimisation and Joint Diagonalisation

0 )wg( s.t. )wf(min :becomes Prob1 and,

]w,...,w,...,w,...,w,w,...,[w

vec(W) w form,in vector or,

1 r and RC:f

RC:W)](gW),...,(gW),([g W)(g

where

Prob1 - 0 W)(g s.t. f(W)min

NMNM1MN212N111

1MN

rMNTr21

C


25

Joint Diagonalisation Criterion

2

FYY

T

1

K

1tW

HVXY

t)],([R-t),(R (W) where

t),(W)(min arg ))(W(

-:FunctionCost

)(Wt),(R-t),(R) W( t),(R

diagF

FJ

Exploiting the non-stationarity of speech signals measured by the cross-spectrum of the output signals,


26

Exterior Penalty Function Approach

.set for the functionspenalty

exterior of sequence a form willW)(Then

.or 2, 1, b R,C:W)(W)( if And

1. let and ,q as ,

and 0such that be let , qFor

0}.W)(|C{W : and

continuous is RC:g(W) If :

q

MN

bqq

q

q1qq

MN

rMN

W

U

gU

N

gW

Lemma


27

Exterior Penalty Function Approach

Typical exterior penalty functions, and the shadow area represents the feasible set.


28

Proposed General Cost Function

.)W(min argbecomes problem separation theThus,

functions. weightingnonneg theare ],...,[ and

))]W(()),...,W(([ )) U(W(where

))U(W()W( )W(

W

Tr1

Tr1

T

new

new

J

UU

JJ

With a factor vector κ to incorporate exterior penalty functions, our cost function becomes:-


29

Numerical Experiments• Use an exterior penalty function

2

F

H I-)()WW(

• Employ a variant of gradient adaptation

• Utilize the filter length constraint to address the permutation problem (Parra & Spence)

• System with two inputs and two outputs (TITO!)

• H(z) = [{1 1.9 -0.75}, z-5{0.5 0.3 0.2}; z-5{-0.7 -0.3 -0.2}, {0.8 -0.1}]; D = 7, T = 1024, K = 5.


30

Convergence Performance of the New Criterion as a function of κ


31

Room Environment Experiment• Use roommix function due to Westner

• Room 10x10x10m3 cube

• Wall reflections calculated up to fifth order, atten. factor 0.5

• SIR is plotted as a function of length of the separating system

ji

2

j

2

ij

i

2

i

2

ii

)(s)(H

)(s)(H10log s]SIR[H,


32

Room Environment


33

Room Environment SIR


34

S1

S2

x1

x2

FDICA

1

2

P

)(ˆ1 S

)(ˆ2 S

S1×0.5

S2×1

S2 ×0.3

S1 ×1.2

S2×0.6

S1×0.4

Permutation Problem in FD-CBSS


35

Summary of Existing Solutions to Permut. Problem in FD-CBSS

• Constraints on the filter models in the frequency domain

• Using special structure contained in signals

• Merging beamforming view to align solutions

• Exploiting the continuity of the spectra of the recovered signals – could coupled hidden Markov Models be used?

• What happens when the sources move, enter/re-enter the environment? What is the way forward?


36

Exploiting Source (Pseudo) -Periodicity

• W. Wang, M.G. Jafari, S. Sanei, and J.A. Chambers, “Blind source separation of convolutive mixtures of cyclostationarity”, to appear in the Special Issue on BSS, International Journal of Adaptive Control and Signal Processing, Guest Editor: Mike Davies, Queen Mary’s College, University of London

• H. Swada, R. Mukai, S. Araki, and S. Makino, “A robust and precise method for solving the permutation problem of frequency-domain blind source separation”, ICA 2003, Nara, Japan, 2003, pp. 505-510.


37

A natural gradient update exploiting cyclostationarity

kkkkkkk yT WRyyfIWW ~21

• The Cyclostationary NGA uses the update equation

where

and p is the cycle frequency of the p-th source

m

p

kjHm

p

pp ekkEkk11

~ yyRR yy


38

A natural gradient update exploiting periodicity

• The Periodic NGA type update equation

where

kk,TTk

k,TTkkkk

iiy

iiHy

WySR

ySRIWW

,

,21

1

iyi

iH

iy

Tkdiagsignk,T

TkkTk

, and

,

RyS

yyR


39

Emerging Applications Biomedical:-

ECG, EEG, MEG and their integration

Microarray time courses

Measurements from the nano-lab

http://www.nmrc.ie/research/transducers-group/trends.htmlhttp://www.nanospace.systems.org/ns_2000/NS00_Sessions.htmhttp://nanomed.ncl.ac.uk/m2l.htm

Star Trek: The Tri-corder


40

Emerging Applications • T. Bowles, J. Chambers, and A. Jakobsson, “Advanced spectral estimation for the identification of cell-cycle regulated genes”, IEEE EMBS UK and RI Postgraduate Conf in Biomedical Engineering and Medical Physics, 2003.

• X. Liao, and L. Carin, “Constrained independent component analysis of DNA microarray signals”, IEEE Workshop on Genomic Signal Processing and Statistics, 2002.

• S-I, Lee, and S. Batzoglou, “Discovering biological processes from microarray data using independent component analysis”, Dept EE/CS, Stanford Univ.


41

Summary • The exploitation of constrained optimisation has been fundamental to the development and application of adaptive signal processing; this process is, however, very much in its infancy in blind source separation (BSS).

• Utilisation of certain a priori knowledge on the mixing matrices and the properties of the sources is likely to yield solutions to real-life SP problems.

• As such, the challenge for DSP engineers in the 21st Century, is to advance the application of BSS methods in line with methods from adaptive signal processing.


42

Other References • A. Cichocki, and P. Georgiev, “Blind source separation algorithms with matrix constraints”, IEICE Trans. Fundamentals, Vol. E86-A(3), March 2003, pp. 522-531.

• J.G. McWhirter, “Mathematics and signal processing”, Mathematics Today, April 2003, pp 47-54.

• W. Wang, S. Sanei, and J. Chambers, “Penalty function based joint diagonalization approach for convolutive blind source separation”, submitted to IEEE T-SP, Sept 2003.


43

Close ???

Mark Twain

“A man who swings a cat by its tail learns things he can learn no other way”

IEEE UKRI Talk, De Montfort Univ., April 2, 04 1 Open Issues in Constrained Blind Source Separation Jonathon Chambers Cardiff Professorial Research Fellow.

Documents

ieee ukri talk

ieee signal processing

lms update slide

adaptive blind signal

blind source separation

bss references

adaptive array processing

adaptive filters