muser

8/2/2019 muser

1/11

Machine Vision and Applications (1993) 6:140-150 M a c h i n e V i s i o n a n dA p p l i c a t i o n s9 Springer-Verlag1993

MUSER: A prototype mus ica l s core recogn i t ion sys temu s i n g m a t h e m a t i c a l m o r p h o l o g yB h a r a t h R . M o d a y u r , V i s v a n a t h a n R a m e s h , R o b e r t M . H a r a l i c k , a n d L i n d a G . S h a p i r oIntelligent Systems Lab oratory, Electrical Engineering D epartment, FF -10, University of Washington, Seattle WA 98195, USA

A b s t r a c t . M u sic r ep resen t a t i o n u t i l i zes a f a i r l y r i ch r ep er -t o i re o f s y m b o l s . T h e s e s y m b o l s a p p e a r o n a s c o r e s h e e t w i t hre l a t i v e ly l i tt l e sh ap e d i s t o r t io n , d i f f e r i n g f ro m th e p ro to ty p esy mb o l sh ap es main ly b y a p o s i t i o n a l t r an s l a t i o n an d sca l ech an g e . Th e p ro to ty p e sy s t em we d esc r i b e i n t h i s a r t i c l e i sa imed a t r eco g n i z in g p r i n t ed mu s i c n o t a t i o n f ro m d ig i t i zedm u s i c s c o r e i m a g e s . T h e r e c o g n i t io n s y s t e m i s c o m p o s e d o ft w o p a r t s : a l o w - l e v e l v i s i o n m o d u l e t h a t u s e s m o r p h o l o g i c a la lg o r i t h ms fo r sy mb o l d e t ec t i o n an d a h ig h - l ev e l mo d u le t h a tu t i l izes p r i o r k n o w led g e o f mu s i c n o t a t i o n t o r easo n ab o u tsp a t i a l p o s i t i o n s an d sp a t i a l s eq u en ces o f th ese s y mb o l s . Th eh i g h - l e v e l m o d u l e a l s o e m p l o y s v e r i f i c a t i o n p r o c e d u r e s t oc h e c k t h e v e r a c i t y o f th e o u t p u t o f t h e m o r p h o l o g i c a l s y m b o lr e c o g n iz e r . T h e s y s t e m p r o d u c e s a n A S C I I r e p r e s e n ta t i o n o fmu s i c sco res t h a t can b e i n p u t t o a mu s i c - ed i t i n g sy s t em.M a t h e m a t i c a l m o r p h o l o g y p r o v i d e s us t h e t h e o r y a n d th eto o l s t o an a ly ze sh ap es . Th i s ch a rac t e r i s t i c o f ma th emat i ca lm o r p h o l o g y l e n d s i t s e l f w e l l to a n a l y z i n g a n d s u b s e q u e n t lyreco g n i z in g mu s i c sco res t h a t a r e r i ch i n we l l -d e f i n ed mu -s i ca l sy mb o l s . S in ce mo rp h o lo g i ca l o p era t i o n s can b e e f f i -c i e n t l y i m p l e m e n t e d i n m a c h i n e v i s i o n s y s t e m s t h a t h a v esp ec i a l h a rd ware su p p o r t , t h e r eco g n i t i o n t a sk can b e p e r -f o r m e d i n n e a r r e a l - t i m e . T h e s y s t e m a c h i e v e s a c c u r a c y i ne x c e s s o f 9 5 % o n t h e s a m p l e s c o r e s p r o c e s s e d s o f a r w i t ha p e a k a c c u r a c y o f 9 9 . 7 % f o r t h e q u a r t e r a n d e i g h t h n o t e s ,d e m o n s t r a t i n g t h e e f f i c a c y o f m o r p h o l o g i c a l t e c h n i q ue s f o rsh ap e ex t r ac t i o n .

K e y w o r d s : M a t h e m a t i c a l m o r p h o l o g y - L o w - l e v e l v i s io n -F e a t u r e e x tr a c t io n - C o n n e c t e d c o m p o n e n t s - M u s i c n o t a t i o n

1 I n t r o d u c t i o nTh e p e r fo rman ce o f e f f i c i en t a l g o r i t h ms t o r eco g n i ze i so l a t edtwo -d imen s io n a l sh ap es d e t e r i o r a t es r ap id ly as t h e sh ap esb e c o m e i n c r e a s in g l y i n t e rc o n n e c te d . M u s i c s c o r e r e p r e s e n -t a t i o n i s o n e su ch d o main wh ere t h e d i f f e r en t sy mb o l s a r e

9 r r an g ed i n a t r u ly two -d im en s io n a l f a sh io n . U t i l iza t i o n o fCorrespondence to: Robert Haralick

p r i o r k n o w l e d g e a b o u t m u s i c r e p r e s e n t a t i o n b e c o m e s n e c -es sa ry t o d ev i se e f f i c i en t me th o d s t o r eco g n i ze t h e mu s i cs y m b o l s .

A mu s i ca l - sco re l ay o u t i s t h e v i su a l man i f es t a t i o n o f i n -t e r r e l a t ed p ro p er t i e s o f mu s i ca l so u n d - p i t ch , i n t en s it y , t ime ,t imb re an d p ace (Read 1 9 6 9 ) . S y mb o l s i n d i ca t i n g t h e t o n es ,t h e i r d u r a t i o n a n d t h e m a n n e r o f p e r f o r m a n c e f o r m t h e m u -s i c n o t a t i o n . T h e b a s i c s y m b o l s w e a r e c o n c e r n e d w i t h i nth i s p ap er i n c lu d e t h e :- S t a f f - an a r r an g em en t o f f i v e p a ra l l e l li n es t o g e th e r wi th

s p a c e s b e t w e e n t h e m- C le f s - sy mb o l s t h a t d e t e rm in e th e p i t ch as so c i a t ed wi th

a p a r t i cu l a r s t a f f- N o t e h e a d s a n d s t e m s - s y m b o l s t h a t d e t e r m i n e a n o t e ' s

t i m e v a l u e- F l a g s a n d b e a m s - o t h e r s y m b o l s th a t d e te r m i n e t h e t i m e

v a lu e o f a n o t e- R e s t s a n d p a u s e s - s y m b o l s th a t in d i c a te m o m e n t s o f s i -l en ce i n a mu s i ca l p i ece

- Acc id e n t a l s an d k ey s i g n a tu res - sy m b o l s th a t a r e p l acedin f ro n t o f a n o t e t o m o d i fy i t s p it ch , i . e ., t o r a i se o r l o w erth e p i t ch

- Bar l i n es - t h in v e r t i ca l l i n es d rawn th ro u g h t h e s t a f f to se to f f t he t i m e l e n g t h o f e a c h m e a s u r e

M a t h e m a t i c a l m o r p h o l o g y p r o v i d e s u s t h e t h e o r y a n d t h eto o l s t o an a ly ze sh ap es . P r i n t ed mu s i c sco re i s r i ch i n we l l -d e f i n ed , b u t i n t e r co n n ec t ed sh ap es . Th e sh ap e-b ased ap -p r o a c h o f m a t h e m a t i c a l m o r p h o l o g y t h u s l e n d s i t s e l f w e l lt o b e in g u sed as an e f f ec t i v e to o l i n u n d er s t an d in g p r i n t edmu s i c sco re .

T h e p r o t o t y p e s y s t e m p r o c e e d s t h r o u g h t h re e d i s t in c ts t ag es t o r eco g n i ze mu s i c sco res . Th ey a r e : l a y o u t e x t r a c -t io n , s y m b o l r e c o g n i ti o n , a n d h i g h - l e v e l r e a s o n i n g . In t h ef i r st s tag e , t h e sk ew an d t h e sca l e o f t h e imag e i s d e t e rmin edb y l o ca t i n g t h e p o s i t i o n o f t h e s t a f f l i n es . Th i s i s ach i ev edb y a c o m b i n a t i o n o f m o r p h o l o g i c a l a n d i m a g e p r o c e s s i n gt ech n iq u es . I n t h e seco n d s t ag e t h e d i f f e r en t sy mb o l s a r e ex -t r a c te d u s i n g s e q u e n c e s o f m o r p h o l o g i c a l o p e r a t i o n s . P r i o rk n o w l e d g e o f m u s i c r e p r e s e n t a t i o n i s u t il i ze d i n m a k i n g t h ed e t ec t i o n a lg o r i t h ms s imp le as we l l a s e f f i c i en t . I n t h e l a s ts t ag e , p r i o r k n o w led g e o f mu s i c r ep resen t a t i o n (Rea d 1 9 6 9 )

8/2/2019 muser

2/11

a n d t h e s h a p e s o f t h e s y m b o l s i s m a d e u s e o f t o r e a s o nabou t the s pa t i a l pos i t ions and s equences o f thos e s ymbols .T h i s h i g h - l e v e l m o d u l e a l so e m p l o y s v e r i fi c a ti o n p r o c e d u r e st o c h e c k t h e v e r a c i t y o f t h e o u t p u t f r o m t h e m o r p h o l o g i -c a l s y m b o l r e c o g n i z e r ( F i g . 1 ). T h e r e m a i n d e r o f t h e p a p e ri s o rgan ized a s fo l lows . Sec t ion 2 ou t l ine s the p rob lem wea re t ry ing to s o lve and the f ina l goa l . Sec t ion 3 b r i e f ly d i s -cus s e s re l a t ed work . Sec t ion 4 g ives a ba s ic in t roduc t ion tothe morp ho log ica l ope ra t ions and the no ta t ions us ed . Sec t ion5 ou t l ine s the p r io r a s s umpt ions . Sec t ions 6 and 7 d i s cus sthe l ayou t ex t rac t ion and the s ymbol recogn i t ion s t ra t egy .Sec t ion 8 g ives a de ta i l ed look in to a few o f the rep re s en-t a t ive a lgor i thms fo r the s ymbol de tec t ion s t age . Sec t ion 9des c r ibe s the h igh- l eve l rea s on ing modu le and exp la ins thep o s t p r o c e s s in g f o r a f e w i m p o r t a n t c l as s e s o f s y m b o l s . S e c -t ion 10 d i s cus s e s the expe r im en t s and the re s u l t s ob ta ined s ofar .

2 T h e p r o b l e m a n d t h e g o a lW e h a v e a k n o w n r e p r e s e n t a t i o n o f m u s i c , a n i m a g e o fprinted score sheet 9 W e a l s o h a v e p r i o r k n o w l e d g e o f t h eva r ious s ymbols tha t c an appea r on the s co re , the s e t o fp o s s i b l e p o s i t i o n s d i f f e r e n t s y m b o l s c a n o c c u p y , a n d s o m ed e f i n it e s e q u e n c e s o f s y m b o l s . T h e p r o b l e m i s t h e n t o r e c o g -n ize a l l t he mean ingfu l s ymbols on the image , the i r s pa t i a lpos i t ions , and the s pa t ia l s equenc e o f the s e s ymbols ( s ee Mo-day ur 1991 fo r an ea r l i e r ve rs ion o f th i s work) 9 I f the re a re Ns y m b o l s $1, $ 2, . . . . SN w h i c h o c c u r i n t h e im a g e i n u n k n o w n

9 9 Nl oca t ions , we can rep re s en t the image a s I = Ui= l (S i )~ , v~ ,whe re the S i do no t in t e r s ec t ( th i s cond i t ion i s re l axed fo rs ym bols l ike s t a f f l i ne and no te s t ems w hich in te r s ec t o the rs ymbols ) 9 T he p urpos e o f the m orpho log ica l s eq uence o fope ra t ions i s to ex t rac t the ind iv idua l s ymbols w i th the i r r e -s pec t ive t rans la tions . L e t us a s s ume tha t the f i r s t s ymb ol , $1h a s b e e n e x t r a c t e d a n d r e m o v e d f r o m I . N o w , t h e r e s i d u eN Simag e I~cs can be rep re s en ted a s I~es = Ui=2( i )~ , u~ . T h es y m b o l e x t r a c t i o n p r o c e e d s u n t il I h a s b e e n d e c o m p o s e dc o m p l e t e l y .

T he goa l i s to rep re s en t ou r recogn i t ion re s u l t a s a s e -q u e n c e o f t e x t s y m b o l s a s f o l l o w s :3 C 4 3 D 8 [ 4 C 4 4 E 4 ]T h i s rep re s en ta t ion ind ica te s a C no te in the th i rd oc tave( th i s i s the C be low midd le C) tha t i s a qua r t e r no te , fo l lowe d

by a D no te in the th i rd oc tave tha t i s an e igh th no te . T henthe re i s a chord cons i s t ing o f a qua r t e r no te C in the four thoc tave and a qua r t e r no te E in the four th oc tave .

S i n c e M U S E R i s d e s i g n e d t o w o r k o n p r i n t e d f o r m s o fmu s ic rep re s en ta t ion , it en joys the advan tage o f encoun te r ings ymb ols o f regu la r s hape . In th i s rega rd the p rob lem i s we l ld e f i n e d . O n c e t h e i m a g e i s n o r m a l i z e d ( w h i c h c a n b e d o n ei f the s pac ing be twe en the s t a f f li ne s i s know n) the re l a t ives ize s o f the va r ious s ym bols can be de te rm ined . T h i s i s he lp -fu l e s pec ia l ly in c rea t ing the appropr ia t e ly s i zed s t ruc tu r inge lemen t s .

1413 R e l a t e d w o r kT he doc to ra l d i s s e r t a t ions o f Prus l in (1966) and Pre rau( 1 9 7 0 ) u s e d c o n t o u r - t r a c i n g te c h n i q u e s f o r s y m b o l s e g m e n -ta t ion and hence s t a f f l ine s epa ra t ion was p rob lema t i c . T hegaps c re a ted a s a re s u l t o f s t a f f l i ne remov a l we re f i l led andt h e a s p e c t r a t io o f s e g m e n t e d s y m b o l s a l o n g w i t h s o m e m u -s ic ru le s we re us ed to recogn iz e the s ymbo ls . Fu j inag a ' s the -s i s work (1988) us ed p ro jec t ion p ro f i l e t e chn iques fo r s t a f fl ine de tec t ion a s we l l a s s ymbol s egmenta t ion and recogn i -t ion . T he p ro jec t io n t echn iques to le ra te s kew in the s t a f f l i neor i en ta t ion . T h i s w ork a s s um ed tha t the s i ze s o f the d i f fe ren tmu s ic s ym bols in a g iven s co re s hee t a re re l a tive . T he re s u l t sw e r e r e p o r t e d f o r f o u r c a r e f u l l y c h o s e n s c o r e s a m p l e s . T h er e c o g n i t i o n a c c u r a c y w a s 7 3 % o n t h e a v e r a g e .

T h e E N G R A V E s y s t e m ( f o r h a n d w r i t t e n m u s i c s c o r er e c o g n i t io n ) r e p o r t e d i n R o a c h a n d T a t e m ( 1 9 8 8 ) e x p l o i t e dthe fac t tha t mus ic rep re s en ta t ion i s a h igh ly s t ruc tu red do-m a i n a n d u t i l iz e d k n o w l e d g e o f m u s i c t o a c h i e v e s i g n i fi c a n ti m p r o v e m e n t s i n l o w - l e v e l im a g e p r o c e s s i n g t o e x t r a c t p r i m -i t ive fea tu re s . T he s ys tem proc es s ed s ix images o f handw r i t -t en s co re s and the re s u l t s we re repor ted to be good . T hes y s t e m r e p o r te d b y B a u m a n n a n d D e n g e l ( 1 9 8 7 ) u s e d a t o p -down s t ra t egy in o rde r to t ake advan tage o f the h ie ra rch i -ca l s t ruc tu re o f mus ic rep re s en ta t ion . Us ing mus ic ru le s andcons t ra in t s , i t app l i ed a dec i s ion t ree c l a s s i f i e r to recogn izem u s i c s y m b o l s w i t h a n a c c u r a c y o f a b o u t 9 0 % .

M U S E R u t i l i z e s m u s i c r u l e s a n d c o n s t r a i n t s t o s o m ee x t e n t i n t h e l o w - l e v e l s y m b o l e x t r a c t i o n s t a g e a n d t o al a rge ex ten t in the h igh- l eve l rea s on ing s t age . T he low- leve ls t age us e s morpho log ica l s equences in a f e a t u r e e x t r a c t i o n -t o p o l o g y m a t c h i n g s t ra t egy ( to be exp la ined l a t e r ) to de tec tthe mus ica l s ymbo ls . S ta f f l i ne ex t rac t ion , wh ich i s a l eng thya n d o f t e n t r o u b l e s o m e p r o c e s s i n o t h e r s y s te m s , i s a c h i e v e db y a s i n g l e m o r p h o l o g i c a l o p e r a t io n . T h e h i g h - l e v e l m o d u l eu s e s p r i o r k n o w l e d g e a b o u t m u s i c r e p r e s e n t a t i o n a n d , u s -i n g t h e o u t p u t o f t h e s y m b o l r e c o g n i z e r , g e n e r a t e s a n A S C I Ir e p r e s e n ta t i o n o f t h e m u s i c s c o re . T h i s m o d u l e a l s o e m p l o y sv e r i fi c a t io n p r o c e d u r e s t o c h e c k t h e v e r a c i t y o f t h e l o w - l e v e lo u t p u t. T h e o v e r a l l s y m b o l r e c o g n i t i o n a c c u r a c y o f t h e s y s -t e m i s i n e x c e s s o f 9 5 % , w i t h a n a c c u r a c y o f 9 9 . 7 % f o r t h equa r t e r and e igh th no te s .4 M o r p h o l o g i c a l o p e r a t io n s a n d n o t a t io n sT here a re f ive bas ic morpho log ica l ope ra t ions tha t a re us edth roughou t th i s work . In th i s s ec t ion , we wi l l g ive a b r i e fdes c r ip t ion o f the ope ra t ions and the i r ma them a t ica l no ta t ion(s ee Hara l i ck e t a l . 1987 fo r a more comple te de s c r ip t ion ) .4 . 1 Di la t ionD i l a t i o n i s t h e m o r p h o l o g i c a l t r a n s f o r m a t i o n w h i c h c o m -b ines two s e t s us ing vec to r add i t ion o f s e t e l emen t s .L e t A a n d / 3 b e s u b s et s o f E n . T h e d i l a ti o n o f A b y Bis deno ted by A | B and is de f ined byA O B = {C E E N [ C = a + b f o r s o m e a E A a n d b E B } .

8/2/2019 muser

3/11

142In p rac t i ce , t h e se t A i s co n s id e red as t h e imag e u n d erg o in gt r an s fo rmat io n an d t h e se t / 3 i s r e f e r r ed t o a s t h e s t ru c tu r i n ge l emen t . To d i l a t e an imag e , we ru n t h e s t ru c tu r i n g e l emen to r ig in o v er a l l t h e b in a ry o n e p ix e l s o f t h e imag e . Th e a r easwep t b y t h e s t ru c tu r i n g e l emen t i s t h e d i l a t ed imag e .

4.2 Eros ionEro s io n , t h e mo rp h o lo g i ca l d u a l o f d i l a t i o n , i s a sh r i n k in go p e r a ti o n . I t i s th e m o r p h o l o g i c a l t r a n s f o rm a t i o n w h i c h c o m -b in es two se t s u s in g v ec to r su b t r ac t i o n o f se t e l emen t s .

L e t A a n d / 3 b e s u b se t s o f E N . T h e e r o s i o n o f A b y Bi s d en o t ed b y A O B an d i s d e f i n ed b yA Q B = { x E E N I x + b E A

f o r e v e r y b E B } .To e ro d e an imag e , we ru n t h e s t ru c tu r i n g e l emen t o r i g ino v e r a l l t h e p i x e l s i n t h e i m a g e . W e m a r k t h o s e p i x e l s a twh ich t h e s t ru c tu r i n g e l emen t o r i g in can s t an d an d wh erei t s en t i r e a r ea co v er s o n ly b in a ry o n e p ix e l s . Th e a r ea o fm a r k e d p i x e l s i s t h e e r o d e d i m a g e .

4.3 O peningT h e o p e n i n g o f a n im a g e A b y a s t r u c tu r in g e l e m e n t K i sd e n o t e d b y A o K a n d is d e fi n e d a s A o K = ( A O K ) | K .

Th e o p e n in g o p era to r ru n s t h e s t ru c tu r i n g e l em en t t h ro u g ht h e a r e a o f b i n a r y o n e p i x e l s , k e e p i n g t h e a r e a o f t h e s t r u c -t u ri n g e l e m e n t c o n t a i n e d i n t h e a r e a o f b i n a r y o n e p i x e l s i nt h e i m a g e . T h e a r e a s w e p t b y t h e s t r uc t u r in g e l e m e n t i s t heo p e n e d i m a g e . O p e n i n g a n i m a g e w i t h a d i s k s t r u ct u r in g e l -e m e n t s m o o t h s t h e b o u n d a r y , b r e a k s n a r r o w i s t h m u s e s , a n de l imin a t es smal l sp eck s .

4.4 Clos ingT h e c l o s i n g o f a n i m a g e A b y a s t r u ct u r in g e l e m e n t K i sd en o t ed b y A * K an d i s d e f i n ed as A 9 K = (A | K ) E3 K .C l o s i n g i s t h e m o r p h o l o g i c a l d u a l o f o p e n i n g . W h a t o p e n -in g d o es t o t h e fo reg ro u n d , c l o s in g d o es t o t h e b ack g ro u n d .C l o s i n g a n i m a g e w i t h a d i s k - s t r u c t u r i n g e l e m e n t s m o o t h sth e b o u n d ary , fu ses n a r ro w b reak s , f i l ls smal l h o l es , an d f i ll sg a p s o n t h e b o u n d a r y .4.5 H i t - or - m is s t rans form at ionT h e h i t - o r - m i s s tr a n s f o r m a t i o n o f a n i m a g e A b y t w o s t ru c -t u r i n g e l em en t s K1 an d / s K1 N K2 = ~ , can b e d e f i n eda s

A | (K1 , K2 ) = ( A Q 1s N ( A c Q K s ) .Th e h i t -o r -mi ss t r an s fo rmat io n i s u se fu l i n l o ca t i n g p a t -

t e rn s i n an imag e . Th e s t ru c tu r i n g e l emen t K1 i s u sed t o

d ef in e t h e fo reg ro u n d p a t t e rn b e in g so u g h t , wh i l e t h e s t ru c -t u r i n g e l emen t K2 i s u sed t o d e f i n e t h e n e ig h b o r in g b ack -g ro u n d p a t t em ( r e f e r t o Co s t a 1 9 9 0 fo r a d e t a i l ed d esc r i p t i o no f v a r i o u s mo rp h o lo g i ca l s eq u en ces fo r p a t t e rn d e t ec t i o n ) .

4.6 S t ruc tur ing e lem ents ~T h e m o r p h o l o g i c a l s y m b o l r e c o g n i z e r u s e s , a m o n g o t h e rsh ap es , t h r ee s t ru c tu ri n g e l em en t s o f s t an d ard sh ap e : (1 )l i n e - , ( 2 ) d i sk - , an d (3 ) b o x - s t ru c tu r i n g e l emen t s . Th e i r n o -t a t i o n s a r e a s fo l l o ws .- A l i n e - s t ru c tu r i n g e l em en t wi th o r i g in a t ( 0 , 0 ) an d ex t en d -

in g f ro m ( x l , y l ) t o (x 2 , y2 ) i s d en o t ed b y l ine ( x 1 , y l , x2 , Y2)- A d i sk - s t ru c tu r i n g e l em en t wi th i t s o r ig in a t th e cen t e r an d

rad iu s r i s d en o t ed b y d i s k r- A b o x - s t ru c tu r i n g e l emen t wi th wid th w an d h e ig h t h i s

d e n o t e d b y b o x ( w , h )

4 .7 I m age as s ignm ent no ta t ionT h e e q u a t io nIo~ig~n~l = [original ~ di sk 5w o u l d m e a n t h e f o l l o w i n g . T h e o r i g i n a l i m a g e Io~gi~al ist o b e d i l a t ed b y a d i sk o f r ad iu s 5 p ix e l s an d t h e r esu l t i n gi m a g e i s t o r e p l a c e t h e o r i g in a l i m a g e .

5 P r i o r a s s u m p t i o n s

T h e f o l l o w i n g a s s u m p t i o n s a r e m a d e a b o u t t h e s t r u c t u r e o fm u s i c s c o r e s t h a t M U S E R c a n h a n d l e .- Th e s i zes o f t h e d i f fe r en t sy mb o l s a r e r e l a t i v e . Th i s e s sen -

t i a l l y mean s t h a t o n ce t h e sca l e o f t h e imag e i s co mp u ted ,a n a p p r o x i m a t e e s t im a t e o f t h e s i z e s a n d s h a p e s o f t h ev a r i o u s s y m b o l s c a n b e d e t e r m i n e d .

- Th e s t a f f l i n es a r e eq u a l l y sp aced . Th e l ay o u t ex t r ac t io nm o d u l e d o e s a m e d i a n c a l c u l a ti o n o n t h e s p a c i n g b e tw e e nth e h o r i zo n t a l s t a f f l i n es t o d e t e rmin e t h e sca l e o f t h e im-ag e . Th u s , w e r eq u i r e t h e s t a f f l i n es t o b e eq u a l l y sp aced .T h e i m a g e d o e s n o t h a v e a l a r g e s k e w . T h i s i s r e q u i r e dt o m a k e t h e s t a f f e x tr a c t i o n a l g o r it h m r e m a i n s i m p l e . T h es t a f f i s ex t r ac t ed b y o p en in g t h e ima g e wi th a l o n g h o r i -zo n t a l l i n e ( ab o u t 4 0 p ix e l s wid e) . Th i s wo u ld d e t ec t h o r i -zo n t a l an d n ea r h o r i zo n t a l l i n es ( i f t h e l i n es a r e mo re t h an1 p ix e l t h i ck ) . I f t h e re i s a l a rg e sk ew , t h e a lg o r i t h m wi l lf a i l t o d e t ec t t h e s t a f f l in es .

T h e f o l l o w i n g a s s u m p t i o n s a r e m a d e a b o u t th e r e p r e s e n -t a ti o n o f m u s i c s c o r e s t h a t M U S E R i s d e s ig n e d t o h a n d le .1 . Th e n u m b er o f p a ra ll e l l i n es i n a s t a f f i s a s su m ed t o

b e f i v e . T h i s m e a n s t h a t w e d o n o t a l l o w f o r p e r c u s s i o ns t av es , cu e s t av es , e t c .

2 . We wi l l d ea l w i th mu l t i p l e v o i ce mu s i ca l s co res .3 . C u r r e n t i m p l e m e n t a t i o n a s s u m e s t h a t t h e p r i n t e d m u s i -

c a l s c o r e f o l lo w s s o m e o f t h e c o n v e n t i o n s s p e c i f i ed b yR o e m e r ( 1 9 8 5 ) .

8/2/2019 muser

4/11

4. The d i s tance be tw een success ive no te heads in p ixe l s i sp r o p o r t i o n a l to t h e n u m b e r o f p i x e l s in t h e m e a s u r e a n dthe le f t no te dur a t ion .

5 . A cc identa l s a r e p laced squar e ly on the l ine or the spacedi r ec t ly in f ron t o f the no te they a l te r . O ur imp lementa t ionj u s t l o o k s f o r n o t e s i n t h e b o t t o m r i g h t q u a d r a n t o f t h eacc identa l w hi le a s soc ia t ing acc identa l s w i th no tes .

6 . S tems , in gener a l , go dow n w he n a t tached to the le f t o f then o t e . T h e y g o u p w h e n a t t a c h e d t o t h e r ig h t o f t h e n o t e .T h e s t e m l e n g t h i s n o r m a l l y t h e l e n g th o f o n e o c t a v e .

T h e r e a r e a n u m b e r o f o t h e r c o n v e n t i o n s a b o u t re s t s y m b o l sa n d a u g m e n t a t i o n d o t s t h a t o u r s y s te m u s e s . F o r e x a m p l e , aquar te r r e s t i s a s sumed to be cen te r ed on the s ta f f . A ha l fr es t touches the th i r d s ta f f l ine above , w hi le a w h ole r es ttouche s the f our th s ta f f l ine be low . Som e of the as sumpt ionsm e n t i o n e d a b o v e p r o v i d e a n s w e r s f o r t h e c h o i c e s o f t h r es h -o lds used in the pos tpr ocess ing s tage tha t f il t e rs ou t spur ioussymbols tha t a r e de tec ted in the f ea tur e ex t r ac t ion s tep .

M u sic c o r ~ 7Scanner~[ I n p u t m a g e I

Layout Extractor

I MorphologicalSymbolRecognizer

[ 4 Feedback

High-Level oduleConnectedComponentAnalysisModule

Reasoning&Verification

Fig. 1. MUSER - system block diagram

AS C I I"lmP'ootpn|

6 L a y o u t e x t r a c to rThe f i r s t s tep tha t MU SER goes th r ough in the r ecogni t ionpr ocess i s de te r min ing the sca le and the skew of the image .T h e s k e w i n t h e i m a g e c a u s e s t h e r e c t a n g u l a r c o o r d i n a te s y s -tem, r e la t ive to w hich the d i f f e r en t symbols a r e loca ted , tobe r o ta ted . This r o ta t ion can be cor r ec ted once the skew hasbeen de te r mined . The sca le of the image i s used to f ix thes izes of the d i f f e r en t s tandar d s ize s t r uc turing e leme nts usedb y t h e s y m b o l r e c o g n i z e r . T h e s c a l in g o f t h e s t a n da r d s t r u c-tur ing e lements can be ach ieved in a s emi- in te r ac t ive w ay,

143p r o v i d e d p o l y g o n a l a p p r o x i m a t i o n s o f t h e o r i g in a l s t r u c tu r -ing e lements a r e ava i lab le .

To ex t r ac t the s ta f f lines , the image i s op ened w i th a hor -izonta l l ine 35 p ixe l s w ide . Le t th i s l ine s t r uc tur ing e lementb e K .I s t a f f = I i z ~ p zt t 0 KA c o n n e c t e d c o m p o n e n t a n a l y s i s o n I s t a f f w o u l d g i v e i n -f or mat ion ab out the ex t r em i t ie s of the hor izonta l l ines in theimage and the i r o r ien ta t ion . The median of the spac ing be -t w e e n a d j a c e n t li n e s c a n n o w b e c o m p u t e d . T h i s w o u l d g i v ean appr oximate es t ima te of the sca le of the image . Le t th i se s t i m a t e d s c a l e b e d e n o t e d b y ~ z . T h e d i m e n s i o n s o f a l lthe s t r uc tur ing e lements tha t a r e used f or symbol ex t r ac t ionw ou ld then be sca led by th i s va lue . The o r ien ta t ion of thel ines w ould ind ica te the skew of the image .

7 M o r p h o l o g i c a l s y m b o l r e c o g n i z e rA p r e l i m i n a r y e x a m i n a t i o n o f t h e s y m b o l s t h a t o c c u r i n m u -s ic repr esen ta t ion r evea ls the f o l low ing . Ther e a r e som e sym -bols w hose f ea tur es a r e exac t ly s imi la r to ( or in som e cases ,a subse t o f ) o the r s . The only f ac tor tha t d i s t inguishes thef o r m e r f r o m t h e l a t e r m a y b e t h e s i ze . A t y p i c a l e x a m p l e o fth i s f ac t i s the gr ace no te . The only th ing tha t d i s t inguishesa gr ace no te f r om a r egula r no te i s i t s s ize . This over lap off ea tur es among the d i f f e r en t symbols necess i ta te s the use ofa p r u d e n t s e q u e n c e o f s y m b o l e x t r a ct i o n .

H o w d o w e d e c i d e w h i c h s y m b o l s t o e x t r a c t e a r l ie r a n dw h i c h o n e s l a t e r ? T h e g u i d e l in e t h a t i s e m p l o y e d i n M U S E Rin dec id ing the pr ior i ty of a symbol in the ex t r ac t ion se -quence i s i t s f o r egr ound a r ea and the lack of d i s t inguish ingconcavi t i e s and cur va tur es . Symbols l ike the r epea t measur ebar , r epea t symbol , c le f , and f u l l no te heads w ou ld thus ge t ah ighe r pr ior i ty w hi le the r es t o f the symbo ls , acc identa l s, andother s tha t have d i s t inguish ing concav i t i e s , w ould com e la te rin the symbo l ex t r ac t ion sequence . A r egula r no te w ou ld thush a v e t o b e e x t r a c t e d b e fo r e t h e g r a c e n o t e s s in c e w e w o u l dbe us ing s imi la r ly shaped s t r uc tur ing e lements ( bu t o f d i f -f e r en t s izes ) to ex t r ac t bo th the symbols .

T h e a l g o ri t h m s u s e d f o r s y m b o l d e t e c t io n f o l l o w a t w o -s tep pr ocess . I n the f i r s t s tep , the pr imi t ive f ea tur es of ag iven symbol a r e ex t r ac ted . The second s tep u t i l i zes pr iork n o w l e d g e o f t h e t o p o l o g i c a l r e la t i o n sh i p a m o n g t h e d i f f e r-en t f ea tur es to de tec t the symbol . P lease no te tha t the s t r uc -t u r in g e l e m e n t s e m p l o y e d t h r o u g h o u t th i s s y m b o l d e t e c t i o np h a s e w o u l d " l o o s e l y " f o l lo w t h e s h a p e o f t h e m e d i a l a x i s( the ske le ton) of the f ea tur e shape be ing sought . This i s toincor por a te a ce r ta in degr ee o f to le r ance in the de tec t ion pr o-ces s . Thus , imper f ec t ions such as a f ew mis s ing f or egr oundpixe ls , b r oken edges , and b lur r ed co r ner s w i l l no t a f f ec t theoutput o f the symbol de tec t ion pr ocess . A lgor i thms tha t u t i -l i ze the backgr ound f ea tur es of symbols a r e l e s s s ens i t ive( le s s tuned) to s t r uc tur ing e lement s izes than the ones tha tuse f or egr ound f ea tur es . Symbols l ike the no tehead , how -e v e r , d o n o t p o s s e s s b a c k g r o u n d f e a tu r e s a n d c o n s e q u e n t l y

8/2/2019 muser

5/11

14 4ar e depend ent on su i tab ly s ized s t r uc tur ing e lements f or the i rex t r ac t ion . This is w her e the r e la t ive - s ize as sumpt ion com esin handy. We as sume tha t w i th the sca le of the image de -te r mined ( by the s ta f f - l ine d i s tance) , the s t r uc tur ing e lem ents izes can be f ixed . This s t i l l does no t imply tha t the s t r uc -tur ing e lement s izes have no r oom f or to le r ance . The s ta f fl ine ex t r ac t ion w or ks as adequ a te ly w i th a hor izonta l l ine 20pixe l s w ide as w i th the 35 - p ixe l l ine tha t w e used in our ex-per iments . The notehead ex t r ac t ion s t i l l f unc t ions w e l l w i tha d i sk- s t r uc turing e lem ent ( 7- p ixe l r ad ius ) a s w i th the e l l ip -t i ca l s t r uc tur ing e lement w e used .

O nce a symbol i s de tec ted , i f i t i s loca l in na tur e i t i sr emoved f r om the input image . This i s to pr event the ex-t r ac ted symbol f r om g iv ing pos i t ive r esu l t s in subsequentpr ocess ing . Except io ns to th i s a r e the s ta f f l ines and no tes tems w hich a r e no t r emoved a f te r de tec t ion .

I n the f o l low ing sec t ion , w e w i l l d i scuss a f ew o f ther epr esen ta t ive a lgor i thms used in the f ea tur e ex t r ac t ion-topology match ing s t r a tegy to de tec t d i f f e r en t symbols . Fora m o r e c o m p l e t e d is c u s si o n o n t h e a l g o ri t h m s s e e M o d a y u rand H ar a l ick ( 1991) .

8 The symbol detect ion a lgorithms8 . 1 C l e f s y m b o l sT h e o n l y c l e f s y m b o l s t h a t M U S E R h a n d l e s c u r r e n t l y a r ethe bas s and t r eb le c le f s .8.1.1 Bass clefThe bass c le f i s the la s t symbo l in our symbol de tec t ionsequence . For conven ience , w e d i scuss i t he r e . The im por tan tf ea tur e he r e i s the head o f the bas s c le f symbol w i th tw o dots( one be low the o the r ) to the r igh t ( F ig . 2 ) . The h ead po r t ionis eas i ly de tec ted by opening the input image w i th a d i sk-shaped s t r uc tur ing e lement K b a ~ s ( of 0 .44SZ p ixe l r ad ius )w i th a d ow n w ar d ta l l ( s ee F ig . 8 ).I b a s s H e a d = ~ i n p u t 0 K b a s s

S i n c e K b ~ has gap s in i t , I b a ~ H r i s c losed w i th a boxto f il l the gaps . This pr oduces a s ing le connec ted com ponen t .I b a ~ H ~ a d = I b ~ s ~ H ~ d Q b o x ( 1 . 6 7 S i , 0 . 3 3 S DThe r esu l t o f th i s oper a t ion i s i l lus t r a ted in F ig . 3 . N ow , thet w o d o t s l y i n g t o t h e r i g h t o f t h e h e a d h a v e t o b e e x t ra c t e d .To begin w i th , a l l the hor izonta l and ve r t i ca l l ines in thei m a g e a r e r e m o v e d .l h = I in p ~ t o l i n e ( O , 0, 1 .11SI, 0)Iv = I i ~ p ~ t o l i n e ( O , 0, 0, 1.11SI)R e m o v e I h a n d I v f r o m t h e i m a g e .

h r = I ~ p ~ t n ( I h w I v )~ n o L i n es = I i n p u t X O R [ c am pFigur e 4 show s the image a f te r the above oper a t ion .

To loca te the ind iv idua l do ts , the r esu l tan t image i so p e n e d w i t h a d i s k . T h i s s h o u l d b e f o l l o w e d b y a n o t h e ropening to ge t r id of b lobs b igger than the r equi r ed s ize .Ido t~ = I~oL~nr o d i sk O . 1 S IT h e n w e g e t r i d o f t h e b i g g e r b lo b s .Ib igBZob~ = Ido t~ o l ine (O , 0, 0.44SI , 0)

[ t e m p = l b ig B l o b s N [ d o E sIdo t~ = h o t ~ X O R h e m p

Since w e seek a pa i r o f do ts w i th one ve r t i ca l ly be low theothe r s epar a ted b y about S~ p ixe l s , the im age Idot~ i s c losedw i th a ve r t i ca l s t r uc tur ing e lement .] d o t P a i r = I d o t s Q l i n e ( O , 0, 0, 1.11S1)[ d o t P a i r = [ d o t P a i r X O R [ do E s

The second oper a t ion pr oduces r esu l t s on ly a t those p lacesw her e pa i r s o f do ts occur , one ve r t i ca l ly be low the o the r andseparated b y less than 1.11S~ pixels . Th is is i l lus trated inF ig . 5 . The r equi r ed f ea tur es have now been ex t r ac ted . W hatf o l l o w s i s a n o t h e r s e q u e n c e o f m o r p h o l o g i c a l o p e r a t io n s t ocheck the topology . S ince the do t pa i r i s expec ted to l i e tothe r igh t o f the bas s head por t ion , the head i s t r ans la ted tothe r igh t and in te r sec ted w i th the do t pa i r .I d i l a te l ~ i g h t = I b a s s H e a d ( ~

l i n e ( 0 . 8 3 S i , 0, 1.11Sl, 0)I b a s s = I d i la t e R i g h t A [ d o t P a i r

The r esu l t o f the above oper a t ion i s show n in F ig . 6 . Theoper a t ion pr oduces r esu l t s in p laces w her e bas s c le f s a r enot p r esen t . I n the ac tua l symbol ex t r ac t ion sequence , bas sc le f s a r e ex t r ac ted la s t. Thus , the o t l l e r symb ols w ou ld haveb e e n r e m o v e d b y t h e n a n d t h e b a s s c l e f e x t r a c ti o n r e su l t s i nf or egr ou nd p ixe l s on ly a t those loca t ions w her e the bas s c le fs y m b o l s a r e p r e se n t . T h e f o l l o w i n g s e q u e n c e i s e m p l o y e d t oge t r id of the bas s symbol f r om the input image .

f b a s s D i l = f b a s s ~ d i s k O . 2 2 S ~I d i l a t e L e f t = [ b a s s D i l O

l i n e ( O , O , - 1 . 5 7 S I , 0 )I d i l a te D o w n = I d i l a te L e f t O

l i n e ( O , 0 , 0 , - 1 . 6 7 S I )l t e m p = [ d i l a t e D o w n N ~ i n p u tl i~ ;~ t = h ~ p X O R I ~ p ~t

Since I b~ss w ould pr oduce output a t the loca t ion of the do tpa i r to the r igh t o f the bas s symbol , 16as~ is di la ted to thele f t and then dow n. Th e in ten t ion i s to cove r the en t i r e bas ssymbol to f ac i l i t a te l a te r r emova l . The r esu l t o f the th r eed i la t ions i s then r emoved f r om the input image in the usua lw a y .

8/2/2019 muser

6/11

145

V ] '

1 2

O

| | ", 9a ,

P lip O t

2 3 4\

t

m r ~ ": . ' ~ .

dbJ ,

I$

t II

5 6

Fig. 2. B ass clef. A sample measure con-taining two occurrences of the bass sym-bo lFig.3. Bass clef extraction. After headextractionFig. 4. Bass c lef extraction. The horizon-tal and vertical lines have been removedFig. 5. Ba ss clef extraction. The dot pairhas been extractedFig. 6. Bass cle f extraction. The head hasbeen translated to the right and intersectedwith the dot pair

8 .1 .2 Tr eb le c le fThe t r eb le c le f symbo l ( F ig . 7 ) i s de tec ted by a s equ enceof openings f o l low ed by an in te r sec t ion . To begin w i th , thenear - c i r cu la r shape f or ming a pa r t o f the body of the t r eb lesymbol i s ex t r ac ted . This i s ach ieved by opening the inputimage w i th a s emi- c i r cu la r s t r uc tur ing e lement K t ~ e b ~ l asshow n in F ig . 8 .

I t r e b l e 1 = I i n p u t o K t r e b l e 1T h e b o d y o f t h e t re b l e c l e f s y m b o l i s e x tr a c t e d b y o p e n i n gw i th a box- shaped s t r uc tur ing e lement Kt~ebt~2 that is 0.44Sxpixe ls w ide and 1 .8S~ p ixe l s t a l l , o r ien ted r oug hly a t a 45 ~angle .I t r e b l e 2 = I i n p u t 0 I ( t r e b l e 2T h e o u t p u t o f t h e l a s t o p e n i n g o p e r a t i o n w o u l d y i e l d t h eb o d y o f t h e t r e b l e s y m b o l t h a t l i e s a p p r o x i m a t e l y a t t h emiddle , w i th the o the r f ea tur e ly ing be low i t . The f o l low -ing sequence i s employed to ve r i f y the spa t ia l r e la t ionsh ipof the tw o f ea tur es ex t r ac ted so f a r . The s t r uc tur ing e leme ntsused a r e K U n e l = l ine ( 0 , 0 . 5 5 S l , 0 , 1 . 1 1 S i ) an d Kline 2 = l ine(0,0,-1.11S~,0).

The s t r uc tur ing e lement K U n ~ l does no t inc lude the or i -g in . Thus the above oper a t ion t r ans la tes Itr~bl~l u p w a r d s b y0 .55SI p ixe l s and then d i la tes the t r ans la ted image b y 0 .55SIp ixe l s v e r t i ca l ly .I d i l a t e L e f t = [ d i la t e U p 0 K l i n e 2This oper a t ion d i la tes the r esu l t o f the pr ev ious d i la t ion tothe le f t . This image i s now in te r sec ted w i th [ t r e b l e 2 , w h i c hw o u l d c o n t a in t h e b o d y o f t h e t re b l e c l e f s y m b o l .h r e b l e = I d i l a t e L e f tA

I t r e b l e 2T h e i m a g e I t r e b l e w o u l d t h e n h a v e f o r e g r o u n d p i x e ls o n l ya t the loca t ions of the t r eb le c le f symbols tha t occur r ed inthe input image . The t r eb le c le f symbols could po ten t ia l lyappear in tw o d i f f e r en t sizes , the sm al le r o f w hich i s used todeno te c le f changes ins ide a measur e . This sm al le r tr eb le c le fsymbol i s appr oximate ly 80% of the r egula r s ize . Thus , thesmal l t r eb le c le f i s ex t r ac ted by an exac t ly s imi la r p r oce dur ebut w i th a l l the s t r uc tur ing e lements s ca led dow n by a f ac torof 0 .8 .

O nce the t r eb le c le f symbol i s de tec ted , i t i s r emovedf r o m t h e i n p u t i m a g e b y t h e f o l l o w i n g s e qu e n c e :] t emp = [ inpu t N 1 t r ebl e[ i n p u t = [ i n p u t X O R I t e m p

[ d i l a t e U p = I t r e b l e l ~ t ( l i n e l

8/2/2019 muser

7/11

146

K t r e b l e l

g \

K b a s s8

V V ~Ik |LT ~

. r I 6 ~ I ' T IAle l~ e l " J I B M i 'l r p ' ~ ' ," ,41 ~ ? M m"U / M I - ~ i.-==d

~: ~ l i* L ' j F m-

/K t r e b l e 2

1 4 1 5

1 2

1 0

1 3

11

| 9

9D | 9

$$Fig. 7. Treble clefFig. 8. Structuring elements fo r cle f symbolsFig. 9. Filled note headFig. 10. Eighth restFig.l l . Hit- or miss structuring elements foreighth restsFig. 12. Note beamsFig. 13. Q uarter restFig. 14. original image o f a sample m easureFig. 15. Recognized symbols of the above m ea-sure with the symbols overlaid

8 . 2 F i l l e d n o t e h e a d sF i l l ed n o t e h ead s (F ig . 9 ) a r e p resen t i n q u ar t e r , e i g h th , an ds ix t een th n o t es . Th e s t ru c tu r i n g e l emen t Kbzob t h a t i s u sedto ex t r ac t t h ese n o t e h ead s i s e l l i p t i ca l , w i th a ma jo r ax i s t omin o r ax i s r a t io o f ap p ro x im ate ly 5 :4 . Th e e l l i p se i s o r i en t edro u g h ly a t a 4 5 ~ an g l e .I b l o b = I i n p u t 0 K b l o bTh i s o p en in g w o u ld ex t r ac t t h e n o t eh ead s o f q u ar te r , e i g h than d s i x t een th n o t es ( an d o th e r l o wer d u ra t i o n n o t es , i fp r e s e n t ) . A s b e f o r e , t h e e x t r a c t e d n o t e h e a d s a r e r e m o v e df ro m th e i n p u t imag e t o p rev en t i n t e r f e r en ce o f t h i s f ea tu rewi th su b seq u en t f ea tu re ex t r ac t i o n seq u en ces .I t e m p = I i n p u t 0 I b l obh n p u t = I te m p X O R [ in p u t

8 . 3 E i g h t h r e s tA care fu l i n sp ec t i o n o f t h e e ig h th r es t sy m b o l (F ig . 1 0 ) r e -v ea l s t h a t t h e re a r e a t l eas t two f ea tu res o n e ca n l o o k fo r . Th i sr e s t s y m b o l h a s a d i s k - s h a p e d h e a d w i t h a n u p - c o n c a v i t y t oi t s r i g h t. Th e h ead i s d e t ec t ed s imp ly b y o p en in g w i th a d i sk -sh ap ed s t ru c tu r i n g e l emen t . Th e u p -co n cav i t y i s d e t ec t ed b ya h i t -o r -mi ss t r an s fo rm.f h e a d = f in p u t 0 d i s k O . 1 7 S iT h e a b o v e s e q u e n c e e x t r a c t s t h e h e a d o f t h is s y m b o l . N e x t ,t h e u p - c o n c a v i t y i s l o c a t e d b y h i t - o r - m i s s t r a n s f o r m . T h e

s t r u c t u r i n g e l e m e n t s u s e d a r e : K s , w h i c h i s w e d g e s h a p e dwi th a t h in b o d y , an d /

8/2/2019 muser

8/11

Th i s o p en in g o b v io u s ly wo u ld ex t r ac t a l l v e r t i ca l l i n es t h a ta r e 2 .2 2 S ~ o r mo re p ix e l s l o n g . Th u s , i n ad d i t i o n t o t h e n o t es t e m s w e w o u l d a l s o b e e x t r a c ti n g p o r ti o n s o f o t h e r s y m b o l s .I n t e r s e c ti o n o f t h e s e s t e m s w i t h t h e n o t e h e a d s , h o w e v e r ,wo u ld g e t r i d o f a l l t h e sp u r io u s n o t e s t ems . Th i s i s ach i ev edb y t h e h i g h - l e v e l m o d u l e w h e n i t r e a s o n s a b o u t t o p o l o g y .

8 . 5 N o t e b e a m sT h i c k b e a m l i n e s ( F ig . 1 2) c o n n e c t i n g t w o o r m o r e n o t es t ems i d en t i fy an e ig h th o r s i x t een th n o t e o r o th e r l o werd u ra t i o n n o t es . F o r an e ig h th n o t e t h e re i s a s i n g l e b eaml in e , f o r a s i x t een th n o t e two b eam l i n es , an d so o n .

T o l o c a t e a b e a m l in e t h a t j o i n s t w o o r m o r e n o t e s o rt h a t wh ich p ro t ru d es f ro m a s i n g l e s t em, t h e fo l l o win g f ac ti s u ti l ized : T h e b e am l i n es co u ld b e o r i en t ed i n j u s t a f i x edn u m b e r o f o r i en t a ti o n s . I n t h e s a m p l e m u s i c s c o r e w e w o r k e dwi th , t h e re we re j u s t two su ch o r i en t a t i o n s . I f t h e se t o f a l lp o s s i b l e o r i e n ta t i o n s t h e b e a m l in e s c o u l d a s s u m e i s k n o w n ,th ey co u ld b e ex t r ac t ed b y o p en in g wi th t h i ck s t r a i g h t l i n es(b o x es ) a t t h o se o r i en t a t i o n s an d t ak in g t h e u n io n o f a ll t h o seo p e n i n g s. T o l o c a t e t h e t h ic k b e a m l i ne s j o in i n g t w o o r m o r en o t e s t e m s , o p e n i n g i s e m p l o y e d w i t h b o x e s t h a t w e r e o r i -e n t e d r o u g h l y a t + 1 1 ~ a n d - 1 1 ~ T h e s t ru c t u ri n g e l e m e n tK b o ~ V p i s abo ut 2S~ p ixe ls w ide, 0 .2S~ p ixe ls ta l l , and or i -e n t e d a p p r o x i m a t e l y a t + 1 1 ~ T h e o t h e r s t ru c t u ri n g e l e m e n tI ( b o x D o w n i s o f t h e s a m e d i m e n s i o n s b u t o r i e n t e d a p p r o x i -m ate ly a t - 1 1 ~

I s l a n t U p = I i n p u t 0 K b o x U pI s l a n t D o w n = I i n p u t o K b o x D o w nN o w , t h e s e n o t e b e a m s a r e r e m o v e d f r o m t h e i n p u t i m a g e .I t e m p = I i n p u t A ( l s l a n tU p U I s l a n t D o w n )I i~ p ~ t = I t , m p X O R I i~ p ~ t

147Th e f i r s t o p era t i o n o p en s t h e i n p u t imag e wi th t h e smal lb o x - s t r u c t u r i n g e l e m e n t K s m , u s z ~ t . T h e s m a l l n o t e b e a m ,a s w e k n o w , l ie s e i t h e r a b o v e o r b e l o w t h e l a r g e r n o t e b e a m .Th e r esu l t o f t h e fo l l o win g seq u en ce o f t h r ee o p era t i o n s i st h e l a r g e r n o t e b e a m s d i l a t e d d o w n w a r d a n d u p w a r d . T h i sd i la t e d i m a g e d e f i n e s r e gi o n s a b o v e a n d b e l o w t h e l a r g e rn o t e b e a m s , w i t h i n w h i c h t h e s m a l l e r n o t e b e a m s c o u l d b eex p ec t ed . Th e l a s t o p era t i o n i n t h e seq u en ce i n t e r sec t s t h eresu l t o f t h e f i r s t o p en in g o p era t i o n wi th t h i s d i l a t ed imag e .T h i s s e q u e n c e t h u s e x tr a c t s t h e s m a l l n o t e b e a m s l y i n g e i t h e ra b o v e o r b e l o w t h e l a r g e n o t e b e a m s .

I t i s p o ss ib l e t h a t t h e ab o v e seq u en ce y i e ld s sp u r io u sb e a m l i ne s ( t h o u g h t h e y h a v e n o t i n o u r e x p e r i m e n t s s o f a r )t h a t d o n o t p ro t ru d e f ro m n o t e s t ems . Hen ce , o n ly t h o se n o t eb e a m s t h a t e m e r g e f r o m n o t e s t e m s s h o u l d b e e x t r a c te d . T h es a m e r e a s o n i n g i s n o t e m p l o y e d i n e x t r a c ti n g t h e l a r g e r n o teb e a m s ( i n t h e p r e v i o u s s e c t i o n ) b e c a u s e w e e x p e c t l a r g e rf ea tu res t o b e mo re r e l i ab l e t h an t h e smal l e r o n es .f t r u e S l a n t s = f s m a l l S l a n t ( 'l 1 s t e r n

I 1 = I t ~ s z ~ n t ~ | d i s k O . 2 2 S tI1 = I1 (9 l ine (O, O, 0.55S~, O)

I ~ ,~ u s z ~ r ~ t = I 1 | l i n e ( O , O, - 0 . 5 5 S ~ , O )Th e f i rs t o p era t i o n i n t h e ab o v e seq u en ce i n t e r sec t s t h e sm al lb e a m l i n e s w i t h t h e n o t e s t e m s t o r e m o v e s p u r i o u s b e a ml in es . S in ce t h i s o p era t i o n y i e ld s o n ly t h e en d p o in t s o f t h esmal l b ea m l i n es ( i n t h e v i c in i t y o f th e n o t e s t em s) , t h e r esu l to f t h e f i rs t o p era t i o n i s d i la t ed w i th a d i sk an d t h en a l l o we dto ex p an d h o r i zo n t a l l y . Th e fo l l o win g t h r ee d i l a t i o n s ach i ev et h is . T h e e x t r a c t e d s y m b o l i s r e m o v e d f r o m t h e i n p ut i m a g ein t h e u su a l way .I t e m p = I i n p u t A I s m a l l S l a n t

8 .5 .1 S m al l n o t e b eam sA f t e r t h e n o t e b e a m s t h a t j o i n n o t e s t e m s h a v e b e e n d e -t ec t ed , t h e r e l a t i v e ly smal l e r n o t e b eams t h a t p ro t ru d e f ro ms i n g l e n o t e s t e m s a r e t a r g e t e d . T h e s e s m a l l e r n o t e b e a m sc o u l d e i t h e r l i e a b o v e o r b e l o w t h e l a r g e r n o t e b e a m s . T h ef o l l o w i n g s e q u e n c e i s u s e d t o l o c a t e t h e s m a l l n o t e b e a m s .A b o x - s t r u c t u r i n g e l e m e n t KsmaUSlan that i s 0 .88S1 p ixe lswid e , 0 .3 9 S~ p ix e l s t a l l, an d o r i en t ed a p p ro x ima te ly a t +1 1 ~is used .I s m a l l S l a n t = I i n p u t o g s m a l l S l a n $

I b~ g S la n t = I s t ~ n t V p U I s z ~ n t D o ~I b i g S l a n t = I b i g S l a n t @

l i n e ( O , O, O, 1.67S 0I b i g S l a n t = I b i a S l a n t @

l i n e ( O , O, O, -1 . 6 7 S I )I s ,~ a U S ~ n t = I s ~ ,~ u s z ~ n t n I b i g S t ~ n t

8 . 6 Q u a r t e r r e s tS y m b o l s t h a t d e n o t e d i f f e re n t d u r a ti o n s o f r e s t a l w a y s o c -cu p y t h e sam e v e r t i ca l p o s i t io n r e l a t i v e t o t h e s t a f f li n es .Th e d i s t i n g u i sh in g f ea tu res o f a q u ar t e r r e s t a r e i t s co n cav i -t i e s (F ig . 1 3) . I t h as a l e f t co n cav i t y , a r i g h t co n cav i t y , a n d ad o w n c o n c a v i t y . T h e g e o g r a p h i c a l p o s it i o n s o f th e c o n c a v -i t i e s wi th r esp ec t t o each o th e r i s a l so an imp o r t an t d i s t i n -g u i sh in g f ac to r . Th e r i g h t co n cav i t y i s t o t h e r i g h t o f th e l e f tc o n c a v i t y a n d t h e d o w n c o n c a v i t y i s b e l o w b o t h t h e l e f t a n dth e r i g h t co n cav i t i e s . To l o ca t e a p a r t i cu l a r co n cav i t y , s ay t h er i g h t c o n c a v i t y , t h e i m a g e i s o p e n e d w i t h a w e d g e - s h a p e ds t ru c tu r i n g e l emen t KrightWedge with i t s acu t e o r i n n er s i d efac in g r i g h t ( l i k e a < sy m b o l ) . A h i t -o r -m i ss t r an s fo rm at io no n t h e r esu l t i n g imag e l o ca t es t h e r i g h t co n cav i t y . Th e s t ru c -t u r i n g e l e m e n t s i n t h i s c a s e w o u l d b e / ( 1 , w h i c h w o u l d b ee x a c t l y t h e s a m e a s K r i g h t W e d 9 e t o d e t ec t t h e fo reg ro u n dp a t t e rn , an d Ks , wh ich wo u ld b e a f i l l ed t r i an g l e su i t ab ly

8/2/2019 muser

9/11

148p o s i t i o n ed wi th r esp ec t t o K1 t o d e t ec t t h e b ack g ro u n d co n -cav i t y .

[ o p e n = f i n p u t o I( . r ig h t W e d g e[ r i e h t C o n c = [ o p e n | (KI, K2)A s imi l a r ap p ro ach i s u sed t o d e t ec t t h e l e f t co n cav i t y . Re l -a t i v e p o s i t i o n s o f t h e co n cav i t i e s can t h en b e u sed t o d e t ec tt h e p resen ce o f t h e q u ar t e r r e s t .

9 H i g h - le v e l r e a s o n in g m o d u l eT h e o u t p u t o f t h e m o r p h o l o g i c a l s y m b o l r e c o g n i z e r i s i nt h e f o r m o f b i n a r y i m a g e p l a n e s . E a c h p l a n e c o n t a i n s o n eo r mo re sy mb o l s t h a t f a l l i n to a p a r t i cu l a r sh ap e ca t eg o ry .T h e h i g h - l e v e l m o d u l e , w h i c h f o l l o w s t h e s y m b o l r e c o g -n i z er , i s c o m p o s e d o f t w o p a r ts : a c o n n e c t e d c o m p o n e n t sa n a l y s i s ( C C A ) m o d u l e a n d a r e a s o n i n g m o d u l e . T h e C C Am o d u l e p e r f o r m s c o n n e c t e d c o m p o n e n t a n a l y s i s o n e a c h b i tp l a n e o u t p u t f r o m t h e s y m b o l r e c o g n iz e r . F o r e a c h c o n n e c t e dco mp o n en t i n a b i t p l an e , d esc r i p to r s su ch as cen t ro id , a r ea ,a n d b o u n d i n g r e c t a n g l e a r e c o m p u t e d . T h i s i n f o r m a t i o n a n dth e sp a t i a l r e l a t i o n sh ip s b e tween t h e sy mb o l s a r e u sed t op er fo rm f i n a l r eco g n i t i o n o f t h e sy mb o l s .

In t h i s s ec t i o n , we f i r s t d esc r i b e t h e g en era l p o s tp ro cess -i n g s t r a t eg y an d fo l l o w i t u p wi th sp ec i f i c d e t a i l s o n p roc e s s i n g a f e w i m p o r t a n t s y m b o l s .

T h e o u t p u t f r o m t h e f e a t u r e e x t r a c t i o n m o d u l e i s o n l ya mo d era t e p a r t i t i o n o f t h e sy mb o l s i n t h e mu s i c sco re i n tosev era l ca t eg o r i es . Th i s p a r t i t i o n h as t o b e r e f i n ed b y u s in gp r i o r k n o w l e d g e a b o u t t h e e l e m e n t s i n a m u s i c s c o re . F o r e x -amp le , t h e b i t p l an e co n t a in in g t h e s t a f f li n es h as ex t r an eo u sin fo rmat io n t h a t n eed s t o b e f i l t e r ed o u t . S in ce t h e d e t ec t i o na lg o r i t h m assu m es t h a t t h e s t a f f li n e i s a l o n g h o r i zo n t a l l i n e ,h o r i zo n t a l l i n es t h a t a r e p a r t o f l o n g a r c t i e s b e tween n o t esa re a l so d e t ec t ed d u r in g t h e d e t ec t i o n s t ep . M o reo v er , a s i n -g l e s t a f f l i n e m ay b e b ro k e n i n to mu l t i p l e p i eces (b ecau seo f t i n y b reak s i n t h e i n p u t d a t a ) . S o fu r t h e r p ro cess in g n eed sto b e d o n e t o g ro u p b ro k en l i n es an d t o f i l t e r o u t h o r i zo n t a ll i n es t h a t a r e n o t p a r t o f t h e s t a f f l i n es i n t h e mu s i c sco re .

F o r e a c h b i n a r y i m a g e , C C A i s p e r f o r m e d a n d w e p r o -d u ce an i n t e rmed ia t e d a t a s t ru c tu re t h a t co n s i s t s o f :- Or ig in a l ima g e- id- T h e c o m p o n e n t n u m b e r- Th e ro w an d co lu m n co o rd in a t es o f th e cen t ro id o f t h e

c o n n e c t e d c o m p o n e n t- T h e a r e a o f t h e c o n n e c t e d c o m p o n e n t- Th e u p p e r le f t an d b o t t o m r i g h t co o rd in a t es o f t h e r ec t an -

g u l a r b o u n d i n g b o x o f e a c h c o n n e c t e d c o m p o n e n t- Th e p a ram ete r s o f t h e b es t - f i t ti n g e l l ip se fo r each co n -

n e c te d c o m p o n e n tTh e im ag e- id i d en t if i e s th e p re l imin ary c a t eg o ry t o wh ich

t h e c o m p o n e n t b e l o n g s. W e c h o s e t h e o t h e r d e s c r i p to r s s u c has cen t ro id an d b o u n d in g r ec t an g l e co o rd in a t es b ecau se t h esed esc r i p to r s ad eq u a t e ly en co d e t h e i n fo rmat io n i n t h e imag e .

9 .1 A lgor i thms for pos tprocess ing and f il t e r ingIn t h i s s ec t i o n we d esc r i b e a lg o r i t h ms t h a t we u se d u r in g t h ep o s tp ro cess in g s t ag e . Th e o u tp u t o b t a in ed a f t e r p o s tp ro cess -i n g an d f i l t e r in g i s an i n t e rn a l d a t a s t ru c tu re en cap su l a t i n g t h ein fo rmat io n i n t h e mu s i c sco re . Th i s sec t i o n d esc r i b es so meo f t h e a lg o r i t h ms u sed a f t e r t h e f ea tu re ex t r ac t i o n s t ep s . Weg iv e sp ec i f i c d e t a i l s fo r ex t r ac t i o n o f s t a f f l i n es , co m p u ta -t i o n o f n o t e d u ra t i o n s an d n o t e p i t ch v a lu es , an d as so c i a t i o no f acc id en t a l s wi th n o t es . A l l su b seq u en t p ro cess in g i s d o n eb y r ep resen t i n g t h e sy mb o l d a t a i n t e rn a l l y i n t h e fo rm o fl i n k ed l i s t s . S ev era l l i n k ed l i s t s a r e fo rmed f ro m th e CCAmo d u le o u tp u t , i n c lu d in g t h e b a r an d s t em l i s t , n o t e l i s t , c l e fl i s t , an d t h e acc id en t a l l i s t . S u b seq u en t p ro cess in g i s d o n eb y mo d i fy in g /v e r i fy in g t h e d a t a i n t h ese l i n k ed l i s t s .Sta f f l i ne process ing . The b i n a r y i m a g e o b t a i n e d a s t h e o u t p u to f t h e h o r i zo n t a l l i n e ex t r ac t i o n a lg o r i t h m co n t a in s ex t r an e-o u s i n fo rm at io n t h a t h as t o b e f i l t e red o u t . Th e d e t ec t i o n s t epassu m es t h a t t h e s t a f f l in e i s a l o n g h o r i zo n t a l l i n e an d t h eo u tp u t i s co mp o sed o f a co l l ec t i o n o f l o n g h o r i zo n t a l l i n es .As men t io n ed ea r l i e r , p a r t s o f a r c t i e s ac ro ss n o t es a r e h o r i -z o n t a l s e g m e n t s a n d h e n c e m a y b e p r e s e n t i n t h e o u tp u t . D u eto n o i se , a s i n g l e s t a f f l i n e ma y g e t b ro k e n u p i n to mu l t i p l eh o r i zo n t a l p i eces .

Th e s t a f f l i n es a r e i d en t i fi ed b y u s in g t h e fo l l o w in gsteps:1 . De t ec t l o n g h o r i zo n t a l l i n es i n t h e i n p u t imag e ( f ea tu re

ex t r ac t i o n s t ep ) .2 . De t e rmin e t h e cen t ro id , a r ea , an d o r i en t a ti o n o f th e co n -

n e c t e d c o m p o n e n t s i n t h e o u t p u t f r o m t h e f e a t u r e e x t ra c -t ion s tage.

3 . D e t e r m i n e d i s c o n n e c t e d h o r i z o n t a l s e g m e n t s t h a t c a n b ep ar t o f t h e same s t a f f l i n e. Th i s i s d o n e b y c o m p u t in g t h ep erp en d i cu l a r d i s t an ces f ro m th e o r i g in ( t o p l e f t p o in t i nth e imag e) t o t h e i n d iv id u a l l i n e seg men t s . Th e p e rp en -d i c u la r d i s t a n c e s h o u l d b e a p p r o x i m a t e l y t h e s a m e f o r a l lt h e p o t en t i a l merg er can d id a t es .

4 . M erg e p o t en t i a l can d id a t es i f t h e d i s t an ces b e twee n t h ead j acen t en d p o in t s o f t h e l i n e seg men t s i s l e s s t h an asp ec i f i ed d i s t an ce t h r esh o ld .

5 . S av e t h e d e t a i l s fo r each s t a f f li n e i n a s t a f f l i n e a r r ay .T h i s a r r a y i s u s e d t o d e t e r m i n e t h e p i t c h o f e a c h n o t e.

T h e s p a c i n g s b e t w e e n a d j a c e n t s t a f f l in e s a r e t h e n c o m p u t e d .T h i s s p a c i n g g i v e s t h e a p p r o x i m a t e s i z e s f o r t h e s y m b o l sex p ec t ed i n t h e mu s i c sco re . A ro b u s t e s t ima t e o f t h e sp ac in gi s o b t a in ed b y t ak in g t h e med ian o f a l l t h e sp ac in g s b e tweenthe s taf f l ines .Dist inguishing measure bars and note stems. M e a s u r e b a r san d n o t e s t ems a r e b o th v e r t i ca l l i n es . M easu re b a r s a r e u su -a l l y l o n g er . F o r a mu s i c sco re t h a t i n v o lv es mu l t i p l e v o i ces ,s u c h a s a p i a n o s h e e t , w e u s e t h e t e r m m e a s u r e b a r t o s i g n i f yth e v e r t i ca l l i n e t h a t ru n s ac ro ss b o th p a i r s o f s t a f f s ( t r eb l eas we l l a s b ass ) . A s i g n i f i can t d i f f e r en ce b e tween c l e f s an dm e a s u r e b a r s i s d u e t o t h e i r p l a c e m e n t i n t h e m u s i c s c o r e .S t e m s a r e a l w a y s a t t a c h e d t o n o te h e a d s . T h e p r o c e d u r e u s e dt o r e m o v e s t e m s f r o m a l i s t c o n t a i n i n g b a r s a n d s t e m s d o e s

8/2/2019 muser

10/11

this by intersecting each item on the list with the boundingrectangles of elements in the note list. All items that do nothave any intersection are potential measure bars. Only mea-sure bars that have significant enough area are retained asmeasure bars. Our current implementation stops at this step,but further checks could be made, such as checks for nearbyclef symbols.Note dur a t ion computa t ion . The output from the feature ex-traction module consists of binary images containing filledblobs and unfilled blobs. In order to determine the durationof each note, a test for intersection of the bounding box foreach notehead and some stem in the stem list was performed.If there is an intersection, then the note duration was halved.Then tests of intersection of the stem with beams, slants andsquiggles are performed in order to identify quarter notes,single eighth notes, etc. The tests are performed in the orderspecified below:1. Determine if note head intersects with some stem.2. If there was an intersection with a stem, then determine

if the stem intersects with some beam.3. If there was an intersection with a beam, then determineif the beam intersects with some slant.

A s s oc ia t ion o f acc iden ta l s w i th no tes . Accidentals such assharp, flat, and natural symbols are associated with notes orstaff lines by using the follo wing steps. Accidentals can beassociated with notes when they occur next to a note. Theycan also be part of a key signature, in which case they occurnext to a clef symbol. Because of this we use the followingsteps to associate accidentals with notes:1. For each accidental in the accidental list, determine notes

close to the accidental. Only notes that are in the bottomright quadrant, defined by coordinate axes with the originas the upper-left corner of the accidental 's bounding box,are potential candidates for association. This is due to thefact that the accidentals are always located to the left andabove or on the same level as the notehead.

2. Compute the distance between the closest clef symboland the accidental. If this distance is greater than theaccidental-to-note-distance, then the accidental is associ-ated with the note. If both the distances are greater than aspecified fraction of the measure length (number of pixelsbetween successive measure bars), then the current imple-mentation flags the accidental as a possible misclassifica-tion. The clef-to-accidental distance is compared with thenote-to-accidental distance because the accidental couldbe part of the key signature or be a modifier for the note.

10 E x p e r i m e n t s a n d resultsThe printed score sheets we worked with were digitized at300 dpi. The system runs on a MVI - Genesis 2000 im-age processing workstation and takes 2 minutes to processa 512x480 image. So far we have processed 74 images(512x480 pixels). There were 176 complete measures inall. There were 1311 occurrences of stock symbols (where

149a stock symbol is defined as a symbol that the system isdesigned to recognize) out of which 1252 symbols were rec-ognized correctly. Thus the overall recognition accuracy is95.5%. Figures 14 and 15 show the recognition results forone of the images with the recognized symbols overlaid. Ta-ble 1 shows the misdetection and misclassification rates forthe different stock symbols. The misdetection and misclas-sification are defined as follows.Misdetection%

number o f correctly identified occurrences= 1 - 7 F o - ~ ~ .,/ xlO0Misclassification of symbol A

number of times a non-A symbol was recognized as ATotal number of occurrences of non-A stock symbolsThe misdetection was highest for the quarter rest and

lowest for the quarter and eighth notes. This can be ex-plained by the fact that the notes appeared with relativelyless shape-size variation. In addition, the distinguishing fea-tures of notes are the note head, stem and the beams, allof which are stable features. For the quarter rest the bot-tom concavity was not a reliable feature. In a few cases, theconcavity merged with the staff lines and was consequentlyobscured. The results can be improved if we include theforeground features of this symbol and exclude the bottomconcavity from the detection sequence.Table 1. Recognition results for the sample music scoreSymbol type Occurrences Correctly Misclassification

identified(%) (%)

Half note 97 89 (91.75) 1 (0.0007)Quarterand eigth notes 967 964 (99.7) 1 (0.0007)Treble clef 24 23 (95.83) 0Bass clef 23 21 (91.3) 0Flat 63 61 (96.83) 4 (0.0031)Sharp 13 13 (100) 0Natural 23 22 (95.65) 0Quarter rest 47 42 (89.36) 2 (0.0015)Eigth rest 18 17 (94.44) 1 (0.0007)

There were in excess of 30 symbols in the scores weworked with, but the algorithms were designed to extractonly around 12 of these symbols (symbols like grace notes,slurs, and note duration extension dots were not dealt with).This did not affect our results at all as is evidenced by thelow misclassification rates.

1 1 C o n c l u s i o nIn this paper we described the essential components ofMUSER, a music score recognition system, and providedalgorithms that are part of the system. The layout extraction

8/2/2019 muser

11/11

15 0a n d s y m b o l r e c o g n it i o n m o d u l e s u t il iz e m o r p h o l o g i c a l o p e r -a t i o n seq u en ces wh i l e t h e mid l ev e l mo d u le u t i l i zes co n v en -t i o n a l C C A m e t h o d s . A t p r e s e n t , o u r i m p l e m e n t a t i o n o f t h ea lg o r i t h ms fo r p o s tp ro cess in g i s co mp le t e an d we a r e i n t h ep r o c e s s o f i m p l e m e n t i n g t h e p r o c e d u r e s n e c e s s a r y t o t r a n s -l a t e t h e l i n k ed l i s t o f mu s i c n o d e s t ru c tu res i n to an AS CI If o r m a t . F o r t h e d a t a s e t us e d , t h e m o r p h o l o g i c a l s y m b o l r e c -o g n i ze r p ro d u ced en co u rag in g r esu l t s , su b s t an t i a t i n g o u r ea r -l i e r c l a i m t h a t m a t h e m a t i c a l m o r p h o l o g y c o u l d b e u s e d a sa p o w e r f u l t o o l t o a n a l y z e a n d d e t e c t s h a p e s . T h e p o s t p r o -c e s s i n g s t ep s e m p l o y e d r e d u c e a m b i g u i t ie s t h a t w e r e p r e s e n ti n t h e o u tp u t . P o s tp ro cess in g wi l l s i g n i f i can t l y imp ro v e t h ep e r f o r m a n c e o f t h e s y s t e m w h e n t h e n o i s e l e v e l i n t h e i n p u timag e i s q u i t e h ig h . Ad d i t i o n a l mu s i c n o t a t i o n ru l es Ro emer(1 9 8 5 ) co u ld b e u sed as co n s t r a in t s i n t h e r easo n in g s t ep t ofu r t h e r r ed u ce amb ig u i t y .Acknowledgements. We would l ike to acknowledge M s. Bei W ang,who helped us with the laboratory experiments.

ReferencesBaumann S, Dengel A (1987) Transforming printed piano musicinto midi. SSPR 90(9):532-550Costa M (1 990 ) A practical guide to task-oriented seq uences of

morphological operations for use in image analysis. Technical

Repo rt EE-ISL-90-01, Department of Electrical E ngineering,University of W ashington

Fujinaga I (1988) Optical music recognition using projections. M.S.Thesis, McG ill University, Faculty of Music, M ontreal, CanadaHaralick RM, Sternberg SR, Zhuang X (1987) Im age analysis math-

ematical morphology. IEEE Trans Pattern Analysis MachineIntelligence 9:532-550

Moda yur BR (1991) Morphological algorithms for printed musicscore recognition. Technical report EE-ISL-91. Department ofElectrical Engineering, U niversity of Wa shington (In prepara-tion)

Mo dayur BR, H aralick RM (1991) Music score recognition usingmathematical morphology. In: Proceedings of the Fifth Interna-tional Conference on Symbolic and Logical Computing. Madi-son, S.D ., April 1991

Prerau DS (1970) Computer pattern recognition of standard en-graved music notation. Ph.D. dissertation, MIT

Pruslin DH (1966) Automatic recognition of sheet music. Sc.D.dissertation, MIT

Read G (1969) Music notation - a manual of modern practice, 2rided. AU yn Bacon, BostonRoach JW, T atem JE (1988) Using domain knowledge in low-levelvisual processing to interpret handwritten music: an experi-ment. Pattern Recognition 21(1):33-44Roemer C (1985) The art of music copying: the preparation ofmusic and performance, 2rid edn. Roederick Mu sic Company,Sherman Oaks