-
8/3/2019 Multi Modal Human Computer Interaction Research
1/2
C H I 9 7 . 2 2 - 2 7 M A R C H 1 9 9 7 O R G A N I Z A T I O N A L O V E R V I E W S
M u l t i m o d a l H u m a n C o m p u t e r I n t e r a c t i o n R e s e a r c ha t T o s h i b a R e s e a r c h a n d D e v e l o p m e n t C e n t e rY o i c h i T a k e b a y a s h i a n d M i w a k o D o iH u m a n I n te r fa c e T e c h n o l o g y C e n te r ,
R e s e a rc h a n d D e v e l o p m e n t C e n te r , T o s h i b a C o r p o r a t i o n1 K o m u k a i - T o s h i b a - c h o , S a i w a i -k u , K a w a s a k i 2 1 0 , J a p a n+81 44 549 2243{ yo ic h i , do i} @ e e l . r dc . t o sh iba . c o . jp
A B S T R A C TT o s h i b a ' s H u m a n I n t e r f a c e R e s e a r c h G r o u p i s p u r -s u i n g m e d i a u n d e r s t a n d i n g a n d i n t e l l i g e n t i n t e r a c -t i o n t e c h n o l o g i e s t o a c h i e v e n a t u r a l m u l t i m o d a l H C I( h u m a n - c o m p u t e r in t e r a ct i o n ) . I n c o l la b o r a ti o n w i t hT o s h i b a ' s o t h e r c o r p o r a t e l a b o r a t o r i e s , e n g i n e e r i n gl a b o r a t o r i e s a n d b u s i n e s s d i v i s i o n s , w e h a v e b e e nd e v e l o p i n g p r a c ti c a l i n t e r a c t i v e s y s t e m s a n d p r o d -u c t s r e l a t e d t o i n f o r m a t i o n s e r v ic e s , c o n s u m e r e le c -t r o n i c s, d o c u m e n t fi li n g a n d i n d u s t r i a l e q u i p m e n t .K E Y W O R D SO r g a n i z a t i o n s , m u l t i m o d a l , H C I , in f o r m a t i o n fi lt e r-i n g, k n o w l e d g e s h a ri n g , m e d i a u n d e r s t a n d i n g .O R G A N I Z A T I O N A N D R E S E A R C HT H E M E SB e s t k n o w n f o r i t s w o r l d ' s fi r s t l e t t e r h a n d l i n g s y s -t e m u s i n g h a n d - w r i t t e n c h a r a c t e r r e c o g n i t i o n a n dJ a p a n e s e w o r d p r o c e s s o r u s i n g K a n a - t o - K a n j i c o n -v e rs i on , T o s h i b a R e s e a r c h a n d D e v e l o p m e n t C e n t e r( R D C ) h a s b e e n d e v e l o p i n g v a r i o u s m e d i a c o n v e r-s i o n / u n d e r s t a n d i n g s y s t e m s a n d n a t u r a l l a n g u a g ep r o c e s s i n g s y s t e m s .W e b e l i ev e t h a t t h e s e t e ch n o l o g ie s p la y i m p o r t a n tr o l e s i n a c h i e v i n g u s e r - c e n t e r e d m u l t i m o d a l h u m a nc o m p u t e r i n te r a c t io n . W h i l e a d v a n c es in c o m p u t -i n g e n v i r o n m e n t s h a v e h e lp e d u s g a t h e r a n d s h a r e al a rg e a m o u n t o f m u l i m e d i a d a ta , t h e y c a u s e t h e s i t-u a t i o n i n w h i c h w e h a v e b e e n f o rc e d t o w o r k u n d e rs t r e s s d u e t o a f l o o d o f i n f o r m a t i o n . T o s o l v e t h i sp r o b l e m , w e a r e f o c u s i n g o u r H C I r e s ea r c h o n i n f o r-m a r i o n r e t ri e v a l a n d k n o w l e d g e s h a r i n g , b a s e d o nm e d i a u n d e r s t a n d i n g t e c h n o l o g i e s . S p e c i fi c al ly , w eh a v e b e e n e x p l o r i n g u s e r' s i n t e n t i o n a n d c o n t e n t s o fm u l t i m e d i a d a t a f r o m t h e t h e v i e w p o i n t o f m e d i ac o n v er s io n a n d u n d e r s t a n d i n g f u n c t i o n s a n d m u l t i -m o d a l i n t e r fa c e , b e c a u s e t h e i r f u ll u n d e r s t a n d i n g i sc r u c i al i n r e t r i e v i n g u s e f u l i n f o r m a t i o n .
T o s h i b a ' s H u m a n I n t er f ac e T e c h n o l o g y C e n t e r ( H IC )w a s e s t a b l i she d in 1995, a s a c o r po r a t e o r ga n iz a -t i o n , a i m i n g t o a c h i e v e h u m a n - c e n t e r e d r e l ia b l e m e -d i a t e c h n ol o g i es i n h a r m o n y w i t h o u r h u m a n s o -c i et y . T o a p p l y t h e s e t e c h n o l o g i e s t o v a r i o u s s y s-t e m s a n d p r o d u c t s , a b o u t 3 0 r e s e a r c h e r s a r e c o l -l a b o r a t i n g w i t h o t h e r o r g a n i z a t i o n s , i n c l u d i n g t h o s ei n c h ar g e o f c o m p u t e r a n d c o m m u n i c a t i o n s y s t e m s ,c o n s u m e r e le c t r o n ic s , p o w e r s y s t e m s , a n d i n d u s -t r i al e q u i p m e n t . O u r w o r k w i d e l y c o v e r s m e d i ac o n v e r s i o n / u n d e r s t a n d i n g f u n c t i o n s , f ro m c h a ra c -t e r r e c o g n i t i o n , t o d o c u m e n t u n d e r s t a n d i n g , n a t u -r al l a n g u a g e u n d e r s t a n d i n g , a s w e l l a s m e d i a i n t er -a c t i o n s u c h a s in f o r m a t i o n f i lt e r in g , k n o w l e d g e / i n -f o r m a t i o n s h a r i n g , s p e e c h d i a l o g u e , v i d e o b r o w s i n ga n d h u m a n f a ct o rs . F i g u r e 1 sh o w s t h e f r a m e w o r ko f i n f o r m a t i o n r e t r ie v i n g a n d s h a r i n g s y s t e m u s i n ga s e t o f m e d i a c o n v e r s i o n / p r o c es s i n g f u n c t i o n s " H I -w a r e " .a n s w e r s q u e s t io n s
" " ~ " ~i l m a n a g e m e n tL - J - - ' ~i i I = "i i
m u l ti m e d i a n f o r m a t io n
Sr a r e s t s tr u c t r iz a ti o n
~ s t r u c t u r ~ z a t i o n 1~.............. .~.l~..r~.~..~]!9~
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ]] / f~ d o c u m e n tD BJ L r g a n i z a t io n D B/ p e r s o n n e l D B
F i g u r e 1 : F r a m e w o r k o f i n f o r m a t i o n r e t r i e v i n g a n ds h a r i n g s y s t e mA P P R O A C HS t r u c t u r i n g M u l t i m e d i a I n f o r m a t i o nI n o r d e r t o c r e a t e u s e r - c e n t e r e d m u l t i m o d a l i n t e r -
7 9
-
8/3/2019 Multi Modal Human Computer Interaction Research
2/2
O R G A N I Z A T I O N A L O V E R V I E W S C H I 9 7 . 2 2 - 2 7 M A R C H 1 9 9 7face, it is vitally important to structure multime-dia information using media conversion. Structu redmultimedia information enables both humans andcomputers to share and retrieve information as theywish and to have a better understanding of eachother.The fundamental basis for this task is knowledge-bases and language dictionaries, which we are cur-rently building.E n h a n c i n g M u l t i m o d a l I n t e r a c t i o nWe need to upgrade intelligent multimodal humancomputer interaction technologies using agents sothat users can find more enjoyment and comfortin working with computers. This means creatinga system which understands users' intentions andsituations from their utterances and gestures andprovides such services as information retrieval, ad-vices and suggestions, and whatever help they need,while directing a natural dialogue with users.D e v e l o p i n g S e n s o r s a n d I n p u t - o u t p u t D e v i c e sFinally, we are also investigating new sensors andinput -outp ut devices. They extra ct information userspresented both voluntarily and unintentionally. Suchinformation facilitates the development of mediaunderstanding technologies and makes it possiblefor computers to understand users' intention andsituations.The h uman brain consists of thousand s of architec-tural types of computers, each of which has variousfunctions including voice understandi ng, scene un-derstanding, language understanding, translation,dialogue, speed reading, and problem-solving. Thus,a future picture for intelligent multimodal inter-faces could be realized by upgrading conversion ofeach multi-level media and integrati ng those mediathrough the organization of knowledge-bases andlanguage dictionaries, thereb y assisting huma ns' aca-demic activities. Now tha t environm ents for digi-tal information are being set, we should acceler-ate our research and development for highly ad-vanced acquisition, sharing, and dispatch of knowl-edge/information.S E L E C T E D P R O J E C T SI n f o r m a t i o n F i l te r i n g S y s t e mWe have developed an information filtering systemfor newspaper articles published every day in digi-tal form. The system c omputes similarities betweenthe user's information need and each article basedon our expanded vector space model, and then se-lects articles suitable to his need. The system alsodetects other similar articles, so tha t it can indicatea cluster of similar articles. The selected newspaperarticles are provided to users by using communica-tion tools in the Internet, e.g., e-mail and WWW
(World Wide Web). This syste m is being used asJapan' s first information filtering service.P e r s o n a l I n f o r m a t i o n P r o v i d e rWe have been developing a multimodal PersonalInformation Provider(PIP), for enhancing informa-tion/knowledge sharing and closer human relationsamong groups. This system employs natura l lan-guage and emotion understanding from speech andkeyboard input, with a user-initiative dialogue man-ager and multimodal response generator. The sys-tem runs in real time on a personal computer withan interface agent to make the user's stored infor-mation open to others under the user's permission.Experiments based on the PIP are being performedon about 300 people for knowledge and know-howsharing in our laboratories. Figure 1 shows the ad-vice/help on demand system for our office knowl-edge/know-how sharing system.H I - w a r e ( C o m m o n H I s e rv ic e e n v ir o n m e n t )We have been developing HI-ware (Com mon HI Ser-vice Environment) where various kinds of HI func-tions such as speech recognition/synthesis, charac-ter recognition, and machine translation are easilyand organically available to develop advanced HCI.As shown in Figure 2, the enviro nment has two fea-tures. One is standardized AP I (Application Pro-grammi ng Interface). The API keeps consistencyamong HI functions so that various kinds of HIapplications can incorporate them in the commonmanner. The other feature is a common dictionaryshared among HI functions; a new word registeredin the common dictionary can be provided to all HIfunctions.
Applications
Speech recognition ]~,,,,,~A C h a r a c t e r r e c o g n i t i o r ]~ _ . . .i Knowledge (
: - e ] - - I C o m % n I, c t , o nAbstrac,on 1 7
Figure 2: Configuration of HI-warePUBLICATIONS1. Aoki, H. et al. "A Shot Classification Methodto Select Effective Key-f rames for Video Browsing,"Proc. ACM Multimedia'96, 19962. Ono, K. et al. :"Abstr act Generation Basedon Rhetorical Struct ure Extract i on," COLING '94,1994.3. Miike, S. et al. : "A Full-Text Retrieval Systemwith a Dynamic Abstract Generation Function,"Proc. SIGIR '94, 1994.
8 0
