Joint AMI/Pascal/IM2/M4/ workshop, June 21-23 2004, Martigny EU research initiatives in multimodal interaction Mats Ljungqvist European Commission Information.

Joint AMI/Pascal/IM2/M4/ workshop, June 21-23 2004, Martigny

EU research initiatives in multimodal interaction

EU research initiatives in multimodal interaction

Mats LjungqvistMats Ljungqvist

European CommissionEuropean Commission

Information Society DG,Information Society DG,Interfaces (Unit E1)Interfaces (Unit E1)

Mats LjungqvistMats Ljungqvist

European CommissionEuropean Commission

Information Society DG,Information Society DG,Interfaces (Unit E1)Interfaces (Unit E1)


Outline of PresentationOutline of Presentation

• IntroductionIntroduction

• Who can participate?Who can participate?

• ““The story so far….”The story so far….”

• FP5 activitiesFP5 activities

• Overview FP6 call 1, 2 and 3Overview FP6 call 1, 2 and 3• Call 1: Multimodal InterfacesCall 1: Multimodal Interfaces• Call 2: Cognitive SystemsCall 2: Cognitive Systems• Call 3: FET - Future and Emerging TechnologiesCall 3: FET - Future and Emerging Technologies

• The IST workprogramme for 2005-2006The IST workprogramme for 2005-2006

• Calls 4 and 5Calls 4 and 5

• Preparation of the 7th EU Framework Programme Preparation of the 7th EU Framework Programme 2007-20112007-2011


Communication &

Computing Infrastructures

Components &

Micro-Systems

Knowledge&

Interface Technologies

Applied IST Research addressing Major Societal & Economic Challenges

IST in FP6 (~4 Billion Euro, 2003-2006)

IST in FP6 (~4 Billion Euro, 2003-2006)


Who can participate?Who can participate?

• All member statesAll member states

• States associated to FP6States associated to FP6: : same conditions as same conditions as member statesmember states

– Bulgaria, Romania, Turkey, Iceland, Israel, Bulgaria, Romania, Turkey, Iceland, Israel, Liechtenstein, Norway, SwitzerlandLiechtenstein, Norway, Switzerland

• "INCO countries“:"INCO countries“: specific earmarked funding budgetspecific earmarked funding budget– Russia, New Independent States, Mediterranean Russia, New Independent States, Mediterranean

Countries, Western Balkans, Developing countries Countries, Western Balkans, Developing countries

• CountriesCountries with co-operation agreementwith co-operation agreement participate participate without EU funding (exceptionally with funding)without EU funding (exceptionally with funding)

– Argentina, Australia, Brazil, Canada, China, Chile, India, Argentina, Australia, Brazil, Canada, China, Chile, India, Japan, Kazakhstan, Russia South-Africa, Ukraine, USAJapan, Kazakhstan, Russia South-Africa, Ukraine, USA

• Other countriesOther countries participate without funding participate without funding (exceptionally with funding).(exceptionally with funding).


Who can participate?Who can participate?

• Another possibilityAnother possibility

• Join a project as unfunded “external member” Join a project as unfunded “external member” sometimes possible according to modalities set up sometimes possible according to modalities set up by the consortium. by the consortium.

• Examples in some language resources projects Examples in some language resources projects (China, Canada, USA, …).(China, Canada, USA, …).

• Examples of funded and unfunded Examples of funded and unfunded participation (from 1participation (from 1stst FP6 MMI call) FP6 MMI call)

• Russia, USA, Senegal, Romania, Turkey, ….Russia, USA, Senegal, Romania, Turkey, ….


Cognition and InterfacesCognition and Interfaces

• Building on activities in earlier framework Building on activities in earlier framework programmes (…,ESPRIT, Telematics, FP5)programmes (…,ESPRIT, Telematics, FP5)

• Human Language TechnologiesHuman Language Technologies– speech, multilinguality, translationspeech, multilinguality, translation

• Multimodal, multisensorial dialogueMultimodal, multisensorial dialogue• Signal processing and codingSignal processing and coding• Machine visionMachine vision• Affective computingAffective computing• RoboticsRobotics• Disappearing computerDisappearing computer• Neuroinformatics, presence researchNeuroinformatics, presence research

• Focus in FP6:Focus in FP6:• Multimodal Interfaces (MMI)Multimodal Interfaces (MMI)• Cognitive Systems (CS)Cognitive Systems (CS)• Beyond robotics, life-like perception systems, Bio-inspired Beyond robotics, life-like perception systems, Bio-inspired

Intelligent Information Systems (in FET)Intelligent Information Systems (in FET)

Interface related work also in FP6 application areasInterface related work also in FP6 application areas


• Projects in FP5 are laying the foundations for FP6 initiatives in the area:Projects in FP5 are laying the foundations for FP6 initiatives in the area:

• KERMIT Kernel Methods for Images and Text • M4 MultiModal Meeting Manager

• Nespole! Negotiating through SPOken Language in E-commerce

• FAME Facilitating Agent for Multicultural Exchange • PF_STAR Preparing Future Multisensorial Interaction Research

• SIRIDUS Specification, Interaction and Reconfiguration In Dialogue Understanding Systems

• ALERT Alert System for Selective Dissemination of Multimedia Information

• CORETEX Improving Core Speech Recognition Technology

• VISATEC Vision-based Integrated Systems Adaptive to Task and Environment with Cognitive abilities

• ECVISION European Research Network for Cognitive AI-enabled Computer Vision System

• LAVA Learning for Adaptable Visual Assistants

• LC_STAR Lexica and Corpora for Speech-to-Speech Translation Technologies

• SPEECON Speech Driven Interfaces for Consumer Applications

• SPEECHDAT Speech Databases for Creation of Voice Driven Teleservices

• ORIENTEL Multilingual access to interactive communication services for the Mediterranean and the Middle East

• NECA A Net Environment for Embodied Emotional Conversational Agents

• HUMODAN An automatic human model animation environment for augmented reality interaction

• SMARTSKETCHES A multimodal approach to improve usability in the early stages of product design

• FET initiatives

• …and more at… www.cordis.lu/ist -->projects

FP5 projectsFP5 projectsFP5 projectsFP5 projects

Vision

Vision

Haptic

s

Haptic

s

Spee

ch/

Spee

ch/

lang

uag

lang

uag

ee

Resou

rces

Resou

rces

Mul

timod

ality

Mul

timod

alityLe

arni

ng

Lear

ning

Emot

ions

Emot

ions


• Interaction between and among human and the virtual and physical environmentInteraction between and among human and the virtual and physical environment

• Multilingual systemsMultilingual systems

• ResultsResults

• 14 proposals selected for negotiations14 proposals selected for negotiations

• Most contracts signed December 2003Most contracts signed December 2003

• Funding Funding

• 73 Meuro73 Meuro

Call 1 overview Multimodal Interfaces (MMI)

Call 1 overview Multimodal Interfaces (MMI)

Develop natural and adaptive multimodal interfaces, that respond intelligently to speech and language, vision, gesture, haptics and other senses

Develop natural and adaptive multimodal interfaces, that respond intelligently to speech and language, vision, gesture, haptics and other senses


MMI call results: MMI call results: NoE NoE (Networks of Excellence)(Networks of Excellence)

MMI call results: MMI call results: NoE NoE (Networks of Excellence)(Networks of Excellence)

• SIMILAR - SIMILAR - The European taskforce creating human-machine The European taskforce creating human-machine interfaces SIMILAR to human-human communication interfaces SIMILAR to human-human communication • multimodal interface research, multiple sensory modalities, fusion of modalities:

visual interaction, interactive speech, interactive haptics, learning, and human human-computer interaction (HCI)

• HUMAINE - HUMAINE - Human-Machine Interaction Network on EmotionHuman-Machine Interaction Network on Emotion• study of systems that can register, model and/or influence human emotional and

emotion emotion-related states - central to future interfaces.

• ENACTIVE - Enactive InterfacesENACTIVE - Enactive Interfaces• interfaces and interaction paradigms based on Enactive Knowledge, i.e.,

knowledge stored in the form of motor responses and acquired by the act of "doing” - a form of cognition inherently tied to actions

• PASCAL - PASCAL - Pattern Analysis, Statistical Modelling and Pattern Analysis, Statistical Modelling and Computational LearningComputational Learning• pattern analysis, statistical modelling, and computational learning as core

enabling technology for discovering, structuring and presenting complex information, or for processing of complex sensory data.

Academic research in key enabling technologiesAcademic research in key enabling technologies


MMI call results: MMI call results: IP IP (Integrated Projects)(Integrated Projects)

MMI call results: MMI call results: IP IP (Integrated Projects)(Integrated Projects)

• CHIL - CHIL - Computers In the Human Interaction LoopComputers In the Human Interaction Loop• create environments in which computers serve humans who focus on interacting

with other humans

• AMI - AMI - Augmented Multi-party InteractionAugmented Multi-party Interaction• new multimodal technologies to support human interaction, in the context of

smart meeting rooms and remote meeting assistants.

• TC-STAR - TC-STAR - Technology and Corpora for Speech to Speech Technology and Corpora for Speech to Speech TranslationTranslation• advanced research in all core technologies for Speech-to-Speech-Translation:

speech recognition, translation, and synthesis.

- supporting human-human interaction- perception in multiple modalities

- supporting human-human interaction- perception in multiple modalities


MMI call results: MMI call results: STREP STREP (Strategic Targeted Research Projects)(Strategic Targeted Research Projects)

MMI call results: MMI call results: STREP STREP (Strategic Targeted Research Projects)(Strategic Targeted Research Projects)

• MATRIS -MATRIS -Markerless Real-time Tracking for Augmented Reality Markerless Real-time Tracking for Augmented Reality Image SynthesisImage Synthesis• a system for markeless tracking, by analysis of the camera images and

exploitation of unobtrusive inertial motion sensors. This will enable the system as a whole to determine its location and orientation mimicking the way a human orients himself.

• DIVINES - Diagnostic and Intrinsic Variabilities in Natural SpeechDIVINES - Diagnostic and Intrinsic Variabilities in Natural Speech• improving speech recognition performance by research into feature extraction

and modelling techniques that would have better capacities, particularly in handling speech intrinsic variabilities

• TALK - Talk and Look, Tools for Ambient Linguistic KnowledgeTALK - Talk and Look, Tools for Ambient Linguistic Knowledge• to advance our understanding of generic technologies that will extend the

Information State Update" (ISU) approach to adaptive multimodal and multi multi-lingual interaction for developing adaptive multimodal dialogue systems.

• T’nD - T’nD - Touch and DesignTouch and Design• shape generation and modification based on novel haptic interaction and

intelligent shape manipulation operators in order to exploit existing manual skill of designers

- small focussed efforts- bootstrapping new research areas

- small focussed efforts- bootstrapping new research areas


MMI call results: MMI call results: STREP, SSA STREP, SSA (Specific Support Action)(Specific Support Action)

MMI call results: MMI call results: STREP, SSA STREP, SSA (Specific Support Action)(Specific Support Action)

• TAI-CHI - TAI-CHI - Tangible Acoustic Interfaces for Computer-Human Tangible Acoustic Interfaces for Computer-Human InteractionInteraction• development of tangible interfaces that employ physical objects and space as

media to bridge the gap between the virtual and physical worlds by exploring how physical objects, augmented surfaces and space can be transformed into tangible acoustic embodiments of natural seamless unrestricted interfaces.

• HIWIRE - Human Input That Works In Real EnvironmentsHIWIRE - Human Input That Works In Real Environments• embedded robust multimodal dialogue systems with flexible speech input in

mobile, open and noisy environments (e.g. aeroplanes, cars, street etc.).

• MWEB - Multimodal Web InteractionMWEB - Multimodal Web Interaction• support for the development and adoption of industry standards (W3C

Recommendations) enabling multimodal Web access using mobile devices.


Call 2 overview Cognitive Systems

Call 2 overview Cognitive Systems

• Artificial systems that combine perception, action reasoning, Artificial systems that combine perception, action reasoning, learning and communicationlearning and communication

• Provide enabling technologies for robotics, natural language Provide enabling technologies for robotics, natural language understanding, man-machine interaction, complex real world understanding, man-machine interaction, complex real world systemssystems

• ResultsResults

• Negotiations with Negotiations with 8 projects8 projects concluded concluded

• Most contracts to be signed in July-August 2004Most contracts to be signed in July-August 2004

• Funding Funding

• Around 25 MeuroAround 25 Meuro

Construct physically instantiated or embodied systems that can perceive (‘see’, ‘hear’, ’smell’), understand the semantics of information conveyed through their perceptual input (‘recognise’, ‘categorise’) and interact with their environment (‘decide’, ‘act’), and evolve in order to achieve human-like performance in activities requiring context specific knowledge.

Construct physically instantiated or embodied systems that can perceive (‘see’, ‘hear’, ’smell’), understand the semantics of information conveyed through their perceptual input (‘recognise’, ‘categorise’) and interact with their environment (‘decide’, ‘act’), and evolve in order to achieve human-like performance in activities requiring context specific knowledge.


What is needed?What is needed?

• Cognitive SystemsCognitive Systems

• Extensive Extensive foundational work of exploratory naturefoundational work of exploratory nature• ““Cover ground” (as we don’t really know yet what is important)Cover ground” (as we don’t really know yet what is important)

– The nature of cognition, knowledge, the role of language The nature of cognition, knowledge, the role of language

– ArchitecturesArchitectures

– Perception, learningPerception, learning

– Autonomy, notion of self, social interactionAutonomy, notion of self, social interaction

• InterdisciplinaryInterdisciplinary partnerships partnerships

• Multimodal InterfacesMultimodal Interfaces

• Significant advances in key areas:Significant advances in key areas:• ““EmotionalEmotional interfaces” interfaces”

• Interfaces that are Interfaces that are anticipatoryanticipatory, learning, evolving and emerging: , learning, evolving and emerging: the “the “Do what I meanDo what I mean” paradigm” paradigm

• Seamless Seamless controlcontrol of IT enabled intelligent environment of IT enabled intelligent environment

• Fusion/synchronisation Fusion/synchronisation of multiple modalitiesof multiple modalities

• Real-time, better than human Real-time, better than human interpretation of everyday languageinterpretation of everyday language


IST in FP6: Work Programme 2005-2006

IST in FP6: Work Programme 2005-2006

• Evolution of running Strategic ObjectivesEvolution of running Strategic Objectives

• Calls in Dec 2004 or 2005

• Multimodal interfaces call in Dec 2004 or in 2005 ~50 m Euro?

• Cognitive systems ~50 m Euro?

• Applications and Services for the Mobile User & Worker

• Open Web Consultation and Consultation Meetings in April 2004

• Reports will be made available www.cordis.lu/ist

• IST-FET: New Research InitiativesIST-FET: New Research Initiatives

• Bio-inspired Intelligent Information Systems• ~20 m Euro?, Call closes on 22 Sept

• Presence and Interaction in Mixed Reality Environments• ~20 m Euro?, Call in Dec 2004?

• Creating and Moulding Ambience• Call in 2005?

• New IST Work programme: autumn 2004New IST Work programme: autumn 2004www.cordis.lu/istwww.cordis.lu/ist


The Roadmap to FP7

The Roadmap to FP7

7th FP

Feb ’04Financial Perspectives

(2007-2013)

2007 – 2011 ...

12 May ’04Preparing the future:

reinforcing European Research Policies

Early ’05Proposal

on FPAdoption FP

2003 2004 2005 20066th FP


FP7: 6+2 axesFP7: 6+2 axes

• CollaborationCollaboration Collaborative ResearchCollaborative Research

• Private/public partnershipPrivate/public partnership Technological platformsTechnological platforms

• Individual research teams Individual research teams Basic Research GrantsBasic Research Grants

• Human resourcesHuman resources Mobility and Life Long LearningMobility and Life Long Learning

• Research Infrastructures Research Infrastructures Access/Networking & support of new Access/Networking & support of new InfrastructuresInfrastructures

• Co-ordination of national & Co-ordination of national & European Research AreaEuropean Research Area

regional research programmesregional research programmes

and policiesand policies

++

• Space (ESA)Space (ESA)

• SecuritySecurity

ICT research maps into all 6+2 axesICT research maps into all 6+2 axes


ContactsContacts

• Next callsNext calls

• Call 3 now open (closes 22 September)Call 3 now open (closes 22 September)

• Call 4 in late 2004, call 5 in 2005Call 4 in late 2004, call 5 in 2005• Based upon 2005-06 Work Programme (Oct/Nov Based upon 2005-06 Work Programme (Oct/Nov

2004)2004)

• ContactContact

• www.cordis.lu/ist/so/interfaceswww.cordis.lu/ist/so/interfaces

• [email protected]@cec.eu.int

IST helpdeskIST helpdeskFax : +32 2 296 83 88Fax : +32 2 296 83 88e-mail : [email protected] : [email protected]

Joint AMI/Pascal/IM2/M4/ workshop, June 21-23 2004, Martigny EU research initiatives in multimodal interaction Mats Ljungqvist European Commission Information.

Documents

countries countries

eu funding

joint amipascalim2m4

funding argentina

fp6 mmi

mediterranean countries

unfunded external member

member states bulgaria