Vision Technologies, Software Architecture & Processing Strategy in the UPC Smart Room Doctorado en Informática y Telecomunicación EPS-UAM, May 19th 2006 Josep R. Casas UPC – Technical University of Catalonia 1 Vision Technologies, Software Architecture & Vision Technologies, Software Architecture & Processing Strategy in the UPC Smart Room Processing Strategy in the UPC Smart Room Josep R. Casas Josep R. Casas UPC UPC – Image Processing Group Image Processing Group Doctorado en Ingeniería Informática y de Telecomunicación Escuela Politécnica Superior – UAM May 19 th 2006 Vision Technologies, Software Architecture & Processing Strategy in the UPC Smart Room EPS-UAM, May 19 th 2006 Service, architecture, integration Service, architecture, integration – Joachim Neumann – Jordi Salvador (Daniel Almendro, Shadi El-Hajj) Video Technologies Video Technologies – Cristian Cantón (Body & Gesture) – Josep R. Casas – Christian Ferran (Object Detection) – Xavi Giró (Object Detection) – José Luis Landabaso (Det/Track) – Miriam León (Text Detection & OCR) – Ferran Marqués (Face Det + ID) – Ramon Morros (Face ID +Det) – Montse Pardás (Activity & Emotion) – Javier Ruiz (software APIs) – Verónica Vilaplana (Face Det) UPC Smart Room Team UPC Smart Room Team Audio Technologies Audio Technologies – Alberto Abad – Mireia Farrus – Javier Hernando – Jordi Luque – Dušan Macho – Climent Nadeu – Carlos Segura – Andrey Temko NLP Technologies NLP Technologies – Pere Comas – Maria Fuentes – Edgar González – Mihai Surdeanu – Jordi Turmo
28
Embed
Vision Technologies, Software Architecture & …arantxa.ii.uam.es/~jms/seminarios_doctorado/abstracts...Computer services supporting Human-Human interaction Human Human Computer Data
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Vision Technologies, Software Architecture & Processing Strategy in the UPC Smart Room
Doctorado en Informática y TelecomunicaciónEPS-UAM, May 19th 2006
Josep R. CasasUPC – Technical University of Catalonia 1
Vision Technologies, Software Architecture & Vision Technologies, Software Architecture & Processing Strategy in the UPC Smart RoomProcessing Strategy in the UPC Smart Room
Josep R. CasasJosep R. CasasUPC UPC –– Image Processing GroupImage Processing Group
Doctorado en Ingeniería Informática y de TelecomunicaciónEscuela Politécnica Superior – UAM
May 19th 2006
Vision Technologies, Software Architecture & Processing Strategy in the UPC Smart Room
EPS-UAM, May 19th 2006
Service, architecture, integrationService, architecture, integration– Joachim Neumann– Jordi Salvador
(Daniel Almendro, Shadi El-Hajj)
Video TechnologiesVideo Technologies– Cristian Cantón (Body & Gesture)– Josep R. Casas– Christian Ferran (Object Detection)– Xavi Giró (Object Detection)– José Luis Landabaso (Det/Track)– Miriam León (Text Detection & OCR)– Ferran Marqués (Face Det + ID)– Ramon Morros (Face ID +Det)– Montse Pardás (Activity & Emotion)– Javier Ruiz (software APIs)– Verónica Vilaplana (Face Det)
UPC Smart Room TeamUPC Smart Room Team
Audio TechnologiesAudio Technologies– Alberto Abad– Mireia Farrus– Javier Hernando– Jordi Luque– Dušan Macho– Climent Nadeu– Carlos Segura– Andrey Temko
NLP TechnologiesNLP Technologies– Pere Comas– Maria Fuentes– Edgar González– Mihai Surdeanu– Jordi Turmo
Vision Technologies, Software Architecture & Processing Strategy in the UPC Smart Room
Doctorado en Informática y TelecomunicaciónEPS-UAM, May 19th 2006
Josep R. CasasUPC – Technical University of Catalonia 2
Vision Technologies, Software Architecture & Processing Strategy in the UPC Smart Room
“We perform high-level, forward looking, long term research…”“This is not good for my PhD…”“This is long term research, and will never be useful for a product in the market”
(company)“I’m sure someone will find it useful for something…” (researcher)
!! Researchers (in Engineering) should envision actual applicationResearchers (in Engineering) should envision actual applicationss……
Vision Technologies, Software Architecture & Processing Strategy in the UPC Smart Room
Doctorado en Informática y TelecomunicaciónEPS-UAM, May 19th 2006
Josep R. CasasUPC – Technical University of Catalonia 3
Vision Technologies, Software Architecture & Processing Strategy in the UPC Smart Room
EPS-UAM, May 19th 2006
Framework: CHIL project Framework: CHIL project
““Computers in Human Interaction LoopComputers in Human Interaction Loop””
Instead of involving humans in the workflow and Instead of involving humans in the workflow and programmatic tasks defined and scheduled by programmatic tasks defined and scheduled by machines (explicit operation, keyingmachines (explicit operation, keying--in commandsin commands……))
The visionThe vision!! putput the Computers in the Loop of humansthe Computers in the Loop of humans
observing humans, observing humans, engaging and interacting with humans,engaging and interacting with humans,predicting and proactively providing services,predicting and proactively providing services,acting on perceived human need, acting on perceived human need, intruding as little as possible intruding as little as possible
(hovering in the background as (hovering in the background as electronic butlerselectronic butlers))
Vision Technologies, Software Architecture & Processing Strategy in the UPC Smart Room
EPS-UAM, May 19th 2006
Framework: CHIL servicesFramework: CHIL services
•• Provide computing services implicitly Provide computing services implicitly … by putting Computers in the Interaction Loop of Humans… by observing humans interacting with humans… by predicting needs and proactively providing services
•• CHIL services instantiated as demonstration prototypesCHIL services instantiated as demonstration prototypes–– ConnectorConnector
Helps people to get in touch (avoids phone tag).It connects people at the right timeright time through the right deviceright device.
–– Memory JogMemory JogReminds you of things. It provides pertinent informationpertinent information at the right timeright time (proactive/reactive, unobtrusive)
–– Socially Supportive WorkspaceSocially Supportive WorkspaceHelps people to work together. It is a Smart TableSmart Table, on which virtual paper is used to increase efficiency in group decisions
–– Relational CockpitRelational CockpitAnalysis of group behavior to improve productivity
Vision Technologies, Software Architecture & Processing Strategy in the UPC Smart Room
Doctorado en Informática y TelecomunicaciónEPS-UAM, May 19th 2006
Josep R. CasasUPC – Technical University of Catalonia 4
Vision Technologies, Software Architecture & Processing Strategy in the UPC Smart Room
•• Expected societal outcomeExpected societal outcome– Reduce preoccupation with technological artifact (techno-clutter)– Improve productivity by use of human context– Improve human experience
•• Expected scientific outcomeExpected scientific outcome– Perception: Full description & understanding of all human communication signals
across multiple modalities (audio, image, speech, language, signs…)!! Functionalities: who, where, what (in/out), how, whyFunctionalities: who, where, what (in/out), how, why……• Robustness in perceptual user interfaces (always on)
– Synthesis: from human-friendly to human-like interfaces!! Functionalities: situation models, strategy, Functionalities: situation models, strategy, proactivityproactivity, politeness, privacy care, politeness, privacy care……• Progress in output interfaces and actuators
•• European Project (6European Project (6thth FP / IST)FP / IST) http://chil.server.dehttp://chil.server.de– 2004 ! 2006 ! 2010 (2nd phase)– 25 M€ (1st phase)– Involves 15 partners from 9 countries
Germany, France, Netherlands, Sweden, Italy, Check Rep, Greece, Spain, US
Vision Technologies, Software Architecture & Processing Strategy in the UPC Smart Room
Perception to Action Perception to Action ——…… Realizing the CHIL VisionRealizing the CHIL Vision
CHIL visionCHIL vision!! putput the Computers in the Loop of humansthe Computers in the Loop of humans
observing humans, observing humans, engaging and interacting with humans,engaging and interacting with humans,predicting and proactively providing services,predicting and proactively providing services,acting on perceived human need, acting on perceived human need, intruding as little as possible intruding as little as possible
(hovering in the background as (hovering in the background as electronic butlerselectronic butlers))
What do we need to realize this vision?What do we need to realize this vision? Instantiated into ServicesInstantiated Instantiated
into Servicesinto Services
Vision Technologies, Software Architecture & Processing Strategy in the UPC Smart Room
Doctorado en Informática y TelecomunicaciónEPS-UAM, May 19th 2006
Josep R. CasasUPC – Technical University of Catalonia 5
Vision Technologies, Software Architecture & Processing Strategy in the UPC Smart Room
EPS-UAM, May 19th 2006
Technology frameworkTechnology framework
•• HypothesisHypothesis“Multimodal interface technologies mature enough to get computers listening, watching,
talking, helping…”!! New generation of computer servicesNew generation of computer services
•• Technology areasTechnology areas– Perception from sensors ! who, where, what, how, why…– Modeling/Understanding ! predict, interpret situation– Managing Interaction ! proactive/reactive, natural, friendly, polite, privacy– Synthesis from actuators ! audio, video, calls, signs, text– Software Architecture ! integration, interoperation!! Specific challenges in audioSpecific challenges in audio--visual technologiesvisual technologies
•• Scientific outcome: Scientific outcome: ““Technology PushTechnology Push””– Objective measures of progress & efficiency through open/well-defined technology
evaluations !! Technology catalogueTechnology catalogue– User studies and User evaluations
Vision Technologies, Software Architecture & Processing Strategy in the UPC Smart Room
•• Video Signals Video Signals –– multiple camerasmultiple camerasEnabling tech: Foreground Detection
– Person Location & Tracking (PLT)Enabling tech: Face Detection
– Face Identification (Face ID)Combined: ID tracking
– Head-Pose Detection/Orientation
•• Other (e.g. text) Other (e.g. text) –– multiple sourcesmultiple sources– Summarization, Question&Answering
•• Multimodality (MM)Multimodality (MM)– MM Location &Tracking (speaking/not)– MM Identification (Visible/speaking)– MM Head-Pose– MM Events– MM Activity
Vision Technologies, Software Architecture & Processing Strategy in the UPC Smart Room
EPS-UAM, May 19th 2006
AudioAudio--Visual PerceptionVisual Perception
•• Challenges for the Challenges for the ““WhoWho””, , ““WhereWhere”” (1(1stst tier technologies)tier technologies)Tracking people in natural, evolving, unconstrained scenariosPersons behave without constraints, unaware of audio/video sensors
– Location and trackingVisual – background subtraction: error-prone (shadows, occlusion), feature based (e.g.
color): difficult to initialize (color histogram)Audio – high reverberation times (seminars & meeting rooms), impossible to rely on a
direct path to microphones
– Identification technologies Audio – far field (noise, overlap)Visual – wide angle (low-res), occlusionsA + V – unconstrained motion of the people, no assumptions on position/orientation to
facilitate well-posed signals (frontal faces or speakers aiming at sensors)
Vision Technologies, Software Architecture & Processing Strategy in the UPC Smart Room
Doctorado en Informática y TelecomunicaciónEPS-UAM, May 19th 2006
Josep R. CasasUPC – Technical University of Catalonia 8
Vision Technologies, Software Architecture & Processing Strategy in the UPC Smart Room
EPS-UAM, May 19th 2006
Facing challenges (I)Facing challenges (I)
•• Sensor fusion: MultiSensor fusion: Multi--viewviewProbabilistic approach: product of
(UKA ISL) M. Wölfel, K. Nickel, J. McDonough, MLMI 2005
Vision Technologies, Software Architecture & Processing Strategy in the UPC Smart Room
Doctorado en Informática y TelecomunicaciónEPS-UAM, May 19th 2006
Josep R. CasasUPC – Technical University of Catalonia 9
Vision Technologies, Software Architecture & Processing Strategy in the UPC Smart Room
EPS-UAM, May 19th 2006
AudioAudio--Visual PerceptionVisual Perception
•• Challenges for the Challenges for the ““WhatWhat”” (2(2ndnd/3/3rdrd tier technologies)tier technologies)Speech Recognition for continuous large vocabularyconversational speech, overlapped, competing acoustic events
– Automatic Speech Recognition (ASR / AVASR)Audio – far field, partly compensated with beamforming (subject to localization/tracking
performance)Audio – non-native English speakersA + V – all the previous challenges for localization & ID
– Summarization(technology initially designed to work from written text input) Unstructured textual input provided from transcriptions (ASR)
Vision Technologies, Software Architecture & Processing Strategy in the UPC Smart Room
Far field sensors (ubiquitous computers) ! natural use of beamforming from a microphone array
Influence of Accurate Localization on the Word Error Rate
(UKA ISL) M. Wölfel, K. Nickel, J. McDonough, MLMI 2005
55.8%55.8%labeled positionlabeled position58.4%58.4%estimated position (Audio & Video)estimated position (Audio & Video)59.1%59.1%estimated position (Video only)estimated position (Video only)59.8%59.8%estimated position (Audio only)estimated position (Audio only)66.5%66.5%single microphonesingle microphone
Fujinon lenses DV2.2x1.4,5SA2: 1/3", 1.4-3.1mm (84º-126º)– 2 Person Cameras: mid walls, head & shoulders views– 1 Active Camera PTZ:
VideoTec PTH300, Pentax H6ZBME
OtherOther– Sync master: MOTU Digital Time Piece (genlock, Timecode
labels for A/V)– Video Selector MOXIE SVA-801: Real-time monitoring– A/V distributor ELPRO (genlock signal, LTC)– Ad-hoc Software for recording control
Vision Technologies, Software Architecture & Processing Strategy in the UPC Smart Room
Vision Technologies, Software Architecture & Processing Strategy in the UPC Smart Room
Doctorado en Informática y TelecomunicaciónEPS-UAM, May 19th 2006
Josep R. CasasUPC – Technical University of Catalonia 16
Vision Technologies, Software Architecture & Processing Strategy in the UPC Smart Room
EPS-UAM, May 19th 2006
Data collection Data collection –– CHIL Evaluation CampaignsCHIL Evaluation Campaigns
•• Evaluations are Key to Assessing and Driving ProgressEvaluations are Key to Assessing and Driving Progress– Benchmarks, Measures of Performance (MOPs)– User Studies, Measures of Effectiveness (MOEs)– Cooperation + Competition = ““coopetitioncoopetition””
•• Functionalities & TechnologiesFunctionalities & Technologies– Working Group in Each Area– Define Metrics, Databases and Benchmarks– Performance Benchmark Evaluations in Each Area
•• Evaluation CampaignsEvaluation Campaigns– First “Dry-Run”: completed June 2004– Year One: Completed January 2005 (Open to external sites)– Year Two: Completed March/April 2006 (Coordination with NIST)
All analysis information is on the flow. Each module “hooks” to the needed flowsNot flexible (flows must be defined at design time)Real timeNo common memory (each module stores any information needed)
Video capture Video capture clientclient
FG FG Segmentation Segmentation
ClientClient
Video flowVideo flow
Segmentation flowSegmentation flowFace Detection Face Detection
ClientClient
Face flowFace flowBody Analysis Body Analysis
ClientClient
Vision Technologies, Software Architecture & Processing Strategy in the UPC Smart Room
EPS-UAM, May 19th 2006
UPC UPC –– Video TechnologiesVideo Technologies
•• General Object/Body detection & trackingGeneral Object/Body detection & trackingJosé Luis Landabaso
•• Face Detection & IDFace Detection & IDVerónica Vilaplana, Ramon Morros, Ferran Marqués
•• Body & Gesture Analysis / Head PoseBody & Gesture Analysis / Head PoseCristian Canton
Vision Technologies, Software Architecture & Processing Strategy in the UPC Smart Room
Doctorado en Informática y TelecomunicaciónEPS-UAM, May 19th 2006
Josep R. CasasUPC – Technical University of Catalonia 19
Vision Technologies, Software Architecture & Processing Strategy in the UPC Smart Room
EPS-UAM, May 19th 2006
Localization & TrackingLocalization & Tracking
•• Motivation / GoalMotivation / Goal– Continuous monitoring of scene: “who-where” from all available sensors (A/V)– Support higher level tasks: ID, Head Pose, Activity Classification…– Fundamental for services: situation model, targeted audio/video… !
elementary component for context awareness•• Task definitionTask definition
– Locate people in scene• Single Person (speaker) / Multiple Person (everyone)
– Track people positions in time (correspondence problem)– Input from 4 cameras (+zenithal) and
•• ShapeShape--fromfrom--silhouette (classic approach)silhouette (classic approach)Foreground camera points define rays in scene space intersecting object at some unknown depth. Union of visual rays for all points in silhouette defines a generalized cone within which the 3D object must lie
•• Contribution: Contribution: Cooperative Background Modeling Cooperative Background Modeling Background models in each view are cooperatively learnt, using evidence from
all cameras, in a Bayesian framework– Advantages
• Better 2D foreground regions extracted• More accurate 3D foreground volumetric models
•• 3D Location and tracking3D Location and trackingSpatially connected foreground voxels are grouped and tracking is done for 3D
blobs
L.-Q. Xu, J.L. Landabaso, M. Pardàs, "Shadow Removal with Blob-based Morphological Reconstruction for Error Correction“, ICASSP 2005, Philadelphia, USA
Vision Technologies, Software Architecture & Processing Strategy in the UPC Smart Room
Doctorado en Informática y TelecomunicaciónEPS-UAM, May 19th 2006
Josep R. CasasUPC – Technical University of Catalonia 20
Vision Technologies, Software Architecture & Processing Strategy in the UPC Smart Room
EPS-UAM, May 19th 2006
UPC UPC –– Video Video Body detection & tracking Results Body detection & tracking Results
Vision Technologies, Software Architecture & Processing Strategy in the UPC Smart Room
Doctorado en Informática y TelecomunicaciónEPS-UAM, May 19th 2006
Josep R. CasasUPC – Technical University of Catalonia 21
Vision Technologies, Software Architecture & Processing Strategy in the UPC Smart Room
EPS-UAM, May 19th 2006
UPC UPC –– VideoVideoFace DetectionFace Detection
Face DetectionFace Detection•• Low resolution images, small faces: use only color, size & shapeLow resolution images, small faces: use only color, size & shape
descriptors, dondescriptors, don’’t use texturet use texture– Color
Constant color model in the (Cr,Cb) subspace, skin color modeled with a Gaussian distribution
– ShapeAspect ratio of bounding box of regionHaussdorf distance (between region contour and a face shape model)
•• Exploiting temporal information: Exploiting temporal information: – For mask correctionmask correction (to detect faces when the body tracking fails)– For face model adaptation (color and shape)
F. Marqués, V. Vilaplana. “Face segmentation and tracking based on connected operators and partition projection”. Pattern Recognition, 35(3):601-614, 2002
Vision Technologies, Software Architecture & Processing Strategy in the UPC Smart Room
EPS-UAM, May 19th 2006
UPC UPC –– VideoVideoPerson IDPerson ID
Face RecognitionFace Recognition•• Two different aspects:Two different aspects:
– Intra-session and Inter-session identification– Model updating
Vision Technologies, Software Architecture & Processing Strategy in the UPC Smart Room
Doctorado en Informática y TelecomunicaciónEPS-UAM, May 19th 2006
Josep R. CasasUPC – Technical University of Catalonia 24
Vision Technologies, Software Architecture & Processing Strategy in the UPC Smart Room
EPS-UAM, May 19th 2006
Person ID Person ID –– Cont.Cont.
•• ProgressProgress– Hard to assess: changed evaluation conditions
• 1 ! 5 sites, 11 ! 26 individuals• Face ID decoupled from face detection/tracking, • 15 sec training sequences instead of 5 training images
– Audio• For 30 sec training / 5 sec test, error rate dropped from 6.86% to 2.19% (CMU)
– Video• For 10 sec test, 30% (AIT/UKA) improved to 23% (AIT)
– Multimodal• Audio helps video when speaker is present
•• RemarksRemarks– Far-field, unconstrained poses affect video performance greatly– Audio can be trusted, especially for long duration– When audio is present, video seems complementary– When audio is not present?
•• ProspectsProspects– Check evaluation conditions to bring them closer to its use for services– Explore further fusion possibilities (multisensor/multimodality/integration)
Vision Technologies, Software Architecture & Processing Strategy in the UPC Smart Room
EPS-UAM, May 19th 2006
•• MultiviewMultiview video analysis to extract body pose and limbs video analysis to extract body pose and limbs position for gesture and scene understandingposition for gesture and scene understanding– Hierarchical human body model: geometry for analysis
Simple body model
Position analysis, simple body action (standing up, walking,…).
Stick body model
Gesture analysis, 3D tracking over multiplecameras,…
C. Canton-Ferrer, J.R. Casas, M. Tekalp, M. Pardàs, "Projective KalmanFilter: Multiocular Tracking of 3D Locations Towards Scene Understanding", MLMI2005, Edinburgh, July 2005
Vision Technologies, Software Architecture & Processing Strategy in the UPC Smart Room
Doctorado en Informática y TelecomunicaciónEPS-UAM, May 19th 2006
Josep R. CasasUPC – Technical University of Catalonia 25
Vision Technologies, Software Architecture & Processing Strategy in the UPC Smart Room
– Pan Correct Classification within neighbour range [%]
New taskNew task: Acoustic (!) IRST demo on : Acoustic (!) IRST demo on Speaker Loc + head orientationSpeaker Loc + head orientation
Vision Technologies, Software Architecture & Processing Strategy in the UPC Smart Room
EPS-UAM, May 19th 2006
Head Pose Estimation Head Pose Estimation –– ContCont
•• Status and progress on Status and progress on Studio dataStudio data– Mean error (pan/tilt): 12° / 15° ! 12° / 10°– Pan Correct classification: 45% ! 51%– Tilt Correct classification: 43% ! 50%
•• Main progress this year: CLEAR Results on Main progress this year: CLEAR Results on CHIL seminar dataCHIL seminar data(estimates with respect to the room coordinates)– Mean error (pan): 49º (34º best system)– Pan Correct classification: 35% (45% best system)
•• ProspectsProspects– Low resolution captures opens a complete new field for head pose estimation– Information from multiple views helpful AND necessary for stabilizing / confirming
hypotheses– Classifiers and feature spaces used for high-resolution pose estimation are not feasible
as standalone systems anymore !multimodal fusion approaches: body posture, tracking, speech detection, …
New Tasks: acoustic & multimodal head orientation (currently pilNew Tasks: acoustic & multimodal head orientation (currently pilot experiments)ot experiments)
D4.7 D4.7 ““3D tracking of several persons from multiple camera views. 3D tracking of several persons from multiple camera views. Head Orientation trackingHead Orientation tracking””
Vision Technologies, Software Architecture & Processing Strategy in the UPC Smart Room
Doctorado en Informática y TelecomunicaciónEPS-UAM, May 19th 2006
Josep R. CasasUPC – Technical University of Catalonia 26
Vision Technologies, Software Architecture & Processing Strategy in the UPC Smart Room
•• Objects such asObjects such as– Electronic devices: Laptop, PDA, mobile phones, etc.– Smart room objects: chairs, cups, bottles, etc.
•• Features such asFeatures such as–– Position, orientation, on/off, open/closePosition, orientation, on/off, open/close, etc.– Owner, connected to, User, etc.
•• Room activity analyzed using Room activity analyzed using Stochastic Context Free GrammarsStochastic Context Free Grammars– A set of rules are manually defined. Parsing is performed over series of events
to effectively detect specific activitiesspecific activities (in particular, static objects, moved chairs, etc.
–– HighHigh--Level informationLevel information is not only seen as the aiming target, but also as a way to reinforce the basic Low-Level Tracking
•• Background Modeling Using Video UnderstandingBackground Modeling Using Video Understanding: : Adaptive background modeling techniques usually fail under certain conditions.
Suppose a person hovering in the background, which then stops, sits, or lays. During a period of time, the corresponding blob will still be active, but, little by little, the pixels of the blob will become part of the backgroundpixels of the blob will become part of the background.
– The process of merging into the background, could be prevented once we we positively know that the object has stoppedpositively know that the object has stopped. The instants when the objects stop could be determined by video understanding techniques.
J.L. Landabaso, M. Pardàs, L.-Q. Xu, "Hierarchical Representation of Scenes using Activity Information“, ICASSP 2005, Philadelphia, USA
Vision Technologies, Software Architecture & Processing Strategy in the UPC Smart Room
Doctorado en Informática y TelecomunicaciónEPS-UAM, May 19th 2006
Josep R. CasasUPC – Technical University of Catalonia 27
Vision Technologies, Software Architecture & Processing Strategy in the UPC Smart Room
EPS-UAM, May 19th 2006
Interesting outcomeInteresting outcome……
•• Change in researchersChange in researchers’’ attitudeattitudePreviously: “This recording/situation/scenario is not good because…”
… the presenter gets out of the camera view… the speaker does not talk to the microphone… participants don’t look at the cameras… bad lighting/shadows, strong reverberation…
Now: “Er… Well, we’ll have to adapt… This is challenging”… cameras should cover the whole area… ID profile views (challenging)… cancel noise (classify acoustic events)… far field, reverberation, wide views, noise, low res, shadows, lights,
!! what if we combine? (signal data, features, scores, decisionwhat if we combine? (signal data, features, scores, decision……))–– exciting anticipation of the challenge exciting anticipation of the challenge ––
!! Promising for Promising for Robust Perceptual InterfacesRobust Perceptual Interfaces
Vision Technologies, Software Architecture & Processing Strategy in the UPC Smart Room
EPS-UAM, May 19th 2006
ConclusionConclusion
•• Framework: the CHIL projectFramework: the CHIL project– The CHIL Vision ! Computers should help, in a “naturally human” way– Proof of concept: instantiating services (demo)
•• Multimodal Interface Technologies aim to fulfill the CHIL visionMultimodal Interface Technologies aim to fulfill the CHIL vision“Putting the Computers in the Loop of Humans” ! instantiated in services– Robust technologies to understand human communication signals across multiple modalities, in
natural, varying, unconstrained human interaction scenarios•• Facing challengesFacing challenges
•• Smart Room and Sensor SetupSmart Room and Sensor Setup– Equipment and Data collection – CHIL evaluation campaigns
•• Software ArchitectureSoftware Architecture– Data flows and distributed processing (CHIL ICE cube)
•• Vision Technologies at UPCVision Technologies at UPC– Person tracking, Person ID, Body Analysis, Object Detection, Text Detection, Activity Analysis,
Emotion Detection– Most techniques published or to be published in 2004/2006