/14 /14 /14 Augmented Human Communication Lab C 2019/5/20 Speech Translation Neural Machine Translation Brain Analysis Spoken Dialog Multi-modal Dialog Why don’t you join our lab! I’m looking for a lab. Information Retrieval QA System Multi-modal Multi-language ASR Speech Synthesis Deep Speech Chain Deep Neural Network Affective Computing Emotion and Environment Recognition Prof. Satoshi Nakamura Assis. Prof. Koichiro Yoshino WEB Information Processing Toward enhancement of human communication abilities Toward enhancement of human communication abilities, AHC lab is studying multilingual speech translation, dialog system, user- adaptive super-human automatic speech recognition/synthesis, and brain analysis related human communication. We have also been managing Data Science Center since 2017. Assoc. Prof. Katsuhito Sudoh Research Assoc. Prof. Sakriani Sakti Assis. Prof. Hiroki Tanaka ( ) Goal-oriented Dialog Non goal-oriented Dialog Incongruity measurement Prediction of feeling Early Detection of Dementia Communication Support Dialog Research Assoc. Prof. Keiji Yasuda Visiting Assoc. Prof. Yu Suzuki
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Affective ComputingEmotion and Environment Recognition
Prof. Satoshi Nakamura
Assis. Prof. Koichiro Yoshino
WEBInformationProcessing
Toward enhancement of human communication abilitiesToward enhancement of human communication abilities, AHC lab is studying multilingual speech translation, dialog system, user-
adaptive super-human automatic speech recognition/synthesis, and brain analysis related human communication. We have also been managing Data Science Center since 2017.
Assoc. Prof. Katsuhito Sudoh
ResearchAssoc. Prof.
Sakriani Sakti
Assis. Prof. Hiroki Tanaka
( )
Goal-oriented DialogNon goal-oriented Dialog
Incongruity measurementPrediction of feeling
Early Detection of DementiaCommunication Support Dialog
• Multilateral translation for 8 Asian languages• Network-based S2ST
2010
•21 multilateral text translation
C-STAR
• Multilateral translation for 7 world languages
IWSLT
• Evaluation Campaign of S2S technologies
2011
VoiceTra
NAIST
ATR ATR
2019/5/20
/14
/42
Riken AIP Tourism Information Analytics TeamIoT2H: (Internet of Things to Human)
2019/5/20 Satoshi NAKAMURA@AHC,NAIST 8
IoT2H is a technology to bridge Internet of Things and human-beings.
What’s happening?
IoT, Social information
ToHuman
Output in language
Congestion factor for tourism spots
Shopping Hospital
Bus
Train
Restaurant
Temples
BeaconBeacon
Beacon Beacon
KinkakujiTemple is
now crowded
Chat bot
Hotel
Tourism Information in KyotoIdea development of Deep Learning
Image captioning image2cap!
Real-time
Assoc. Professor Katsuhito Sudoh
Background2000 Bachelor of Engineering, Kyoto University2002 Master of Informatics, Kyoto University2015 Ph.D. (Informatics), Kyoto University2002-2017 NTT Communication Science Laboratories2017- Associate Professor, Graduate School of
Information Science, NAIST
Machine TranslationSpoken Language Processing
I went Nara last night at noon
Information extraction from speech(using recognition “confidence”)
長尺矩形のオイルストレーナ74が溝条72aに略鉛直姿勢で嵌合される。
A long rectangular oil strainer 74 is fitted within the grooves 72a in a substantially vertical posture.
Translation with accurate word order (re-ordering)Translation of technical terms
2019/5/20
Toward a Language Barrier-free Future!Translation & Language Understanding
Evaluation of Natural Language Generation (Katsuhito Sudoh)
Semantic Automatic Evaluation of Translation (Kosuke Takahashi)
Reference TranslationOriginalRussia can
declarevictory
Evaluate the meanings of translations Encode the sentences as same vector
端子は互いに接触しないように配置されている
Terminals are placed not to be in contact with each other.
It’s fluent but gives wrong meaning...
IT’S POSSIBLY MISUNDERSTOOD
Focusing on risks of Misunderstanding
Style transfer for natural language(Kosuke Futamata)
Apply arbitrary stylistic features.
The chicken was delicious.
The chicken was Terrible.
Style(Positive)
Style(Negative)
Past ・Simultaneous optimization of speech recognition and machine translation ・Translating normal style into honorific styleResearch: ・Small and accurate translation models ・Evaluation of simultaneous speech translation systems
・Machine translation error analysis ・Pivot translation strategies ・Automated programing ・Multilingual machine translation ・Code efficiency prediction based on OJS data
relation between nouns domainToritaniKinami knowledge
relation between verbs preferencehitswing how to say
Assistant ProfessorKoichiro Yoshino
Background2009 Bachelor of Arts in Environmental Information,
Keio University2014 Ph.D. (Informatics), Graduate School of Informatics,
Kyoto University2014- JSPS Research Fellow (PD)2015- Assistant Professor, Graduate School of Information Science, NAIST
Did Tortani hit a home-run?
Toritani who got the start in 1st line-up hit 2 doubles.
Toritani hitsubject
focus→ retrieve from news text
Toritani who got the start
in 1st line-up
hit 2 doubles
subject
object
Web Text
subject
2019/5/20
Understanding
Recognition
Management
ASR, Para-linguistic recognition (SP, CC)
Understanding
Management
GenerationAction
TTSAction & behavior generation (SP, CC)
Generation
What is understanding?• Materialization of utterances• Dialogue act tagging• Knowledge acquisition • Knowledge extension• Relations between events
NLP techniques (NL)
NLP techniques (NL)
PRESTO: incremental
knowledge acquisition
PRESTO: incremental processing and
knowledge acquisition
affective computing
Spoken Dialogue Group- Toward cooperative systems through interactions -
How do we realize systems?• Decision making with reinforcement learning• Using a variety of information:
• Algorithm of reinforcement learning• Evaluations of dialogue systems
How systems have effects?• Generate responses
according to manager decisions, contexts, personality, etc
• Image and interaction• End-to-end systems w distillation
Confirm?
Use emotion?
Ask a question?
There are 3 Italian restaurants at Ikoma. Do you have any preference?
Kiyomizu-temple is very crowded due to the high-
seasonAIP: Touristic
information analysis
AP Yoshino
D 品川 D 杉山
M Mai M1
M 隆辻
D 河野
D Tung M 浅井
M1
D 村瀬 M 池内
M 田中 M1
Italian near by Ikoma
DA: questionDomain: restaurant{
type=Italian, …}Obs.
Assistant ProfessorHiroki Tanaka
Background Bachelor of Engineering,
Ph.D. (Engineering), NAIST
2019/5/20
Cognitive Communication Group
Assistant Prof.Hiroki Tanaka
Assessment and training of social communication skills (based on cognitive behavior therapy: SST)・Task: speaking, listening, small talks ・Feedback regarding eye gaze, speech and image・Now tested in clinics
Human - humanHuman - machine
Communication
Automatic assessment / Feedback(Medical and educational system)
Estimation of Cognitive &Psychological States
Current and Ongoing Researches
Automated social skills training EEG measuring during Simultaneous translation
Tourism Information Analysis using Tensor Decomposition
D3Haruko Yagura
Predicting Objective Speech Quality Score
Various modalities
EEG Face / Eye Voice
Previous Research Topics
• Speech recognition using EEG signals
• Detection of dementia from responses
• Prediction of depressive tendency from lifestyle
Anomalous Sentence Detection using EEG
Measuring Empathy from EEG Signals
M1Ivan Halim P.
Application:Objective quality measurement of synthesized speech
M2Taiki Kinoshita
Empathy
Inter-BrainSynchronization
EEG
Statistical analysis
Measuring empathy
Application:Evaluation of human-machine empathy
Taro eats an apple
Taro runs an apple
Speech EEG Prediction
Correct
Incorrect
Prediction whether the speech sentence is correct or incorrect using EEG based on machine learning model
Application:Evaluation of machine outputs, adaptive dialogue system
M2ShunnosukeMotomura
M2Motoi Kubo
Apply tensor decomposition to a variety of (=high dimensional) tourism information and analyze trends of tourist's tourist routes and popular spots.
Tourismdata
Tensordecomposition
Loc
atio
n
+ ・・・ +
Trend of migration pathway, popular spot
/14
Education2005-2008 Doctorate degree (Dr.-Ing)
in Engineering Science, University of Ulm, GERMANY
2000-2002 Master degree (MSc ) in Communication Technology, University of Ulm, GERMANY
1995-1999 Bachelor degree (BSc) in Informatics, Bandung Institute of Technology, INDONESIA
Work Experience2018 – Research Assoc. Professor, Augmented Human Communication Labs, NAIST, JAPAN
Research Scientist, RIKEN Advanced Intelligence Project AIP, JAPAN 2011 – 2017 Assistant Professor, Augmented Human Communication Labs, NAIST, JAPAN2009 – 2011 Visiting Professor, Faculty of Computer Science, University of Indonesia, INDONESIA 2006 – 2011 Expert Researcher, Spoken Language Communication Research Groups, NICT, JAPAN 2003 – 2009 Research Engineer - Researcher, Spoken Language Communication Research Labs, ATR, JAPAN 2001-2002 Masterarbeit, Speech Understanding Dept,
Daimler Chrysler Research Center, GERMANY1999-2000 Junior Software Consultant, Sumarno Pabotingi
Associate, INDONESIA
Research Assoc. Prof. Sakriani Sakti
2019/5/20
Speech Processing Research and Applications~ Let’s make a machine that can hear and speak as human ~