Information School of Tsinghua University Research …robotics.nus.edu.sg/sge/seminar/SunFuChun2009-1.pdf · Information School of Tsinghua University & Research Work ... Environmental

2009-2-3 1

Introduction to

Information School of Tsinghua University &

Research Work Conducted in Our Group

Fuchun Sun

Tsinghua UniversityState Key Lab of Intell.Tech. & Syst.

National Univ. of Singapore, 01.19, 2009

2009-2-3 2

Introduction to Tsinghua University

2009-2-3 3

Started as Tsinghua Preparatory SchoolFounded in 1911, the last year of Qing Dynasty, as Tsinghua Preparatory School for sending Chinese students to studyabroad by the “Boxer Indemnity（庚子赔款）”

“Tsinghua” comes from the name of the royal “TsinghuaGarden”, which is currently the headquarter of the university

2009-2-3 4

Became National Tsinghua University in 1928

3 Schools, 15 Depts in 1928HumanitiesSciencesLaw

School of Engineering in 1932CivilMechanical Electrical

The Original GateThe Big Hall The Tsinghua School

2009-2-3 5

Transitions since 1937

Moved to Changsha (1937) and then Kunming (1938) together with Peking Univ. and Nankai Univ. to form Sorthwest Associated Univ. Returned to Beijing in 1946 with 5 Schools

Humanities, Science, Law, Engineering, Agriculture

Became a Polytechnic Univ. in 1952Learned from Russia with restructuring of high education Established new disciplines such as Radio Engineering

2009-2-3 6

Resumed as a Comprehensive Univ since 1980’s

School of Economics and Management (‘84)

Humanities and Social Science (‘93)

School of Law (‘99)

Arts and Design (1999) Public Policy & Administration (‘00)

Journalism and Communication(’02)

2009-2-3 7

Currently, 13 Schools and 54 Departments13 Schools (49 departments)

SciencesArchitectureMechanical EngineeringInformation Science & TechnologyCivil EngineeringAerospace (’04)Medicine (‘06, Peking Union Medical College, Tsinghua University)

Economics and Management (’84)Humanities and Social Science (’93)Law (’99)Arts and Design (‘99)Public Policy & Administration (‘00)Journalism and Communication(’02)

5 Independent DepartmentsEnvironmental Science and Engineering Electrical Engineering (电机工程与应用电子技术）

Engineering Physics Chemical EngineeringMaterials Science and Engineering

1 Research InstituteInstitute of Nuclear and New Energy Technology

Campus (hectares) 405.5Beijing 306others 100

Floor (million square meters) 2.56Beijing 2.23others 0.33

2009-2-3 8

Faculty Members 2,857Professors 1,201

Associate professors 1,061under the age of 45 1,925 (67%)academicians of CAS 34academicians of CAE 30

Post-docs 957Degree Students 31,395

Undergraduates 13,900Master candidates 12,179Academic degree 5,055Professional degree 7,124

PhD Candidates 5,316

Faculty Members and Students (2007)

Foreign Students 2,021 Ph.D Students: 101Master Degree: 254Undergraduates: 784 (5.7%)Training Students: 882 (come from 75 countries)Female 943 (47%)

HK/MC/TW Students 300Ph.D Students: 59Master Degree: 96Undergraduates: 108Training Students: 37

http://www.au.tsinghua.edu.cn/

2009-2-3 9

Professional Degrees“Master of Professional” Programs

Tsinghua’s efforts to bridge the gap between universities and industryStudents from industry with working experience

Part-time, off-campus

Intensive courseworkMath, engineering, and also

management, ……Thesis topics from industry

Solve real problems from

engineering practice

Advisors from both Tsinghua and

industry

Degree Programs and Student Distribution (Aug. 2007)

(2717) Master of Professional(616) Master of Business

Administration (MBA)(784) Executive MBA(332) Master of Public

Administration (MPA)(176) Master of Professional

Accounting (MPAcc)(118) Juris Master(30) Master of Fine Arts (MFA)

(4) Master of Lansscape Architecture (MLA)

2009-2-3 10

Research Fund and Patents

3.70 4.15.50

7.308.509.40

10.5011.7

13.814.9

19.3

0

2

4

6

8

10

12

14

16

18

20

1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007

Total

Industry

Year 1998 1998 2000 2001 2002 2003 2004 2005 2006 2007Applied China

Oversea149 189 344 441 583 758 824 872 875 985

200 230Granted China 64 121 135 187 164 501 537 521 568 601

Oversea 57 52

RM

B in 100X

million

Oversea

SingaporeNUS (Student Exchange, Joint Master Program in Logistics & EMBA, Joint Workshop between NUS ECE and Tsinghua CS)NTU (Student Exchange, EMBA)

University-University Collaboration

2009-2-3 12

University-University CollaborationUSA

Harvard (Public Administration, Business Administration, Environmental Engineering, Architecture)MIT (Business Administration, Architecture)Yale (Environmental Engineering)Stanford (Student Exchange, Business Administration)Cornnel University (Scholar Exchange, semi-annual Joint Workshop)

UKCambridge-MIT-Tsinghua Distance Education ProgramSino-UK Network Education Program (Camb.-Tsinghua)Tsinghua-Camb. IT Talents Training Program

JapanKeio (3E, cooperative degree program)Kyoto (environmental engineering, Core University Program)Waseda (Law)Tokyo Institute of Technology (Joint Master Program)

2009-2-3 13

University-University CollaborationGermany

RWTH Aachen (Industrial Engineering, Automotive Engineering)Munich University of Technology (one of the 5 global strategic partner universities

FranceSino-France 4+4 double degree programSino-France Exchange Foundation programSino-France Energy and Environment Center (cooperation between Tsinghua and French Ministry of Education)

The NetherlandsTsinghua-TU_Delft Joint Workshop (annual)Research Collaboration in Micro-electronics area

KoreaTsinghua-Samsung/LG-KAIST Joint WorkshopsAsia Research Center funded by The Korea Foundation for Advanced Studies)

Internationalization in globalization

Joint degree program

Internationalization in globalizationInternational Research Training Group

Cross-modal Interaction in Natural and Artificial Cognitive Systems (CINACS)

Lectures for CINACS

Internationalization in globalization

2009-2-3 17

Introduction to Information School S &T

2009-2-3 18

Milestones of SIST

1933: Electrical Engineering Dept Founded1952: Radio Engineering Dept Founded by merging the telecom labs of Tsinghua and Peking Univ.1955: Automatic Control Discipline Founded in EE Dept1956: Electronic Computer Discipline Founded in EE Dept1958: Automatic Control Dept Founded (including Computer, Control, and Operation Research Disciplines)1970: Computer Science & Technology Dept Founded by merging Automatic Control Dept with the computational mathematics discipline1970: Industrial Automation Dept Founded 1980: Institute of Micro-electronics Founded1994: School of Information Science & Technology Founded 2001: School of Software Founded2003: Research Institute of Information Technology Founded

Current Structure of SIST

19

Information School (SIST)

Electronic Engineering (EE) Tsinghua National Lab (TNList)

Network Research CenterComputer S&T (CS)

Research Institute of IT (RIIT)Micro- and Nano-Electronics

Institute of Micro-ElectronicsAutomation (AU)

Theoretical Computer Science Research CenterSchool of Software (SS)

Departments Institutes

Master of Professionals

Master of Professionals

http://www.ee.tsinghua.edu.cn/

http://dns.ime.tsinghua.edu.cn/

2009-2-3 20

Dept of Computer Science and Technology

5 Divisions and a State Kay Lab

Computer Software and Computing Theory Division

Human-Computer Interaction and Media Integration Division

High Performance Computing Division

Computer Networks Division

Intelligent Technology and Systems Division

Bo Zhang Sanli Li Andrew Chi-Chih Yaoxue Zhang

2009-2-3 21

Dept of Electronic Engineering

5 Divisions and Two State Kay LabsInformation Opto-electronics DivisionCircuits and Systems DivisionElectromagnetic Field and Microwave DivisionCommunication and Information System DivisionSignal and Information Processing DivisionInformation Network and Complex System Div.

Youshou Wu Bingkun Zhou

2009-2-3 22

Dept of Automation

5 Divisions and 1 National Engineering CenterDetection and Electronics Div.Process Control Engineering Div.Information Processing Div.Control Theory and Technology Div.System Engineering Div.National Engineering Research Center for Computer Integrated Manufacturing System

Yanda Li Cheng Wu

2009-2-3 23

Dept. of Micro-electronics and Nano-electronics (Institute of Micro-Electronics)

3 Divisions and 1 Pilot Line (1) Devices Research Division(2) IC and System Design Division(3) Device Physics and CAD Technology Division(4) Sub-micron Pilot Line (the only one university in China that holds such a pilot line)

Zhijian Li

2009-2-3 24

School of Software Engineering

4 Divisions and 1 National Engineering CenterSoftware theory and systemSoftware engineering and managementInformation system and engineeringImage processing and graphicsNational Engineering Research Center for Enterprise Information Software

Jiaguang Sun

2009-2-3 25

Research Institute of Information Technology (RIIT)

Advisory Committee

Director

Wireless and Mobile Communication Technology R&D Center

Digital Television Technology R&D Center

Operating System and Middleware Technology R&D Center

Web and Software Technology R&D Center

Electronic Packaging Technology R&D Center

Future Information Technology R&D Center

Microprocessor and SoC Technology R&D Center

2009-2-3 26

Tsinghua National Laboratoryof Information Science and Technology (TNList)

Computer Science & Artificial Intelligence

Next-Generation Internet

Communication Systems and Networking

Opto-electronic Devices and Applications

Micro- and Nano-Electronics

Complex Network Systems

Bio-informatics

Academic Committee

Director

Adversary Committee

Academic Committee

Pervasive Computing

Technology Innovation Branch (RIIT)

Managing Board

Academia Programs (1/2)

2009-2-3 27

Ranked #1 (2008)#1 (2006) #1 (2001)

Ranked#1(2008) #1 (2006)#1 (2001)

Academia Programs (2/2)

2009-2-3 28

Ranked #1 2008#1 (2006)#3 (2001)

Ranked #1 (2008)#1 (2006)#2 (2001)

Restructuring for Better Interdisciplinary Researches and Industrial Collaboration

2009-2-3 29

Dept. of Electronics Engineering

Dept. of Computer S&T

Dept. of Automation

School of SoftwareInstitute of Micro-

Electronics

Research Institute of Information

Technology

Tsinghua National Lab of Information Science and

Technology

2009-2-3 30

Faculties and Students (2007)

Total Bachelor Master Doctor

School of Information Science 6971 2701 3078 1192

Electronic Engineering 2245 1021 801 423

Computer Science & Technology 1929 645 868 416

Automation 1463 660 532 271

Micro-electro & Nano-electronic 569 155 332 82

Software Engineering 765 220 545 0

Faculties 486Professors 166Asso. Prof. 178 Postdocs: 116

Oversea Visitors in 2006 (from 30 countries or regions, not include visitors to professors)

Nation（area）

Amount of delegation

Amount of people

America 157 425Japan 44 128

HongKong 20 77France 18 69Holand 13 48Canada 13 18Britain 13 23korea 12 83

Australia 11 23Germany 9 48Singapore 7 12

Taiwan 6 38Italy 5 33

Finland 5 16India 3 4

Norway 3 23Spain 2 9

New Zealand 2 3Philipine 2 20

Brazil 2 3Colombia 1 10

Cyprus 1 1Russia 1 3Cuba 1 4

Sweden 1 1Pakistan 1 3Vietnam 1 8

Switzerland 1 1Portugal 1 6Danmark 1 2

Total 356 1142

Visiting Outside in 2007

EE CS AU ME SE Total

Short term 116 203 57 53 11 440Long term（over 6 months） 1 5 3 1 0 10

Total 117 208 60 54 11 450

Professors visited abroad

Students visited abroadEE CS AU ME SE Total

Short term 119 158 55 27 14 373Long term（over 6 months） 11 11 7 2 0 31

Total 130 169 62 29 14 404

Tsinghua Information ForumAn Industry Sponsored Distinguished Lecture Series on Advanced Topics in IT Field (since 2004)Encourage professors to invite highly distinguished speakers from all over the worldSupported 40+ Distinguished Lectures so far under the name “Tsinghua Information Forum – A/B/C Lectures” (A=NTT DoCoMo ; B=Nokia,C=IBM)Lecture information will go to the sponsor companyVery well attendedExpecting more supports from 2007 academic year!

33

2009-2-3 34

International Student Exchange

THU-Stanford Student Exchange Program (2005~)China-experience for Stanford students and America-experience for Chinese studentsNot only course study, but also (more) industry intern, and further research collaborationMutually waive the tuition fee and recognize the credits10-12 students alternatively per year and stay for 3 months

THU-Cornnel Scholar Exchange (2005~)2 Scholar exchanged in 2006Tsinghua-Cornell Joint Workshop

THU-Waterloo Scholar/Student Exchange Program(2005~)2 Scholars and 2 students exchange each yearDouble degree program in computer science

THU-USC Scholar/Student Exchange Program (2007~)5-7 undergraduate students exchangeJoint workshop (May 24-25, 2007)

THU Teachers and Students in NUS

THU Students in Stanford

THU Students in USC

Special Fund supporting Graduate Students to Int’l Conf

Dept.2004 2005 2006 2007

DoctorMaster Total DoctorMaster Total DoctorMaster Bachelor Total DoctorMaster Total

EE 1 2 3 20 7 27 26 19 45 28 21 49

CS 4 4 26 4 30 53 14 67 54 15 69

AU 2 2 7 2 9 14 6 1 21 16 6 22

ME 1 2 3 6 3 9 7 3 10

SE 7 7 7 7 8 8

RIIT 2 2 1 1 2 2 2

Total 7 2 9 54 24 78 100 50 1 151 105 55 160

Statistic of students sponsored by SIST:

Currently supported by an alumnus, but industry support is needed for sustainable development

Students have Traveled to 30+ Countries/Regions Nation 2004 2005 2006 2007 Total

Austria 2 1 3Australia 3 7 10poland 2 1 3Macao 1 1

Bahamas 1 1Brazil 1 1 2Germany 1 3 4

UAE 1 1France 1 7 4 12Finland 1 1Korea 7 4 1 12Holand 1 1 2Canada 2 4 2 8Czechic 1 1Malaysia 1 1 2Mauritius 1 1America 2 31 52 18 103

Mexico 1 1Norway 2 2Portugal 1 1 2 4Japan 1 2 10 6 19Sweden 2 2

Switzerland 4 4Taiwan 2 6 8Thailand 1 1Turkey 11 11Spain 3 2 5Greece 2 8 10Hongkong 1 6 11 18Singapore 1 5 10 1 17

Italy 1 1 2India 1 1 2

Britain 1 5 3 9Vietnam 4 4China 1 5 1 7

合计 9 78 151 55 293

Research Work Conducted in Our Group

Intelligent System Theory &Space Application Technologies

Prof. Fuchun SunDistinguished Young Scholar in 2006Excellent Dissertation Prize in 2000Associate Editor《IEEE Trans. On Neural Networks》

Member of the Editorial Board of《Soft Computing》and …

Associate Prof. Huaping LiuMore than Papers Published in IEEE Trans. on Automatic Control, Fuzzy Systems, Circuits

and Systems, Automatica, etc.



8 Ph.D Candidates

6 Master Candidates

4 Postdoctors

1

一、Spacecraft Dynamics and Control

(1) On-orbiting identifications for space robots or vehicles?

(2) Engineering oriented controller design?(3) Optimal trajectory planning with

vibration reduction?(4) Optimal docking strategy for manned

maneuvering units or two spacecrafts? (5) Position/force hybird control for object

graping?

（1）A Space Robot with Flexible Dual Arms

Stanford Space RobotA Space Robot Test –bed in Tsinghua Univ.


（2）A Space Robot with One Arm

A Space Robot with One Arm


（3） Manned Maneuvering Unit

Manned Maneuvering Units and Models


（1）Modeling the satellite networks and performance evaluation.

（2）Dynamic Routing algorithm.（3）Transmission Protocol and

optimization.（4）Satellite management and control.（5）QoS of satellite networks.

二、 Satellite Networks

In low load（link availability 90％），Transmission delay dominates，single layer network is better than two-layer one.

In high load（link availability 98％ )，processing and queue delay dominates，the two-layer network is better than the single-layer one.

三、Tele-operation

目标星服务星

地面站地面站

卫星

通信网

通信

通信网测控中心

通信卫星通信卫星

地面操作中心

Space Teleoperation System

Serving Sat、Target Sat、Communication Sat

Ground-based Control Center，Measuring/Control Center，Ground Station

Function Demonstration 一 Capturing the Client Satellite

Initialization Gripper moves to Client Gripper moves to Client

Grasp the clientThe client rotates 90 degreePull the client to move

三、Tele-operation

Initialization Gripper moves to unit

Grasp the ORUMove the ORUPut the ORU in preassigned place

四、Tele-operation

Function Demenstration一ORU

Gripper moves to unit

THMR-I：Indoor (1986-1990)

THMR-II: Indoor

(1986-1990)

四、Mobile Robots

THMR-V：Outdoor（1999-2005）

Optical coder

Compass

DGPS

Laser radar

CCD camera

Control system

Data radio

THMR-III：Outdoor(1991-1998)

四、Mobile Robots

Key technologies:

(1) Road detecting for mobile robots.(2) Location for mobile robots.(3) Path planning and obstacle

avoidance.(4) Vision serving control.

四、Mobile Robots

5.1 What is the multi-scale model ?

Space scale: The size of the airplane, the distance between the atomsMulti-scale Systems (MUSS)

Scale

Time scale: Vary from the time of flight to the vibrations of the electrons Multi-time scale systems (MTSS)

五、Modeling & Control for multi-time scale systems

Example:Reactor, Neurobiology

5.2 What are multi-time-scale dynamical systems?

A class of systems whose dynamics are described by

mathematical models in different time scales.

For example: A space robot


Some typical multi-time scale

dynamical systems

Aerospace

Manufacturing

Airplane

Process Industry

RealNetwork

Networked systems

Space Physics

5.3 The description for multi-time-scale systems?

The multi-time scale system can be described as

Whereεis the small parameter. However, in the past

decades, most achievements are restricted to

1 1 2

2 1 2

( ) ( ( ), ( ), ( ), )( ) ( ( ), ( ), ( ), )

t t t tt t t t

εε ε

=⎧⎨ =⎩

x f x x ux g x x u&

&(5.1)

1 1 2

2 1 2

( ) ( ( ), ( ), ( ))( ) ( ( ), ( ), ( ))

t t t tt t t tε=⎧

⎨ =⎩

x f x x ux g x x u&

& (5.2)


5.4 Problems concerned with multi-time-scale models?

(1) The ill-conditioned property of the multi-time scale dynamics?

(2) Computational complexity?Not suitable for system simulation and controller design.

(3) ε-independent control problem is enough? In some cases,ε-independent control solution do not exist.

Furthermore, for more precise control, ε-dependent control problem is required.

1 1 2

2 1 2

( ) ( ( ), ( ), ( ), )( ) ( ( ), ( ), ( ), )

t t t tt t t t

εε ε

=⎧⎨ =⎩

x f x x ux g x x u&

& (5.3)


A flexible single-link manipulator, space robot, etc. Linear singularly perturbed model

A flexible multi-link manipulator, large-scale space structure, etc.

Nonlinear singularly perturbed model

T-S fuzzy model

Fuzzy singularly perturbed model (FSPM)

5.5 Fuzzy Singularly Perturbed Models (FSPMs)


FSPM approach shows the following merits: (1) The FSPM simplifies model representation for a class of

multi-time scale dynamical systems, such as flexible-link manipulator, etc.;

(2) Well-developed techniques for linear singularly-perturbed systems are available for the controller design of a FSPM, so that the upper-bound of the small parameter can be obtained;

(3) The stability and control performance can be guaranteed.

1 1i p ipIf is and and isξ μ ξ μL

11 12 1

21 22 2

( ) ( ) ( ) ( ) ( ) ( ) ( )( ) ( ) ( ) ( ) ( ) ( ) ( )

i i i

i i i

t t t tThen

t t t tε ε ε

ε ε ε ε⎧ = + +⎨

= + +⎩

x A x A z B uz A x A z B u&&

( ) ( ) ( ) ( ) ( )i it t tε ε ε= +E y A y B u&

(5.5)


Theretical research conducted:

(1) Sufficient conditions and Necessary conditions of FSPM as universal Approximator to multiple scale nonlinear systems.

(2) Controllability of FSPM.

(3) Adaptive and Robust Control for Multiple Scale Nonlinear Systems based FSPM.

(4) FSPMs with markov jump parameters are used to model the networked control systems, some related theoretical results for controller design are hopefully obtained.

(5) ε-dependent Robust control and adaptive control Based FSPM.


Related works ------ Our group

Controller design for Markovian jumping systems subject to actuator saturation, Automatica, 2006, 42(3): 459-465.

Stability analysis and synthesis of fuzzy singularly perturbed systems, IEEE Transactions on Fuzzy Systems, 2005, 13(2):273-284.

Hinf control for fuzzy singularly perturbed systemsFuzzy Sets and Systems, 2005, 155(2): 272-291.

Comments on "H-infinity fuzzy control design for nonlinear singularly perturbed systems with pole placement constraints: An LMI approach" IEEE Transactions on Systems, Man and Cybernetics, Part B, 2004, 34(6):2422

Comments on ‘constrained controller design of discrete Takagi-Sugeno fuzzy models’, Fuzzy Sets and Systems, 2004, 146(2): 473-476

Reduced-order H-infinity filtering for linear systems with markovian jumpParameters, Systems and Control Letters, 2005, 54(8):739-746

80%

20%

Visual Perception &Brain Intelligence

High Performance of Computers

Visual cognition & Machine Intelligence

Unstructured Information (USI)

Structured Information (SI)

USI

SI

Visual Cognition Principle and Model

Brain Intelligence

五、Unstructured Information Processing

……

Topological PerceptionTheory

Marr Feature Analysis Theory

Feature Space Theory：PCA, ICA

Scale Space, MRF Feature ExtractionFeature SelectionImage Analysis

Visual Path ModelTarget RecognitionTracking & LocationInformation Fusion

What & Where Path

Selective AttentionMechanism

Feature ExtractionTarget Detection

Image Compression

Receptive Field (RF) Model DOG (Difference of Gaussian)

Feedback ConnectionPath

Feature Association Information Interaction

Feature Extraction

Learning Mechanism Self-organized、

Unsupervised

Feature ExtractionTarget Recognition

Visual Cognition Mechanism & Computation Model for USI Processing

Stimulus: on off


KnowledgeReasoning

Integration Decision

Expression and Computation from USI to SI

Knowledge(SI/ Formalization)

decision-making

Data(USI)

Information(Semi-SI)Content

Understanding

Data Mining

Information Refining

Knowledge Mining


Multi-modal information Brain-Computer Interaction (BCI) and fusion are important approaches for robust feature processing, target detection, target recognition, Information Fusion, etc.

Multi-modal information BCI

Multi-modal fusion based auto-navigation for intelligent vehicles

Applications for Multi-modal Information and Fusion

Multi-sensor target imaging, fusion and recognition

http://images.google.cn/imgres?imgurl=http://mil.eastday.com/m/20060905/images/00756077.jpg&imgrefurl=http://mil.eastday.com/m/20060905/u1a2299120.html&h=479&w=550&sz=58&hl=zh-CN&start=14&tbnid=Al0WGkOMFr9AdM:&tbnh=116&tbnw=133&prev=/images%3Fq%3D%25E5%2585%2589%25E5%25AD%25A6%25E5%258D%25AB%25E6%2598%259F%26gbv%3D2%26svnum%3D10%26complete%3D1%26hl%3Dzh-CN%26newwindow%3D1%26sa%3DG

Evidences for Multi-modal Information Interaction and Fusion

1 Cat’s visual cortex can also receive auditory information;

2 Cat’s epithalamus can receive visual, auditory and touch information;

3 Rattlesnake’s coping area receives visual and infrared information;

4 Hawk’s night vision cells combine the level error and time error of concha.

Multi-modal perceptive information congregates in many traditional areas determined

by single perception signal.

Why multi-modal Fusion?

1 Improve the receiver’s detection, recognition, discrimination and memory ability;

2 Multi-stimulation is more effective than single stimulation.

Physiological Evidences:


1. Image Feature

Multi-modal Feature


场景

文本

人工

文本

3. Text FeatureScene Text Artificial Text

2. Audionary Feature


4. Biological Feature (Face Feature and Gait Feature)


（Ghazanfar & Schroeder, 2006）

Spontaneous level indicated by the white line

Receptive Fields of a Ventral Intraparietal Area Neuron

Multisensory Integration and Multi-modal Fusion

Superposition ofthe receptive fields from the two modalities


A Biological Instance of Multi-Modal Information Fusion

Target

Heat Eyes

Light Eyes

Feature Fusion

Feature Extraction

Perceptive Image

………………

……

……

…...

……

……

…...

……

……

…...

……

……

…...

……

……

…...

…………

……

……

…...

……

……

…...

……

……

…...

Reverse Projection

Fusion

Rattlesnakes: Target Recognition through Multi-modal Information Fusion

Reverse Projection

Fusion


http://www.zskxg.com/scienceworld1/dd1.asp

Multi-sensory Remote Sensing Image Fusion Based on Biological Multi-modal Fusion Mechanism

Feedback

Fusion

Recognition

Optical Sensor

SAR

Target

1A

1( )j A 2A

3A

4A

2( )j A 3( )j A

4( )j A

Ω

U

Feature Extraction & Mapping

Feature Extraction & Mapping

Feature Fusion

………………

…..

…..

…..

…..

…..

…..

…..

…..

…..

……

Feedback

Fusion

A Direct Application with Multi-Modal Information Fusion


http://images.google.cn/imgres?imgurl=http://mil.eastday.com/m/20060905/images/00756077.jpg&imgrefurl=http://mil.eastday.com/m/20060905/u1a2299120.html&h=479&w=550&sz=58&hl=zh-CN&start=14&tbnid=Al0WGkOMFr9AdM:&tbnh=116&tbnw=133&prev=/images%3Fq%3D%25E5%2585%2589%25E5%25AD%25A6%25E5%258D%25AB%25E6%2598%259F%26gbv%3D2%26svnum%3D10%26complete%3D1%26hl%3Dzh-CN%26newwindow%3D1%26sa%3DG

Feature Point Matching

Remote Sensing Image Registration and Fusion


Remote Sensing Image Registration and Fusion


Optical Image SAR Image Fusion Image

Optical Image SAR Image Fusion Image

Multi-sensory Remote Sensing Image Fusion


Road Line Detection

Multi-modal Information Fusion for Intelligent Vehicle’s Autonomous Navigation


Radar and Visual Information Fusion



Feasible Region Detection



Feasible Region Detection



Pedestrian Detection

Pedestrian Detection with Integrated Detector and Shapelet Feature


Pedestrian Detection with Integrated Detector and Shapelet Feature

Pedestrian Detection


Vehicle Tracking

Pedestrian Tracking

Target Tracking with Kalman Filter and Sequence Monte Carlo Filter


Pedestrian Tracking in Multi-camera Environment


Some Application Instances Based on Multi-modal Video

1. Video Segmentation and Classification


Some Application Instances Based on Multi-modal Video

2. News Video Processing System

3. Sports Video Analysis


4. Key Figure Discovery

5. Speaker Detection


6. Pedestrian Detection, Tracking and Recognition


Three Aspects for Visual Cognition ComputationStudy

On?

Technical Realization

MechanismModels

Theoretical

Method

Visual cognition mechanism and computational model for USI

processing

Modeling and machine learning for USI

Content Comprehension and semantic representation

Multi-modal information interaction and knowledge reasoning

Feature extraction and conformity for USI

Environmental perception, target detection and target recognition

based on visual information

Target location and tracking based on multi-sensory information fusion


Anthropopathic analogyHow to interact human-human？

seeing, hearing, taste, touch, smell,reading, speaking, writing …

Our goal: change the computer’s quasi deaf-mute-blind state, and let it

hasseeing, hearing, taste, touch, smell, perception ability learning, memory and emotion cognition abilityspeaking, writing expression abilityproprioception, self-consciousness

Human-Computer Interaction

A Higher Level: Multi-modal Fusion and Interaction

Human-Human Interaction


Thank you for attention!

Information School of Tsinghua University Research …robotics.nus.edu.sg/sge/seminar/SunFuChun2009-1.pdf · Information School of Tsinghua University & Research Work ... Environmental

Documents