From Human Machine Interaction to Human Machine ... · From Interaction to Integration and Intelligence Interaction •Stimulus-response •Machine simply take task orders from human

From Human Machine Interactionto Human Machine Integration and Intelligence

Le Thanh Ha, Ph.DAssoc. Prof. at University of Engineering and Technology,

Vietnam National [email protected]; [email protected]; +84983 692 592

mailto:[email protected]

mailto:[email protected]

About myself

• Full name: Le Thanh Ha

• 2005-2010: Ph.D at Korea University, Korea

• 2010-now:

– Assoc. Prof. at University of Engineering and Technology (UET), VNUH

– Head of HMI Laboratory

• Expertise: Computer vision, Image/video processing and analysis, Machine learning

5/14/2018 2Le Thanh Ha, HMI Laboratory

HMI Laboratory

IntelligenceIn

terf

ace

Standardization

Standardization

Standardization

ISO/IEC JTC 1 Information Technology

Interaction

Integration

Intelligence

Inte

rfac

e https://hmiuet.wordpress.com

Computer Vision

Artificial Intelligence

Computer Graphics

Video Compression

Single intelligent humanoid

They look like machines. But there is human soul inside!!!

5/14/2018 Le Thanh Ha, HMI Laboratory 4

NS-5 Eva R2-D2 Terminator

Swamp and crowd inspired from nature

5

Collective transportation Flock of birds School of fishes

Seasonal migrationCollective hunting Celebrating crowd

5/14/2018 6

Human and Machine senses

Le Thanh Ha, HMI Laboratory

Human and Machine collaborations

Interface Interaction

Integration Intelligence


Human Machine Interface

• How do humans manipulate with machine and how information exchanged between human and machines?

• Answer: Standardization


Keyboard layout

ISO/IEC 9995-3 Information technology: the Complementary Latin group layout according to ISO/IEC 9995-3:2010


Video standards

ISO standards

243 Committee in ISOIT Committee publish 3004/21347 (14%) ISO standards

5/14/2018 Le Thanh Ha, Lab of HMI 11

April 2017

April 2018: 3149

Information technology

Committee TitlePublished standards

Standards under development

ISO/IEC JTC 1 INFORMATION TECHNOLOGY 3004 546

ISO/IEC JTC 1/SC 17 Cards and personal identification 114 31

ISO/IEC JTC 1/SC 2 Coded character sets 54 3

ISO/IEC JTC 1/SC 22 Programming languages, their environments and system software interfaces 108 20

ISO/IEC JTC 1/SC 23 Digitally Recorded Media for Information Interchange and Storage 138 2

ISO/IEC JTC 1/SC 24 Computer graphics, image processing and environmental data representation 79 8

ISO/IEC JTC 1/SC 25 Interconnection of information technology equipment 188 27

ISO/IEC JTC 1/SC 27 IT Security techniques 165 63

ISO/IEC JTC 1/SC 28 Office equipment 48 2

ISO/IEC JTC 1/SC 29 Coding of audio, picture, multimedia and hypermedia information 574 (20%) 103

ISO/IEC JTC 1/SC 31 Automatic identification and data capture techniques 117 30

ISO/IEC JTC 1/SC 32 Data management and interchange 77 17

ISO/IEC JTC 1/SC 34 Document description and processing languages 80 11

ISO/IEC JTC 1/SC 35 User interfaces 67 16

ISO/IEC JTC 1/SC 36 Information technology for learning, education and training 38 10

ISO/IEC JTC 1/SC 37 Biometrics 118 28

ISO/IEC JTC 1/SC 38 Cloud Computing and Distributed Platforms 10 5

ISO/IEC JTC 1/SC 39 Sustainability for and by Information Technology 6 12

ISO/IEC JTC 1/SC 40 IT Service Management and IT Governance 19 9

ISO/IEC JTC 1/SC 6 Telecommunications and information exchange between systems 352 38

ISO/IEC JTC 1/SC 7 Software and systems engineering 171 425/14/2018 Le Thanh Ha, Lab of HMI 12

April 2018: 3149

ISO/IEC JTC 1 - Information technology

5/14/2018 Le Thanh Ha, Lab of HMI 13

From Interaction to Integration and Intelligence


IntelligenceIntegrationInteraction

implies stimulus-response

implies partnerships WORKING together

implies partnerships THINKING together

From Interaction to Integration and Intelligence

Interaction

• Stimulus-response

• Machine simply take task orders from human

• Like client-server architecture

Integration

• Machine and human work as partners

• Work done by exchange sequence of information between Machine and Human

• Engage all kinds of physical sensors: Geometric-Visual, Sound-Auditory, Haptic-Tactile

Intelligence

• Information exchanged just like human and human

• Engage higher abstraction of information exchanged between Machine and Human

• Machine can be able to draw both physical and emotional meaning from human


From Interaction to Integration

Interaction can be described as stimulus-response

Integration implies partnership WORK together


Desktop applications: MS Offices.doc, .rtf, .html

Online applications: Google drivers.html version 5.0

From Interaction to Integration


U. Farooq, et al., Human-Computer Integration, Interaction vol. 23, iss. 6, pp. 26-32, 2016

From Integration to Intelligence

• There is a continuum from Integration to Intelligence. Intelligence extends but does not replace interaction an integration.

• Engage higher abstraction of information exchanged between Machine and Human

• Machine can be able to draw both physical and emotional meaning from human


Integration implies partnerships WORKING

together

Intelligence implies partnerships THINKING

together

From Integration to Intelligence


Integration implies partnerships WORK

together

Intelligence implies partnerships THINK

together

Open issue in Computer science:+ How to present level of abstract of information?+ How to autonomous create new thing? (Unsupervised learning)+ Knowledge presentation and discovery?+ …

Abstraction level of visual information


Physical Abstract

Low level Middle level High level

Light frequenciesBrightness

emotion

Edges

Lines

ContoursColors

Regions

Depth

Pixel Intensity

anomaly events

a facesa pen

a person

attention

quality assessmentan animal

Intelligence degree

a banknote

social awareness

Standardization for AI


Intelligence

Inte

rfac

e

Standardization

Standardization

Standardization

ISO/IEC JTC 1 Information Technology

ISO/IEC JTC 1/SC 42 - ArtificialIntelligence

Recently created in 2017

Interaction

Integration

Intelligence

SOME WORKS FROM OUR GROUP

Vietnamese banknote speaker

Smart camera for healthcare

Human visual system based Image quality assessment


Vietnamese banknote speaker

Money speaker helps visual impaired people to identify Vietnamese banknotes (iOS & Android)

Make use of Compact descriptors for Visual Search (CDVS):• A standard proposed by the Moving Picture Experts Group [3] (MPEG) organization• Implemented with SIFT at its core• CDVS operators:

– Extract – extract and encode compact descriptors from a set of images– Make index – generate a tree-structured codebook from descriptors extracted from a set of images– Pairwise matching – compare the descriptors extracted from query and reference images– Retrieval – search and discovery of images contained within a large collection, some of which depict

the same objects or scenes as those depicted by a query image.

CDVS


Extraction pipeline

Pairwise Matching pipeline

Retrieval pipeline

a SIFT vector represented by 128 bytes

Image database


Smart camera for healthcare

• Fall detection

• Video source from IP cameras

• Detection accuracy from testing dataset: 90%.

• Speed of detection on Rasberrypi 2: 20fps

• Automatically send sms or notifyVN Patent pending Number: 1-2017-02136


Detection pipeline




HVS-based image quality assessment• Objective measures do

not meet subjective• All current IAQs grades

images as a whole– Can’t be aware of

subject context– Can’t be applied to

current block based video coding

➢ Needs method assess quality of images locally


The same amount of noise has been added to these two images, such that their PSNR isidentical. High-frequency noise was inserted into the bottom region of the left image, whereas band-pass filtered noise was inserted into the top region of the right image. The noise is hardly visible in the left image due to our low sensitivity to high-frequency stimuli and the strong masking by highly textured content in the bottom region.

Subjective IQA database

• Since there is now benchmark available

• We need to create that database for our own purposes.

• Adapt ITU-R BT.500-11 – “Methodology for the Subjective Assessment of the Quality of Television Pictures,” 2002.


Create new tool for subjective testing



IQA databases

Simple IQA database• 20 subjects• 20 images• Distortion levels: 5 QPs• Image patch size: 64 × 64.• 2 resolution levels• 3 types of image patches

– Edge– Smooth– Texture (complex)

• Total pairs of image patchs: 600

Huge IQA database• 1480 subjects• 308 images (HD and FHD)• Distortion levels: random• Image patch size: 64 × 64• Types of image patches: Spatially

random• Total pairs of image patches: 61 600


Experiment results on 1st database


• Relationship between MOSs and QP values are approximated by a logistic function

where 𝑎1 and 𝑎2 are estimated from data.


• Applied on H.264/AVC video coding standard and obtain good results.

• A patent has been filed for this HVS-based video coding method.





Works on 2nd database

• Data cleaning to remove noise and outliers

• Deep neuron network architectures for learning how HVS assess image quality (locally)

• Applied to the HVS-based IQA DNN to video coding scheme H.265


Reference DNNs


Jongyoo Kim, Sanghoon Lee, "Deep Learning of Human Visual Sensitivity in Image Quality Assessment Framework," CVPR 2017

THANK YOU VERY MUCH!


From Human Machine Interaction to Human Machine ... · From Interaction to Integration and Intelligence Interaction •Stimulus-response •Machine simply take task orders from human

Documents