Top Banner
Visual Intelligence Prof. Rita Cucchiara AimageLab, Dipartimento di Ingegneria «Enzo Ferrari» Università di Modena e Reggio Emilia, Italy Director of the National CINI Lab AIIS
56

Presentazione standard di PowerPoint · 2019-11-05 · SAM Saliency /context detection Soft-attentive LSTM Text generation LSTM. Toward an explainable AI: What the Machine pays attention

Apr 07, 2020

Download

Documents

dariahiddleston
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Presentazione standard di PowerPoint · 2019-11-05 · SAM Saliency /context detection Soft-attentive LSTM Text generation LSTM. Toward an explainable AI: What the Machine pays attention

Visual Intelligence

Prof. Rita Cucchiara

AimageLab, Dipartimento di Ingegneria «Enzo Ferrari»Università di Modena e Reggio Emilia, ItalyDirector of the National CINI Lab AIIS

Page 2: Presentazione standard di PowerPoint · 2019-11-05 · SAM Saliency /context detection Soft-attentive LSTM Text generation LSTM. Toward an explainable AI: What the Machine pays attention

“Like the steam engine or electricity in the past, AI is transforming our world, our society and our industry.

Growth in computing power, availability of data and progress in algorithms have turned AI into one of the most strategic technologies of the 21st century.”

Artificial Intelligence for Europe - Brussels, 25.4.2018

Artificial Intelligence

Page 3: Presentazione standard di PowerPoint · 2019-11-05 · SAM Saliency /context detection Soft-attentive LSTM Text generation LSTM. Toward an explainable AI: What the Machine pays attention

“AI refers to systems that display intelligent behaviour by analysing their environment and taking actions – -with a certain degree of autonomy- to achieve a specific goal.”

Artificial Intelligence for Europe - Brussels, 25.4.20182018

Artificial Intelligence

Page 4: Presentazione standard di PowerPoint · 2019-11-05 · SAM Saliency /context detection Soft-attentive LSTM Text generation LSTM. Toward an explainable AI: What the Machine pays attention

Machine Learning

Deep Learning

Game theory

Knowledge representation

Automated Reasoning

Logics

Computer Vision

Pattern Recognition

Natural Language Processing

Cognitive Robotics

IntelligentIoT

Speech Recognition

Multi-Agents

Fuzzy systems

…systems that display intelligent behaviour by analysing their environment and taking actions

Page 5: Presentazione standard di PowerPoint · 2019-11-05 · SAM Saliency /context detection Soft-attentive LSTM Text generation LSTM. Toward an explainable AI: What the Machine pays attention

National Lab CINI AIIS:

51 nodes ( 47 universities, CNR, IIT, FBK)910 members

>100 Labs>700 projects>80 spinoff

National CINI Lab AIISArtificial intelligence andIntelligent systems

Page 6: Presentazione standard di PowerPoint · 2019-11-05 · SAM Saliency /context detection Soft-attentive LSTM Text generation LSTM. Toward an explainable AI: What the Machine pays attention

UNIMORE Modena , Italy

AImageLab Dipartimento di Ingegneria «Enzo Ferrari» & Modena Technopole

AIRI AI Research & Innovation Center; 36 People working in AI, ML and CV

Page 7: Presentazione standard di PowerPoint · 2019-11-05 · SAM Saliency /context detection Soft-attentive LSTM Text generation LSTM. Toward an explainable AI: What the Machine pays attention

ComputerVision

PatternRecognition

MachineLearning

Deep Learning with (Artificial)

Neural NetworksIntelligentInference

Action

Display intelligent behaviour

Analyse the environment

Take actions

Visual Intelligence

Page 8: Presentazione standard di PowerPoint · 2019-11-05 · SAM Saliency /context detection Soft-attentive LSTM Text generation LSTM. Toward an explainable AI: What the Machine pays attention

Where do you put your attention?

What do you predict while driving?

Saliency or task-driven attention?

Visual intelligence

Page 9: Presentazione standard di PowerPoint · 2019-11-05 · SAM Saliency /context detection Soft-attentive LSTM Text generation LSTM. Toward an explainable AI: What the Machine pays attention

Visual Intelligence for a better human and machine mutual Comprehension

Page 10: Presentazione standard di PowerPoint · 2019-11-05 · SAM Saliency /context detection Soft-attentive LSTM Text generation LSTM. Toward an explainable AI: What the Machine pays attention

..l’arte suprema di saper vedere..

Page 11: Presentazione standard di PowerPoint · 2019-11-05 · SAM Saliency /context detection Soft-attentive LSTM Text generation LSTM. Toward an explainable AI: What the Machine pays attention

Visual intelligencecan helpmachines

Page 12: Presentazione standard di PowerPoint · 2019-11-05 · SAM Saliency /context detection Soft-attentive LSTM Text generation LSTM. Toward an explainable AI: What the Machine pays attention

Interacting with AI

Page 13: Presentazione standard di PowerPoint · 2019-11-05 · SAM Saliency /context detection Soft-attentive LSTM Text generation LSTM. Toward an explainable AI: What the Machine pays attention

Viual Intelligence

Interacting with AI

by collaborative robotics

Page 14: Presentazione standard di PowerPoint · 2019-11-05 · SAM Saliency /context detection Soft-attentive LSTM Text generation LSTM. Toward an explainable AI: What the Machine pays attention

Interacting with AI

Page 15: Presentazione standard di PowerPoint · 2019-11-05 · SAM Saliency /context detection Soft-attentive LSTM Text generation LSTM. Toward an explainable AI: What the Machine pays attention

AI, data understanding and visual Intelligence

can helphumans in controlling Machines in cyber

Page 16: Presentazione standard di PowerPoint · 2019-11-05 · SAM Saliency /context detection Soft-attentive LSTM Text generation LSTM. Toward an explainable AI: What the Machine pays attention

Visual Intelligencefor secuirty

Page 17: Presentazione standard di PowerPoint · 2019-11-05 · SAM Saliency /context detection Soft-attentive LSTM Text generation LSTM. Toward an explainable AI: What the Machine pays attention
Page 18: Presentazione standard di PowerPoint · 2019-11-05 · SAM Saliency /context detection Soft-attentive LSTM Text generation LSTM. Toward an explainable AI: What the Machine pays attention

Imagine to haveImagination

Fakes from Art

Art2Realby GANs

[M. Tomei, M. Cornia, L. Baraldi, R. Cucchiara. “Art2Real: Unfolding the Reality of Artworks via Semantically-Aware Image-to-Image Translation”. CVPR 2019]

Page 19: Presentazione standard di PowerPoint · 2019-11-05 · SAM Saliency /context detection Soft-attentive LSTM Text generation LSTM. Toward an explainable AI: What the Machine pays attention
Page 20: Presentazione standard di PowerPoint · 2019-11-05 · SAM Saliency /context detection Soft-attentive LSTM Text generation LSTM. Toward an explainable AI: What the Machine pays attention

AI recognizes

YOU and your

ancestors!

Page 21: Presentazione standard di PowerPoint · 2019-11-05 · SAM Saliency /context detection Soft-attentive LSTM Text generation LSTM. Toward an explainable AI: What the Machine pays attention

Visual Intelligence is Imagination and Hallucination

DeepFakes

https://www.cnn.com/interactive/2019/01/business/pentagons-race-against-deepfakes/

Page 22: Presentazione standard di PowerPoint · 2019-11-05 · SAM Saliency /context detection Soft-attentive LSTM Text generation LSTM. Toward an explainable AI: What the Machine pays attention

REAL FAKE

AI can helpdesigners

Page 23: Presentazione standard di PowerPoint · 2019-11-05 · SAM Saliency /context detection Soft-attentive LSTM Text generation LSTM. Toward an explainable AI: What the Machine pays attention

AI can helpall of usin security and smart cities

PrEVUE

Page 24: Presentazione standard di PowerPoint · 2019-11-05 · SAM Saliency /context detection Soft-attentive LSTM Text generation LSTM. Toward an explainable AI: What the Machine pays attention

..l’arte suprema di saper vedere..

Saliency

Page 25: Presentazione standard di PowerPoint · 2019-11-05 · SAM Saliency /context detection Soft-attentive LSTM Text generation LSTM. Toward an explainable AI: What the Machine pays attention
Page 26: Presentazione standard di PowerPoint · 2019-11-05 · SAM Saliency /context detection Soft-attentive LSTM Text generation LSTM. Toward an explainable AI: What the Machine pays attention

SALIENCY [Itti Koch PAMI 1989]

[ Itti and Koch PAMI ’89, Nature Reviews 2001]“Saliency map”: an image map representing areas of saliency

Page 27: Presentazione standard di PowerPoint · 2019-11-05 · SAM Saliency /context detection Soft-attentive LSTM Text generation LSTM. Toward an explainable AI: What the Machine pays attention

Saliency: data-driven, perceptual or semantics driven?

Page 28: Presentazione standard di PowerPoint · 2019-11-05 · SAM Saliency /context detection Soft-attentive LSTM Text generation LSTM. Toward an explainable AI: What the Machine pays attention

• SAM

Saliency Attentive Model (SAM) @ AImageLab

M.Cornia, L.Baraldi, G.Serra, R.Cucchiara

Predicting Human Eye Fixations via an LSTM-based Saliency Attentive Model

IEEE Transactions on Image Processing, 2018Ranked #1 at LSUN Competition CVPR2017

Page 29: Presentazione standard di PowerPoint · 2019-11-05 · SAM Saliency /context detection Soft-attentive LSTM Text generation LSTM. Toward an explainable AI: What the Machine pays attention

SAM

Refine with an iterative (LSTM-based) model the saliency detection

Page 30: Presentazione standard di PowerPoint · 2019-11-05 · SAM Saliency /context detection Soft-attentive LSTM Text generation LSTM. Toward an explainable AI: What the Machine pays attention

Trained with generic images

(SALICON MIT300).. Now

SAM can explore the world

Page 31: Presentazione standard di PowerPoint · 2019-11-05 · SAM Saliency /context detection Soft-attentive LSTM Text generation LSTM. Toward an explainable AI: What the Machine pays attention

Saliency and Attention

Page 32: Presentazione standard di PowerPoint · 2019-11-05 · SAM Saliency /context detection Soft-attentive LSTM Text generation LSTM. Toward an explainable AI: What the Machine pays attention

[A.Palazzi,D. Abati, Davide S. Calderara, and R.Cucchiara Predicting the Driver's Focus of Attention: the DR(eye)VE Project IEEE Transactions on Pattern Analysis and Machine Intelligence 2018]

Page 33: Presentazione standard di PowerPoint · 2019-11-05 · SAM Saliency /context detection Soft-attentive LSTM Text generation LSTM. Toward an explainable AI: What the Machine pays attention

Tell me what you see

Saliency and Attentionin image captioningfrom Vision to Language

Page 34: Presentazione standard di PowerPoint · 2019-11-05 · SAM Saliency /context detection Soft-attentive LSTM Text generation LSTM. Toward an explainable AI: What the Machine pays attention

Saliency and Captioning

[M Cornia, L. Baraldi, G. Serra, R. Cucchiara Paying More Attention to Saliency: Image Captioning with Saliency and Context Attention ACM TOMM 2018 ]

ResNet -50, trained with Imagenet;

SAM Saliency /context detectionSoft-attentive LSTM

Text generation LSTM

Page 35: Presentazione standard di PowerPoint · 2019-11-05 · SAM Saliency /context detection Soft-attentive LSTM Text generation LSTM. Toward an explainable AI: What the Machine pays attention

Toward an explainable AI: What the Machine pays attention of when is describing the scene

Page 36: Presentazione standard di PowerPoint · 2019-11-05 · SAM Saliency /context detection Soft-attentive LSTM Text generation LSTM. Toward an explainable AI: What the Machine pays attention

Attention to details

Page 37: Presentazione standard di PowerPoint · 2019-11-05 · SAM Saliency /context detection Soft-attentive LSTM Text generation LSTM. Toward an explainable AI: What the Machine pays attention

Where people are, what people are doing, what the people see.

Page 38: Presentazione standard di PowerPoint · 2019-11-05 · SAM Saliency /context detection Soft-attentive LSTM Text generation LSTM. Toward an explainable AI: What the Machine pays attention
Page 39: Presentazione standard di PowerPoint · 2019-11-05 · SAM Saliency /context detection Soft-attentive LSTM Text generation LSTM. Toward an explainable AI: What the Machine pays attention

Ball in hand?

Learning to put Attention in details

Page 40: Presentazione standard di PowerPoint · 2019-11-05 · SAM Saliency /context detection Soft-attentive LSTM Text generation LSTM. Toward an explainable AI: What the Machine pays attention
Page 41: Presentazione standard di PowerPoint · 2019-11-05 · SAM Saliency /context detection Soft-attentive LSTM Text generation LSTM. Toward an explainable AI: What the Machine pays attention

People detection

Page 42: Presentazione standard di PowerPoint · 2019-11-05 · SAM Saliency /context detection Soft-attentive LSTM Text generation LSTM. Toward an explainable AI: What the Machine pays attention

People Join detection

Page 43: Presentazione standard di PowerPoint · 2019-11-05 · SAM Saliency /context detection Soft-attentive LSTM Text generation LSTM. Toward an explainable AI: What the Machine pays attention

People Join detection

Page 44: Presentazione standard di PowerPoint · 2019-11-05 · SAM Saliency /context detection Soft-attentive LSTM Text generation LSTM. Toward an explainable AI: What the Machine pays attention
Page 45: Presentazione standard di PowerPoint · 2019-11-05 · SAM Saliency /context detection Soft-attentive LSTM Text generation LSTM. Toward an explainable AI: What the Machine pays attention
Page 46: Presentazione standard di PowerPoint · 2019-11-05 · SAM Saliency /context detection Soft-attentive LSTM Text generation LSTM. Toward an explainable AI: What the Machine pays attention

[M. Fabbri, F. Lanzi , S. Calderara, R. Cucchiara, Learning to Detect and Track Visible and Occluded Body Joints in a Virtual World ECCV2018]

Hallucinating occluded joints

Page 47: Presentazione standard di PowerPoint · 2019-11-05 · SAM Saliency /context detection Soft-attentive LSTM Text generation LSTM. Toward an explainable AI: What the Machine pays attention

Hallucinating third dimension of (occluded) joints

Page 48: Presentazione standard di PowerPoint · 2019-11-05 · SAM Saliency /context detection Soft-attentive LSTM Text generation LSTM. Toward an explainable AI: What the Machine pays attention
Page 49: Presentazione standard di PowerPoint · 2019-11-05 · SAM Saliency /context detection Soft-attentive LSTM Text generation LSTM. Toward an explainable AI: What the Machine pays attention

For Machine and Human Mutual Comprehension

Human Behavior Understanding

Page 50: Presentazione standard di PowerPoint · 2019-11-05 · SAM Saliency /context detection Soft-attentive LSTM Text generation LSTM. Toward an explainable AI: What the Machine pays attention

Future HMI

ControllableExplainableCorrectable

Captioning for Explainable Reasoning

Page 51: Presentazione standard di PowerPoint · 2019-11-05 · SAM Saliency /context detection Soft-attentive LSTM Text generation LSTM. Toward an explainable AI: What the Machine pays attention

A penny for your thoughts

Captioning for Explainable Reasoning

Page 52: Presentazione standard di PowerPoint · 2019-11-05 · SAM Saliency /context detection Soft-attentive LSTM Text generation LSTM. Toward an explainable AI: What the Machine pays attention

A penny for your thoughts

Captioning for Explainable Reasoning

Page 53: Presentazione standard di PowerPoint · 2019-11-05 · SAM Saliency /context detection Soft-attentive LSTM Text generation LSTM. Toward an explainable AI: What the Machine pays attention

Imagine to understandwhat the robot sees

Captioning for Explainable Reasoning

L. Baraldi, R.CucchiaraExplainable Robot-World interactionArxiv 2019.

Page 54: Presentazione standard di PowerPoint · 2019-11-05 · SAM Saliency /context detection Soft-attentive LSTM Text generation LSTM. Toward an explainable AI: What the Machine pays attention

AI ALGORITHMS & ARCHITECTURESAI DATAAI HARDWARE

What do you need?

Page 55: Presentazione standard di PowerPoint · 2019-11-05 · SAM Saliency /context detection Soft-attentive LSTM Text generation LSTM. Toward an explainable AI: What the Machine pays attention

What you don’t need..

1 ignorance2 negligence3 malevolency4 skepticism

Page 56: Presentazione standard di PowerPoint · 2019-11-05 · SAM Saliency /context detection Soft-attentive LSTM Text generation LSTM. Toward an explainable AI: What the Machine pays attention

Ma c’e’ una magia che e’ opera divinaLa’ dove la scienza di Dio si manifestaattraverso la scienza dell’uomo…

(U.Eco 1984)

THANKS

Thanks to all AImageLab UNIMORE