Artificial Intelligent Sensors: The core of Cyber-Physical ... · • Embedded Real-Time Fall Detection with Deep Learning on Wearable Devices; Euromicro DSD/SEAA 2018, August 29

Artificial Intelligent Sensors:

The core of Cyber-Physical-Systems

From Theory to Practice

Danilo Pau

Advanced System Technology

Agrate Brianza

Introduction

2

The Cyber-physical Systems

Physic

al

Dom

ain

Cyber

Dom

ain

Obje

ct

Dom

ain

Physical

SensingActuation

and Control

3

Cyber-physical Systems applications

Cyber-physical Systems

Cities

Home/Building

Industry 4.0

Autonomous CarsDatacenters

Agriculture 2.0

4

Opportunities and Challenges

5

Computers were big 6

• Intel 8086

• 8 MHz

• 128 KB RAM

• 16 KB ROM

• 1.84 W

• 360 $

Olivetti M241983

Computers were big 7

Olivetti M241983

• Intel 8086

• 8 MHz

• 128 KB RAM

• 16 KB ROM

• 1.84 W

• 360 $

• STM32 MCU L4

• 80 MHz

• 128 KB RAM

• 1 MB Flash

• < 20mW

• < 4 €

Sensors were big • First accelerometer (1923)

• Credits: McCollum and Peters

• Commercialized by 1927 in the US

• Resistance Bridge type

• E-shaped frame containing 20 to 55 carbon rings in a tension-

compression Wheatstone

• Half-bridge between the top and center section of the frame

• Dimensions: ~ 28 c𝑚3

• Resonant frequency < 2 kHz

• Application in bridges, dynamometers, and aircraft

• Major revision (1936)

• 2-axis with up to 100g range

• Applications vastly increased

• Price: $420 ($6,275 at todays rate)Credit http://www.sandv.com/downloads/0701walt.pdf

http://www.egr.msu.edu/classes/ece480/capstone/fall12/group07/techpres.pdf

8

STMicroelectronics 6-axis IMU evolution

SiP 3D digital accelerometer and gyroscope

Sensors are miniaturized

Credit http://semieurope.omnibooksonline.com/2014/semicon_europa/International_MEMS_Forum/13_Romain_Fraux_System_Plus_Consulting.pdf

9

Digital Camera Kodak’s Steven Sasson 1973

• 50 ms to capture the image

• 23 s to record on a tape

• 3.6 kg, 10K pixels.

• black-and-white images.

• Electronic still camera, US patent

4131919 A

10

http://www.cmosis.com/technology/technology_overview/endoscopy_ultra_small_area_imaging_modules

Ultra Small Imaging Modules 11

With Machine Learning

Backup camera

Side mirror camera

12

Multi and Heterogeneous sensors hub

• 4x MP34DT04-C1 – 64dB SNR Digital MEMS microphone

• LSM6DSM – iNEMO inertial module: 3D accelerometer and

3D gyroscope

• LSM303AGR – ultra-compact high-performance eCompass

module: ultra-low

• LPS22HB – MEMS nano pressure sensor: 260-1260 hPa

absolute digital output

• BlueNRG-MS – Bluetooth low energy network processor

• STBC03JR – linear battery charger with 150 mA LDO 3.0 V

• STM32F446 – 32-bit high-performance 180 MHz MCU

(ARM® Cortex®-M4 with FPU)

13

IoT with 100s of Billions of Sensors

Fonte: https://www.ncta.com

15

16

Unit

Sen

sors

Co

mm

Unit

Sen

sors

Co

mm

Unit

Sen

sors

Co

mm

Gateway

Unit

Sen

sors

Co

mm

Gateway

Gateway

Server Cloud

Application

Intelligent Mechanisms

Scalability

Responsiveness

Dumb units

Designing too centralized CPS 16

Example: Cloud based Voice Recognition

• Average person’s daily utterances as 16,000 words[1] and an average

speech rate of 163 words per minute[2], ≈ 98 minutes of speech per day.

• Multiplying that number by 128 kbps, the result would be 94 MB of voice

data per person per day. 1 million (≈94TB), 10 million (≈1PB) and 100

million (≈9 PB)

17

1. https://www.researchgate.net/publication/6223260_Are_Women_Really_More_Talkative_Than_Men

2. http://sixminutes.dlugan.com/speaking-rate/

Voice RecognitionAudio coding

Predictive Maintenance 18

Time- Cycle

Based

Maintenance

Condition-

Based

Maintenance

• Predefined lifetime for replacement

• Unexpected failures

• Adaptively raise alert based on the actual

condition of the product and environment

• Focus on critical event prediction

1. Anomaly detection: How to classify the

present condition into normal and abnormal

2. Sensor based detection: How to

recognize change-points of the system

1. https://iot.ieee.org/images/files/pdf/phm2017/06-19-2017-Rick-Durham_IEEE-PHM_Presentation20170610.pdf

Example: Cloud based Predictive

Maintenance• 2 X,Y accelerometers on each bearing, times 4

• 20 KHz sampling rate @ 16 bits per axis 320 Kbytes/s 27.648

Gbytes/day

19

https://iot.ieee.org/images/files/pdf/phm2017/06-19-2017-Rick-Durham_IEEE-PHM_Presentation20170610.pdf

loaded rotating shaft

with a constant speed

of 2000 rpms

Scalability

Responsiveness

Dumb units

EquipmentUs

e c

as

es

Local Processing• STM32F469AI 32-bit ARM Cortex-M4 MCU

• 180MHz, 2MB FLASH

• 384+4 KB of SRAM including 64-KB of CCM

• ART for 0-wait state from FLASH

• DSP Instructions

Motors Environment

Vibration and Environmental• ISM330DLC 6-Axis digital MEMS axel + gyro

• MP34DT05-A Microphone

• LPS22HB MEMS Pressure sensor

• HTS221 Humidity & Temperature Sensor

Sensing

Processing

Wired• L6362A IO-Link communication transceiver

device IC

Connectivity

STEVAL-BFA001V1B for Intelligent Edge CM and PdM

The STEVAL-BFA001V1B includes:

1. STEVAL-IDP005V1- industrial sensor

board

2. STEVAL-UKI001V1 - Adapter board

for ST-LINK/V2-1

3. 0.050” 10-pin flat cable

4. 4 Pole cable mount connector plug,

with male contacts

5. M12 female connector with 2m cable

20

Designed for:

• Condition Monitoring (CM)

• Predictive Maintenance (PdM)

The STEVAL-BFA001V1B is based on 3D digital accelerometer, environmental and acoustic MEMS sensors

1

2

3

4

5

https://www.st.com/en/evaluation-tools/steval-bfa001v1b.html

The effects on the Applications

Cyber-physical Systems

Performance reduction, cascade effects on Applications

Faults, Errors, Uncertainty, Malfunctioning,Intrusions

Changes:

Nonstationary,

seasonality, periodicity

21

IoT DDoS, screwing Dyn, Oct 2016

• Many IoT devices e.g. refrigerators,

thermostats, and toasters were the

attackers.

• From 09:30 to 18:00 ET, Dyn’s servers

were attacked in three DDoS waves. It

was based on Mirai code

• Cyberattack, affected Twitter, Amazon,

Reddit, Netflix, and more since they used

Dyn DNS provider.

• A group called “New World Hackers“ has

claimed responsibility for the attack.

22

https://readwrite.com/2016/10/22/the-internet-of-things-was-used-in-fridays-ddos-attack-pl4/

Unit

Sen

sors

Co

mm

Unit

Sen

sors

Co

mm

Unit

Sen

sors

Co

mm

Gateway

Unit

Sen

sors

Co

mm

Gateway

Gateway

Server Cloud

Application

Intelligent Mechanisms

Scalability

Responsiveness

Intelligent units

Designing Intelligent and distributed CPS 23

Environment

Sensors

Detection

Adaptation

Application / Service

User

Sensors

Detection

Adaptation

Application / Service

User

Application / Service

Adaptation

IoT need to achieve hyper scalability24

25

https://www.computer.org/csdl/mags/co/2017/12/mco2017120007.pdf

26

27

Smart vs Intelligent 28

Approximating a function

• Given

• Assuming we can calculate the function's derivatives at a single point in a range of 𝒙 and

given a Taylor series:

• For 𝑁 → ∞ 𝑎𝑐𝑐𝑜𝑟𝑑𝑖𝑛𝑔𝑙𝑦 𝑡𝑜 Cantor-Bernstein-Schröder Theorem

𝑓 − 𝑇 < 𝜀

• 𝑇 approximates 𝑓 with a small error at will

29

No more Analytical Expressions

Just raw data

• Given

30

𝒙 = 𝒚 = 𝐿𝑢𝑙𝑢′

These are just colored Pixels

This is just a label

Neural Networks as powerful approximator

• Universal approximation theorem (Cybenko1989, Hornik1991)

• Given

• MLP Approximator is

31

𝑓∗ − 𝝎 ∗ 𝑔 𝑾𝑥 + 𝑐 + 𝑐 < 𝜀

MLP approximates 𝑓 with a small

error at will

Break-through on Artificial Neural Networks

1. 𝐶𝑜𝑚𝑝𝑢𝑡𝑖𝑛𝑔 𝑀𝑎𝑐ℎ𝑖𝑛𝑒𝑟𝑦 𝑎𝑛𝑑 𝐼𝑛𝑡𝑒𝑙𝑙𝑖𝑔𝑒𝑛𝑐𝑒, 𝑂𝑥𝑓𝑜𝑟𝑑 𝑈𝑛𝑖𝑣𝑒𝑟𝑠𝑖𝑡𝑦 𝑃𝑟𝑒𝑠𝑠, 𝟏𝟗𝟓𝟎Alan Turing on Artificial Intelligence. Imitation Game

2. Universal approximation theorem, Cybenko, 1989 A MLP can be the right approximator of any function

3. Reducing the Dimensionality of Data with Neural Networks, by Hinton & Salakhutdinov, Science, 2006 end to end Autoencoder trained on data based on probabilistic neurons (RBM) approximated faces

32

Reservoir Computing: Recursive Neural Gas 33

• Neurons Compete, Coordinate and Adapt while Self

Organizing to find optimal data approximation.

• In RNG, the set of neuron Units U becomes

https://www.youtube.com/watch?v=XtC1M7nrDk0

https://arxiv.org/abs/1807.09510

34

Credit: Universita degli Studi Pavia and STMicroelectronics

On Line Self Organizing Neural Network

Human Activity Recognition

t-SNE view

35

Credit: Universita degli Studi Pavia and STMicroelectronics

On Line Self Organizing Neural Network

Human Activity Recognition

t-SNE view

36

Credit: Universita degli Studi Pavia and STMicroelectronics

On Line Self Organizing Neural Network

Human Activity Recognition

t-SNE view

37

Credit: Universita degli Studi Pavia and STMicroelectronics

On Line Self Organizing Neural Network

Human Activity Recognition

t-SNE view

38

Credit: Universita degli Studi Pavia and STMicroelectronics

Case study: Human Activity Classification

Running Activity

Case study: Human Activity ClassificationWalking Activity

39

Credit: Universita degli Studi Pavia and STMicroelectronics study

State Model N

Neural Netowork Model 2

AI based Anomaly DetectionOperating in remote and harsh environments

Self

Learning

Artificial

Neural

Network

Machine

Learning

Self

Learning

Statistical

models

40

0 2000 4000 6000 8000 10000 120000

10

20

30

40

50

Samples

Tem

pera

ture

(°C)

Sensor 1

Sensor 2

Sensor 3

0 2000 4000 6000 8000 10000 120000

2

4

6

Samples

Num

ber o

f miss

ing

data

Time varying signals

Operational Phase, Data predictionTraining Phase

Neural Netowork Model 1

Event

Detection and Validation

On line Re–training and Adaptation

AI Model State identification

Event

http://ieeexplore.ieee.org/document/7966066/

STMicroelectronics CPS related

publications• Intelligent Cyber-Physical Systems for Industry 4.0, to be published @ 1st IEEE International Conference on Artificial

Intelligence for Industries, Septembre 26-28 2018

• Detecting changes at the sensor level in cyber-physical systems: Methodology and technological implementation,

Neural Networks (IJCNN), 2017 International Joint Conference on, 14-19 May 2017, DOI: 10.1109/IJCNN.2017.7966066

• Event-Driven Cooperative-Based Internet-of-Things (IoT) System, Proceedings of International Conference on IC

Design and Technology, June 4–6, 2018, Otranto Italy

• Testing a Mobile Visual Search application using a novel open source CPS simulator, GTTI Thematic Meeting on

Multimedia Signal Processing 2017, January 29-31 2017, http://www.isip40.it/gtti.mmsp2017/

• Accurate Cyber Physical System Simulation for Distributed Visual Search Applications, Proceedings of 3°

International Forum on Research and Technologies for Society and Industry, Modena, Italy, September 11-13 2017

• An Open-Source Extendable, Highly-Accurate and Security Aware Simulator for Cloud Applications, Proceedings

of 21st Conference on Innovation in Clouds, Internet and Networks (ICIN 2018), 20-22 February 2018 Paris, France

• Designing a Mobile Visual Search application with the help of a novel open source CPS simulator, GTTI Thematic

Meeting on Multimedia Signal Processing 2018, January 22-23 2018, http://webmagazine.unitn.it/evento/disi/28651/gtti-

thematic-meeting-2018-on-multimedia-signal-processing/

• Invited talk - COSSIM: An Open-Source Integrated Solution to Address the Simulator Gap for Systems of

Systems; Euromicro DSD/SEAA 2018 August 29 – 31, 2018, Prague | Czech Republic http://dsd-

seaa2018.fit.cvut.cz/main/DSD_2018_DetailedProgram.pdf

41

STMicroelectronics A.I. related publications• A 2.9 TOPS/W Deep Convolutional Neural Network SoC in FD-SOI 28nm for Intelligent Embedded Systems, IEEE ISSCC February

2017 and 17th INTERNATIONAL FORUM ON MPSoC for software-defined hardware, http://www.mpsoc-forum.org/agenda.html

• Intelligent Embedded and Real-Time ANN-based Motor Control for Multi-Rotor Unmanned Aircraft Systems, Proceedings of 25th

IFIP/IEEE International Conference on Very Large Scale Integration (VLSI-SoC) Abu Dhabi, UAE October 23 - 25, 2017

• Efficient Light Harvesting for Accurate Neural Classification of Human Activities, Proceedings of 2018 IEEE International Conference

on Consumer Electronics (ICCE), Las Vegas, USA, January 12-14, 2018

• Complexity and Accuracy of Hand-Crafted Detection Methods Compared to Convolutional Neural Networks, Proceedings of 19th

international Conference on Image Analysis and Processing, ICIAP 2017, 11-15 Sept 2017Pre-trainable Reservoir Computing with

Recursive Neural Gas; L Carcano, E Plebani, DP Pau, M Piastra - arXiv preprint arXiv:1807.09510, 2018

• Pre-trainable Reservoir Computing with Recursive Neural Gas; L Carcano, E Plebani, DP Pau, M Piastra - arXiv preprint

arXiv:1807.09510, 2018

• Artificial Intelligent Sensors: the core of Cyber-Physical-Systems From Theory to Practice, 18th International Forum on MPSoC for

Software defined Hardware, 7/29 – 8/3 Snowbird, Utah

• Embedded Real-Time Fall Detection with Deep Learning on Wearable Devices; Euromicro DSD/SEAA 2018, August 29 – 31, 2018,

Prague | Czech Republic

• Studying the Effects of Feature Extraction Settings on the Accuracy and Memory Requirements of Neural Networks for Keyword

Spotting, Consumer Electronics Berlin (ICCE-Berlin), 2018. ICCEBerlin 2018. IEEE 8th International Conference on

• A CNN Architecture for Efficient Semantic Segmentation of Street Scenes, Consumer Electronics Berlin (ICCE-Berlin), 2018.

ICCEBerlin 2018. IEEE 8th International Conference on; Conference 2nd Best Paper

• Automated generation of Single Shot Detector C library from a high level Deep learning framework, 4th International Forum on

Research and Technologies for Society and Industry; Palermo, Italy, September 10-13 2018

• Parallelized Convolutions for Embedded Ultra Low Power Deep Learning SoC, 4th International Forum on Research and Technologies

for Society and Industry; Palermo, Italy, September 10-13 2018

42

43𝐼, 𝑟𝑜𝑏𝑜𝑡 2004

[email protected]

44

https://www.nextrembrandt.com/ , 2016

Isn’t ironic ?