This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Internet-of-Things and Cyber-physical Systems- Exploratory Research in Signal Processing, Communication and Computing
Dr. Arpan PalPrincipal Scientist and Research HeadInnovation Lab, KolkataTCS
Apr 15, 2023
2
Cyber-physical Systems – Internet-of-Everything
Humans
Physical Objects
and Infrastruct
ure
Computing Infrastruct
ure
Peo
ple
Con
text
Dis
cove
ry
PhysicalContext Discovery
INTERNET OF EVERYTHING
Physical Context
Discovery
What is happening, where and when
People Context
DiscoveryWho is doing what, where and when,
who is thinking what
Internet of
Digital
Internet of
Things
Internet of
Humans
ABI Research. May 7, 2014
"In the next century, planet earth will don an electronic skin. It will use the Internet as a scaffold to support and transmit its sensations.“ - Neil Gross 1999
3
It’s a Connected Universe
NEVER FORGET YOUR PILLS MONITOR YOUR ACTIVITYMONITOR THE AGED
Click to edit Master title styleProgram: IoT Platform Solutions
6
Integrated Platform for Intelligent Enterprise
People Feedback & Emotions
Social Media
Integrated Services
Sensors & IoTPlatform Legacy Monitoring & Control Systems Enterprise Data
Smart Integration Platform
Transportation Human Resources Energy
OperationsSafety Asset Tracking
Smart Integrated Services
Sense
Analyze
Extract
Respond
Intelligence
Smart Domain Services
Supply Chain
Security and Surveillance
Sense: People Context, Appliances, Building, Plant, Utility Infrastructure
Sync Transportation with Remote Operations Link Asset Tracking and Safety with Surveillance
Employee Wellness and Energy Preservation as Community Initiatives
Intelligent Integration Platform
Integrated Intelligent Services
7
Requirements and Challenges for IoT – Need for a Platform
Applications need support for
Visibility Capture & store data from sensors
InsightsPatterns, relationships and models
Control Optimize and actuate
TCUP – TCS Connected Universe PlatformA horizontal platform for addressing the IoT Software and Services market
Model-driven DevelopmentModel the Domain Knowledge
Model the Infrastructure – Network, Storage, Compute
Model the Analytics – map to Domain Requirements
Model the Architecture – Device and Cloud
TCUP Platform
Model the Sensor – Semantics, Phenomenon
8
TCUP Design and Architectural Highlights
18 patents filed, Standard Body Contribution - IETF and Singapore ITSC
• Fog Computing – Utilize unused compute power of edge devices
Distributed Computing on Edge Devices
• To reduce network congestion• Adequate Security and Reliability
Adaptive, Lightweight yet Secure Communication Protocols
• For economical scaling of sensor data store
Efficient Compression
• Statistical and Information-theoretic measure to find out potential privacy-breaching content
Sensitivity Measurement and Privacy Preservation
• Semantic annotation of sensors• Sensor Search Engine
Semantic Enabled Sensor Explorer
• Algopedia – Algorithm Repository, Search and Recommendation• Semantic Sensor Web
Model-driven Development
ManageScale,
Reduce Cost
Handle Privacy
Manage Diversity and Inter-
opEase of
Development
9
Horizontal operators(semantic integration) operates on data from heterogeneous sources to created integrated data streams.
Semantic Sensor Web - From Data to Wisdom
temperature
humidity
odor
image
high temperature
gaseous odor
light
concentrated light
high temperature indicates fire
gaseous odor indicates gas discharge
Fire from Gas Leak, evacuate
immediately, send fire fighting team
equipped with gas leakage
data
information
knowledge
wisdom
Vertical operators(semantic abstraction) operates on artifacts at each level and transcends them to the next level
F PCS(Data, KB*) → Information
F PCS(Knowledge, KB) → Wisdom
F PCS(Information, KB) → Knowledge
KB: Knowledge base
Adopted from: Physical-Cyber-Social Computing: An early 21st Century Approach, Amit Sheth et. al.
10
Research Outcomes – Some of the Results
Publications in ACM Sensys, Ubicomp, Infocomm, Middleware
7 times less network load
10 times less latency
Lightweight Security – 5
times less overhead
Improved Privacy via Sensitivity Detection
Improved Compute through Edge Devices
11
Program: Human Behavior Modeling and Data Collection
12
Human Data Collection and Behaviour Modelling - Program Overview
Research Goals - Given a context, predict behaviour. Given behaviour, find out context.Focus Domains – Organization Behavior, Consumer BehaviorCurrent Available Models – Statistical, Need of the Day – Models based on physical data
Meetings – Group or One-on-one
Digital Communication
Individual @Work or Leisure
Individual – Day in the Life Of
• Strength and Polarity of Relationships
• Meeting Flow• Emotive State• Outcome
• Formal / Informal, Business / Social
• Strength and Polarity of Relationships
• Tenor of Communication
• Nature of Communication
• Location and Time spent
• Mood and Physical State
• Engagement Level
• Cognitive Load
• Location and Time spent
• Activity• Behavioral
Routine• Social
Interaction• Moment of
decision makingMicrophone
Email / Knome
Mobile Phone, Desktop Mounted Cameras, EEG/GSR
Mobile Phone, Kinect, EEG/GSR,Smart MeterSurveillance Camera
13
Human Identification and Activity Detection using Kinect
Human Identification– Skeleton Model Based / Depth
based– 20 joints of skeleton data
• 2D Camera with IR depth sensor
• Excitation by IR light pattern
Human Identification • Gait cycle detection• Feature extraction from
skeleton joints• Training• Recognition
Papers in IEEE Fuzz, CEC, IEEE SMC, UbiComp and ECCV
Activity • Sitting• Standing• Walking
Human activity recognition using RGB-DAccuracy is above 90%
14
Activity Detection using Mobile Phone Inertial Sensors
Activity Detection– Uses Accelerometer Data– Gyroscope and Magnetometer for orientation
Irfan Essa (1994), “Analysis, interpretation and synthesis of facial expressions“, PhD Thesis, MIT, Cambridge, MA, USA. (Advisor: Alex (Sandy) Pentland)
Robust Solutions using camera, microphone and accelerometer
20
PoC Approach and Expected Novelty
HR, HRV, RRPPG extractionRealtimeVideo
Audio from Mic
Accelerometer
Cardiovascular Model
SpO2
BP, ECG
Spirometry
Breathing Rate
Image of eyePupillary Reflex
21
Photo-plethysmography (PPG) using Mobile Phone Camera
Subject1 Subject2 Subject3
Actual Detected Actual Detected Actual Detected
68 66 66 63 85 84
2.9% 4.5% 1.1%
Papers at Mobihoc, IEEE BIBE, SenSys, ICASSP
Data set Pd Ps PP-diff < 15
Standard dataset (14 features) 92.9% 74.7% 77.9%
TCS dataset - add height, weight, age
99.3% 82.7% 85.5%
22
Program: Mobile Interactive Remote Sensing
23
Sensing the Physical World
Mobile phone based crowd sensing
Robot assisted sensing
www.popularmechanics.com
www.engadget.com
www.allthingssd.com
apollo2.cs.illinois.edu
Camera based sensing
24
Intelligent Transportation – Vehicle Model Driven Sensor Data Analysis
KNOWN PARAMETERS EFFECTS TARGET INDUSTRYVehicle Type & Driving Behavior Road Condition Monitoring City MunicipalityRoad Condition & Driving Behavior Car Prognosis Automotive
Road Condition & Vehicle Type Driving Behavior Analysis Insurance
Acceleration a(t) = f (H(t), v(t), R(t), D(t))
H(t)
Papers in ICST, Percom
25
Phone Microphone based Sound Scaping
Solution OverviewEvent driven with participatory sensing aided audio surveillance system • Classification of Traffic Noise (Honk Detection) and Crowd Noise
Papers in CODIS, ISSNIP, ISDA
26
Phone Camera based 3D Reconstruction from 2D images
Input Images
Dense Reconstruction without using mobile inertial sensors
- 20 images, compute time (4 core, 1GPU) ~ 20 min
- 120 images, compute time (16 core, 1GPU) ~ 30 min
- Bandwidth saving ~ 8 times, if done on mobile
Sparse Reconstruction using Mobile Inertial Sensors for Camera Position Estimation
• 20 images, compute time (4 core, 1GPU) ~ 3 min (without using inertial sensors)
• 20 images – compute time (4 core, 1GPU) ~10 sec. (with inertial sensors)
• Bandwidth saving ~ 200 times, if done on mobile
• Sparse good enough for many applications• Mobile Sensing and ACCV (submitted)• Dense Reconstruction with mobile
inertial sensors under progress with more number of images target < 1min
Dense Reconstruction -120 images
Dense Reconstruction - 20 images
• Low cost solution for 3D reconstruction from multiple 2D images captured from mobile device.
• Motion information from the inbuilt inertial sensors – for camera position estimation
• Applications in Agro-advisory service, Remote Diagnostics, Remote Healthcare
27
Multi-sensor Fusion for Robot-assisted Sensing
Application in remote sensing in hazard-prone areas• Robot carries 2D camera and heat / chemical sensors on a rotating arm• 3D reconstruction from the 2D vision• Estimation of Heat / gas leak / sound Source (direction and range) through passive
directional signal processing• Fusion of heat / gas / sound map on reconstructed 3D vision map
www.ese.wustl.edu
Ongoing Work Possible reuse from 2D-3D reconstruction and sound