Using Video Analysis and Tracking and Managing Resources

Using Video Analysis and Ad-hoc Wireless Connection Technologies to

Detect Specific Objects for Tracking and Managing Resources

Turning Videos into LBS

LSCM LBS Forum October 20, 2015

Dorbin Ng, PhD

Department of Systems Engineering and Engineering Management The Chinese University of Hong Kong

3

LBS for Timely Resource Management

Otherwise… Shortages

Bottlenecks

Cascading Delays

Service Disruptions

Resources E.g., staff & equipment

Supporting consumption for facilitating operations

Timely replenishment for ensuring seamless operations

Knowing the shortages Triggering timely actions

Equipment in Hospital

4

Will wheelchairs or stretchers be available when needed for patient care?

Trolleys in Airport

5

Looking for trolleys?

Replenishing trolleys in timely manner?

Stroller Services in Malls & Airports

6

Are they available when needed?

What are the Possible LBS Solutions for Tracking & Managing Resources?

7

RF Solutions?

Tagging all items?

Reader network infrastructure?

Maintenance cost?

Setup cost?

Visual/image solutions?

Setup cost?

Identification?

Camera network infrastructure?

Maintenance cost?

Robotic solutions? Proximity solutions? … Multi-modal solutions? What would be the right mix of modals? …

Video Analytics for Resource Tracking & Management

Camera/CCTV Infrastructure

Enabling Technologies Real-time

Notifications

Turning Videos into Tracking Data & Actionable Intelligence

Resource Analytics

8

Video Sensing Management & Analytics Architecture Framework

9

Video Sensing Management & Analytics Architecture Framework

Video Stream

Feature Extraction

Object Detection

Object Learning Module

Sensor Devices

Alert Management

Monitoring Visualization

Resource Visualization

Content Analysis

Application & Presentation

Video & Sensor Input Sources

Event Detection for Alert Notification

Object & Event Monitoring

Management Module Analytics Module

Resource Modeling & Management

Monitoring Modeling & Management

Monitoring & Management

Platform

Making use of computation power, machine learning, image processing, & system engineering

Providing monitoring & counting capability for resource management

Image Feature Extraction: Motivation

A subjective decision on defining similar images

Different people interpret similarity differently

Even the same person interprets similarity differently in different situations

Difficulties on searching similar images

Choosing features to represent images

Color, texture, shape, object, semantics, …

Accessing images from the defined, extracted features

High dimensional feature vectors with continuous values

Efficiency in finding similar feature vectors in a high dimensional space

Enabling multi-mode search on multimedia data 10

Image Features

Images Containing Similar Colors

11

Images Containing Similar Shapes

12

Images Containing Similar Content Soccer game

Mountains as prominent scenery

13

Challenges in Image Similarity

14

• Similar color distribution

• Similar texture pattern

• Similar shape/pattern

• Similar real content

Deg

ree

of

dif

ficu

lty

Histogram matching

Texture analysis

Edge, contour analysis

Life-time goal :-)

• Similar object Image Segmentation, Pattern recognition

Image Feature Extraction Techniques

Color histogram

Histo144 image feature extraction

Relying only on color information

Color and texture histograms

MCP image feature extraction

Combining texture and color information

Color coding

Cuebik image feature extraction

Coding a color to each of the pre-defined image region

More features available via OpenCV APIs

http://docs.opencv.org/modules/refman.html

OpenCV (Open Source Computer Vision Library) is an open source computer vision and machine learning software library.

15

Color Histogram: Histo144

Dividing an image into 3x3 regions

Possible to have 5x5, 7x7, … regions

For each region, calculating a 16-bin color histogram using

Munsell Color Space

Which uses hue, chroma, and value instead of red, blue, and green

In which close colors are perceptually similar.

Combining all regions’ color histograms to form

A 144-dimensional feature vector

i.e., 144 doubles representing an image

Reducing the dimension to 50 by a Single Value Decomposition (SVD) matrix, trained by corpus-specific images 16

Histo144 Feature Extraction Results

17

Texture & Edge Detections

18

19

Slide from http://images.slideplayer.com/14/4246341/slides/slide_4.jpg

Object Detection: Motivation

Human perceives objects (e.g., faces, trolleys) easily

But mysteriously in terms of how it is being done

Many different object types exist in environment

Groups of people, vehicles, chairs, etc.

Objects in each object type may have variations but have common features.

Automated object detection can be useful in

Supporting image retrieval

Labeling and categorizing images

Enhancing photography for automatically focusing, color balancing, and zooming on a specific object

Building systems for automatic object identification

Counting number of objects for management and decision making purposes

20

Object Detection & Machine Learning

Challenges in Object Detection

Amount of variation in visual appearance Color, texture, contour, shape, etc.

Orientation and rotation with respect to the camera

Distance from the camera

Variation in surrounding environment Light sources

Varying intensity, color, and location with respect to the object

Nearby objects Casting shadows on the object

Reflecting additional light on the object

21

Object Detection Techniques

A two-part strategy to cope with all this variation

View-based approach

Multiple detectors to deal with variation in orientation

Statistical modeling

Within each detector to account for the remaining variation

22

View-based Detectors

Independent development of individual detectors

Each detector specialized to

A specific orientation of the object; e.g.:

Specialized to the different rotating views of objects

A specific size within a rectangular image window

Operation

Re-applying detectors for all possible positions

Re-applying detectors for all resized images

Combining individual detectors’ results

Choosing the strongest detection if multiple detections happening at the same or adjacent locations

23

Statistical Form of Detector

Two statistical distributions are modeled within each view-based detector:

The statistics of the given object is modeled as P(image | object)

The statistics of the rest of the visual world, called “nonobject” class, is modeled as P(image | nonobject)

Detection decision using the likelihood ratio test

If the likelihood ratio (the left side) is greater than the right side, the object is decided to be present

24

P( image | object ) >

P( nonobject )

P( image | nonobject ) P( object )

Learn

Detect

Training Object Detectors via Machine Learning Techniques

25

Training Image Data

Testing Image Data

New Image Data

Machine Learning

Feature Extraction

Object Detector

Result Evaluation

Trained Object

Detector

Detected Objects

Feature Extraction

Feature Extraction

Support Vector Machine (SVM) A supervised learning technique to define a discriminative

classifier by a separate hyperplane

Given a set of labeled data, SVM yielding an optimal hyperplane, which is able to categorize new examples

26

Multiple green lines offer a solution to separate 2 groups of objects. Is any of them better than the others?

SVM gives the largest minimum distance to the training data by having the optimal separating hyperplane maximizing the margin of the training data.

Graphics from http://docs.opencv.org/doc/tutorials/ml/introduction_to_svm/introduction_to_svm.html

Trolley Detection

27

Transform Detect

Filter Transform

Input

Real-time Notification for Trolley Shortage

28

Image processing Feature extraction Feature grouping

Machine learning Feature selection Ensemble methods

Difficulties Different viewing

angles, sizes, orientations, & lighting conditions

Leveraging on Large quantity of

same type of rigid objects

Pilot Project: Trolley Management at HKIA

18 Trolley Stations in HKIA Baggage Reclaim Hall

2 3 4 5 9 8 7 6 10 11 12 13

Custom Hall

Immigration

Trolley pick up point

5.2B

5.2A

5.4A

5.4B 5.5B

5.5A

5.6B

5.6A

5.8B

5.7A 5.7B

5.8A

5.9B

5.9A

5.10B

5.10A 5.12A

5.12B

Trolley Stations in HKIA Baggage Reclaim Hall

29

About another 100 trolley stations in airside Departure & Arrival Halls and at curbside outside Terminals 1 & 2

Which station(s) is out of trolleys? Or, having shortage? What is the efficient way to orchestrate limited staff resources to timely replenish trolleys to the right station(s)?

30

Trolley shortage

notification

Real-time Notification

via Mobile App

(iOS & Android)

Out of trolley notification

Notification history

31

1. Photo viewing for the particular message being selected in the “Message” screen

2. Photo view supporting zoom-in & zoom-out functions

1. Default view with a full photo fitting on the screen

Message

32

Performance

Hour-box view for each location

Chart view showing

details on performance

“Daily Record” & “History Record”

on following pages

33

Performance

1. “Daily Record” view with hourly timeline information for a selected date

2. Current date being the default selected date

34

Performance

1. Selecting date 2. Clicking “Add” to the date list 3. Repeating #1 & #2 for selecting

more dates 4. When done with selecting

date(s), clicking “Generate” to see a chart view

Chart view showing

details on performance

on selected date(s)

Clicking “Save” to capture the chart views to

an image file

35

Map

Select Location Menu Button

36

Oct 2 Fri

Oct 3 Sat

Oct 4 Sun

Oct 5 Mon

Oct 6 Tue

Oct 7 Wed

Oct 8 Thu

Typhoon Mujigae being within 400

km from HK

Normal day pattern

Flight schedule disruption 1

Flight schedule disruption 2

Flight schedule disruption 3

Flight schedule disruption 4

Normal day pattern being

resumed

An

alytics: Re

sou

rce M

on

itorin

g B

usin

ess In

tellige

nce

Acknowledgement

37

• Video Analytics for Resource Management

2015.09.22 – 2017.09.21, ITP/048/15LP, Prof. CH Cheng, Dr. Dorbin Ng, Mr. Tim Chan

Asian Institute of Supply Chains & Logistics, CUHK • Resources Management via Real-time RFID Tracking Data Analytics – A Case in the

Healthcare Industry 2014.01.01 – 2015.12.31, Prof. CH Cheng, Dr. Dorbin Ng

• Image-based Drug Verification for Risk Management in Drug Dispensing 2014.10.01 – 2016.09.30, Prof. CH Cheng, Dr. Dorbin Ng

http://www.se.cuhk.edu.hk/~ssi/