CLASSIFICATION OF ABNORMAL CROWD BEHAVIOR USING IMAGE PROCESSING AND STATE MACHINES NG TZE JIA A thesis submitted in fulfilment of the requirements for the award of the degree of Master of Engineering (Electrical) Faculty of Electrical Engineering Universiti Teknologi Malaysia AUGUST 2015
26
Embed
CLASSIFICATION OF ABNORMAL CROWD BEHAVIOR USING …eprints.utm.my/id/eprint/53842/1/Ng,TzeJiaMFKE2015... · awam seperti stesen kereta api, lapangan terbang dan pusat membeli-belah
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
CLASSIFICATION OF ABNORMAL CROWD BEHAVIOR USING IMAGE
PROCESSING AND STATE MACHINES
NG TZE JIA
A thesis submitted in fulfilment of the
requirements for the award of the degree of
Master of Engineering (Electrical)
Faculty of Electrical Engineering
Universiti Teknologi Malaysia
AUGUST 2015
iii
An appreciation to all who made this possible
especially my parents, supervisor and fellow friends.
iv
ACKNOWLEDGMENT
It is a valuable experience for me in working on this project and many people
have contributed towards my thoughts and understanding in making this a success. I
would like to express my gratitude to the people who have offered their precious help
and advices to me.
First of all, I would like to express my deepest gratitude to my project
supervisor, Dr. Usman Ullah Sheikh for guidance, motivations, advices and
friendship. This thesis would not be presented here without his continuous
assistances in overcoming all problems and difficulties.
Next, I wish to convey my sincere appreciation to my family for their love,
blessing and support both mentally and financially. Last but not least, I would like to
thank all my fellow friends who were involved directly or indirectly for their
knowledge, support, kindness and the time we spent together. Their views and tips
suggested were handy indeed.
v
ABSTRACT
The study of crowd behavior in public areas or during public events such as
subway station, airport and shopping mall had been started two decades ago. In this
thesis, an automated video surveillance to detect abnormal activities in a crowd using
the concept of state machine is proposed. This method is divided into three stages
which are pre-processing, feature extraction and behaviour classification. In pre-
processing, frame differencing is used for segmentation while optical flow is
performed to estimate the crowd motion. Extracted features consist of global and
local features. Global features will consider the features on the whole frame whereas
local features only consider the features on each detected object. Based on extracted
features, abnormal crowd behaviour can be classified using state machines. The
proposed state machine contains four states which will evaluate different features in
different states respectively. The frames that are able to reach the final state of the
behaviour in its state machine will be classified as the behaviour. The behaviours that
can be detected are walking, running, crowd formation, crowd splitting and panic
crowd. The method is validated using UMN data set and PETS 2009 data set. The
result of the classification has achieved an accuracy of 96.3%.
vi
ABSTRAK
Kajian tentang kelakuan orang ramai di kawasan awam atau dalam acara
awam seperti stesen kereta api, lapangan terbang dan pusat membeli-belah telah
dikaji sejak dua dekad yang lalu. Dalam tesis ini, sistem pengawasan video
automatik yang boleh mengesan aktiviti orang ramai yang tidak normal dengan
menggunakan konsep mesin keadaan telah dicadangkan. Kaedah ini dibahagikan
kepada tiga peringkat iaitu pra-pemprosesan, pengekstrakan ciri-ciri dan pengelasan
kelakuan. Dalam pra-pemprosesan, penolakan gambar digunakan untuk segmentasi
manakala aliran optik digunakan untuk menganggar gerakan orang ramai.
Pengekstrakan ciri- ciri meliputi ciri-ciri global and ciri-ciri tempatan. Ciri-ciri global
merupakan ciri-ciri yang diambil daripada seluruh gambar manakala ciri-ciri
tempatan merupakan ciri- ciri yang diekstrak daripada objek-objek yang dikesan.
Berdasarkan ciri-ciri yang diekstrak, kelakuan orang ramai yang tidak normal boleh
dikelaskan dengan menggunakan mesin keadaan. Mesin keadaan yang dicadangkan
mengandungi empat keadaan yang akan menilai ciri-ciri yang berbeza dalam
keadaan masing-masing. Gambar yang dapat mencapai keadaan yang terakhir bagi
mesin keadaan kelakuan akan dikelaskan dalam kelakuan tersebut. Kelakuan yang
dapat dikelas dalam sistem ini ialah berjalan, berlari, orang ramai berkumpul, orang
ramai berpecah dan orang ramai panik. Kaedah ini telah disahkan dengan
menggunakan set data UMN dan set data PETS 2009. Ketepatan hasil pengelasan
telah mencapai 96.3%.
vii
TABLE OF CONTENT
CHAPTER TITLE PAGE
DECLARATION ii
DEDICATION iii
ACKNOWLEDGEMENT iv
ABSTRACT v
ABSTRAK vi
TABLE OF CONTENTS vii
LIST OF TABLES x
LIST OF FIGURES xii
LIST OF SYMBOLS xiv
1 INTRODUCTION 1
1.1 Introduction 1
1.2 Background 2
1.3 Research Questions 4
1.4 Problem Statement 5
1.5 Objectives of Study 5
1.6 Scope of Work 6
1.7 Contributions 6
1.8 Thesis Organization 7
2 LITERATURE REVIEW 9
2.1 Introduction 9
2.2 Pre-processing 11
2.3 Feature extraction 12
viii
2.4 Object Tracking 19
2.5 Summary 28
3 METHODOLOGY 30
3.1 Introduction 30
3.2 Pre-processing 31
3.2.1 Image Resize 31
3.2.2 Convert RGB to Grayscale Image 32
3.3 Segmentation 33
3.3.1 Obtain Region of Interest(ROI) 33
3.3.2 Removing Small Components 34
3.3.3 Dilation 34
3.3 Feature Extraction 35
3.3.1 Optical Flow 35
3.3.2 Global Features 36
3.3.3 Local Features 39
3.4 Crowd Behavior Recognition using Simple State
Machine Concept
41
3.4.1 Classify Running and Walking Components 42
3.4.2 Classify Crowd Formation Event 43
3.4.3 Classify Crowd Splitting Event 48
3.4.4 Classify Crowd Evacuation/Panic Event 49
3.6 Summary 53
4 RESULT & DISCUSSION 55
4.1 Introduction 55
4.2 Dataset 55
4.2.1 UMN Dataset 56
4.2.2 PETS 2009 Dataset 57
4.3 Simple Graphical User Interface (GUI) 62
4.3.1 Load Image 62
4.3.2 Behavior Selections 62
4.4 Frame Distribution 64
ix
4.5 Calculation of Error Rate and Accuracy 64
4.6 Categories of Crowd Behaviors 66
4.7 Ground Truth 67
4.7.1 UMN Dataset Ground Truth 69
4.7.2 PETS 2009 Dataset Ground Truth 70
4.8 Results 70
4.8.1 Walking and Running 70
4.8.2 Crowd Formation 72
4.8.3 Crowd Splitting 73
4.8.4 Evacuation or Panic Crowd 74
4.8.5 Result Summary 77
4.9 Discussion 78
4.9.1 Result Comparisons 78
4.9.2 Reasons of Fail Detections 80
4.9.2.1 Illumination 80
4.9.2.2 Occlusion 82
4.9.2.3 Crowd Formation and Splitting Exist but not
Obvious
82
4.9.2.4 Uncertainties during Transition of Behaviors 82
4.10Summary 83
5 CONCLUSION 84
5.1 Conclusion 84
5.2 Recommendations and Future Improvements 85
REFERENCES 87
x
LIST OF TABLES
TABLE NO. TITLE PAGE
2.1 Different types of feature extraction analysis and related
works.
14
2.2 Shape feature descriptors and related works. 15
2.3 Appearance-based feature descriptors and related work. 16
2.4 Feature selection for object tracking. 17
2.5 Types of Optical Flow Method. 18
2.6 Categories of object tracking approach and related works. 20
2.7 Categories of event recognition approach and related
works.
22
2.8 Examples of learning models and its related work. 23
4.1 Shows Calibration data of setting up the cameras. 59
4.2 Cameras used to film the datasets. 61
4.3 UMN dataset frames distribution in each video. 64
4.4 PETS 2009 dataset frames distribution in each video. 65
4.5 The ground truth for UMN datasets. 69
4.6 The ground truth for PETS 2009 datasets. 70
4.7 Error rate for walking behavior detection in UMN dataset. 71
4.8 Error rate for walking behavior detection in PETS 2009
dataset.
72
4.9 Error rate for crowd formation detection in UMN dataset. 73
4.10 Error rate for crowd formation detection in PETS 2009 74
xi
dataset.
4.11 Error rate for crowd splitting detection in UMN dataset. 73
4.12 Error rate for crowd splitting detection in PETS 2009
dataset.
75
4.13 Error rate for panic event detection in PETS 2009 dataset. 76
4.14 Error rate for panic event detection in UMN dataset. 77
4.15 Summary of results. 77
4.16 PETS 2009 dataset experimental result compared with
other approaches.
79
4.17 UMN dataset experimental result compared with other
approaches.
79
xii
LIST OF FIGURES
FIGURE NO. TITLE PAGE
2.1 A bar chart which shows number of papers related to
human or crowd behavior analysis published each year.
10
2.2 Types of approach in crowd analysis. 11
3.1 General flow of the method proposed. 30
3.2 Groups of directions. 38
3.3 Example to find inward and outward vectors. 38
3.4 Two cropped objects in the scene with their respective
centroid.
46
3.5 A scene that is being divided into nine partitions. 46
3.6 The crowd formation state machine. 47
3.7 Three different situations when there are 2 objects
moving in the same region in different directions.
47
3.8 The crowd splitting state machine. 49
3.9 The evacuation state machine. 51
3.10 Flow chart of the overall proposed system. 53
4.1 Scenes in UMN datasets. 58
4.2 Eight different views from PETS 2009 dataset. 58
4.3 Location of the 8 cameras installed. 61
4.4 Simple graphical user interface. 63
4.5 A frame loaded into the interface. 63
4.6 The result of a selected frame. 63
xiii
4.7 The examples of results obtained. 68
4.8 Poor lighting in (a) indoor and (b) outdoor scene. 81
4.9 Strong lighting causing solid shadows and unusual
brightness.
81
4.10 An example of a scene with occlusion. 81
xiv
LIST OF SYMBOLS
𝑨 - Image
𝐴𝑎 - Average acceleration
Ar - Range of average acceleration
Atp - Total area of objects detected in the partition
𝑩 - Structuring element
𝑩𝒔 - Symmetric of 𝐵 defined by Eq. 3.2 below.
b - Number of magnitude of optical flow vectors found in an object
𝐶𝑎 - Acceleration constant for user to control its sensitivity
𝐶𝑐 - Crowd constant
𝐶𝑣 - Average velocity constant for user to control its sensitivity
𝐶𝑤𝑟 - Walk/run constant set by user to control its sensitivity
𝑬 - Euclidean space
ℎ - Height of the image
𝐿𝑎 - Local average vector length
Ldmv - Direction with majority vectors
𝐿𝑗 - Magnitude of the optical flow vectors found in an object
𝐿𝑚𝑎𝑥 - Longest vector in found in an object
Loutward - Ratio of the outward vectors in an object
m - Number of outward vectors
n - Number of optical flow vectors found in a frame