Video Processing and Analysis for Surveillance Applications

7/30/2019 Video Processing and Analysis for Surveillance Applications

http://slidepdf.com/reader/full/video-processing-and-analysis-for-surveillance-applications 1/4

2009 International Symposium on Intelligent Signal Processing and Communication Systems (ISPACS 2009) December 7-9, 2009

WP1-D-5

978-1-4244-5016-9/09/$25.00 c2009 IEEE – 607 –

Video Processing and Analysis for Surveillance Applications

Supavadee Aramvith*, Suree Pumrin*, Thanarat Chalidabhongse**, Supakorn Siddhichai****Department of Electrical Engineering,Faculty of Engineering Chulalongkorn University, Thailand

Email: [email protected] , [email protected]

**Faculty of Information Technology, King Mongkut’s Institute of Technology Ladkrabang, ThailandE-mail: [email protected]

***National Electronics and Computer Technology Center (NECTEC), ThailandE-mail: [email protected]

Abstract — Presently researches in networked surveillance

system grow continuously and substantially. One reason is

because of the insecurity incidents such as terrorism acts in

Thailand and many countries around the world. This results in

the need of intelligent surveillance and monitoring system

consisting of real-time image capture, transmission, processing,

and surveillance information understanding. This information

will be vital to people safety and indeed national security. Video

Information Processing and Analysis for Surveillance System is

a pilot project to realize such intelligent surveillance system

developed by Thai researchers. The developed system consists of

6 parts, namely, super-resolution image reconstruction, suspectdetection, suspect tracking, suspect appearance extraction,

suspect activity analysis and level of awareness decision. The

output information from the proposed system is suspect features,

suspect activity pattern and direction, and level of suspicious

incident. This would be very useful for the users to further

analyze this information to use in practical surveillance system.

I. I NTRODUCTION Nowadays researches in networked surveillance experiencecontinuous growth. Parts of the reason are from the instabilityincidents happened all around the world. Thus, there is an urgent

need of such intelligent monitoring or surveillance system which

consists of real time capturing of images, transmission, processing,and understanding the information related to those monitoring. This

information will be utilized later on for the nation’s security. Thedevelopment of such system requires the knowledge of various

disciplines such as signals, image processing, computer vision,communication and networking, pattern recognition, sensors and

fusion, etc. Nevertheless the knowledge on signal and image processing are considered as the important components of

surveillance systems. The integration of such knowledge will enablethe users to analyze the information extracted from several videocameras for further usage and processing.

The urge to develop intelligent surveillance systems partly comes

from the limitation of human perception system. Even human has asound capability in analyzing images combined with experiences andcan certainly indicates whether that incident is significant or not.But physical condition of each human varies and may limit the

effectiveness in the analysis process. The proposed video processing

and analysis of surveillance system will play very important role toaid the deficiency of human’s role in such system.

Many of the video surveillance system in the Thailand’s market

are expensive and yet lack the capability of intelligent system suchas no image analysis function. This makes the system lack theability to send warning signal to pre-alarm before the incidentshappen. Also it is difficult and might take a long time for the officer

to locate the suspects in the video after the incidents did happen.

The problem may get worse in the larger scale surveillance system.The researches in intelligent video surveillance system have long

been conducted in several countries in the world. In United States,the project called Video Surveillance and Monitoring (VSAM) [1]

studied and solved several issues in video surveillance such asautomatic camera calibration, multicamera systems [2-4]. There are

also several other works developed by leading universities andlaboratories as can be seen in [5]. The next generation videosurveillance system will not only solve the issues of detection and

tracking but also solve the issue of event searching [6]. Other references in development of sophisticated video surveillance systemcan be found in [7-10]. In Thailand, there are several researches that

lead to the development of such intelligent video surveillance such as

face and hand tracking [11], person detection and tracking usingcooperative cameras [12, 24], video segmentation [13-15],

development of multipoint video monitoring system [16], anddevelopment of intruder tracking video camera [17]. The proposed

and developed video processing and analysis for surveillance system

presents in this paper is the first pilot project in Thailand supported by the Thailand Research Fund.

The paper is organized as follows. The overall system is

described in Section II. Each sub-system and experimental results,namely super-resolution image reconstruction, suspect detection andtracking, suspect appearance extraction, suspect activity analysis,

level of awareness decision making and working system prototype

are described in Section III. The conclusion is presented in SectionIV.

II. PROPOSED VIDEO I NFORMATION PROCESSING AND

A NALYSIS SYSTEM

Fig. 1 Overall picture of intelligent video surveillance system

The diagram of the whole picture of video surveillance system is

shown in Fig. 1. The input video comes from either wired videocameras or wireless video cameras connected to the network. Thevideo images will be transmitted to video server at the ground station

to store in the video database. Each video image will then be sent toour proposed system for processing and analysis. Various importantinformation are then extracted from the video images such as date,




– 608 –

time, camera coordinate, suspect features, and level of situationawareness. This information could be further utilized for ordering theimportance of each video image and to keep as metadata for video

indexing in the video database. The graphic user interface will

display the level of situation awareness of each video image suchthat the users can make a query in real-time.

Fig. 2 Video information processing and analysis system

The process of our proposed video information processing and

analysis can be described as follows. The input to the system is the

uncompressed video images. The suspect detection sub-system willdetect the human portions of the video images by segmenting thehuman foreground portion from the background scene. The output is

in the form of group of suspect mask. The suspect mask will be aninput to suspect appearance extraction, suspect tracking, and suspectactivity analysis sub-systems. Prior to the analysis of suspect

appearance, the segmented pixels of human portion will be enhanced

the resolution by the super-resolution image reconstruction sub-

system. The enhanced resolution image is further analyzed thefeatures such as the height, skin color, dress color, other suspiciouscharacteristics such as wearing sunglasses, cap, and backpack,velocity, and acceleration. In addition, the suspect mask is processed

by suspect tracking sub-system such that the activity information in

the scene is obtained and uses as an input to suspect activity analysissub-system to find the pattern and directional information of a particular suspect. Finally, the output of the system which are the

suspect’s features and pattern and directional information of suspect’s activities will be kept as raw data and XML format. The

raw data will be used as an input to level of situation awareness sub-system where the final analysis is in the form of high, medium, low

alarm and no alarm of level of situation awareness. For data kept inthe form of XML format, it is used for both stored format andinterchange with other systems. The proposed system can process

data according to the user’s query on-demand or the scheduleanalysis. The system diagram of the proposed system is shown in

Fig. 2. Each sub-system will be described in details as follows.

A. Image Super Resolution Reconstruction

This sub-system plays more and more important roles today as theinput video images are usually from low resolution video cameras,

which is typically used in CCTV video surveillance system. To be

able to enhance the quality of low resolution images, image super-resolution reconstruction method is needed.

To reconstruct high resolution image (HR) from low resolutionimage (LR) is a method to increase the spatial resolution of image by

the enhancement factor of r . For example, input original image of size 60x80 pixels is enhanced by factor of 4 will result to higher resolution image of size 240x320 pixels. The general methods used

for image super-resolution are interpolation function such as bilinear

or bicubic interpolation. However, those methods use theinformation from only 1 image to reproduce higher resolution inwhich technically is insufficient. Therefore, other methods that useseveral of lower resolution images to reproduce the higher resolution

images are the popular methods. In this system, we apply non-

uniform interpolation method with iterative deterministic regularizedmethod [18] which can solve the problem of undetermined case andcan handle automatic registration. We use consecutive 8 low

resolution video images to reproduce a higher resolution image. Thesystem will process in several iterations until achieving the bestresults. Fig. 3 shows an example of resulting image.

Fig. 3 Example result of super-resolution image reconstruction

B. Suspect DetectionTo analyze an image, the techniques of segmentation are needed

such that one could segment foreground or interested objects, i.e.,

human, from the background scene. In this work, we chose background subtraction method as it has low complexity and yeteffective thus is suitable to use in real time system. We implementedadaptive parametric statistical background subtraction method [13].

We further removed noises from the resulting segmented images byusing morphological operators such as dilation and erosion. The

results of background subtraction are shown in Fig 4. After that eachsegmented regions will be classified into each group or objects by

using connected component. The objects will be covered by squared

boundary, as seen in Fig. 5.

Fig. 4 Example result of background subtraction

C. Suspect Tracking In this sub-system, after persons are identified, the system will

check whether this is the new person entering the scene or not. Each person entering the scene will be registered into a system and being

labeled a person number, as shown in Fig. 5. Then the system will

track each person’s movement in the scene by using the method of window search. Current system can handle up to 256 persons in ascene. For each person, the system will store the features explainedin the next sub-section. In current system implementation, there still

exists a problem of tracking in terms of occlusion among several

persons where split and merge of two or more persons occurred in ascene. Nevertheless, we have designed an algorithm to handle suchcases.




– 609 –

Fig. 5 Example results of person registration and tracking .

D. Suspect Appearance Analysis

As already mentioned, the features of each person will beextracted from this sub-system as explained below.

• Height - The computation of each person’s height can be done by measuring reference distance of each person to video camera withthe assumption that the location where video camera installed is

fixed and known in advance. The measured distance will be used to

calculate the height with reference to world and image coordinates ineach video frame and we obtained average height. Neverthelessheight calculation is very sensitive to several factors such as location

of each person in different viewing angle and postures of each person in a scene.

• Velocity and Acceleration - Movement speed of each person in

a scene can be an indicator to signify abnormal activities. Thus ineach video frame, the velocity can be computed the first derivative of the person’s position over time. Correspondingly, the acceleration

can be computed from the second derivative of the person’s position

over time.

• Body Part Segmentation of a person - To track activity of a

person, we need to track activities of each part of the body such as

head, arm, body, leg, etc. In this work, we apply cardboard model[19] to model position and size of human body. The proportion of each component of human body is shown in Fig. 6 where 50% of the

height denotes upper body and the lower 50% of the height denoteslower body portion. The upper 20% portion of upper body denotes

head and the rest 30% is the main body. The lower body will bedivided into equal portion in width for left and right legs.

Fig. 6 Body classification

• Backpack Detection - With the assumption that the person

posture is symmetric when they are standing, walk, and run, thischaracteristic would indicate whether a person holds backpack or

box or not. Thus if a person holds backpack, the symmetrical of that

person posture will change. The algorithm will search for thesymmetrical and asymmetrical areas of that person and will comparethe calculated results with a predefined threshold to decide whether a person holds backpack or not. Note that current algorithm does not

work when a person holds backpack in such a way that thesymmetrical of human body remains the same.

• Sunglasses Detection - To detect whether a person wears

sunglasses, we consider only the face area and the face has to be

frontal or nearly frontal. The face image will be converted fromRGB to grayscale. Then we use circularity analysis to check the

areas in the face that is the most similarity to the sunglasses.

• Skin Color Detection - In this work, we apply skin detectionwith ellipse model. This method constructs parametric skin

distribution model [20] such that the skin color model can beobtained and used in skin detection.

• Dress color detection - From body classification process, wecan locate the middle part of the body and leg areas in which we canlocate areas of shirt and pant/blouse. Finally the color of the dress of

a person can be obtained.

E. Suspect Activity Analysis

Fig. 7 Example results of trajectory paths

To be able to analyze activities, such as pattern and direction, of

each person in a scene is an important indicator to compare thoseactivities with the suspicious ones we would like to search for.There are many algorithms related to recognition of motion pattern

in [21]. In this work, we apply linear regression method [22] to

analyze and predict the movement of each person with an

assumption that the movement of a person in a scene is linear. Theexperiments have been done to find the parameters to indicate and predict the movement in clockwise, counter-clockwise, diagonal,

horizontal, and vertical directions. The system can apply to detect

several cases such as detection of a person crossing predefined linewhere the line can be in any directions. Example of the results isshown in Fig. 7.

F. Level of Awareness Decision Making

(a) (b) (c) (d)

Fig. 8 Level of situation awareness (a) no alarm (b) low alarm(c) medium alarm (d) high alarm

The features of each person which obtained from the above

sections will be used to evaluate whether a particular scene needs

attention or not. The feature vector that is considered in this scope of work can be shown as follows.

1. External features such as luggage, sunglasses, headgear or cap.2. Activity pattern such as the total appearance time of a person

and a period of time that a person stays in a critical area.

There are 4 levels of awareness, namely, no alarm, low alarm,medium alarm, and high alarm. Those alarms will be shown with itscorresponding color in a scene, as shown in Fig 8. The settings of

level of awareness in particular scene can be done by user’s settingof the above feature vector parameters. In special incidents such asevent that involves the handover of object, a person stays still in a

critical area beyond a normal period of time, and a person appears ina scene beyond a normal period of time, the algorithms designed can

also handle the warning of such cases.

III. WORKING SYSTEM PROTOTYPE To demonstrate the performance of the proposed system, we

designed and developed working system prototype which can receivethe input from testing video sequences and real-time input from

video camera. Example of the GUI of the system is shown in Fig. 9.

The system can monitor and alarm for 2 cases.




– 610 –

1. Region of interest area monitoring where users can define thearea of interest in a scene through graphical tools. If personsstay in these areas longer than the time specified by users,

system will generate sound alarm.

2. Line crossing monitoring where users can define the linecrossing path in a scene through graphical tools. If person walkscross the specified line at the specified direction, system willgenerate sound alarm.

The working system prototype is implemented using C/C++

language on Microsoft Visual Studio.NET 2005 with OpenCVlibrary [23]. Current implementation on Pentium D dual core processor can process the 320x240 video frame rate up to 20

frames/second.

Fig. 9 Example of GUI of working prototype system

IV. CONCLUSIONS This paper presents the summary of video information processing

and analysis for surveillance system. The system consists of 6 sub-

systems as mentioned above. We also developed working system prototype to demonstrate the capabilities of our proposed systemwhen applied to practical surveillance system. The informationextracted from video images in this system will be valuable

resources for helping with suspicious incident analysis such as pre-alarm warning, locate suspect in video scenes, and building suspect

information knowledge database for crime investigation. The system proposed still needs continuous refinement, nevertheless, the system

can be used in real-time monitoring and can be easily fine-tuned touse with any traditional CCTV or surveillance systems.

ACKNOWLEDGMENT This research project cannot be succeed without the continuous

and dedicated efforts of research assistants, Mr. Pichai

Amnuaykanjanasin, Mr. Danuwat Sawangpol, Mr. Kosol

Punyasoponlert, Acting SubLt. Chaiyaporn Chatchawarnkitkul, andMr. Nattachai Watcharapinchai. We also would like to thank for thevaluable input from Major Decha Boonyaruk from Armed ForcesSecurity Center, Supreme Command Headquarters of Thailand.

Lastly we would like to thank for the financial support of this projectfrom The Thailand Research Fund.

R EFERENCES [1] R. Collins et. Al, “A system for video surveillance and monitoring,”

VSAM Final Report, Carnegie Mellon Univ., Pittsburgh, PA, Tech.

Rep. CMU-RI-TR-00-12, May 2000.

[2] Haritaoglu, D. Harwood, and L. Davis, “W4: Real time surveillanceof people and their activities,” IEEE Trans. Pattern Anal. Machine

Intell., vol. 22, no. 8, pp. 809-830, Aug. 2000.

[3] T. Horprasert, D. Harwood, and L. Davis, “A statistical approach for

real-time robust background subtraction and shadow detection,” in

Proc. IEEE Frame-Rate Workshop, Kerkyra, Greece, 1999.

[4] G. Stauffer, “Learning patterns of activity using real-time tracking,”

IEEE Trans. Pattern Anal. Machine Intell ., vol. 22, no. 8, pp. 747-757, Aug. 2000.

[5] Remagnino, Jones, Paragios, and Regazzoni, Video Based Surveillance Systems Computer Vision and Distributed Processing .

Norwell, MA: Kluwer, 2002.

[6] VACE: Video Analysis and Content Exploitation [Online]. Available:

http://www.ic-arda.org/InfoExploit/vace/

[7] S. Chaudhuri and D. R. Taur, “High Resolution Slow-MotionSequencing: How to generate a slow-motion sequence from a bit

stream,” IEEE Signal Processing Magazine, pp. 16-24, March 2005.

[8] G. L. Foresti, C. Micheloni, L. Snidaro, P. Remagnino, and T. Ellis,

“Active Video-Based Surveillance System: The low-level image andvideo processing techniques needed for implementation,” IEEE Signal

Processing Magazine, pp. 25-37, March 2005.

[9] A. Hampapur, et. al., “Smart Video Surveillance: Exploring theconcept of multiscale spatiotemporal tracking,” IEEE Signal

Processing Magazine, pp. 38-51, March 2005.

[10] H.-M. Chen, S. Lee, R. M. Rao, M.-A. Slamani, and P. K. Varshney,“Imaging for Concealed Weapon Detection: A tutorial overview of

development in imaging sensors and processing,” IEEE Signal Processing Magazine, pp. 52-61, March 2005.

[11] N. Soontranon, S. Aramvith, and T. H. Chalidabhongse, “Face andHands Localization and Tracking for Sign Language Recognition,” Proceeding of International Symposium on Communication and

Information Technologies (ISCIT 2004) , Sapporo, Japan, October 2004.

[12] P. Amnuaykanchanasin, S. Aramvith, and T.H. Chalidabhongse,

“Face Tracking using Two Cooperative Static and Moving Cameras,” Proceeding of International Conference on Multimedia & Expo

(ICME) 2005, Amsterdam, Netherland, July 2005.

[13] T. Thongkamwitoon, S. Aramvith, and T. H. Chalidabhongse, “An

Adaptive Real-Time Background Substraction and Moving Shadows

Detection,” Proceeding of IEEE International Conference onMultimedia and Expo (ICME 2004), Taipei, Taiwan, June 2004.

[14] T. Thongkamwitoon, S. Aramvith, and T. H. Chalidabhongse, “Non-Linear Learning Control Factor for Statistical Adaptive Background

Subtraction Algorithm,” Proceeding of IEEE International Symposium

on Circuits and Systems (ISCAS) 2005, Kobe, Japan, May 2005.

[15] T. Thongkamwitoon, S. Aramvith, and T. H. Chalidabhongse,

“Adaptive Background Subtraction Algorithm with Auto BrightnessControl for Consumer-type Cameras,” Journal of Broadcast Engineers,

Korea Society of Broadcasting Engineer, June 2005.

[16] P. Waiyamujjamai, S. Toeyon, and S. Aramvith, “Remote Multi-point

Video Monitoring System for Small Sized Factory,” Technical Report, Industrial Projects for Undergraduate Students (IPUS), Thailand

Research Fund,March 2004.

[17] P. Tappayutpijarn, S. Teerapraditpol, and S. Aramvith, “Intruder-

Tracking Video Camera,” ,” Technical Report, Industrial Projects for Undergraduate Students (IPUS), Thailand Research Fund ,March 2006.

[18] Sina Farsiu, M. Dirk Robinson, Michael Elad, and Peyman Milanfar,

“Fast and Robust Multiframe Super Resolution”, IEEE Transactions on

Image Processing , Vol.13, No.10, Octorber 2004.

[19] S. Ju, M. Black, Y. Yacoob. “Cardboard People: A ParameterizedModel of Articulated Image Motion”, International Conference on Face

and Gesture Analysis.1996.

[20] Lee, J. Y. and Yoo, S. I. “An Elliptical Boundary Model for Skin Color

Detection”. International Conference on Imageing Science, Systems,

and Technology. 2002

[21] W. Hu, T.N. Tan, L. Wang, and S.J. Maybank. A survey on

visualsurveillance of object motion and behaviors. SMC-C , 34(3):334– 352, August 2004.

[22] Jaan Kiusalaas. Numerical Methods in Engineering with Python. Cambridge University Press. NY, USA,2005: 124-126.

[23] OpenCV library http://opencvlibrary.sourceforge.net [24] N. Bumrungkiat, S. Aramvith, and T.H. Chalidabhongse, “Continuous

Person Tracking Aross Multiple Cameras using Color and Shape Cues,”

Proceeding of International Workshop on Advanced Image Technology(IWAIT) 2009, Seoul, Korea, January 2009.

Video Processing and Analysis for Surveillance Applications

Documents