Page 1
Introduction Analysis Methods Tracking Tools Conclusion
Hopalong CasualtyCapabilities and Limitations of Visual Surveillance
Ingo Lutkebohle
Computational Perception LabApplied Computer Science Group
Bielefeld University
27. Dezember 2005
Ingo Lutkebohle Hopalong Casualty 1
Page 2
Introduction Analysis Methods Tracking Tools Conclusion Motivation and Overview Problem Sketch Surveillance
Visual Motion Analysis
Goal: Compact description of motion.
Various levels:
body configuration
motion path
“operate on block”
Application Areas
Human-Computer Interaction
Games (e.g., PS2 EyeToy)
Motion Capture (for movies)
Surveillance
Ingo Lutkebohle Hopalong Casualty 2
Page 3
Introduction Analysis Methods Tracking Tools Conclusion Motivation and Overview Problem Sketch Surveillance
Contents of the talk
1 IntroductionMotivation and OverviewProblem SketchSurveillance
2 Analysis MethodsLocating Humans
3 TrackingInterest PointsResultsAnalysis
4 ToolsSystems
5 Conclusion
Ingo Lutkebohle Hopalong Casualty 3
Page 4
Introduction Analysis Methods Tracking Tools Conclusion Motivation and Overview Problem Sketch Surveillance
Our scenario
Ingo Lutkebohle Hopalong Casualty 4
Page 5
Introduction Analysis Methods Tracking Tools Conclusion Motivation and Overview Problem Sketch Surveillance
Why this is difficult
Ambiguity Low Resolution Occlusion
Ingo Lutkebohle Hopalong Casualty 5
Page 6
Introduction Analysis Methods Tracking Tools Conclusion Motivation and Overview Problem Sketch Surveillance
The Roadrunner problem
when you see it, it’s too late already
Appearance is not enough
1 Take visual experience
2 Add world knowledge
3 Predict activity
Ingo Lutkebohle Hopalong Casualty 6
Page 7
Introduction Analysis Methods Tracking Tools Conclusion Motivation and Overview Problem Sketch Surveillance
Human Visual Analysis
model-based vision
resolves visual ambiguity
learn from visual and
motor experience
Ingo Lutkebohle Hopalong Casualty 7
Page 8
Introduction Analysis Methods Tracking Tools Conclusion Motivation and Overview Problem Sketch Surveillance
Human Visual Analysis
model-based vision
resolves visual ambiguity
learn from visual and
motor experience
Ingo Lutkebohle Hopalong Casualty 7
Page 9
Introduction Analysis Methods Tracking Tools Conclusion Motivation and Overview Problem Sketch Surveillance
Human Visual Analysis
model-based vision
resolves visual ambiguity
learn from visual and
motor experience
Ingo Lutkebohle Hopalong Casualty 7
Page 10
Introduction Analysis Methods Tracking Tools Conclusion Motivation and Overview Problem Sketch Surveillance
Human Visual Analysis
model-based vision
resolves visual ambiguity
learn from visual and
motor experience
Ingo Lutkebohle Hopalong Casualty 7
Page 11
Introduction Analysis Methods Tracking Tools Conclusion Motivation and Overview Problem Sketch Surveillance
Surveillance Applications
Restricted Areas
Little activity
Presence detection
Use cases:
Alarm triggerForensic use
needs storage for weeks
Public Areas
Continuous activity
Separation, classification
use cases
deterrentinvestigative
needs storage for days
Ingo Lutkebohle Hopalong Casualty 8
Page 12
Introduction Analysis Methods Tracking Tools Conclusion Motivation and Overview Problem Sketch Surveillance
Surveillance Specifics
Conditions
low resolution
low frame rate
long stretches of nothing going on
Goals
Categorize behaviour
Levels1 regular vs. irregular2 run - fight - chase
Ingo Lutkebohle Hopalong Casualty 9
Page 13
Introduction Analysis Methods Tracking Tools Conclusion Locating Humans
Task Sketch
Computer View
image: block of pixels (numbers)
everything the same
Goal
Teach a computer to detect relevant image parts.
Interpret it
Ingo Lutkebohle Hopalong Casualty 10
Page 14
Introduction Analysis Methods Tracking Tools Conclusion Locating Humans
First Approach: Motion Detection
Look for large enough changes from one frame to the next.
Pro
easy and fast
gets rid of static parts
Cons
purely intensity/color→ homogenous parts acquire holes
overlaps create ambiguity
Ingo Lutkebohle Hopalong Casualty 11
Page 15
Introduction Analysis Methods Tracking Tools Conclusion Locating Humans
First Approach: Motion Detection
Look for large enough changes from one frame to the next.
Pro
easy and fast
gets rid of static parts
Cons
purely intensity/color→ homogenous parts acquire holes
overlaps create ambiguity
Ingo Lutkebohle Hopalong Casualty 11
Page 16
Introduction Analysis Methods Tracking Tools Conclusion Locating Humans
Prevent holes: Learn how background looks like
Reference Image
Input Image
Result Image
Gotcha
Ingo Lutkebohle Hopalong Casualty 12
Page 17
Introduction Analysis Methods Tracking Tools Conclusion Locating Humans
Prevent holes: Learn how background looks like
Reference Image
Input Image
Result Image
Gotcha
Ingo Lutkebohle Hopalong Casualty 12
Page 18
Introduction Analysis Methods Tracking Tools Conclusion Interest Points Results Analysis
Tracking to resolve ambiguities and overlap
Tracking Procedure
1 First frame: Find interest points
2 Compute unique description3 Subsequent frames: Rediscover by
similarityproximity to expected location
Ingo Lutkebohle Hopalong Casualty 13
Page 19
Introduction Analysis Methods Tracking Tools Conclusion Interest Points Results Analysis
Similarity: Color
color distribution
can focus on hands & face
large variation→ silhouette as constraint
rediscover by proximity→ not robust
Ingo Lutkebohle Hopalong Casualty 14
Page 20
Introduction Analysis Methods Tracking Tools Conclusion Interest Points Results Analysis
Similarity: Color
color distribution
can focus on hands & face
large variation→ silhouette as constraint
rediscover by proximity→ not robust
Ingo Lutkebohle Hopalong Casualty 14
Page 21
Introduction Analysis Methods Tracking Tools Conclusion Interest Points Results Analysis
Similarity: Appearance
“looks like” (face image)
Look for best match
Generalization:Collection of generic patches
Very (sometimes too) specific
Problems with rotation
Ingo Lutkebohle Hopalong Casualty 15
Page 22
Introduction Analysis Methods Tracking Tools Conclusion Interest Points Results Analysis
Similarity: Model prediction
Estimate possible positions
Look for best match
How to start?
Large views only
Ingo Lutkebohle Hopalong Casualty 16
Page 23
Introduction Analysis Methods Tracking Tools Conclusion Interest Points Results Analysis
Tracking Results
Associated Postures Trajectories Summaries
No intrinsic meaning
Ambiguous
Ingo Lutkebohle Hopalong Casualty 17
Page 24
Introduction Analysis Methods Tracking Tools Conclusion Interest Points Results Analysis
Machine Learning Approach
General Approach
1 Gather examples for training
2 Categorize as desired
3 Compare new images to examples
4 Assign most likely category
Challenges
Appearance 6= function
Duration varies
Context matters
What is a category anyway?
Ingo Lutkebohle Hopalong Casualty 18
Page 25
Introduction Analysis Methods Tracking Tools Conclusion Interest Points Results Analysis
Posture
Idea: Some postures are unique
Find these key postures
Self-occlusion problematic
Context big part of interpretation
Ingo Lutkebohle Hopalong Casualty 19
Page 26
Introduction Analysis Methods Tracking Tools Conclusion Interest Points Results Analysis
Motion History Images
Inspired by human peripheral vision
Compare to example images
Only for large motions
Requires sufficient resolution
View-angle specific
Ingo Lutkebohle Hopalong Casualty 20
Page 27
Introduction Analysis Methods Tracking Tools Conclusion Interest Points Results Analysis
Trajectories
Position (center of mass)
Velocity, duration
Low resolution OK
Not much information left
Ingo Lutkebohle Hopalong Casualty 21
Page 28
Introduction Analysis Methods Tracking Tools Conclusion Interest Points Results Analysis
Task Scripts: Recognizing abstract activities
Event Triples
Capture context
Fixed sample size
Event types selected manually
EventVocabulary
{ E, S, M, H }
ExampleSequence
{EMHMS..}
Eventn-Grams
{EMH, MHM, HMS, ... }
n-GramHistograms
Ingo Lutkebohle Hopalong Casualty 22
Page 29
Introduction Analysis Methods Tracking Tools Conclusion Interest Points Results Analysis
Tracking Summary
State of the Art
Tracking associates objects over time
Fails relatively often (even in humans)
Robust approaches yield little information
No clear decision between relevant and irrelevant
Results
Hard problem for recognition
State-of-the-art progresses fast
Sequences not learned, yet
Ingo Lutkebohle Hopalong Casualty 23
Page 30
Introduction Analysis Methods Tracking Tools Conclusion Systems
System Summary
Digitize Locate Segment
TrackSummarizeClassifywalking
Scope
For more details on camera technology, see “Hacking CCTV”,right after this talk.
Ingo Lutkebohle Hopalong Casualty 24
Page 31
Introduction Analysis Methods Tracking Tools Conclusion Systems
Cautious note on implementations
production software not available→ use research implementations, where available
quality, robustness and speed vary
often very particular about input data
integration of approaches is difficult
Ingo Lutkebohle Hopalong Casualty 25
Page 32
Introduction Analysis Methods Tracking Tools Conclusion Systems
OpenCV
Open Source Computer Vision Library
Intel Corporation and contributors
Comprehensive algorithm supports
Pretty fast, can use Intel Performance Primitives (x86)
Written in ’C’, bindings for Python
Supported on Win32 and Linux
Main drawback: Just a library
http://www.intel.com/technology/computing/opencv/
Ingo Lutkebohle Hopalong Casualty 26
Page 33
Introduction Analysis Methods Tracking Tools Conclusion Systems
iceWing
Open source integrationenvironment for algorithms
Basic algorithms included
Extension via plugins, operatingin a processing chain
FireWire, V4L, AVIs, PNGs, . . .
Plugins in ’C’, C++, Python orMatlab
Various unices and Mac OS X
http://icewing.sf.net/
Ingo Lutkebohle Hopalong Casualty 27
Page 34
Introduction Analysis Methods Tracking Tools Conclusion
Conclusion
Indoor presence detection works
The rest is a world full of edge cases
Current methods are not robust enough for public areas
Human-like results require a lot of human help
The Roadrunner problem will be with us for a while
“I wouldn’t stake my life on this technology and Iwouldn’t pay for it either.”
Ingo Lutkebohle Hopalong Casualty 28
Page 35
Introduction Analysis Methods Tracking Tools Conclusion
Outlook: Where is it going?
Research
Integration
30 pixel man, i.e. coping with bad resolution
Interaction analysis
Congress
Maybe a hands-on workshop? Talk to me afterwards!
Ingo Lutkebohle Hopalong Casualty 29
Page 36
Introduction Analysis Methods Tracking Tools Conclusion
Acknowledgements
Thank you for the attention!
Credits to Frank Lomker (iceWing), Joachim Schmidt (motioncapture), Britta Wrede (experimental data) and Julia Luning(22C3).
Ingo Lutkebohle Hopalong Casualty 30