8/12/2019 Introduction to Acitivity Recognition
1/63
Week 2 Introduction to ActivityRecognition
RICHARD DAVIES COM815ACTIVITY MODELLING & RECOGNITION
8/12/2019 Introduction to Acitivity Recognition
2/63
Week 1 - Summary Activity Recognition
Why is this field expanding?
P e r v a s i v e
T e c h n o
l o g y
Activity Modelling & Recognition
P r o c e s s o r
p o w e r
B a t t e r y S i z e
C o s t
Increased Functionality
Environment
Wearable
8/12/2019 Introduction to Acitivity Recognition
3/63
Coursework 1
Investigative Study Report
Identify new and emerging approach Potential Barriers Summary of results
Topics Health Security Sport Science
8/12/2019 Introduction to Acitivity Recognition
4/63
Coursework 1 - Specification
8/12/2019 Introduction to Acitivity Recognition
5/63
Coursework 1 - FormatPaper Title* (use style: paper titl e )
Subtitle as needed (paper subtitle)
Authors Name/s per 1st Affiliation ( Author )line 1 (of Affiliation ): dept. name of organizationline 2-name of organization, acronyms acceptable
line 3-City, Countryline 4-e-mail address if desired
Authors Name/s per 2nd Affiliation ( Author )line 1 (of Affiliation ): dept. name of organizationline 2-name of organization, acronyms acceptable
line 3-City, Countryline 4-e-mail address if desired
Abstract This electronic document i s a live template andalready defines the components of your paper [title, text, heads,etc.] in its style sheet. *CRI TI CAL: Do Not Use Symbols, SpecialCharacters, or Math in Paper Titl e or Abstract . ( Abstract )
Keywords component; formatting; style; styling; insert (keywords)
I. I NTRODUCTION ( Heading 1 )This template, modified in MS Word 2007 and saved as a
Word 97 -2003 Document for the PC, provides authors withmost of the formatting specifications needed for preparingelectronic versions of their papers. All standard papercomponents have been specified for three reasons: (1) ease ofuse when formatting individual papers, (2) automaticcompliance to electronic requirements that facilitate theconcurrent or later production of electronic products, and (3)conformity of style throughout a conference proceedings.Margins, column widths, line spacing, and type styles are built-in; examples of the type styles are provided throughout thisdocument and are identified in italic type, within parentheses,following the example. Some components, such as multi-leveled equations, graphics, and tables are not prescribed,although the various table text styles are provided. Theformatter will need to create these components, incorporatingthe applicable criteria that follow.
II. EASE OF USE
A. Selecting a Template (Heading 2)First, confirm that you have the correct template for your
paper size. This template has been tailored for output on the A4 paper size. If you are using US letter-sized paper, please closethis file and download the file MSW_USltr_format .
B. Maintaining the Integrity of the SpecificationsThe template is used to format your paper and style the text.
All margins, column widths, line spaces, and text fonts are prescribed; please do not alter them. You may note peculiarities. For example, the head margin in this templatemeasures proportionately more than is customary. This
measurement and others are deliberate, using specificationsthat anticipate your paper as one part of the entire proceedings,
and not as an independent document. Please do not revise anyof the current designations.
III. PREPARE YOUR PAPER BEFORE STYLING Before you begin to format your paper, first write and save
the content as a separate text file. Keep your text and graphic
files separate until after the text has been formatted and styled.Do not use hard tabs, and limit use of hard returns to only onereturn at the end of a paragraph. Do not add any kind of
pagination anywhere in the paper. Do not number text heads-the template will do that for you.
Finally, complete content and organizational editing beforeformatting. Please take note of the following items when
proofreading spelling and grammar:
A. Abbreviations and AcronymsDefine abbreviations and acronyms the first time they are
used in the text, even after they have been defined in theabstract. Abbreviations such as IEEE, SI, MKS, CGS, sc, dc,and rms do not have to be defined. Do not use abbreviations inthe title or heads unless they are unavoidable.
B. Units Use either SI (MKS) or CGS as primary units. (SI units
are encouraged.) English units may be used assecondary units (in parentheses). An exception would
be the use of English units as identifiers in trade, suchas 3.5-inch disk drive.
Avoid combining SI and CGS units, such as current inamperes and magnetic field in oersteds. This often leadsto confusion because equations do not balancedimensionally. If you must use mixed units, clearlystate the units for each quantity that you use in anequation.
Do not mix complete spellings and abbreviations ounits: Wb/m2 or webers per square meter, notwebers/m2. Spell units when they appear in text: ...afew henries, not ...a few H.
Use a zero before decimal points: 0.25, not .25. Usecm3, not cc. (bullet list )
Identify applicable sponsor/s here. If no sponsors, delete this text box( sponsors).
8/12/2019 Introduction to Acitivity Recognition
6/63
Systematic Literature Review
Background
Search Strategy
Selection Criteria
Research Question
Discussion
8/12/2019 Introduction to Acitivity Recognition
7/63
Research Question
Systematic Reviews
Clinical TrialsExperimental Studies
Qualitative Studies
Observational Studies
8/12/2019 Introduction to Acitivity Recognition
8/63
Systematic Research Questions
1. Is strength training effective, ie, dostrengthening interventions increase strengthin people who are suffering the effects ofacute and chronic stroke?
2. Is therapeutic exercise of benefit inimproving activity and increasing societalparticipation for people who would beexpected to consult a physiotherapist?
8/12/2019 Introduction to Acitivity Recognition
9/63
Clinical Trials Questions
1. Does eight hours of stretch per day for threemonths reduce thumb web spacecontractures in neurological conditions?
2. Is the Mapleson C circuit more effective thanthe Laerdal circuit in removing secretions andimproving ventilation and gas exchangeduring manual hyperinflation?
8/12/2019 Introduction to Acitivity Recognition
10/63
Your Research Question
Is there any potential in the use ofaccelerometers to improve dental health?
8/12/2019 Introduction to Acitivity Recognition
11/63
Background
Define the problem Sign, symptoms etc.
Highlight importance of problem Cite research papers or government reports with
statistics of incidence levels. How were things normally managed in the
past. Where is the gap between the old technique
and the new one.
8/12/2019 Introduction to Acitivity Recognition
12/63
Search Strategy Specify your keyword search terms Select sources to be included
INSPEC IET IEEE etc.
Specify search dates 2003 onwards.
automated searches, identify resources to beused (digital libraries and search engines) manual searches, identify the journals and
conferences to be searched
8/12/2019 Introduction to Acitivity Recognition
13/63
Search Strategy
identify any ancillary search procedures, e.g.asking leading researchers or research groups,or accessing their web sites; or checkingreference lists of primary studies
8/12/2019 Introduction to Acitivity Recognition
14/63
Selection Criteria
You should have a bunch of papers. Identify inclusion criteria
Identify exclusion criteria Only use the titles and abstracts
Type of study
Number of participants Type of patients
8/12/2019 Introduction to Acitivity Recognition
15/63
Discussion
A handful of papers Read the full paper
Comparable outcomes Quantitative e.g. accuracy.
Potential
Barriers Results
8/12/2019 Introduction to Acitivity Recognition
16/63
Activity Recognition
Activity recognition is the process whereby an
actors behaviour and their situatedenvironment are monitored and analysed to
infer the undergoing activities
Activity Recognition in Pervasive Intelligent Environments, Luke Chen et al
8/12/2019 Introduction to Acitivity Recognition
17/63
Activity Recognition Tasks
Activity Modelling
Behaviour & Environment Monitoring
Data Processing
Pattern Recognition
8/12/2019 Introduction to Acitivity Recognition
18/63
Behaviour & Environment Monitoring
Behaviour & Environment Monitoring
Vision-based Sensor-based
8/12/2019 Introduction to Acitivity Recognition
19/63
Single Camera - Vision
Object
Camera
8/12/2019 Introduction to Acitivity Recognition
20/63
8/12/2019 Introduction to Acitivity Recognition
21/63
Stereo Vision - Active
Object
Laser
CameraA
CameraB
8/12/2019 Introduction to Acitivity Recognition
22/63
Camera Configurations
Object
Camera
Camera
Inward Looking
8/12/2019 Introduction to Acitivity Recognition
23/63
Camera Configurations
Camera
Outward Looking
Scene
8/12/2019 Introduction to Acitivity Recognition
24/63
Camera Configurations
Moving camera
Scene
Camera
8/12/2019 Introduction to Acitivity Recognition
25/63
Infrared Camera
8/12/2019 Introduction to Acitivity Recognition
26/63
Electromagnetic Spectrum
8/12/2019 Introduction to Acitivity Recognition
27/63
Microsoft Kinect
8/12/2019 Introduction to Acitivity Recognition
28/63
Microsoft Kinect
8/12/2019 Introduction to Acitivity Recognition
29/63
Microsoft Kinect
RGB camera 1280x960 resolution, 12 fps 640x480 resolution, 30 fps
IR emitter, IR depth sensor Image depth
Multi array microphone Direction /location of sound
3 Axis accelerometer Orientation of sensor
8/12/2019 Introduction to Acitivity Recognition
30/63
Kinect SDK
Supports human understanding Skeletal Facial recognition Gesture recognition Voice recognition
8/12/2019 Introduction to Acitivity Recognition
31/63
Healthcare Applications
Stroke Recovery
with Kinect
Activity Detection by PR2
Koppula et al, 2013
Learning Human Activities andObject Affordances from RGB-D
Videos
Microsoft Research
8/12/2019 Introduction to Acitivity Recognition
32/63
Vision-based
Learning Human Activities and Object Affordancesfrom RGB-D Videos
Hema Swetha Koppula, Rudhir Gupta, Ashutosh Saxena
Cornell University
8/12/2019 Introduction to Acitivity Recognition
33/63
MIT Vision Research
Eulerian Video Magnication for Revealing Subtle Changes in the World
8/12/2019 Introduction to Acitivity Recognition
34/63
Radial Artery Application
Eulerian video magnication used to amplify subtle motions of blood vessels
arising from blood ow
8/12/2019 Introduction to Acitivity Recognition
35/63
MIT Invisible Motion Technique
MIT Computer Program RevealsInvisible Motion in Video
8/12/2019 Introduction to Acitivity Recognition
36/63
Core Technology
Human Activity Recognition
ObjectSegmentation
FeatureExtraction &
Representation
ActivityDetection &Classification
Algorithms
Human Activity Recognition Systems
SingleMultiple Person
& Crowd
AbnormalActivity
Recognition
Human Activity Recognition Systems
Surveillance &Security
Healthcare Sports
Low
Medium
High
8/12/2019 Introduction to Acitivity Recognition
37/63
Computer Vision
ObjectDetection
Behaviourtracking
Activityrecognition
High levelactivityevaluation
8/12/2019 Introduction to Acitivity Recognition
38/63
Segmentation or Grouping Tokens
Whatever we are grouping. Pixels, points, surface elements etc.
Which pixels/edges/textures are useful and whichare not? Top down segmentation
Tokens belong together because they lie on the same
object. Bottom up segmentation
Tokens belong together because they are locallycoherent
8/12/2019 Introduction to Acitivity Recognition
39/63
Segmentation or Grouping
Why do these tokens belong together?
It is very difficult to tell whether a pixel (token) lieson a surface by simply looking at the pixel.
8/12/2019 Introduction to Acitivity Recognition
40/63
Gestalt Theory
Gestalt definition 1
A physical, biological, psychological, or symbolic
configuration or pattern of elements so unified as awhole that its properties cannot be derived from asimple summation of its parts .
8/12/2019 Introduction to Acitivity Recognition
41/63
Gestalt Theory
Gestalt definition 2
A perceptual pattern or structure possessingqualities as a whole that cannot be describedmerely as a sum of its parts .
8/12/2019 Introduction to Acitivity Recognition
42/63
Gestalt Theory
Gestalt definition 3
A form or configuration having properties thatcannot be derived by the summation of itscomponent parts .
Gestalt means shape in German.
8/12/2019 Introduction to Acitivity Recognition
43/63
Gestalt Theory Gestalt means when parts identified
individually have differentcharacteristics to the whole (Gestaltmeans "organised whole")
e.g. describing a tree - it's parts aretrunk, branches, leaves, perhapsblossoms or fruitBut when you look at an entire tree, you
are not conscious of the parts, you areaware of the overall object - the tree. Parts are of secondary importance even
though they can be clearly seen.
8/12/2019 Introduction to Acitivity Recognition
44/63
Which of these two pictures is easierto remember?
8/12/2019 Introduction to Acitivity Recognition
45/63
Principles of Gestalt Perception
Identify the figurefrom the background
This text on the slideis the figure and thegrey space is thebackground.
Rubens vase
Figure/ground
8/12/2019 Introduction to Acitivity Recognition
46/63
Principles of Gestalt Perception
Proximity
1 2 431+2 = one group3+4 = another group
3 groups of dots in a row
Proximity or contiguity states as things arecloser together will be seen as belongingtogether. Last example is one group.
8/12/2019 Introduction to Acitivity Recognition
47/63
Six Principles of Gestalt Perception
Similarity
Similarity means there is a tendency to see groupswhich have the same characteristics so in this example,
there are three groups of black squares and threegroups of white squares arranged in lines.
The principle of similarity states that things whichshare visual characteristics such as:
Shape Size Color Texture Value Orientation
will be seen as belonging together.
Anomaly
8/12/2019 Introduction to Acitivity Recognition
48/63
Six Principles of Gestalt Perception
Common fateSuppose both principles of proximity and similarity arein place - then a movement takes place - the dots beginto move down the page.
They appear to change grouping.
8/12/2019 Introduction to Acitivity Recognition
49/63
Six Principles of Gestalt Perception
Good continuation
Seeing things as whole lines (sequential) is clearlyimportant. But 'being in wholes means' that fewinterruptions change the reading of the wholelines.A to O and Oto D are two lines. Similarly,C to O and O to B are two lines.
The principle of continuity predicts the preference forcontinuous figures. We perceive the figure as twocrossed lines instead of 4 lines meeting at the center.
C
DA
B
O
8/12/2019 Introduction to Acitivity Recognition
50/63
Six Principles of Gestalt Perception
Closure Related to principle of good continuation, there isa tendency to close simple figures, independent ofcontinuity or similarity.
This results in a effect of filling in missing information
or organising information which is present to make awhole.
In the circle at the top its seen easily. In the other tofigures it's a little more complex.
The second figure can be read as two overlappingrectangles (the gestalt) whereas it can also be seen asthree shapes touching; a square and two otherirregular shapes.
The final shape can be seen as a curve joining three
squares or as three uneven shapes touching.
8/12/2019 Introduction to Acitivity Recognition
51/63
Examples
8/12/2019 Introduction to Acitivity Recognition
52/63
Background subtraction
Naive Approach
Subtract
8/12/2019 Introduction to Acitivity Recognition
53/63
Background subtraction
Provided the objectintensity/colour issufficiently different from
the background. Objects that enter the
scene and stop willcontinue to be detected.
New objects that pass infront of them will bedifficult to detect
8/12/2019 Introduction to Acitivity Recognition
54/63
Background subtraction
If part of the assumedstatic backgroundstarts moving, both
the object and itsnegative ghost (therevealed background)
are detected
8/12/2019 Introduction to Acitivity Recognition
55/63
Background subtraction
Background subtraction is sensitive to anychanges even unimportant ones:
Lighting e.g. sunlight Wind e.g. trees moving.
The camera cannot be moved Vibrations Unwanted behaviour
8/12/2019 Introduction to Acitivity Recognition
56/63
Frame Differencing
Subtract
B(t-1)
Object(t)
8/12/2019 Introduction to Acitivity Recognition
57/63
Frame Differencing
Frame differencing is very quick to adapt tochanges in lighting or camera motion.
Objects that stop are no longer detected. Objects
that start up do not leave behind ghosts. However, frame differencing only detects the
leading and trailing edge of a uniformly coloured
object. As a result very few pixels on the objectare labelled, and it is very hard to detect anobject moving towards or away from the camera
8/12/2019 Introduction to Acitivity Recognition
58/63
Frame Differencing
Object(t) Diff(t-1) Diff(t-3) Diff(t-5) Diff(t-9) Diff(t-15)
We now get a more complete silhouette
But we have two copies, a ghost. One where the object is now
One where the object used to be 15 frames ago.
8/12/2019 Introduction to Acitivity Recognition
59/63
3 Frame Differencing
ANDDiff(t-15)
Diff(t)
Diff(t+15)
Choice of good frame-rate for three-frame differencing depends on the sizeand speed of the object.
8/12/2019 Introduction to Acitivity Recognition
60/63
Background Subtraction
Adaptive Background Estimation - Parking Lot
8/12/2019 Introduction to Acitivity Recognition
61/63
Colour segmentation
Computer Vision - Colour Segmentation
8/12/2019 Introduction to Acitivity Recognition
62/63
Vision
Open discussion on ethics
8/12/2019 Introduction to Acitivity Recognition
63/63
Computer Vision
A Review on Video-Based Human ActivityRecognition