Geoffrey Samuel PhD Researcher Intelligent Systems and Robotics Research Group (ISR) Creative Technologies University of Portsmouth.

Comparison of complex background

subtraction algorithms using a fixed camera

Geoffrey Samuel

PhD Researcher

Intelligent Systems and Robotics Research Group (ISR)

Creative Technologies

University of Portsmouth

Geoffrey Samuel www.GeoffSamuel.com

Comparison of complex background subtraction algorithms using a fixed camera

Intro

Background subtraction is a important and vital step for computers to understand and interpreter a real-world scene

It allows a computer to ignore a background so to concentrate on a foreground object



Hypothesis

Each background subtraction algorithm will have its advantages and disadvantages, and that looking and comparing these with a real-world situation, it would be possible to pick one algorithm or a method of combining algorithms to produce a algorithm capable of balancing speed with quality.



The Goal

Test and evaluate the quality and speed of existing background subtraction algorithms on a complex background with different everyday motions, and to compare the results with those of the extracted “Ground Truth”



Complex Background

Static Background:-Background does not contain any secondary “unwanted” motion. Controlled environment.

Complex Background:-Background contains secondary “unwanted” motion such as the winds effect on trees or blinds.Real-world data.



Synthetic Test Data

Advantages:• Automatically got the “Ground Truth”.• More control over each test clip.

Disadvantages:• Manual frame by frame “Ground Truth”

extraction.• Added artefacts from the Chroma keying

and compositing.



The Experiment

To Create a set of synthetic data with the “Ground Truth”

To test different motions with each background subtraction algorithm

To Compare the results of each algorithm with that of the “Ground Truth”



The Motions 7 everyday motions were chosen:

DrinkingJoggingPicking up walletScratching headSitting downStanding upWalking

Each motion started on the left of the screen and concluded on the right.


Comparison of complex background subtraction algorithms using a fixed cameraCreating the test scenarios

Green Screen

Back Ground

Green Screen with actor

Final Composite “Ground Truth”



Back Plate Difference

│framei – backplate│>Ts

The Algorithms

50



Frame Difference

│framei – framei-1│>Ts

The Algorithms

50



Approximate median

(x = ( framei - framei-1 – framei-2 . . .framei-n ) >

Ts )

→ {background += 1}→ {background -= 1}

The Algorithms



Mixture of Gaussians

frame(it = μ) = Σi=1 ωi,t

.ț(μ,o)

The Algorithms

k



Measuring the Quality

Compare the Per-Pixel value of

each frame with the “Ground Truth”

(0,0) (768,0)

(768,576)(0,576)

(0,0) (768,0)

(768,576)(0,576)



Results - Quality

Test Motions

Backplate Difference Frame Difference Approximate Median Mixture of Gaussian

% of image # of pixels % of image # of pixels % of image # of pixels % of image # of pixels

Drinking 90.78% 401577.3019 82.12% 363282.5031 89.52% 396024.7107 83.78% 370625.2327

Jogging 88.24% 390349.3529 88.88% 393194.9412 92.14% 407602.3824 88.20% 390146.7941

Picking up Wallet 91.26% 403717.114 88.22% 390256.9035 83.40% 368940.5088 90.19% 398979.9737

Scratch head 88.18% 390065.7255 84.87% 375422.2549 90.56% 400599.9216 86.15% 381117.049

Sitting down 88.51% 391528.6796 80.07% 354204.932 82.28% 363994.2039 81.68% 361327.3981

Standing up 89.40% 395491.6311 83.82% 370787.165 80.99% 358290.4563 83.78% 370631.6893

Walking 88.47% 391373.5094 89.81% 397309.3396 94.22% 416820.1321 90.01% 398195.3396

Most correct pixels Most incorrect pixels



Results - Quality

Drinking Jogging Picking up Wallet

Scratch head Sitting down Standing up Walking70.00%

75.00%

80.00%

85.00%

90.00%

95.00%

100.00%

Percent of correctly identified pixels

Backplate Difference

Frame Difference

Approximate Median

Mixture of Gaussian

Test Motions

Ave

rag

e P

erce

nt

of

corr

ectl

y id

enti

fied

pix

els

per

fra

me



Results - Speed

Test MotionsBackplate Difference(Average of 100 times)

Frame Difference(Average of 100 times)

Approximate Median(Average of 100 times) Mixture of Gaussian

Drinking 0.0507 0.0004 0.3301 10.6954

Jogging 0.0507 0.0025 0.0691 10.8219

Picking up Wallet 0.0492 0.0819 0.0730 12.2895

Scratch head 0.0450 0.0850 0.0718 10.6132

Sitting down 0.0420 0.0692 0.0662 10.8503

Standing up 0.0416 0.0747 0.0529 12.7196

Walking 0.0319 0.0129 0.0541 10.5202

“Fastest” Algorithm “Slowest “Algorithm



Results - Speed


Scratch head Sitting down Standing up Walking0.0000

2.0000

4.0000

6.0000

8.0000

10.0000

12.0000

14.0000

Average time to process per frame


Frame Difference

Approximate Median

Mixture of Gaussian

Test Motions

Ave

rag

e p

roce

ssin

g t

ime

per

fra

me

in S

eco

nd

s (r

un

100

tim

es)



Results - Speed


Scratch head Sitting down Standing up Walking0.0000

0.0500

0.1000

0.1500

0.2000

0.2500

0.3000

0.3500

Average time to process per frame


Frame Difference

Approximate Median

Test Motions

Ave

rag

e p

roce

ssin

g t

ime

per

fra

me

in S

eco

nd

s (r

un

100

tim

es)

...now ignoring the Mixture of Gaussian speed results



Conclusion

Backplate difference was the fastest and produce the highest results in 4 out of 7 tests.

Frame difference was the ONLY algorithm to correctly remove the complex background, but couldn't correctly identify the foreground element.



Conclusion

Frame Difference :-Correctly Removed Complex BackgroundIncorrectly Removed inside of Subject

Backplate Difference :-Correctly Identified SubjectIncorrectly kept Complex Background



Taking it furtherA new method that incorporated both the

speed of updating to remove the

background and yet the knowledge of the

background to properly extract the wanted

foreground element.

Theory Framework idea:

Frame Difference Backplate Difference

ƒComplex background removed



Where can this lead?

Application of this technology could be used in:

GamesSurveillanceMesh reconstruction and silhouette

extractionVarious computer vision tasks



Any Questions?



Acknowledgments

UK Engineering and Physical Science Research Council

Seth Benton for his Matlab code



Thank you for your time

[email protected]

www.GeoffSamuel.com

Geoffrey Samuel PhD Researcher Intelligent Systems and Robotics Research Group (ISR) Creative Technologies University of Portsmouth.

Documents

fixed camera hypothesis

extracted ground truth

frame ground truth extraction

manual frame

plate difference frame

times frame differen

ground green screen

different motions