Sravanthi Kota Venkata, Ikkjin Ahn, Donghwan Jeon ...cseweb.ucsd.edu/~mbtaylor/vision/SD-VBS.pdf · Sravanthi Kota Venkata, Ikkjin Ahn, Donghwan Jeon, Anshuman Gupta, Chris Louie,

SravanthiKotaVenkata,IkkjinAhn,DonghwanJeon,AnshumanGupta,ChrisLouie,SaturninoGarcia,SergeBelongie,

MichaelB.Taylor

ComputerScienceandEngineeringUniversityofCalifornia,SanDiego

hIp://parallel.ucsd.edu/vision

VisionisanexciMngapplicaMondomainformany‐coreandmulM‐coresystems

•  “Enablingcomputerstosee”willhaveatangibleandimmediateimpactonpeople’slives

•  LimitlessthirstforcomputaMon–  Largerimagesizes–  Moreaccurateanalyses–  Matchandsearchagainstlargerdatabases–  Runthealgorithminreal‐Mme,orevensuper‐real‐Mme

•  Fullofparallelismtoputthoseidlecorestowork–  i.e.driveuserdemandforfuturemulM‐coreprocessorsandkeep

Moore’sLawgoing

•  Enormousprogressincomputervisionresearchoverthelastdecade,andmoretocome–  Someproblemsarenowevenconsidered“solved.”

AFewExamplesofVision’sCurrentandPotenMalImpactonOurLives

•  Auto‐focuscamerasandcell‐phonesthroughfacedetecMon

•  HelpdoctorsperformsurgeryonpaMentsthousandsofmilesaway

•  Allowplanestoflythemselves(e.g.,UAVs)

•  Enable“carsthatcan’tcrash”

•  EnablemachinesthatautomaMcallyeducateourkids(!)

SD‐VBS:TheSanDiegoVisionBenchmarkSuite•  Aim:MakevisionmoreaccessibletomulM‐coreresearchers

throughacomprehensiveandeasytousesuite.

•  9benchmarksin4differentareas.

Benchmark Concentra.onArea

DisparityMap MoMon,TrackingandStereoVision

FeatureTracking MoMon,TrackingandStereoVision

ImageSegmentaMon ImageAnalysis

ScaleInvariantFeatureTransform(SIFT) ImageAnalysis

RobotLocalizaMon ImageUnderstanding

SupportVectorMachines(SVM) ImageUnderstanding

FaceDetecMon ImageUnderstanding

ImageSMtch ImageProcessingandFormaMon

TextureSynthesis ImageProcessingandFormaMon

DesigngoalsofSD‐VBS

•  CleanimplementaMonsforeaseofanalysisandtransformaMonbyresearchersortheirtools

–  “Clean”CandMATLABversions

–  Nodependenceoncustomlibraries

•  ComprehensivecollecMonofrepresentaMvebenchmarks

•  LowcomplexityofporMngandparallelizingtodifferentpla_orms

SD‐VBS:9applicaMons,>25kernels

Sort

SSD

CorrelaMon

AdjacencyMatrix

FilterBanks

Eigensolver

QRFactorizaMon

ParMcleFilter

PhysicalModel

PCA

Sampling

ConjugateMatrix

MatrixOps

AffineTransforms

MatrixInvert

AreaSum

Gradient

FeatureTracking

DisparityMap

LSSolver

SVD

ANMS

GaussianFilter

ImageS.tch

RectFilter

Adaboost

IntegralImage

FaceDetec.on

ImageSegmenta.on

RobotLocaliza.on

TextureSynthesis

SupportVector

Machines

SIFT

InterpolaMon

RichReuseofKernelsinVision

Sort

SSD

CorrelaMon

AdjacencyMatrix

FilterBanks

Eigensolver

QRFactorizaMon

ParMcleFilter

PhysicalModel

PCA

Sampling

ConjugateMatrix

MatrixOps

AffineTransforms

MatrixInvert

AreaSum

Gradient

FeatureTracking

DisparityMap

LSSolver

SVD

ANMS

GaussianFilter

ImageS.tch

RectFilter

Adaboost

IntegralImage

FaceDetec.on

ImageSegmenta.on

RobotLocaliza.on

TextureSynthesis

SupportVector

Machines

SIFT

InterpolaMon

OverviewofTalk

•  IntroducMontoSD‐VBS•  VisionBenchmarksOverview

–  foreach{FeatureTracking,Disparity,ImageSMtch,SIFT,SegmentaMon,SVM,RobotLocalizaMon,

TextureSynthesis}:BriefDemo

AlgorithmDescripMonAnalysisofCharacterisMcsandHotspots

•  Results•  RelatedWork

•  Conclusion

Intro B1 B2 B3 B4 B5 B6 B7 B8 B9 Results RelatedWork Conclusion

FeatureTracking:Overview

•  ProcessoflocaMngmovingobject(s)acrossframes

ApplicaMons:•  CruiseControl•  PedestrianTracking•  InteracMveRobots

MoMonofthetrackedfeatures


FeatureTracking:Algorithm

Noisefilteredimage Horizontaledgeimage VerMcaledgeimage


Highlyparallel:Eachoutputpixelcanbecomputedinparallelwithothers

ImagePreprocessing

GradientEdge

DetecMon


GradientEdge

DetecMon

FeatureExtracMon


Σ

Σ

Σ

Sort

Features:{x,y,strength}EdgeImages AreaSum

ImagePreprocessing


GradientEdge

DetecMon

FeatureExtracMon

FeatureMatching


Frame1Features

Correlated,withVector

ImagePreprocessing

<x,y,strgth>

<0,0,25><…>

<x,y,strgth>

<6,6,21><1,1,12><0,2,22>

SelectNearbyFeatures(e.g.,withinarangeof5x5)thenpickonewithcloseststrength.

Frame2Features


1 2 4Pe

rce

nta

ge

Exe

cu

tio

n T

ime

0

10

20

30

40

50

Feature Tracking

Gradient

GaussianFilter

IntegralImage

Areasum

RelaMveinputsize1X,2X,4X‐(12k,25k,100kpixels)


GradientEdge

DetecMon

FeatureExtracMon

FeatureMatching

ImagePreprocessing

(ImagePreprocessing)

(FeatureExtracMon)

(EdgeDetecMon)

DisparityMap:Overview

•  ComputesrelaMveposiMonsofobjectsinascenecapturedbystereocameras–depth


DisparityMap:Algorithm

1 2 4Pe

rce

nta

ge

Exe

cu

tio

n T

ime

0

10

20

30

40

50

60

70

Disparity Map

Correlation

IntegralImage

Sort

SSD

RelaMveinputsize1X,2X,4X


ImagePre‐Processing

Correlatelesandrightpairs

FindDisparityMap

ImageSMtch:Overview•  CombiningmulMpleimageswithoverlappingview


ImageSMtch:Overview

•  CombiningmulMpleimageswithoverlappingview

ApplicaMons

•  Moviemaking•  Googleearth,GPSapplicaMonsetc


ImageSMtch:Analysis

1 2 4Pe

rce

nta

ge

Exe

cu

tio

n T

ime

0

20

40

60

80

100

Image Stitching

LSSolver

SVD

Convolution

NonKernelWork

RelaMveinputsize1X,2X,4X


Imagepre‐processing

FeatureDetecMon MatchFeatures SMtchImages

(MatchFeatures)(FeatureDetecMon)(Imagepro‐processing)

ScaleInvariantFeatureTransform(SIFT):Overview

DetecMonanddescripMonofrobustlocalimagefeatures

ApplicaMons•  ObjectrecogniMon•  MoMonAnalysis•  GestureRecogniMon


SIFT:Algorithm

ImageProcessing

andPyramid

GeneraMon

FeaturePyramid

KeypointDetecMon

Removelow

contrastkeypoints


DifferenceofGaussiansPyramid–EdgeImages

CornerdetecMon Removelowcontrastkeypoints

ImageSegmentaMon:Overview

ProcessofparMMoninganimageintomeaningfulsegments,typicallyusedtolocateobjectsandboundaries


ImageSegmentaMon:Algorithm

PyramidGeneraMon

and

GradientImage

ExtractconMnuousedges

Extracthighcontrastedges

Markdiscretesegments


UsingtheconMnuousedgeimageandhighcontrastmatrix,wemarksegments

SegmentedimageConMnuousedgeimage

SupportVectorMachine(SVM):Overview

SeparaMonofagivensetof

dataintotwocategorieswithmaximalgeometricmargin.

ApplicaMons•  MachineLearning

•  Neuralnetworks•  DataclassificaMon


SVM:Algorithm


Traintheclassifierbasedonthetraining

data

EvaluatetheclassifierbasedonthetesMng

data

QualifyofdataclassificaMonImproveswithnumberofIteraMonsusedinlearningstage.

TheexecuMonMmedependsonthenumberofdatapointsandnumberofiteraMons.

RefinetheclassifierbasedonthetesMng

data

RobotLocalizaMon:Overview

ProcessofevaluaMngpathbasedonobstaclesandasetofgoals


xxx

xxxx

xxxxxxx

xxxx

xxxxApplicaMons• SpaceexploraMonprograms• Autonomousvehicles/systems• MobileroboMcs

xxx

RobotLocalizaMon:Algorithm

GlobalPosiMonEsMmaMon LocalPosiMonTracking

ProbabilisMcmodelstoesMmateposiMon

1 2 4Pe

rce

nta

ge

Exe

cu

tio

n T

ime

0

10

20

30

40

50

60

70

Robot Localization

ParticleFilter

Sampling

NonKernelWork


(Tracking)

(EsMmaMon)

TextureSynthesis:Overview

ConstrucMnglargeimagesfromsmallimageusingstructuralcontent

ApplicaMons•  Moviemaking•  Graphics•  ComputaMonalPhotography


TextureSynthesis:Algorithm

ImagePre‐processing

TextureAnalysis TextureSynthesis

text Reg. nearReg. Irreg. nearStoch.Pe

rce

nta

ge

Exe

cu

tio

n T

ime

0

10

20

30

40

50

Texture Synthesis

Sampling

ConjugateMatrix

PCA

NonKernelWork


(TextureAnalysis)(TextureAnalysis)(TextureSynthesis)

FaceDetecMon:Overview

DetermineslocaMonsandsizesofhumanfaces

ApplicaMons•  Objecttracking•  DigitalCamera,Facebook•  Contentbasedimageretrieval


Results:ExecuMonTimes

Benchmark 12kpixels 25kpixels 100kpixels

Featuretracking 2.77 5.0 19.4

DisparityMap 0.8 1.8 6.2

ImageSMtch 0.7 10.1 23.4

SIFT 17.4 44.5 131

TextureSynthesis 18.5

ImageSegmentaMon 8.3 9.2 8.4

SVM UptoThousands

SecondsperFrame


TheseMmesarefor1or2framesofcomputaMononasinglecore.

LotsofpotenMalformakinguseofmulM‐coreprocessors!Farenoughawaythatwecanuselotsofcores,butcloseenoughthatit’saIainablewithmorecores.

Results:ParallelisminVisionKernelsParallelismcalculaMon:‐  WeimplementedatransformaMonpassin

LLVMinfrastructuretrackoperandsthroughtheprogramanddeterminethelongestdependencechainthroughmemory,controlandinstrucMondependences.

‐  Parallelism=#Instrs/CriMcalPathLength

Benchmark Kernel Approx.Parallelism

Disparity CorrelaMon 502

IntegralImage 160

Sort 1,700

SSD 1,800

FeatureTracking Gradient 71

GaussianFilter 637

IntegralImage 1,050

AreaSum 425

MatrixInversion 171,000

SIFT SIFT 180

InterpolaMon 502

IntegralImage 16,000

ImageSMtch LSSolver 20,900

SVD 12,300

ConvoluMon 4,500

SVM MatrixOps 1,000

Learning 851

ConjugateMatrix 502

Thebenchmarksshownontherighthavelotsofparallelism!

(Ifyouarecurious,weencourageyoutogoaheadanddownloadourbenchmarksuite!)


RelatedWork:Othervisioncodes

•  Performance‐orientedbenchmarks–  PARSEC:BodyTracking– MediaBench–image/videocompressionalgorithms–  Spec2000facerec

•  Accuracy‐orientedbenchmarks(forvisionresearch)–  BerkeleySegmentaMonDatabaseandBenchmark

–  PEIPA– Muscle

–  ImageCLEF

•  VisionLibraries–  OpenCV‐Highlytunedvisiontoolbox


RelatedWork:IntelOpenCVvsSD‐VBS


OpenCV SD‐VBS

Goal Tuned,Feature‐loadedimplementaMonsforcommercialandacademicvisionapplicaMons

“Pure”versionsforeasyanalysis,transformaMonandparallelizaMoninmulM‐corearchitectureresearch

SourceCode C++ CorMATLAB,yourchoice

Pla\ormspecificop.miza.ons

HighlyTunedforbestperformanceonIntelandsupporMngarchitectures

AcMvelyremovedopMmizaMonsthatincreasecodecomplexity

Codingstyle

Example:FILTER

Highlyflexible;fullofopMonstochangebehaviorofeachfuncMon

Cleancodewithoutfeaturesthatdeteranalysis

2000linesofcode 50linesofcode

204condiMonalstatements NocondiMonalstatements

PointerOperaMons ArrayOperaMons

Easeofanalysis Harder Easier,becausesimplerimplementaMons

Conclusions

•  ComputerVisionisanexciMngdomainwithimmensepotenMal

•  Visionalgorithmsarefullofparallelism,andcanbenefitfromprocessorswithgreaterandgreaterperformance;whichmakethemidealformulM‐core

•  SD‐VBSisacomprehensiveandcleanbenchmarksuiteforvision,wellsuitedformulM‐coreandmany‐coreresearch.

Publicreleasehttp://parallel.ucsd.edu/vision


WewouldliketothankthefollowingVisionresearchers:

•  AndreaVedaldi•  AthanasoisNoulas•  TomMitchell•  JavierPorMlla•  EeroSimoncelli•  JianboShi

ForcontribuMngtheircodeanddatasetstothebenchmarksuite.

SupportedbyNSFCAREERAward0846152

http://parallel.ucsd.edu/vision

*****

Sravanthi Kota Venkata, Ikkjin Ahn, Donghwan Jeon ...cseweb.ucsd.edu/~mbtaylor/vision/SD-VBS.pdf · Sravanthi Kota Venkata, Ikkjin Ahn, Donghwan Jeon, Anshuman Gupta, Chris Louie,

Documents