ShadowDraw - University of Texas at Austinvision.cs.utexas.edu/381V-fall2016/slides/priyadarshi-paper.pdf · • Shadow Creation by Weighting. Image Matching • Obtain Candidate

ShadowDrawReal-TimeUserGuidanceforFreehandDrawing

HarshalPriyadarshi

Demo

ComponentsofShadow-Draw

• InvertedFileStructureforindexing• Databaseofimages• CorrespondingEdgemaps

• Querymethod• Dynamicallyretrievesmatchingimages• Alignsthemtoevolvingdrawing• Weighsthembasedonmatchingscore,toformshadow

• UI• Displaysashadowofweightededgemapsbeneaththeuser’sdrawing

DatabaseCreation

• ImageAcquisition• EdgeExtraction• PatchDescription• Min-hashEncoding

ImageAcquisition

• 30kimagesspanning40categoriesobtainedfrominternet• Scaledtoobtain300x300images

ProblemsHandled:1. DirectSketchimagesarenotabundant2. DiverseBackground– Stillhasgood

edgesyoumightwanttodraw3. Photographerbiastorescue

DatabaseCreation

• ImageAcquisition• EdgeExtraction• PatchDescription• Min-hashEncoding

EdgeExtraction(Step1)

• Computethelocaledgemagnitude(pm)andorientation(po)ateachpixelusingsteerablefilters

Input Output

EdgeExtraction(Step2)• Normalizetheedgemagnitude• Needtodetectlong,coherentedgesevenwhenfaint(i.e.,notsimplyedgeswithstrongmagnitude)

Input Output

EdgeExtraction(Step3)• MessagePassingforlengthestimation

Input Output

Seethedifference

CannyEdgeDetection(NonMaxSuppression)

Credits- KristenGrauman (bothimages)

DatabaseCreation

• ImageAcquisition• EdgeExtraction• PatchDescription• Min-hashEncoding

PatchDescriptors

• LowDimensionalBiCE descriptor• Encodesahistogramofedgepositionsandorientations

• Doneoveroverlapping60x60patcheswith50%overlap• AsmappinganedgeimageEtoincompleteandevolvingdrawing.

• SIFT/DaisyvsBiCE• Formerreliesonrelativestrengthofedgemagnitudestoprovidediscriminability• ThusreducedperformancecomparedtoBiCE onourtask,whereedgestrengthsarenotimportant.

BiCE descriptor(Steps)1. LocalNormalizationofimagepatchgradients• Removevariationinrelativegradientmagnitudes

2. Binningofnormalizedgradients• usingposition,orientation,andlocallinearlength ofimage

3. Binarization ofnormalizedgradienthistogram• Encodesthepresenceofedges

Presence/absenceofEdgeispreservedacrossmatchingpatches,Theremagnitudemightnot.

Step1 NormalizedGradient(g_cap)

OriginalGradient(g)

Step2

OrientationalignedbinningRobustnesstoorientationchanges

InitialBinningw.r.t.justpositionandorientation.

IncreasingDiscriminabilityLongCoherentEdgesvsShorterTexturedEdges

CalculateEdgeLength

Discretizationinto2binsbyweightbasednormalizedgradientsplitting

Alpha,betaà Tunablethresholdingparams

Withgaussian Bluralongx,y,thetadimension

WHY??

Step3(SubsamplingandBinarization)

• Subsampletodiscretesetofvaluesforx,y,thetaandlength

• Value=1(topTpercentofbinswithhighestfrequency)• Value=0(rest)

• FlattentogettheBiCE desciptor

DatabaseCreation

• ImageAcquisition• EdgeExtraction• PatchDescription• Min-hashEncoding

RetrievalandClusteringEfficiency

• Reducedimensionà ImproveClusteringà ImproveRetrieval

PreservesMaximumJaccard Similarity

Minhashing (Iapologizeformyterribleanimation)

Index VectorA VectorB

1 1 0

2 1 0

3 0 1

4 0 0

5 1 0

6 0 0

H(Index)4

6

1

5

3

2

Index VectorA VectorB

1 1 0

2 1 0

3 0 1

4 0 0

5 1 0

6 0 0

Index VectorA VectorB

1 5 3

Whatarethesevectors?

Sketches(n,k)andInvertedIndexing• Toincreaseprecision

Index VectorA VectorB

1 5 3

2 2 1

3 1 1

4 4 6

5 3 2

6 1 6

Khashfunctions

• ToincreaserecallIndex VectorA VectorB

1 5 3

2 2 1

3 1 1

4 4 6

5 3 2

6 1 6

Index VectorA VectorB

1 5 3

2 2 1

3 1 1

4 4 6

5 3 2

6 1 6

Index VectorA VectorB

1 5 3

2 2 1

3 1 1

4 4 6

5 3 2

6 1 6

Index VectorA VectorB

1 5 3

2 2 1

3 1 1

4 4 6

5 3 2

6 1 6

Index VectorA VectorB

1 5 3

2 2 1

3 1 1

4 4 6

5 3 2

6 1 6

NsetsofKhashfunctions

ComponentsofShadow-Draw

• InvertedFileStructureforindexing• Databaseofimages• CorrespondingEdgemaps

• Querymethod• Dynamicallyretrievesmatchingimages• Alignsthemtoevolvingdrawing• Weighsthembasedonmatchingscore,toformshadow

• UI• Displaysashadowofweightededgemapsbeneaththeuser’sdrawing

QuerySteps

• DynamicallyRetrievingMatchingImages

• Aligning MatchingImagestoDrawnSketch

• ShadowCreationbyWeighting

ImageMatching

• Obtain CandidateMatches• Align CandidateMatcheswiththepartiallydrawnsketch• Assignweighttoeachcandidate’sedgeimage• ConstructShadowImage

Votecountforpatchforeachcandidateimage

CandidateMatchFinding

EdgeImageofSketch

BiCE sketchdescriptorforeachpatch

Top100imagesandcorrespondingoffsetforthehighlyvotedoffsetNote:Not100bestpatches

Xoffsetofthecandidatepatchfromtheusersketchpatch

ResultantCandidateFormat(ImageId,patchoffset-xdirection,patchoffset-ydirection)

QuerySteps

• DynamicallyRetrievingMatchingImages

• Aligning MatchingImagestoDrawnSketch

• ShadowCreationbyWeighting

AligningCandidateMatches

QuerySteps

• DynamicallyRetrievingMatchingImages

• Aligning MatchingImagestoDrawnSketch

• ShadowCreationbyWeighting

ImageWeightingWeightImage

ShadowImageConstruction

ShadowImage

EdgeCandidateImage

BlendingWeightImage

GlobalmatchingtermSpatiallyvaryingmatchterm

NormalizationTerm

3variables:• Globalmatchingterm(v)• Spatiallyvaryingmatchingterm(V)• VisibilityEnhancer(alpha) VisibilityEnhancer

ObtainingGlobalMatchingTerm(V)

UserSketch(8orientations)

CandidateImage(8orientations)

PositiveCorrelation

NegativeCorrelation

Imagenotoriented,theedgesitcapturesareoriented

GlobalMatchingSpatialMatching

Averageof5highesthi fromthecandidateset

GaussianBluronthepositivecorrelationimage

VisibilityEnhancer(alpha)

Whyisitavisibilityenhancer?

ExperimentalFindings

Robustnessto________ ????

PoorvsAveragevsGood

WHY?? WHY??

Complexity

• Strengths• Canhelpdrawingstructurallycomplexobjects• Helpspreservetheuniquestyleoftheusers• Isareal-timealgorithm

•Weaknesses• Leadstogoodshadowsonlyiftheinitialusersketchisnotallovertheplace.Otherwisemightconfusetheuser.• Atusslebetween guidanceand freedom.• Sketchingflowbias– Thewaywestartdrawingthesketchmightaffecttheshadowretrieved,andthusleadtoconfusioninitially,iftheuserisnotverycertainofeachdetail.

References

• OriginalPaper• http://vision.cs.utexas.edu/projects/shadowdraw/ShadowDrawSiggraph11.pdf

• SupportingPapers• LongEdgeDetector

• http://grail.cs.washington.edu/projects/gradientshop/demos/gs_paper_TOG_2009.pdf• BiCE descriptor

• http://larryzitnick.org/publication/BiCE_ECCV10.pdf

ThankYou