Using Computer Vision to Test Web Display

Using Computer Using Computer Vision to Test Web Vision to Test Web DisplayDisplayXu LiuXu Liu

liuxu@cs.umd.eduliuxu@cs.umd.edu

Why test web display?Why test web display? Display BugsDisplay Bugs

Bug

“Opera 7.54”

Normal

IE6

More bugsMore bugs

BUG - Opera Normal - Firefox

Normal - IE Normal - Mozilla

Where comes these Where comes these bugs?bugs?

Different parsers on DHTML, CSSDifferent parsers on DHTML, CSS Web designer doesn’t follow W3C Web designer doesn’t follow W3C

standardstandard IE tolerant buggy HTMLIE tolerant buggy HTML Java Script, ActiveX, FlashJava Script, ActiveX, Flash

Which kinds of bugs do Which kinds of bugs do we have? we have?

Text/Image OverlapText/Image Overlap Incorrect blank areaIncorrect blank area Missing Text/ImageMissing Text/Image

Generally they are all layout problems

How do we detect these How do we detect these bugsbugs

Is HTML source helpful?Is HTML source helpful? Yes, but we need a correct parser which is Yes, but we need a correct parser which is

being testedbeing tested

HTML does not have straight forward HTML does not have straight forward relationship with displayrelationship with display

Eyes always tell the truth – directly use Eyes always tell the truth – directly use the image of snapshotthe image of snapshot

Find the outlierFind the outlier

Let them vote:Let them vote: IE, Firefox, Mozilla, Netscapte, Opera, MyIEIE, Firefox, Mozilla, Netscapte, Opera, MyIE

…… Anyone seems distinct from others is probabAnyone seems distinct from others is probab

ly an outlierly an outlier Assumption : Major are correct, Minor are inAssumption : Major are correct, Minor are in

correctcorrect

Basic QuestionBasic Question

How do we compare 2 images? Are they same?How do we compare 2 images? Are they same?Missing

Front and end

These 2 look different but they should be considered the same

Simpler QuestionSimpler Question

How do we compare 2 sequences? How do we compare 2 sequences?

S1: 1 2 3 4 5 6 8 7

S2: 1 2 4 5 5 5 6 7

Direct Compare

|S1-S2|=0+0+1+1+0+1+2+0=5

Dynamic Time Warping !!

In fact ||S1-S2||=0,

they should be considered the same

Dynamic Time Warping Dynamic Time Warping (DTW)(DTW)

A widely used technique in signal processing A widely used technique in signal processing Speech recognition, Image matchingSpeech recognition, Image matching

Compare S1,S2

1 2

(0,0) 0

( , ) min { ( , 1) | 1( ) 2( ) |}

| 1 2 | ( 1. (), 2. ())i k i

Diff

Diff i j Diff k j S i S j

S S Diff S length S length

Can we directly apply Can we directly apply DTW to compare web DTW to compare web pages?pages?

No!No!

Segmentation First!!

If we directly compare 2 pages:

Most of the error will be omitted

Local VS Global

A Segmented PageA Segmented Page

How do we derive How do we derive segmentationssegmentations

Edge Detection FirstEdge Detection First

Over SegmentOver SegmentMergeMerge

NextNext

For every pageFor every page We have all its segmentsWe have all its segments

For every segmentFor every segment If it CANNOT be found in other pages, If it CANNOT be found in other pages,

possibly it is an errorpossibly it is an error

ResultResult

http://www.microsoft.com/smallbusiness/default.mspx by Opera 7.54

ResultResult

http://www.microsoft.com/learning/default.asp by Opera 7.54

Shortage and Future Shortage and Future WorkWork

Make segmentation more accurateMake segmentation more accurate Make the system completely automaticMake the system completely automatic