Automated Visual Testing That Doesn't Suck!

Adam Carmi (@carmiadam)Co-Founder and VP R&D at Applitools

adam.carmi@applitools.com

AUTOMATED VISUAL TESTING THAT DOES

NOT SUCK

YOU CAN AND SHOULD AUTOMATE YOUR VISUAL

TESTS!

AGENDA

Why automated visual testing?Tools & TechnologyWhere does it fit?

WHAT IS VISUAL TESTING?

A quality assurance activity aimed to verify that a

Graphical User Interface appears correctly to users

A VISUAL BUG

WHY SHOULD IT BE AUTOMATED?

THE TEST MATRIX IS TOO BIG TO APPROXIMATE MANUALLY

Web browsers Operating systems Screen resolutions Responsive designs L10n

WHY SHOULD IT BE AUTOMATED?

MANUALVISUALTESTINGISERRORPRONE

WHY SHOULD IT BE AUTOMATED?

WHY SHOULD IT BE AUTOMATED?

Many are already doing it (and sharing)…

PhantomCSS Fighting Layout Bugs

CSSCritiqueWraith

Needle

Grunt PhotoBox

dpdxt

WebdriverCSS

EyesHuxley

FBSnapshotTestCase

GeminiSelenium Visual Diff

VisualCeption

Specter

Snap And Compare

AGENDA

Why automated visual testing?Tools & TechnologyWhere does it fit?

THE WORKFLOW Drive the AUT and take screenshots

Compare screenshots with baseline images

Report differences

Update the baseline

DEMOhttps://github.com/webdriverio/webdrivercss

THE WORKFLOW Drive the AUT and take screenshots

Compare screenshots with baseline images

Report differences

Update the baseline

DRIVING AND CAPTURING

OTHER

PhantomCSS (CasperJS)

Fighting Layout Bugs (Java) CSSCritique (URLs)Wraith (URLs)

Needle (Python)Grunt PhotoBox (URLs)

WebdriverCSS (JS)

Eyes (All)

Huxley (RP)

FBSnapshotTestCaseSelenium Visual Diff (Java)

VisualCeption (PHP)

Specter (JS)

Wraith-Selenium (Ruby)

Snap And Compare (URLs)

Eyes (All)

Gemini (JS)

dpdxt (URLs)

Gemini (JS)

SCREENSHOTS

Real browsers?Full page?Frames?Regions?Viewport size?Page stabilization?Page preparation?

THE WORKFLOW Drive the AUT and take screenshots

Compare screenshots with baseline images

Report differences

Update the baseline

WHY NOT?

FALSE POSITIVES

ANTI-ALIASING?

ANTI ALIASING 1/2

ANTI ALIASING 2/2

BRIGHTNESS 1/2

BRIGHTNESS 2/2

SUB PIXEL SHIFTS 1/2

SUB PIXEL SHIFTS 2/2

SUB PIXEL SHIFTS 1/2

SUB PIXEL SHIFTS 2/2

AND MORE…1 pixel offsets in element positioning

Dynamic contentMoving elementsImages of different size

Image Comparison APIs

ImageMagick A powerful command line tool for image processing. APIs are available for most programming languages. Fuzzing is used to eliminate slight color differences An error ratio is usually used to determine a match

$ compare –metric AE –fuzz 5% img1.png img2.png diff.png

2246

Resemble.js An image comparison tool

implemented in Javascript Used by PhantomCSS and other

tools. Good de-antialiasing An error ratio is usually used to

determine a match http://huddle.github.io/Resemble.js/

Applitools Eyes A specialized image processing stack

designed to compare computer generated UI images Anti-aliasing Partial and full pixel offsets Images of different size Dynamic content Moving elements Layout matching

Available as a cloud service

DEMO

THE WORKFLOW Drive the AUT and take screenshots

Compare screenshots with baseline images

Report differences

Update the baseline

REPORT DIFFERENCESAs files on the file system (combined with source control)

REPORT DIFFERENCESAs a Gallery (example from Selenium Visual Diff)

THE WORKFLOW Drive the AUT and take screenshots

Compare screenshots with baseline images

Report differences

Update the baseline

UPDATE THE BASELINE

Rename or commit individual image files

Overwrite mode

UPDATE THE BASELINE

GUI (Gemini GUI)

Automatic maintenance (demo)

AGENDA

Why automated visual testing?Tools & TechnologyWhere does it fit?

WHERE DOES IT FIT?

• Components

• Code review

• Developers

• Designers• QA

WHERE DOES IT FIT?

• Pages• Page

sections

• Developers

• Designers• QA• Others

WHERE DOES IT FIT?

• Staging vs. Production

• Ops• QA

WHERE DOES IT FIT?

• Monitoring

• Ops• QA

QUESTIONS?

Adam Carmi (@carmiadam)Co-Founder and VP R&D at Applitools

adam.carmi@applitools.com

THANK YOU