Tests for Every Branch Using CircleCI & Sauce Labs to Continuously Test CS Curriculum at Code.org @bcjordan Brian Jordan, software engineer at Code.org
Jan 09, 2017
Tests for Every Branch Using CircleCI & Sauce Labs to
Continuously Test CS Curriculum at Code.org
@bcjordanBrian Jordan, software engineer at Code.org
Code.org
non-profit
expanding participation in CS
Hour of Code
full curricula, district partnerships,
professional development, policy change
K-12 CS Curriculum
43,000 K-12 teachers trained
courses: 400k teachers 12mm students
how did Code.org start automated testing?
development in 2013-2014
why test? what is Code.org's testing context?
the context
the context
the context
the context
the context
RTL LTR
the context
the context
so how do we test all that?
how we started Selenium testing
how our tests are organized and run
selenium-webdriver cucumber
takeaways
Build a library of steps
Build a library of test levels
Build a library of annotations
@no_mobile@no_ie9
@skip
@db_access@as_student
@eyes
who writes tests?
who/what runs tests?
2015: one pipeline
2016: every commit!
what challenges did we end up facing?
challenge:multiple browsers
solution:selenium-webdriver
Sauce Labs
challenge:
testing new changes
locally:chromedriver Sauce Connect
challenge:
big buckets of changes
pull request tests in
❤
pull request tests incircle.yml
github.com/code-dot-org/code-dot-org
tips and tricks
github.com/code-dot-org/code-dot-org
⚡
tips and tricks
github.com/code-dot-org/code-dot-org
⏰
tipsadding more [commit tags]
tips
use Sauce Connect for Circle <=> Sauce Labs
tips
use Sauce Connect for Circle <=> Sauce Labs
tips
use Debug via SSH and Manual Sessions for debugging
tipsdebugging failures—APIs to the rescue!
tipsdebugging failures—APIs to the rescue!
challenge:
visual, responsive changes
hackathon solution?
not easy...
actual thing!
using eyes
what do tests look like when running?
Speed Run!
(not) wasting time
speeding up test runs
2014: 60-90 min of tests
parallelizationat browser level
2015: 60-90 min of browser tests
to 20-30 min
parallelizationby containers
parallelizationby containers
justifying tests
the bug collection
missing button
button height change
extra margin
overflow: hidden added
duplicate button
stray pixel
Firefox getBBox()
Chrome 50 (deprecated SVGElement.offsetWidth)
my favorite bug ever
who investigates failures?
dev of the day
dev of the day😭
dev of the day+
what's next?
further flakiness investigations
(less flakiness: more parallelization with less pain)
💪
Applitools Eyes test results as GitHub PR comments
takeaways
start small
everybody tests
periodically invest in speed-ups
In memory of Laurel Fan
Thanks to the Code.org team who spread the burden of f leshing out, monitoring and improving our test infrastructure, especially Laurel Fan, Brendan Reville, and Brad Buchanan.
Thanks to the kind folks at Sauce Labs, CircleCI and Applitools who helped us get set up, and to Dave Haeffner @tourdedave for sharing his advice on testing.
code.org/help
github.com/code-dot-org
Thanks!
@bcjordanBrian Jordan, software engineer at Code.org
Q&A