Einstein - Australia and New Zealand Testing Board · a web implementation of ... automatically controlled automated unit tests using RSpec w/ mocks automated functional tests using

Einsteinbuilding a minesweeper robot using tests

Alister ScottThoughtWorks Australia

Mini skirt presentation

minesweeper.github.com

pure CoffeeScript

347 lines of CoffeeScript =>

478 lines of JavaScript

minesweeper.github.comJasmine specs

74 specs in 0.3 seconds

687 lines of CoffeeScript

(nearly double prod code)

how do we create an Einstein robot?

tests!

how?

outsidein

"Are you facing an issue that's too big to wrap your mind around? Break it down. Keep dividing problems into smaller and smaller pieces until you're able to digest them."

~ Getting Real - 37Signals

[http://gettingreal.37signals.com/ch06_Shrink_Your_Time.php]

Break it down

stage one: guess

success rate: < 0.1%

stage two: identify obvious mines

stage two: identify obvious mines

success rate:< 2%

stage three: guess better

success rate: 5%

stage four: 'clusters'

success rate: 13%

stage five:'derived clusters'

success rate: 20%

progressive intelligence: law of diminishing returns?

detect some safe cells

detect some obvious mines

clustersguessing by field probability

derived clusters

related clusters

a minesweeper robot:separation of concerns

can play any minesweeper implementation game using in

built intelligence

a web implementation of minesweeper that can be automatically controlled

automated unit tests using RSpec w/ mocks

automated functional tests using Cucumber & Watir-

Webdriver

minesweeper robot testing pyramid

automated unit tests using RSpec w/ mocks

automated functional tests using Cucumber &

Watir-Webdriver

48 specs in 0.2 seconds

1 feature: 4 scenarios35 seconds

exploratory testing

automated unit tests using RSpec w/ mocks

?manual exploratory testing:scan logs

look at screenshots on guesses

automated functional tests using Cucumber &

Watir-Webdriver

eliminate unnecessary guesses!

● Let robot play heaps of games ● Study each guess (screenshot)● Was it absolutely necessary to guess? If

not, it's a bug!● Write failing spec from screenshot● Then make it pass!

Exploratory Testing

example: gets hard

describe 'safe cells to click' doit 'should detect safe cells taking multiple non-adjacent cell clusters into consideration' do

analyse <<-EOF 3 * 3 1 2 1 * * 4 * 3 * 3 4 . . . 2 * 2 . 2 . 1 EOF analyser.safe_cells_to_click.should == [[3, 2],

[3, 4], [2, 3]]end

end

Performance TestingHow do you test performance of an algorithm that tries to solve a non-deterministic problem?

After thousands of runs, I found an expert mine field that can be solved without guessing (if you start by clicking 0,0)

Can always win this without guessing (start at 0,0)

Ensure we never degrade performance

Einstein'sPerformance MetricsBeginner

Games won: 85%Best time: 2 seconds

Intermediate:Games won: 65%Best time: 8 seconds

Expert:Games won: 20%Best time: 21 seconds

Enough talk alreadyLet's see Einstein in action!

[http://soundcloud.com/blaketothefuture/nyan-cat-the-movie]

Bonus time ~Lessons Learned● Break big things down into little things● Design for testability● Test in the right places ● Supplement your automated testing● Build in performance measures● Change your mind● Be young, be foolish, be happy