Acceptance testing in the land of the startup
Post on 25-May-2015
488 Views
Preview:
Transcript
In the land of the startup
Joseph Wilk
Acceptance Testing
What kind of land is this?
What kind of land is this?
Do startups need acceptance tests?
Measure
Learn
Build
Context is king
Toolset
Rails, MongoDB, Mysql, RabbitMQ, Memcache
Cucumber, Rspec
Cucumber
• Behaviour Driven Development framework
• Plaintext
• Promotes communication
Given /^I have trained monkeys$/ do @test_pilots << Monkeys.newend
Feature: Filling Cucumbers with rocket fuel
Scenario: Rocket fuel Given I have trained monkeys When I click the launch button Then the monkeys should not die
Why Acceptance?
• Developer centric tool
• Behaviour Driven Development
• Conversations
Where acceptance tests are born
ProductOwner
Developer
QAUser
interaction
Where acceptance tests are born
ProductOwner
Developer
QA
Just In TimeTalking
User interaction
Non-technical people writing
Gherkin?
Feature: Ajax pagination for upcoming events In order to reduce friction finding concerts I want to go to As a logged in user I want an Ajax button which loads more concerts
Scenario: Events 60 days in the futures Given ... When ... Then ... Scenario: No events outside of the next 60 days Given ... When ... Then ... Scenario: No events Given ... When ... Then ...
Are we done yet?
“A story isn't done until it is being used by real users in production and has been validated to be a useful part of a product.”
Joshua Kerievsky
LearningsGood and bad
Asynchronous Testing is messy
Scenario: Upload image ... Given all the messages have been processed ...
Feature Overload
254 Feature files
1257 Scenarios/Examples
10807 Steps
Slicing Features
Tagging
Filenames
$ cucumber --tags @media
Subsystems
$ cucumber visitor_*
$ cucumber features/admin/*
Profiles $ cucumber --profile admin
@media, @publicFeature: Visitor views artist’s media
@feed, @adminFeature: Logged in user views views their activity feed
Acceptance testsUnit tests
vs
The Test Build
254 Features, 1257 scenarios, 10807 steps
The Test Build
254 Features, 1257 scenarios, 10807 steps
1 build server ~ 4 hours
The Test Build
254 Features, 1257 scenarios, 10807 steps
1 build server ~ 4 hours
19 EC2 build servers ~ 13:40 minutes
The Test Build
254 Features, 1257 scenarios, 10807 steps
$2068.99/$3000
1 build server ~ 4 hours
19 EC2 build servers ~ 13:40 minutes
Divide and Concuquer
Divide and Concuquer
Divide and Concuquer
Divide and Concuquer
Divide and Concuquer
Divide and Concuquer
Scenario: Related artists Given an artist “SYGC” And “M.Bison” is related to “SYGC” When I visit the artist Then I should see within Similar Artists a link to “M.Bison”
Divide and conquer
Rails
Scenario: Related artists Given an artist “SYGC” And “M.Bison” is related to “SYGC” When I visit the artist Then I should see within Similar Artists a link to “M.Bison”
Divide and conquer
RelatedArtistsService
Rails
Scenario: Related artists Given an artist “SYGC” And “M.Bison” is related to “SYGC” When I visit the artist Then I should see within Similar Artists a link to “M.Bison”
Divide and conquer
related(artist)
RelatedArtistsService
Rails
Scenario: Related artists Given an artist “SYGC” And “M.Bison” is related to “SYGC” When I visit the artist Then I should see within Similar Artists a link to “M.Bison”
Divide and conquer
related(artist)
RelatedArtistsService
HTMLRails
Scenario: Related artists Given an artist “SYGC” And “M.Bison” is related to “SYGC” When I visit the artist Then I should see within Similar Artists a link to “M.Bison”
Divide and conquer
related(artist)
RelatedArtistsService
HTMLRails
Scenario: Related artists Given an artist “SYGC” And “M.Bison” is related to “SYGC” When I visit the artist Then I should see within Similar Artists a link to “M.Bison”
Divide and conquer
related(artist)
RelatedArtistsService
HTMLRails
Scenario: Related artists Given an artist “SYGC” And “M.Bison” is related to “SYGC” When I visit the artist Then I should see within Similar Artists a link to “M.Bison”
unit testunit test
Tests that never fail
Tests that never fail
Tests that never fail
Tests which regularly fail
Daily Build
Tests that never fail
Tests which regularly fail
Daily Build Nightly Build
Tests that never fail
Tests which regularly fail
Daily Build Nightly Build
Tests that never fail
Tests which regularly fail
$cucumber --tags ~@nightly $cucumber --tags @nightly
Flickering Confidence
Flickering Confidence
FAIL
Flickering Confidence
FAIL
FAIL
Flickering Confidence
FAIL
PASS
FAIL
Flickering Confidence
FAIL
PASS
FAIL
Main Flicker@flicker
PairwiseScenario: Testing Cucumber against different environments Given I have a rails app <Rails version> And I’m using Ruby <Ruby version> And I am using the <Test Framework> And I am using the driver <Driver> Then Cucumber should install and play nicely Examples: | Rails version | Ruby version | test framework | Driver | | 3.0 | 1.8.7 | Rspec | Webrat | | 3.0 | 1.8.7 | Testunit | Capybara | | 3.0 | 1.8.7 | Rspec | Capybara | | 3.0 | 1.8.7 | Testunit | Webrat | | 3.0 | 1.9 | Rspec | Webrat | | 3.0 | 1.9 | Testunit | Webrat | | 2.3.1 | 1.9 | Rspec | Webrat | ....
| Rails version | Ruby version | test framework | Driver | | 2.3.1 | 1.8.7 | Rspec | Webrat | | 2.3.1 | 1.8.7 | Rspec | Capybara | | 2.3.1 | 1.8.7 | Testunit | Webrat | | 2.3.1 | 1.8.7 | Testunit | Capybara | | 2.3.1 | 1.9 | Rspec | Capybara | | 2.3.1 | 1.9 | Rspec | Webrat | | 2.3.1 | 1.9 | Testunit | Capybara | | 2.3.1 | 1.9 | Testunit | Webrat |
....
“most faults are caused by interactions of at most two factors”
Run Just Enough Tests
# * Test files must be stored in test/# * Test files names must start with test_# * Test class names must start with Test# * Implementation files must be stored in lib/# * Implementation files must match up with a # test file named# test_.*implementation.rb
@analyzer = Rcov::CodeCoverageAnalyzer.new
Cucover
Autotest
The Art of War Plaintext
, , @"===, ,_____cctI "?AAAAAAAAAAAAAAAA,,,,,,,,,,,,,,,,,,,,,;LLLLLLLLLL ~",,, 1""""""""""""###OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO '"EEEEE, !'"***"~~~~~~"OOOIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII ,EEEEE)>"'''???????"WWW!MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM "E.,)+="WWW~~~~~~#"OOO1OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO ,~:#")LLL!"+++???????"$$$1==========##/ &LLLLLLLLL;;;;;;;;;;;;,,,/ 1#LLLLLLLLLLLLLLLLLLLLLL! ,!###LLLLLL"'EEEE,'"LLLLL! !######LLL" "EEE" "LLLL" !#########L! "EEJ. "LL! !##########1 "JJ*,l" !############"! ,l" 1##########" 1"~~,~~" !##########" !###########! !###########1 !############! 1############# !"#############" !##############! 1##########"' A magnum, By calendron 1#####"' """""
Don’t force structure
Feature: Title In order to <value> As a <role> I want <feature>
Avoid Noise! Given I am able to loginGiven I am able to login
Avoid Inconsistency
Given I loginGiven I authenticateGiven I gain access
BalanceAbstraction
Given I go to the login pageAnd I fill in "username" with "cuke"And I fill in "password" with "cuker"And I click "login"
Given I'm logged in
Use Language Building blocks
Given /I’m logged in/ do User.create!(:user => 'josephwilk', :password => "pass") Given 'I fill in "password" with "josephwilk"' Given 'I fill in "password" with "pass"' Given 'I click "login"'end
Continuous Integration(WIP)Work in progress Done
list movies
add movies
edit movies
delete movies
Pending PassFail
$ cucumber --wip --tags @WIP features/
Limiting Tags in FlowWork in progress Done
add movies
add movies
add movies
add movies
add movies
add movies
add movies
editmovies
add movies
add movies
add movies
deletemovies
$ cucumber --tags @WIP:3 features/
Further reading
• http://cukes.info
• http://wiki.github.com/aslakhellesoy/cucumber
• http://www.cuke4ninja.com/
• http://blog.josephwilk.net
top related