Automate Your Build Acceptance Testing for Mobile Apps Rachel Obstler VP Mobile Testing Chris Karnacki Solutions Lead
Automate Your Build Acceptance Testing for Mobile Apps
Rachel ObstlerVP Mobile Testing
Chris KarnackiSolutions Lead
Your Participation
Open and close your control panel
Join audio:• Choose Mic & Speakers to use VoIP• Choose Telephone and dial using the
information provided
Submit questions and comments via the Questions panel
Note: Today’s presentation is being recorded and will be posted to keynote.com fro on-demand viewing.
Your Participation
Automate Your Build Acceptance Testing for Mobile Apps
Rachel ObstlerVP Mobile Testing
Chris KarnackiSolutions Lead
Mobile and Agile Change the Test Process
Software Diversity
Hardware Diversity
Faster Release Cycles
Agile Processes
Today’s users expect apps to just work on their device
2-week sprints monthly releases
Automation
A. >75%B. 50-75%C. 25-50%D. <25%E. Not sure
Poll: Of your organization’s mobile development projects, what % have build acceptance testing fully automated?
Why Aren’t More Companies Automating Mobile Testing?
ComplexitiesDesktop Skillsets
Access to Devices
Time
Geographical Diversity
% Automation (mean across responses)
Unit tests 53%
Functional tests (GUI) 45%
Functional tests (Beyond GUI, e.g., APIs) 42%
Continuous testing — tests are kicked off at each code check in and integration (CI) 42%
We automate nonfunctional (performance) 38%
We automate nonfunctional (integration) 35%
We automate test environment provisioning 30%
Base: 155 professionals with knowledge of their firm’s Agile practices. Source: Forrester's Q2 2015 Global Agile Software Application Development Online Survey
“Using your best estimate, how much do you automate the following?”
CurrentRelease
FutureReleasePlanning
Automating Build Acceptance Testing Potential Gains Over 2 weeks – Savings of 67
hours of person/testing time (equal to a 200 test case regression test)
Ongoing you could allocate 83% of a person to another project, or reduce your release time by up to 1 week
And… Reduced the time to give
development feedback on the release
Reduced rework for the development team
Small contained project Highest return on automated testing Huge benefit in a short period of time
# of Testers 0 1 2 3 4Elapsed Hours to Validate
Build (Manual) 6.7 3.3 2.2 1.7
Elapsed Hours to Validate Build (Automated) 1.0
Simple Example20 # of Build Acceptance Tests4 # of Target Platforms5 Average # of minutes to manually execute a test case3 Average # of minutes to automatically execute a test caseDaily Frequency of a BAT run
Kick-Start Your Mobile Automation Program
Define the Build Acceptance
Program
Expand When Appropriate
Continuous Integration
Architecture
Execution – Script and Test
Kick-Start Your Mobile Automation Program
Define the Build Acceptance
Program
Defining the Program – Creating Optimized Device ListAdvantages
Ensures coverage across device OSs and form factors
Minimizes the set of devices necessary to test across
iOS Android
Why Real Devices vs Emulators? Different screen sizes/resolutions
cause blurring, sizing issues, items that fall off screen
On-screen, physical controls that function differently
Impacts of low memory or other services running on device
Defining the Program – Build Acceptance Test DefinitionCreate your BAT list, making sure:1. All tests are automatable on real
devices2. All tests validate major functional
areas, such that failure means the build is rejected
3. Most, if not all, basic functionality is covered – use the 80/20 rule
4. The BAT will not run >1 hour
Goals of BAT Save test team effort to setup and
test a build when major functionality is broken
Give development team immediate feedback on the build so they can fix it immediately
Kick-Start Your Mobile Automation Program
Define the Build Acceptance
Program
Continuous Integration
Architecture
Continuous Integration Architecture – The Basics Architecture must enable true
continuous integration Determine what works best based
on current systems and skills: Which build system is utilized Where the app resides Notification plan for BAT completion Team programming or automation skills
And don’t forget to consider security!
BAT Must HavesFor BAT automation to work, the build must trigger both the app to be automatically loaded on the target devices, and the automated tests to run.
CI Architecture and Data Flow DiagramDevelopment Test
Review CodePush Code
Bug Reports
BAT Test Results
Build AutomationVersion Control Real Mobile DevicesBAT Build on pushNightly
regression
A. JenkinsB. Bamboo C. TeamCityD. OtherE. None
Poll: What build automation application do you use for continuous integration?
Example Integration with Jenkins
Kick-Start Your Mobile Automation Program
Define the Build Acceptance
Program
Continuous Integration
Architecture
Execution – Script and Test
Execution – Implement Automated BATs & CI IntegrationThe implementation phase includes:
Writing scripts Testing scripts Integration with your build process End-to-end validation of app build to
report
Object-Level Scripting Easily re-useable Works across devices Remain resilient to ongoing app
changes Creates a framework for future
regression test automation
Sample Report Quick view of
overall BAT success By test pass/fail Ability to dive into
failed test results quickly and share results with the agile team
Kick-Start Your Mobile Automation Program
Define the Build Acceptance
Program
Expand When Appropriate
Continuous Integration
Architecture
Execution – Script and Test
Ongoing Management and ExpansionOnce the project is implemented, you must:
Review the test plan Maintain the scripts Ensure multiple team members are trained Review of CI architecture and process with
the entire agile team Review reports
Then consider expansion, but continue to be prudent on defining manageable projects!
Success includes Training so team can easily maintain
and add to existing scripts Framework that enables easy
expansion of automation program Freeing tester time!
Demo
Keynote Mobile TestingMobile app testing platform for agile development teams Integrates into Your Build Process Comprehensive Cloud-Based
Library of Real Mobile Devices Manual or Fully Automated options Flexible Test Scripting Options Powerful Test Automation Engine Intelligent Reporting
Interactive and automated testing for any mobile app or website on hundreds of mobile phones and tablets
300+ shared devicesPrivate devices
Keynote Mobile Testing Build Acceptance Testing Program
Program Stage Details
1. Define BAT plan and Goals Review or recommend target devices Review or help define list of Build Acceptance tests Create project schedule
2. Define CI Environment and Requirements
Document the different components of the build process, and how BAT automation fits in
Document the data flow and information sharing of results3. Implement Automated BAT Integrate with existing build process
Build and test automated scripts Test end-to-end process
4. Training and Handoff for Ongoing Management
Training / handoff to internal team to maintain, expand program
Questions?
Learn More
About Keynote Mobile Testing: www.keynote.com BAT Automation Program: www.keynote.com/now/bat More questions?
[email protected] +1 (650) 403-2451