The Mobile DevOps Company
Ville-Veikko Helppi Head of Demand Generation
Exercising and Scaling Up Mobile DevOps in the Enterprise
WEBINAR
Mobile DevOps
• The ‘Mobile DevOps’, Process and Tools
• Infrastructure & Architecture • Tips & Tricks for Scaling Up • Troubleshooting Device Lab • Demo • Summary – Q&A
Agenda
More information about Mobile DevOps can be found at bitbar.com
©BitbarTechnologies2017–h7p://bitbar.com
• On-Demand Mobile Device Infrastructure • Support Any Development & Testing Framework • Automate, Automate, Automate – To Save Time, Money and Hassle • Provide Unprecedented Scale and Throughput • Allow For Creative Use (Open API, Customizable Integrations) • Adopt Mobile Development Thought Leadership and Expertise
Support all development stages:
Prototyping Development Testing Production Monitoring
Successful Adoption of Mobile DevOps
• Modern mobile app development is Agile, but infrastructure does not support Agility well
• DevOps practices are emerging in to the intersection of Development, QA and Operations
• DevOps practice emphasizes the collaboration and communication of both software developers and operations while automating the process of software delivery and infrastructure changes
• DevOps practice establishes a culture and environment where building, testing, and releasing software, can happen rapidly, frequently, and more reliably.
Mobile DevOps and Agile
Development QA
Operations
Dev Ops
©BitbarTechnologies2017–h7p://bitbar.com
SPEED WINS It’s a simple fact. The faster you can deliver new applications, features and upgrades to market, and collect feedback, the better your business performs. For that you need higher development pace. And a lot of automation. THERE IS A DEMAND FOR MOBILE DEVOPS DevOps practices and tools have revolutionized the pace and quality of service delivery – but only for traditional server-based apps. FRAGMENTATION WITH TOOLS In mobile the delivery pipelines are more complex – in terms of tools and used methods. Cohesive and well-integrated development tool flow derives from Mobile DevOps practices. MANUAL DOING IS NOT EFFECTIVE Every stage requires a lot of manual work and silo’ed solutions from various vendors and/or open source tools.
Why Mobile DevOps
©BitbarTechnologies2017–h7p://bitbar.com
Delivery Pipeline for Mobile Apps
Development Source Code Management
SCM Build Test Deploy Production
Dev Tools: • IDEs • JIRA • Slack • etc.
SCM: • Git/GitHub/
GitLab • Bitbucket • MSFT • etc.
Continuous Integration and Build
Automated Testing
Automation for app store deployment, beta testers & enterprise app stores
Production Monitoring and feedback to Dev
©BitbarTechnologies2017–h7p://bitbar.com
Delivery Pipeline with Mobile DevOps
Development Source Code Management
SCM Build Test Deploy Production
Dev Tools: • IDEs • JIRA • Slack • etc.
SCM: • Git/GitHub/
GitLab • Bitbucket • MSFT • etc.
Continuous Integration and
Build
Automated Testing Automation for app store deployment,
beta testers & enterprise app
stores
Production Monitoring and feedback to Dev
TheScopeofMobileDevOps
©BitbarTechnologies2017–h7p://bitbar.com
Mobile DevOps - User Experience
Development/IDE SCM
Build
Automated Testing
Deploy / Release
“Production”
Automated builds for every code
change
Built binaries automatically tested
Tested applications, Screenshots for various devices
and languages, certificates, metadata ready to push to App stores
Deploy To Devices
Test apps in real world (Synthetic)
Apps Used by Users • Synthetic Monitoring • Real User Monitoring
Beta Testing (Humans)
©BitbarTechnologies2017–h7p://bitbar.com
Exercising Mobile DevOps
Test
Deploy
Monitoring
SCM
Development
Build
©BitbarTechnologies2017–h7p://bitbar.com
The ‘Shift-Left’ – What People Think It Is
Shi>-Le>TesAng(happensearlier)
TradiAonalTesAng(happensbeforerelease)
©BitbarTechnologies2017–h7p://bitbar.com
• Parallelism, and Lots of Real Mobile Devices (Plug-and-Play) • Mobile Test Automation Frameworks and Appropriate Use of Those • Easy to Adopt, Deploy and Use Admin Configurability • Use of Virtual Machines & Containers (Cloud, On-Premise) • Open API and Ability to Integrate with Any Tools • Efficient Use of Resources
Support all development stages:
Prototyping Development Testing Production Monitoring
What Enables Scalability?
Three Deployment Options for Test Automation with Devices
PublicCloud PrivateCloud
On-Premise
Reserved&User-Specific
Devices
Shared&PubliclyAvailableDevices
HostedbyVendor
HostedLocally©BitbarTechnologies2017–h7p://bitbar.com
UNPRECEDENTED SCALE By automating the creation, validation and deployment of the containers we are able to deploy large number of identical device servers in matter of minutes. ALWAYS CLEAN EXECUTION ENVIRONMENT New pristine container is started for each test session so that the environment is always identical from one session to another. FAST AND EASY TO UPDATE AND MAINTAIN Any updates need to be made only to the scripts that create the container images and the changes are orchestrated to all device servers ALLOWS RUN TIME CUSTOMIZATION OF THE ENVIRONMENT Because the tests are executed inside the container we can allow users to customize the environment any way they like during the test session.
Why Containers
Host
Container1
Container2
Container3
Containern
…
©BitbarTechnologies2017–h7p://bitbar.com
Because the tests are executed inside the container we can allow users to customize the environment any way they like during the test session. This is done by using the run.sh script that is used to launch the the container session. Any shell command or tool that is installed on the container can be called from run.sh script. For instance you can build anything with Maven: mvn clean test –DexecutionType=serverside or install some library or tool with Brew brew install python
Customizing The Environment
Host
Container1
Container2
Container3
Containern
…
©BitbarTechnologies2017–h7p://bitbar.com
#!/bin/bash startAppium(){
if [ "$(uname)" == "Darwin" ]; then startAppiumOSX elif [ "$(expr substr $(uname -s) 1 5)" == "Linux" ]; then startAppiumLinux else echo "Unknown OS system, exiting..." exit 1 fi
} executeTests(){
echo "Extracting tests.zip..." unzip tests.zip if [ "$(uname)" == "Darwin" ]; then echo "Running iOS Tests..." mvn clean test -Dtest=IosAppiumExampleTest -DexecutionType=serverside elif [ "$(expr substr $(uname -s) 1 5)" == "Linux" ]; then echo "Running Android Tests..." mvn clean test -Dtest=AndroidAppiumExampleTest -DexecutionType=serverside fi echo "Finished Running Tests!" cp target/surefire-reports/junitreports/TEST-*.xml TEST-all.xml
} startAppium executeTests
Example of run.sh Script
Host
Container1
Container2
Container3
Containern
…
©BitbarTechnologies2017–h7p://bitbar.com
Example: iOS Virtual Machine Components
1.3.6,1.4.3,1.4.13,1.4.16,1.6.0,1.6.3+Appium_Helper:1.3.6,1.4.3,1.4.13
0.20.4
TestAutomaAonFrameworks
Dependenciesand‘Middleware’iTunes
ProgrammingLanguagesCoreTools
©BitbarTechnologies2017–h7p://bitbar.com
Example: iOS Virtual Machine Components
1.3.6,1.4.3,1.4.13,1.4.16,1.6.0,1.6.3+Appium_Helper:1.3.6,1.4.3,1.4.13
WGETideviceinstaller
Ios-webkit-debug-proxy
usbmuxd/iproxy
0.20.4
AddiAonalframeworks
2.4.9
8.x/7.x
Settings & Config to VM • Disable software updates,
Disable screensaver, Disable sleeping, Disable Spotlight mds agent, Remove dock items, Disable hibernation, Remove sleepimage
• Remove swap files, Zero out free space, Shrink disk
• Set up /etc/hosts, Remove old Keychains, Import keychain
• Create ~/Library/MobileDevice directory, Create ~/Library/MobileDevice/Provisioning Profiles directory
• Import mobileprovision
©BitbarTechnologies2017–h7p://bitbar.com
Mobile Test Automation Frameworks
XCTEST/XCUITEST
UIAUTOMATOR ESPRESSO APPIUM CALABASH
ANDROID No Yes Yes Yes Yes
IOS Yes No No Yes Yes
MOBILE WEB Yes
Limited to x.y clicks No Yes No
LANGUAGE Obj-C/Swift Java Java Almost any Ruby
TOOL FOR TEST CREATION
Xcode UIAutomator viewer
Hierarchy viewer
Appium.app CLI (Human-Readable syntax)
COMMUNITY Apple Google Google Community Community
©BitbarTechnologies2017–h7p://bitbar.com
• Easy to setup & easily extendable • Time-independent • Backward compatibility (supporting
various API levels) • A custom Instrumentation TestRunner
with special privileges • Works on any API levels • Thin layer on top of Android
Instrumentation Framework
©BitbarTechnologies2017–h7p://bitbar.com
• Integral part of Xcode, works with both real devices and simulator
• XCTest allows developers to write tests for components at any level
• XCUITest provides rich UI testing capabilities for apps
• Objective-C / Swift • Nearly all frameworks (Appium, EarlGrey)
are working in conjuction with XCTest • Bitbar Testing products support XCTest +
XCUITest with all deployment options
XCTest / XCUITest
©BitbarTechnologies2017–h7p://bitbar.com
• Different Deployment Options o Hosted in various locations (data centers) o Co-location o On-premise
• Infrastructure Support o Wireless carrier support o Rooted devices (optional) o Support for hardware add-ons (i.e. Bluetooth,
Wearables, etc)
• Security (seriously!) o Isolated network o Dedicated hardware resources o VPN Support o OAuth/SSO
What Also Helps Scaling Up?
©BitbarTechnologies2017–h7p://bitbar.com
Private Cloud / Enterprise Support
• Network Test Support o Network Performance Optimization o Charles Proxy o Fiddler
• Performance o Built-in CPU / Memory tracking
• Graphics Benchmarking Integration o FPS, GPU utilization, battery usage, etc
• Value-Added Services o API Implementations o On-demand scripting o Video recording o Custom reporting / formatting
©BitbarTechnologies2017–h7p://bitbar.com
iOS Device Cluster Server Mac OS 1.9 or later, i5 or greater CPU, 4GB RAM or 16GM Minimum for Parallel Device Runs, 256GB Hard Disk Minimum Android Device Cluster Server Ubuntu Linux 12.04, i5 or Greater CPU, 4GB RAM Minimum, 500MB Hard Disk, 10 Port USB 3.0 Hub or 1USB 3.0 Port per Device, Motherboard with BIOS that supports turning off USB Power Saving. Up to 10 Android devices per Machine. Web Front End Server Ubuntu Server 12.04 LTS 64-Bit (Virtual or Physical), 4-Core CPU Minimum, 16GB of RAM Minimum, 500GB Hard Disk Space, 1GBit Ethernet Connectivity Database Server Ubuntu Server 12.04 LTS 64-Bit (Virtual or Physical), 4-Core CPU Minimum, 8GB of RAM Minimum, 2TB Minimum of Hard Disk, Storage in RAID Array, 1GBit Ethernet Connectivity Other Equipment To Consider Server Racks, USB Cables, PDUs, WiFi Routers, UPS
Hardware Infrastructure
n
©BitbarTechnologies2017–h7p://bitbar.com
• Devices o So many things can go wrong …
• Infrastructural Hardware o Device Control Hardware
• ADB and Instruments/Tools • Hard Disks/Storage • RAM
o USB • Hubs • Cables
• Network o WiFi o Cellular
• Software o Jenkins/CI o SCM o Test Automation Frameworks
80%The Most Typical Problems
10%
5%
5%©BitbarTechnologies2017–h7p://bitbar.com
The most typical problems • Device doesn’t show up
o CI/On-Premise Solution/etc • Device doesn’t react • Device doesn’t start tests/run • Device is out of battery/battery level low • Device shows notifications
Troubleshooting Mobile Devices Daily Check-Ups for Mobile Devices In order to maintain device farm, environment and all aspects of SW & HW, the following daily checks are recommended: • Are devices powered up? • Are all devices WiFi connected? • Do all devices have an active USB connection? • Are battery levels at least 50% (for every
device)? • Are there any pop-ups, system notifications, or
other system dialogs open on device screen? • Are screens unlocked?
We recommend checking every device 3 times/day.
©BitbarTechnologies2017–h7p://bitbar.com
Too Many Devices Connected / Device Server • Too large device cluster • Maximum of ...
o 3 iOS devices per Mac Mini o 10 Android devices per Linux server
• Enough harddisk/storage for all servers We recommend checking & cleaning harddisks (with a shell script) every day.
Typical Mistakes (and How To Tackle) Device Server Check-Ups In addition to checking logs on server machines, it’s highly recommended to check following things: • Diskspace on all machines (script) • Available RAM (script) • Do all devices have an active USB connection? • Available and consumption of CPU for peaks • Network conditions
We recommend automated scripts for checking above-listed items every day.
©BitbarTechnologies2017–h7p://bitbar.com
Network Not Working Properly • Need to check if network is working • Network for ALL connected devices • Take WiFi down and up – try connection
with your browser
We recommend checking network connection 6-9 times every day.
Typical Mistakes (and How To Tackle) Other Software Not Working Properly Again, check all logs, that contain important information about the system and health of the environment: • Syslog (on all server machines) • Catalina out (on application server) • Jenkins logs (on device servers) • MySQL errors (and their logs)
We recommend checking these items every day. In addition, it’s recommended to implement alarm script to notify about any occurring problems.
©BitbarTechnologies2017–h7p://bitbar.com
PROBLEM: If none of the devices connected to one device server are running tests (and you can’t initiate any manual session) SOLUTION: Majority of issues seen with non-running tests are related to Jenkins • Check Jenkins log (https://hostname:8080/log/all) • Restart Jenkins – https://hostname:8080/safeRestart • Restart the device server
Troubleshooting - Mobile Devices PROBLEM: Device is not running any tests. SOLUTION: First, try if you can establish a manual test session with any of your devices. If not, follow these steps: • Check that device is powered and visible on Jenkins
o https://hostname:8080/android-devices or http://hostname:8080/ios-devices
• Check that device is available in Bitbar On-Premise o https://cloudservername:9080/#service/admin/configuration
©BitbarTechnologies2017–h7p://bitbar.com
PROBLEM: Tests fail without running the test script successfully. Fail can happen at init or any phase of the script execution. SOLUTION: Find the exact error and follow these steps: • Retrying test run is a good idea to see if the error is random or reproducable • Check test run and device run views for logs and errors • Check console.log for the test run • Common failures: 1) app installation failed, 2) test zip structure is invalid, 3) Xcode didn’t start (iOS)
Troubleshooting - Mobile Devices PROBLEM: None of devices are running any tests. SOLUTION: Start with logs. In all cases, the logs contain information about errors, wrong behavior and root cause of issues can be found in those. Start with these: • First check application server log (catalina.out) • Check available diskspace • Then restart Application server (sudo reboot now)
©BitbarTechnologies2017–h7p://bitbar.com
Test On Real Devices Automation - Quickly and Efficiently
Use Existing Assets Now and Future Tools
Platform Designed For Agile Development
• Test real world scenarios on real, non-emulated devices with 100% certainty
• See exactly what the user experiences
• Reduce time and headache of device acquisition and management
The Key Things to Boost Mobile DevOps
• Use any automation framework or test exercisers (e.g. Testdroid App Crawler) to get instant reports and results
• Hit the ground running and get actionable results on a wide range of devices
• Run more tests across more devices in less time, with fewer resources
• Bring your own hardware, tools and frameworks and the Bitbar team will integrate them into the cloud
• Bitbar solutions will not lock you in to any tool or automation framework
• No vendor lock-in means that you have a long term and future proof platform for all of your mobile development and testing needs
• Bitbar integrates seamlessly to your development tools and CI process
• Instant results give immediate feedback to the development team
• Faster iterations mean more efficient development, faster time to market and more satisfied users
©BitbarTechnologies2017–h7p://bitbar.com