rial Presented by: Max Saperstone rought to you by: 340 Corporate Way, Suite Orange Park, FL 32073 888‐2 MB AM Tuto 4/7/2014 8:30 AM “Mobile Applications Testing: From Concepts to Practice” Coveros, Inc. B 300, 68‐8770 ∙ 904‐278‐0524 ∙ [email protected]∙ www.sqe.com
70
Embed
Mobile Applications Testing: From Concepts to Practice
As applications for smartphones and tablets become incredibly popular, organizations encounter increasing pressure to quickly and successfully deliver testing for these devices. When faced with a mobile testing project, many testers find it tempting to apply the same methods and techniques used for desktop applications. Although some of these concepts transfer directly, testing mobile applications presents its own special challenges. Max Saperstone says if you follow the same practices and techniques as you have before, you will miss critical defects. Learn how to effectively test mobile applications, and how to add more structure and organization to generate effective test ideas to exploit the capabilities and weaknesses of mobile devices. Max shares first-hand experiences with testing mobile applications and discusses how to address various challenges. Work on real problems on your own device and learn firsthand how to be productive while testing mobile applications.
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
rial
Presented by:
Max Saperstone
rought to you by:
340 Corporate Way, Suite Orange Park, FL 32073 888‐2
MB AM Tuto4/7/2014 8:30 AM
“Mobile Applications Testing: From Concepts to Practice”
or almost a decade, Max Saperstone has been a test engineer focusing on test automation
ng
Max Saperstone Coveros, Inc.
Fand the continuous integration/continuous delivery process. Max specializes in open source tools—Selenium, JMeter, AutoIT, Cucumber, and Chef. He has led several testing automationefforts, including developing an automated suite focused on web-based software to operate over several applications. Max also headed a major project developing an automated testing structure to run Cucumber tests over multiple test interfaces and environments, while developing a system to keep test data “ageless.” He is currently developing a new testiarchitecture for SecureCI to allow testing of multiple interfaces, custom reporting, and minimal test upkeep.
documentation may be reproduced, translated, stored in any electronic retrieval system,
transmitted in any form or by any means, electronic, mechanical, photocopying, recording or
otherwise, without prior written permission of the copyright owner.
Coveros, Inc.
4000 Legato Rd
Suite 1100
Fairfax, VA 22033
Phone: 703-349-6109
www.coveros.com
Mobile Application Testing Tutorial Welcome
Copyright 2014 Coveros, Inc. All rights reserved. Page 1
Welcome
Slide 1
Welcome to the Mobile Testing for Test Professionals training course. In this course, you will learn all about how to apply Mobile Testing to your Software Testing efforts.
Jeffery Payne is CEO and founder of Coveros, Inc., a software company that helps organizations accelerate the delivery of secure, reliable software. Coveros uses agile development methods and a proven software assurance framework to build security and quality into software from the ground up. Prior to founding Coveros, Jeffery was Chairman of the Board, CEO, and co-founder of Cigital, Inc. Under his direction, Cigital became a leader in software security and software quality solutions, helping clients mitigate the risk of software failure. Jeffery is a recognized software expert and popular speaker at both business and technology conferences on a variety of software quality, security, and agile development topics. He has also testified before Congress on issues of national importance, including intellectual property rights, cyber-terrorism, Software research funding, and software quality.
Alan Crouch is a Senior Software Security Specialist with Coveros, Inc. a Virginia-based firm focusing on agile, software quality, and application security. Alan has worked closely with federal agencies and private companies to advise, audit, and support IT security and governance teams. In addition to his cybersecurity experience, he has a strong background in software engineering, test analysis, test automation, and security testing. With a passion for software and security, Alan has focused his career on building secure software and developing better software security practices.
Mobile Application Testing Tutorial Course Trainers
Copyright 2014 Coveros, Inc. All rights reserved. Page 3
Max Saperstone has been working as a Software and Test Engineer for almost a decade, with a focus on Test Automation and the CI/CD process. He specializes in open source tools, including the Selenium Tool Suite, JMeter, AutoIT, Cucumber, and Chef. Max has lead several testing automation efforts, including developing an automated suite focusing on web-based software to operate over several applications for Kronos Federal. He also headed a project with Delta Dental, developing an automated testing structure to run Cucumber tests over multiple test interfaces and environments, while also developing a system to keep test data 'ageless.' He is currently developing a new testing architecture for SecureCI™ to allow testing of multiple interfaces, custom reporting, and minimal test upkeep.
Richard Mills has more than 20 years of experience in software engineering specializing in software development with a concentration on pragmatic software process and tools. Rich has a specific focus in Agile development methods, static and dynamic software analysis tools, configuration management, engineering, automated software build management, and continuous integration. He currently works as a Senior consultant with Coveros, Inc. Helping his customers build software better, faster and more securely. As a consultant, Rich helps customers improve software delivery by coaching and mentoring in the Agile development methodologies, automating software build, test, deployment and employing secure development techniques. Prior to joining Coveros, Rich spent 16 years with Cigital, Inc. fulfilling a wide range of roles including consulting, training, and software engineering.
Mobile Application Testing Tutorial Course Trainers
Copyright 2014 Coveros, Inc. All rights reserved. Page 4
Jeff Pierce is a Managing Consultant with Coveros, Inc., a software company that helps organizations accelerate the delivery of secure, reliable software. Coveros uses agile development methods and a proven software assurance framework to build security and quality into software from the ground up. Jeff is a leader with over twenty years experience in information technology (IT) program/project management and client management in all phases of business investment and system development lifecycle. He has proven technical excellence in development and deployment of strategic web-based enterprise information systems utilizing cloud computing and continuous integration technologies. Jeff also has experience with strategy and roadmap development, policy and procedure development, program management office implementation, IT and business management, AGILE SCRUM and XP management, systems engineering, technical team building and leading large and small on-shore and off-shore full-time and contract staff with an outstanding record of accomplishment.
Mobile Application Testing Tutorial About Coveros
Copyright 2014 Coveros, Inc. All rights reserved. Page 5
About Coveros
Slide 7
Coveros is a consulting company that helps organizations build better software. We provide software development, application security, QA/testing, and software process improvement services. Coveros focuses on organizations that must build and deploy software within the constraints of significant regulatory or compliance requirements. The primary markets we serve include: DoD, Homeland Security & associated critical infrastructure companies, Healthcare providers, and Financial services institutions
What Do You Need to Know to Be a Good Mobile Tester?
Types of Mobile Applications
Mobile Application SDLC
Mobile Application Testing Tutorial Definition of Mobile Application Testing
Copyright 2014 Coveros, Inc. All rights reserved. Page 8
Definition of Mobile Application Testing
Slide 11
Companies everywhere are rapidly building and deploying mobile applications. Testing these applications must be accurate, reliable, usable and rapid – keeping up with the pace of development and changing mobile platforms. Additionally MAT will look at testing areas that have typically only been tested by hardware engineers and will need to testing focus in order for mobile applications to be built, delivered, and executed. Title Month Year
Mobile Application Testing is a process by which application software developed for mobile devices is tested for functionality, security, usability and consistency.
Although Mobile Application Testing is a relatively new aspect of software testing, all existing standard testing practices and techniques can be applied.
What is Mobile Application Testing?
Mobile Application Testing Tutorial Mobile Application Testing Importance
Copyright 2014 Coveros, Inc. All rights reserved. Page 9
Mobile Application Testing Importance
Slide 12
App stores can reduce the time in the feedback loop, and customers can be utilized for early and frequently feedback. Without it, you may be developing an application your customers don’t want
As with all software, code quality verification is important. Users are quicker than ever to abandon an application that doesn’t work perfectly the first and every time.
Users have high expectations and are App stores also make it extremely easy to get software out to the public at a rapid pace. You want to ensure your company is putting their best foot forward to maintain a good reputation Having a consistent, predictable release process will ensure good software is released every time
The extensive choice of browsers, operating systems, devices and applications make it possible for enterprises to develop new, interactive and highly personal ways for customers and employees to access products, services and applications. Mobile devices are at the same time sources of rich information, powerful entertainment and purchasing platforms, and important work tools.
Companies that successfully implement mobile strategies engage both customers and employees in new ways, and create an environment of personal interaction that enhances sales and productivity. What makes mobile so attractive to consumers—instant access to a multitude of apps; a wide range of rapidly evolving devices and functions; a choice of operating systems; new, cloud-based services that require no storage on the device—also makes it tricky for companies to deploy effectively.
The fourth annual World Quality Report—from Cap Gemini, Sogeti and HP—reveals that only 31% of organizations surveyed currently have formal processes in place to test their mobile applications, while just 18% of those have a focus on security
Responsive Web design is an approach that suggests that an app should respond to the user’s behavior and environment based on screen size, platform and orientation.
The practice often consists of flexible screens, grids, and layouts, images and an intelligent use of CSS media queries, automatically switching to accommodate for resolution, image size and scripting abilities.
Testing a mobile web app is fairly similar to how you would test any other web application. In fact, you can often use the same tools.
Using emulators and user-agents you can often replicate resolutions and device types on your laptop.
However, there are some things you won’t be able to replicate (gestures, shakes, etc.), so don’t think that you can avoid testing on an actual mobile device.
For responsive web apps, you can often reuse tests suites for various resolutions, but no good test plan for a responsive app will have one set of identical tests for each resolution.
Mobile Application Testing Tutorial
Copyright 2014 Coveros, Inc. All rights reserved. Page 24
This Page Intentionally left blank
Mobile Application Testing Tutorial Native Mobile Applications
Copyright 2014 Coveros, Inc. All rights reserved. Page 25
Device-Specific Applications are native apps built for not only one platform (e.g. iOS) but also a device (e.g. iPhone).
Responsive Applications
Responsive Applications are platform-specific native apps that are built to respond to various screen size, resolutions and other built-in device capabilities based on the mobile device the application is installed upon.
Mobile Application Testing Tutorial Hybrid Applications
Copyright 2014 Coveros, Inc. All rights reserved. Page 27
By a strict definition, Hybrid apps are technically native apps. However, they are part native apps, part web apps; they live in an app store and can take advantage of the many device features available but rely on HTML being rendered in a browser embedded within the app itself.
Hybrid apps allow an organization to have consistency between different device platforms.
They can also provide presence in the app store without having to build much of a native app.
They can provide some cross-platform development, because HTML components can be reused to reduce costs.
Testing a native app is a mix of software and hardware testing. A tester has to test the app in all the ways a user may use the app (not just standard user behavior).
Using emulators you can often replicate resolutions, gestures and device platforms on your laptop.
However, there are some things you won’t be able to replicate, so don’t think that you can avoid testing on an actual mobile device.
Since each of the selection points are determined by the developer, test that there is enough space to perform actions and that you are properly applying OS Gestures.
Mobile Application Testing Tutorial Testing a Hybrid App
Copyright 2014 Coveros, Inc. All rights reserved. Page 28
Arguably, testing a hybrid app is the most complicated to plan and strategize. A good tester must know how to test both native app and mobile web app components to test a hybrid app successfully.
You can test many components like you would a mobile web app (even the same tools).
Testing the components built with the native code, should be performed just like a typical native app.
However, don’t forget about your integration points!
There are native mobile browser functionality that can have features that can cause unintended behavior in the embedded browser.
Mobile Application Testing Tutorial Unique Mobile Testing Considerations
Copyright 2014 Coveros, Inc. All rights reserved. Page 29
Jailbreaking (iOS) or Rooting (Android) is the process of removing the limitations on mobile devices through the user of software or hardware exploits.
Jailbreaking permits root access to the operating system, allowing download of non-approved apps, extensions and themes not available otherwise.
Large populations of users utilize Jailbroken/rooted mobile devices.
Corporate policy may permit devices from being Jailbroken
Providing escalated privileges by overcoming limitations in a computer system or device by deliberating overriding security, administrative and marketing restrictions by the creator
Non-Approved Applications
Malware, Worms– First iOS Worm, named iKree, was released to affect Jailbroken
iPhones in 2009
Unique Mobile Testing Considerations
Mobile Application Testing Tutorial Hardware Challenges
Copyright 2014 Coveros, Inc. All rights reserved. Page 35
Desktop browsers can be extremely useful when developing and testing mobile web apps.
Desktop browsers like Chrome and Firefox have many tools to effectively use user-agents to simulate multiple mobile browser types with little or no cost.
Leveraging desktop browsers does not replace device testing but they can provide robust debugging options and can be used to simulate multiple screen sizes.
Mobile Application Testing Tutorial User-Agent Switcher for Chrome
Copyright 2014 Coveros, Inc. All rights reserved. Page 39
Emulators offer the less expensive testing option, but they have many limitations for mobile testing and may not give an accurate depiction of the mobile user experience.
However, as part of the mobile testing process the use of emulation has an important role. There is certain testing that you can conduct with emulators that provides value. For example, testing for screen size or visual form factor.
However, emulation is often an approximation of the rendering and can only get you so far.
Leveraging emulators is a positive step towards real device testing, however, not sufficient for comprehensive testing needs.
Cloud Testing platforms allow testers and developers to run their applications on cloud-based emulators and simulators in virtual environments.
This eliminates the need to maintain emulator, simulators for all your testing/development teams.
You can easily test on all types of OS versions and device models.
Some cloud-based tools allow mobile teams to receive real-time alerts against any measurement criteria, allowing them to address issues before they impact end-users
There are often APIs that allow automated testing through these platforms.
Testing on a real, physical device with full functionality to the tester.
Physical device testing will always provide the most accurate testing results, but is also the most costly.
Without access to the real processor and hardware to understand the quirks that some real device have, testers may not discover some real life issues that do not present themselves in emulators
Crowdsourcing is the practice of obtaining testing services by soliciting contributions from a large group of people, and especially from an online community, rather than from traditional employees, outsourcing or suppliers.
It combines the efforts of numerous self-identified volunteers or part-time workers, where each tester of their own initiative adds a small portion to the greater testing efforts.
Crowdsourcing is different from outsourcing in that the work comes from an undefined public rather than being commissioned from a specific, named group.
Crowdsourcing
Mobile Application Testing Tutorial What are Remote Device Access (RDA) Service?
Copyright 2014 Coveros, Inc. All rights reserved. Page 44
RDA services enable access to a live device over the Internet. As a Mobile Apps Tester, you should be aware of such services and should suggest your managers about the capabilities of such services.
Due to the large number of devices available in the market, it is not feasible to buy a new device every time. That makes RDA a much more cost effective solution.
Mobile apps are often supposed to scale to massive volumes. We have to delve into lots of different realistic scenarios for performance over more than just WI-FI.
Performance testing should cover real world mobile networks (Not all networks are created equal).
– 4G LTE, 3G, 2G, EDGE, GPRS
Don’t forget to test middle-tier and back-end for performance
If you’re not testing your mobile app out in the real world like a client would experience, your performance testing is not complete
Mobile Testing Techniques
Mobile Application Testing Tutorial Performance and Stress
Copyright 2014 Coveros, Inc. All rights reserved. Page 53
Testing for Compatibility:-Testing the compatibility of your application with native device features (i.e. To make sure your application is not hampering native device functionality)
Platform
Device
Other Applications
Application backwards compatibility
Browsers
Networks
Compatibility Testing
Mobile Testing Techniques
Mobile Application Testing Tutorial Submission Guideline Testing
Copyright 2014 Coveros, Inc. All rights reserved. Page 54
Submission Guideline Testing
Slide 92
Just an excerpt from the Apple Guidelines:
2. Functionality 1. Apps that crash will be rejected
2. Apps that exhibit bugs will be rejected
3. Apps that do not perform as advertised by the developer will be rejected
4. Apps that include undocumented or hidden features inconsistent with the description of the app
5. will be rejected
6. Apps that use non-public APIs will be rejected
7. Apps that read or write data outside its designated container area will be rejected
8. Apps that download code in any way or form will be rejected
9. Apps that install or launch other executable code will be rejected
10. Apps that are "beta", "demo", "trial", or "test" versions will be rejected
11. iPhone apps must also run on iPad without modification, at iPhone resolution, and at 2X iPhone
12. 3GS resolution
13. Apps that duplicate apps already in the App Store may be rejected, particularly if there are many
14. of them
15. Apps that are not very useful or do not provide any lasting entertainment value may be rejected
16. Apps that are primarily marketing materials or advertisements will be rejected
– Generates pseudo-random streams of user events such as clicks, touches, or gestures, as well as a number of system-level events.
– Can run an automated start-to-finish test of an Android application. You provide input values with keystrokes or touch events, and view the results as screenshots.
Robotium– Support for native and hybrid applications
– Can write function, system and acceptance test scenarios, spanning multiple Android activities
– Integrates smoothly with Maven or Ant to run tests as part of continuous integration.
UIAutomation– Lets you run an automated set of tests against an application– Can be used to perform automated tests on real devices and on the
iPhone Simulator– It enables you to quickly track regressions and performance issues
NUnit– Unit level testing for iOS apps
Test Studio for iOS– Lets developers test native iOS apps as well as hybrid and Web apps– The tool can test common iOS controls and record test scripts for
gestures such as taps, multiple finger gestures such as zooms, swipes and taps.
iPhone test automation– Fonemonkey– UISpec– Bromine (not recommended)