Are You Missing Critical Mobile Tests?

JeanAnn Harrison & XBOSoft

Are You Missing Critical Mobile Tests?

XBOSoft info

• Founded in 2006• Dedicated to software quality

• Software QA consulting• Software testing services

• Offices in San Francisco, Beijing, Oslo and Amsterdam

On the XBO side:

Amy Jan •

Housekeeping• Everyone except the speaker is muted

• Questions via the gotowebinar control on the right side of your screen

• Questions can be asked throughout the webinar, we’ll try to fit them in when appropriate

• General Q & A at the end of the webinar

• You will receive info on recording after the webinar

Reminders

Make sure your drained device is available, as well as another device to send email to the drained device and a charger.

When you see the heart “pumping iron” slides JeanAnn will walk us through a test on your device.

Please type questions in the Chat box.

We are looking for 1-2 volunteers to take the microphone during their device testing to let everyone know what is happening.

About the Speaker

JeanAnn Harrison regularly speaks at industry conferences and is published in software testing magazines.

She has years of experience with mobile testing and is a well-known figure in the QA and software testing community.

Besides software testing , JeanAnn really likes baseball & ice skating.

WEBINAR: EXERCISES IN EXPLORATORY TESTING

JEAN ANN HARRISONSoftware Test Consultant for Project Realms.

FOR MOBILE AND EMBEDDED TESTERS

Jean Ann Harrison Copyright 2013

AGENDA• Introduction

• Note taking & use of slide information

• Quick overview of Mobile/Embedded Software

• Types of Mobile Apps

• Exploratory Testing Definition

• Defining Skill Set for the Mobile Tester

• Mindmap getting started with Mobile Testing

• Good Exploratory Testing Skills/Mobile ET Skills


• Keys to Mobile Exploratory Testing

• Testing Beyond the GUI

• Learning the Architecture

• Organizing Tests/Stories

• Common tools for ET

• Some Test Case Ideas

• Exercises & Test Case Examples

• Summation & Conclusion

Not covered – User Experience, Usability, Trainability, Web Application & Security Tests


MOBILE/EMBEDDED SOFTWARE & DEVICES

• Mobile phones, tablets

• Proprietary devices: mobile heart monitors, law enforcement ticket generators, restaurant personnel taking customer orders on devices, GPS devices, PDAs

• Embedded Software examples: automobile’s computer system, air traffic control systems, airplane navigation system, drug infusion pumps in hospital rooms, elevators, cameras, robots

TYPES OF MOBILE APPS

• Native Applications• Local to device

• Hybrid Applications• Local to device but

interacts w/internet

• Web Applications• Not local to device.

All interactions on internet


EXPLORATORY TESTING - DEFINITION

• Quoting James Bach: “The plainest definition of exploratory testing is test design and test execution at the same time. This is the opposite of scripted testing (predefined test procedures, whether manual or automated). Exploratory tests, unlike scripted tests, are not defined in advance and carried out precisely according to plan.”

http://www.satisfice.com/articles/what_is_et.shtml





DEFINING SKILL SET FOR THE MOBILE/EMBEDDED

TESTER

• Some exposure or knowledge about products from the domain in which you are testing: aerospace, medical, automobile manufacturing, airplanes, factory systems, robotics, regulated environments, etc.

• Some knowledge of: hard sciences: math, physics, electronics, engineering, etc for logical thought process.

• Software sciences: psychology, philosophy, sociology, human factors (human machine interface) for creative & conceptual thought process

• These testers use skills from many knowledge domains, patterns of errors, and basic testing skills to create tests. They are using their intuition, critical thinking, and their mental models.

Chapter 1 – Software Test Attacks to Break Mobile & Embedded Devices



WHAT YOU NEED TO BE A GOOD EXPLORATORY TESTER

• Awareness• Discipline• Focus



KEYS TO MOBILE EXPLORATORY TESTING

• Look beyond the GUI• Timing• Domain knowledge• Knowledge of System

(hardware/firmware/software)• Characteristic of bugs behavior, formulate

patterns• Use life experiences for inspiration

TESTING BEYOND THE GUI = SYSTEM INTEGRATION TESTING


SIT for Mobile/Embedded contains elements:

Software + Hardware + Software Variants + Hardware Variants + Timing + Operating System Drivers + Network Protocols

WHAT DO WE TEST?


LEARNING THE ARCHITECTURE

Work closely with development to learn more about how the mobile application under test works.


EXERCISE FOR GROUP

• In the question box, enter 1 test idea of what you want to explore. The point of this exercise is you do NOT know what the application is, so you can’t tell me what kind of functional test to perform.

• I’m testing YOU to see what kinds of non-functional tests you think might be good to explore.


COMMON TOOLS FOR ORGANIZATION

• Sticky notes per conditions, per variables, per test case idea

• Keep a log of what you are doing, record conditions

• Utilize the device’s log or find an open source logging tool

• Use tables to record all variables/conditions per test

• Document situation of the test, list out your test conditions


”WE BECOME MORE EXPLORATORY WHEN WE CAN'T TELL WHAT TESTS SHOULD BE RUN, IN ADVANCE OF THE TEST

CYCLE, OR WHEN WE HAVEN'T YET HAD THE OPPORTUNITY TO CREATE THOSE TESTS.” JAMES BACH

• Create your description or theory. “What happens when…?”

• … battery runs low?

• … battery is charging from a dead battery?

• … when the device gets too hot?

• … when the battery is replaced due to defective battery?

• … device memory is full?

• … I receive a notification while using the phone or another application?

• … when there is a time/date change?

• … device searches are too slow?


DOCUMENTING YOUR TEST CONDITIONS FOR THE CHARGING OF BATTERY TEST


TEST CASE STEPS

1. Pre-select any application to download onto device

2. Drain your device’s battery charge completely

3. Notate time

4. Plug in charging device

5. Immediately power up device

6. Record times for first condition where app is not installed, device powered up


7. Drain your device’s battery charge completely

8. Plug in charging device

9. Immediately power up device

10. Notate time

11. Start to download pre-selected app

12. Record times for each battery level while downloading

EXERCISE: EXPLORATION FOR MORE TESTS


• What other kinds of tests can you apply based on the condition of starting to charge from a dead battery?

• Record your tester stories

ORGANIZE YOUR THOUGHTS/TESTS


Battery level

Time/Temp Application Installing

Application Installed/not in use/started

App in use (note – various conditions apply here)

0%-5% Time each level increment of charge

Timing of download?

Would notifications factor?

Notifications?Memory usageCPU speed

6%-15% Check the temp at each increment

Completed installation?

Note CPU speed, memory usage

Searches with application DB, online DB?

16%-30% Interaction with data transfer/network communication

31%-50%

CONTROL SYSTEM CONSIDERATIONS DEVELOPING TESTS:

HARDWARE TO SOFTWARE SIGNAL INTERFACE

• Know the input and output connections to the software

• Consider differences in variations between devices like calibrations, physical noise, electron, light, cold, water, dust device, response time, wear & tear on the hardware.

• Identify input & output devices with ranges and resolutions of values (how is the software installed onto the device?)

• Define the full range of environmental input disturbances (unexpected inputs)

• Define possible environmental output disturbances (unexpected outputs)

• Determine what is or is not possible in the test lab

• Conduct a risk analysis of what is acceptable and what is not

• Understand device limitations: CPU processing capabilities, memory, time.

• The Aerospace Industry use the term, “Test like you fly, fly like you test” to combine all facets of how a device’s software should be tested.

Originally written by Jon D Hagar Copyright 2013 Jean Ann Harrison Copyright 2013


• Download either Twitter or Facebook onto device

• Start either downloaded app

• From another device, send an email to the device’s email account

• Immediately send a tweet or post a status

• Continue to do engage Twitter or Facebook app for at least 1 minute

• Record email notification and time when sent and received.

• What other observations occurred?

EXERCISE: NOTIFICATION TEST

EXERCISE: EXPLORING MORE FOR THE INTERRUPT TEST

• What other kinds of interrupts could be tested?• What other tests would you consider if something

goes wrong?


TEST CASE EXAMPLE: INTERRUPTS ON MOBILE PHONES

• Go to your App store and choose an application to download

• While the downloading is occurring, call the mobile phone

• Record observations with the download

• You may need to rely on observing a log file while implementing these tests

• If it fails, what kind of error recovery occurs?


LET’S TALK DATA AND STORAGE

• Mobile devices contain limited amount of storage space.

• Testing for mobile apps storing data locally on a mobile device means testing the boundaries of the app’s design. How will developers handle the limitations?

• Work with your developers to understand those boundaries.

• If your app doesn’t have a boundary set, then the app can fill up your device’s storage capabilities.

• Testing the limits might mean testing how the software polices that boundary by having a specific limit before automatically downloading only so much information.

• Per usage, email programs only download so much data and most recent data to the device.


TEST CASE EXAMPLE: TESTING STORAGE LIMITS

• Download several large applications (ex: Angry Birds & Facebook are large apps)

• Go to your Settings and then to “Manage applications”.

• Review the amount of total storage free.

• Keep downloading apps until you reach below 5KB

• Then start an application where you would save data on the device.

• Check your free storage amount.

• Keep saving data until you reach a point where you receive an error message or a “can’t save” message.

• Check the remaining free storage in the Settings again to see what the threshold is for free space is maxed out.

• Delete the last data which was just added.

• Send data though the internet, but use the app under test (using the application where you first started saving data) and not just email.

• What other tests can you do where storage is maxed out?


SUMMARIZATION OF TECHNIQUES

• Combining tests can help shorten project schedules but what also happens is you can witness results. Automation results do not give you all you need to witness as it happens.

• Ask yourself “what if I … “ and consider tests not related to functional behavior or conditions.

• Using tools, especially log files, can reveal memory usage, getting statistics while conducting tests.

• Documenting includes use of video, writing/development of tests in story formats, user action formats


CONCLUSION

• Remember: In mobile and embedded just poking at the GUI will leave bugs “hiding”

• Different tests by a variety of testers (developer and titled testers) is commonly needed

• Context matters: there is no one set of tests, approaches, testers, or attacks which will work all the time

• Limited resources on device can make testing challenging and requires an entire team effort


JEAN ANN HARRISON’S CONTACT INFORMATION

• EMAIL: [email protected]• Twitter: https://twitter.com/JA_Harrison Or @JA_Harrison• Project Realms Inc website:

www.projectrealms.com

• 8/10/2013, I will be facilitating a Weekend Testing Session on Risk Based Testing for Mobile Devices… details can be found through my Twitter updates.


mailto:[email protected]

https://twitter.com/JA_Harrison

https://twitter.com/JA_Harrison

http://www.projectrealms.com/


REFERENCES

• Software Test Attacks to Break Mobile & Embedded Devices by Jon D Hagar to be published and released fall 2013.

• 2013 STPCon Presentation slides from “Testing Beyond the GUI” by Jon D Hagar & Jean Ann Harrison

• http://www.ministryoftesting.com/2012/06/getting-started-with-mobile-testing-a-mindmap/ Getting started with Mobile Testing Mindmap

• http://karennicolejohnson.com/ Karen Johnson’s website

• http://www.satisfice.com/articles/what_is_et.shtml by James Bach, Satisfice, Inc

• http://www.softwaretestpro.com/Item/5567/ A Different Take on Mobile Testing – Test Beyond the GUI by Jean Ann Harrison

http://www.ministryoftesting.com/2012/06/getting-started-with-mobile-testing-a-mindmap/



http://karennicolejohnson.com/

http://karennicolejohnson.com/




http://www.softwaretestpro.com/Item/5567/

http://www.softwaretestpro.com/Item/5567/


IMAGES REFERENCES

• Slide 3 - Embedded Systems examples

• http://www.jijesoft.com/en/?option=com_content&view=article&id=18&Itemid=45

• http://www.mseedsystems.com/products/view/268/aldelo-for-restaurants-pos-wireless-edition

• http://blog.laptopmag.com/wpress/wp-content/uploads/2013/02/Pandora-Limit-Free-Listening.jpg

• slide 4 - types of Apps

• http://myeventapps.com/whats-the-difference-native-vs-web-apps/2012/794/

• Slide 5 – Exploratory Testing Definition

• http://www.testandtry.com/wp-content/uploads/explore.jpg

• slide 6 - Defining skill set

• http://cdn.cutestpaw.com/wp-content/uploads/2012/11/l-The-Thinker.jpg

IMAGES REFERENCES

• Slide 7 - mindmap used with permission from originators Karen Johnson & James Bach

• www.ministryoftesting.com

• slide 8 - What you need to be a good Exploratory

• Testerhttp://www.wallcoo.net/paint/kagaya_celestial_exploring/Kagaya_art_Celestial_Exploring_INSPIRATION1.html

• Slide 9 - puzzling together pieces

• http://www.aspire2develop.co.uk/images/People%20development%20page%20&%20HOME%20page%204%20piece%20jigsaw%20iStock_000002077956Small.jpg

• slide 10 - Testing beyond the GUI

• http://courses.cs.vt.edu/csonline/OS/Lessons/Introduction/onion-skin-diagram.gif


• Slide 12 – Leaning the architecture

• http://2.bp.blogspot.com/-gO3DIydtaQA/UBoUj4afwsI/AAAAAAAAAvk/cwPAtXkcgJ4/s1600/12152009_muppets7.jpg

• Slides 13, 18, 21 22

• http://www.duvalschools.org/aprtech/student_life/BPA_Team_2/Exercise.html

• Slide 14 – Common organizational tools

• http://www.dislive.com/wp-content/uploads/2013/07/architect-interior-designer-or-carpenter-workplace-with-desk-design-my-room-interior-design-tool-936x624.jpg

• Slide 15 – James Bach quote of Exploratory Testing

• https://admissionblog.usc.edu/files/2013/05/Question-Mark.jpg


IMAGES REFERENCES


• Slide 24 – Let’s Talk Data Storage

• https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcR5pxcFFYJILFm1pg20ttHxc-wPTm_05O4Yr7uhEP--PUZhy-bX

• Slide 27 – Conclusion

• http://www.mobileapptesting.com/wp-content/uploads/2011/03/Mobile-Apps-300x188.jpg

IMAGES REFERENCES

Follow us: @XBOSoftQuestions at: [email protected]

Are You Missing Critical Mobile Tests?

Technology

device testing

software testing community

software testing magazines

software test attacks

basic testing skills

software variants

software sciences

mobile application