Top Banner

Click here to load reader

Are You Missing Critical Mobile Tests?

May 11, 2015




Expert tester and speaker JeanAnn Harrison reveals critical mobile tests that can be used on many mobile devices, even proprietary. Exercises and ideas for testing that you can use and build on. Sponsored by XBOSoft.

  • 1.JeanAnn Harrison & XBOSoft Are You Missing Critical Mobile Tests?

2. XBOSoft info Founded in 2006 Dedicated to software quality Software QA consulting Software testing services Offices in San Francisco, Beijing, Oslo and Amsterdam 3. On the XBO side: Amy Jan 4. Housekeeping Everyone except the speaker is muted Questions via the gotowebinar control on the right side of your screen Questions can be asked throughout the webinar, well try to fit them in when appropriate General Q & A at the end of the webinar You will receive info on recording after the webinar 5. Reminders Make sure your drained device is available, as well as another device to send email to the drained device and a charger. When you see the heart pumping iron slides JeanAnn will walk us through a test on your device. Please type questions in the Chat box. We are looking for 1-2 volunteers to take the microphone during their device testing to let everyone know what is happening. 6. About the Speaker JeanAnn Harrison regularly speaks at industry conferences and is published in software testing magazines. She has years of experience with mobile testing and is a well-known figure in the QA and software testing community. Besides software testing , JeanAnn really likes baseball & ice skating. 7. WEBINAR: EXERCISES IN EXPLORATORY TESTING JEAN ANN HARRISON Software Test Consultant for Project Realms. FOR MOBILE AND EMBEDDED TESTERS Jean Ann Harrison Copyright 2013 8. AGENDA Introduction Note taking & use of slide information Quick overview of Mobile/Embedded Software Types of Mobile Apps Exploratory Testing Definition Defining Skill Set for the Mobile Tester Mindmap getting started with Mobile Testing Good Exploratory Testing Skills/Mobile ET Skills Jean Ann Harrison Copyright 2013 Keys to Mobile Exploratory Testing Testing Beyond the GUI Learning the Architecture Organizing Tests/Stories Common tools for ET Some Test Case Ideas Exercises & Test Case Examples Summation & Conclusion Not covered User Experience, Usability, Trainability, Web Application & Security Tests 9. MOBILE/EMBEDDED SOFTWARE & DEVICES Mobile phones, tablets Proprietary devices: mobile heart monitors, law enforcement ticket generators, restaurant personnel taking customer orders on devices, GPS devices, PDAs Embedded Software examples: automobiles computer system, air traffic control systems, airplane navigation system, drug infusion pumps in hospital rooms, elevators, cameras, robots Jean Ann Harrison Copyright 2013 10. TYPES OF MOBILE APPS Native Applications Local to device Hybrid Applications Local to device but interacts w/internet Web Applications Not local to device. All interactions on internet Jean Ann Harrison Copyright 2013 11. EXPLORATORY TESTING - DEFINITION Quoting James Bach: The plainest definition of exploratory testing is test design and test execution at the same time. This is the opposite of scripted testing (predefined test procedures, whether manual or automated). Exploratory tests, unlike scripted tests, are not defined in advance and carried out precisely according to plan. Jean Ann Harrison Copyright 2013 12. DEFINING SKILL SET FOR THE MOBILE/EMBEDDED TESTER Some exposure or knowledge about products from the domain in which you are testing: aerospace, medical, automobile manufacturing, airplanes, factory systems, robotics, regulated environments, etc. Some knowledge of: hard sciences: math, physics, electronics, engineering, etc for logical thought process. Software sciences: psychology, philosophy, sociology, human factors (human machine interface) for creative & conceptual thought process These testers use skills from many knowledge domains, patterns of errors, and basic testing skills to create tests. They are using their intuition, critical thinking, and their mental models. Chapter 1 Software Test Attacks to Break Mobile & Embedded Devices Jean Ann Harrison Copyright 2013 13. Jean Ann Harrison Copyright 2013 14. WHAT YOU NEED TO BE A GOOD EXPLORATORY TESTER Awareness Discipline Focus Jean Ann Harrison Copyright 2013 15. KEYS TO MOBILE EXPLORATORY TESTING Look beyond the GUI Timing Domain knowledge Knowledge of System (hardware/firmware/software) Characteristic of bugs behavior, formulate patterns Use life experiences for inspiration Jean Ann Harrison Copyright 2013 16. TESTING BEYOND THE GUI = SYSTEM INTEGRATION TESTING Jean Ann Harrison Copyright 2013 SIT for Mobile/Embedded contains elements: Software + Hardware + Software Variants + Hardware Variants + Timing + Operating System Drivers + Network Protocols 17. WHAT DO WE TEST? Jean Ann Harrison Copyright 2013 18. LEARNING THE ARCHITECTURE Work closely with development to learn more about how the mobile application under test works. Jean Ann Harrison Copyright 2013 19. EXERCISE FOR GROUP In the question box, enter 1 test idea of what you want to explore. The point of this exercise is you do NOT know what the application is, so you cant tell me what kind of functional test to perform. Im testing YOU to see what kinds of non-functional tests you think might be good to explore. Jean Ann Harrison Copyright 2013 20. COMMON TOOLS FOR ORGANIZATION Sticky notes per conditions, per variables, per test case idea Keep a log of what you are doing, record conditions Utilize the devices log or find an open source logging tool Use tables to record all variables/conditions per test Document situation of the test, list out your test conditions Jean Ann Harrison Copyright 2013 21. WE BECOME MORE EXPLORATORY WHEN WE CAN'T TELL WHAT TESTS SHOULD BE RUN, IN ADVANCE OF THE TEST CYCLE, OR WHEN WE HAVEN'T YET HAD THE OPPORTUNITY TO CREATE THOSE TESTS. JAMES BACH Create your description or theory. What happens when? battery runs low? battery is charging from a dead battery? when the device gets too hot? when the battery is replaced due to defective battery? device memory is full? I receive a notification while using the phone or another application? when there is a time/date change? device searches are too slow? Jean Ann Harrison Copyright 2013 22. DOCUMENTING YOUR TEST CONDITIONS FOR THE CHARGING OF BATTERY TEST Jean Ann Harrison Copyright 2013 23. TEST CASE STEPS 1. Pre-select any application to download onto device 2. Drain your devices battery charge completely 3. Notate time 4. Plug in charging device 5. Immediately power up device 6. Record times for first condition where app is not installed, device powered up Jean Ann Harrison Copyright 2013 7. Drain your devices battery charge completely 8. Plug in charging device 9. Immediately power up device 10. Notate time 11. Start to download pre-selected app 12. Record times for each battery level while downloading 24. EXERCISE: EXPLORATION FOR MORE TESTS Jean Ann Harrison Copyright 2013 What other kinds of tests can you apply based on the condition of starting to charge from a dead battery? Record your tester stories 25. ORGANIZE YOUR THOUGHTS/TESTS Jean Ann Harrison Copyright 2013 Battery level Time/Temp Application Installing Application Installed/not in use/started App in use (note various conditions apply here) 0%-5% Time each level increment of charge Timing of download? Would notifications factor? Notifications? Memory usage CPU speed 6%-15% Check the temp at each increment Completed installation? Note CPU speed, memory usage Searches with application DB, online DB? 16%-30% Interaction with data transfer/network communication 31%-50% 26. CONTROL SYSTEM CONSIDERATIONS DEVELOPING TESTS: HARDWARE TO SOFTWARE SIGNAL INTERFACE Know the input and output connections to the software Consider differences in variations between devices like calibrations, physical noise, electron, light, cold, water, dust device, response time, wear & tear on the hardware. Identify input & output devices with ranges and resolutions of values (how is the software installed onto the device?) Define the full range of environmental input disturbances (unexpected inputs) Define possible environmental output disturbances (unexpected outputs) Determine what is or is not possible in the test lab Conduct a risk analysis of what is acceptable and what is not Understand device limitations: CPU processing capabilities, memory, time. The Aerospace Industry use the term, Test like you fly, fly like you test to combine all facets of how a devices software should be tested. Originally written by Jon D Hagar Copyright 2013 Jean Ann Harrison Copyright 2013 27. Jean Ann Harrison Copyright 2013 Download either Twitter or Facebook onto device Start either downloaded app From another device, send an email to the devices email account Immediately send a tweet or post a status Continue to do engage Twitter or Facebook app for at least 1 minute Record email notification and time when sent and received. What other observations occurred? EXERCISE: NOTIFICATION TEST 28. EXERCISE: EXPLORING MORE FOR THE INTERRUPT TEST What other kinds of interrupts could be tested? What other tests would you consider if something goes wrong? Jean Ann Harrison Copyright 2013 29. TEST CASE EXAMPLE: INTERRUPTS ON MOBILE PHONES Go to your App store and choose an application to download While the downloading is occurring, call the mobile phone Record observations with the download You may need to rely on observing a log file while implementing these tests If it fails, what kind of error recovery occurs? Jean Ann Harrison Copyright 2013 30. LETS TALK DATA AND STORAGE Mobile devices contain limited amount of storage space. Testing for mobile apps storing data locally on a mobile device means testing the boundaries of the apps design. How will developers handle the limitations? Work with your developers to understand those boundaries. If your app doesnt have a boundary set, then the app can fill up