HUMAN TESTING OF MOBILE APPS FOR VISTACON 2013 CONFERENCE 25 APRIL 2013 JULIAN HARTY Contact me: [email protected] Rev: 29 April 2013 Creative Commons.

HUMAN TESTING OF MOBILE APPSFOR VISTACON 2013 CONFERENCE

25 APRIL 2013

JULIAN HARTYContact me: [email protected]: 29 April 2013

Creative Commons LicenseHow to design your mobile apps by Julian Harty is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.

http://creativecommons.org/licenses/by-sa/3.0/deed.en_US

AUTOMATION

DRIVE FOR EFFICIENCY

STATE OF AFFAIRS

The industry is driving towards automated tests

• ‘Manual testing’ is downgraded

• Agile projects need automated tests to provide fast & frequent feedback

• Development environments for Mobile Apps (SDKs) include Test Automation frameworks

AUTOMATED TESTS

Unit Tests

Continuous Builds

UI Test Automation

Scale out to run tests on multiple devices

• Virtual devices, remote devices, local devices

Complex, sometimes fragile, ecosystems

• Challenges: device security models, limitations in frameworks, connectivity, immature domain, lack of support for multi-modal interfaces, etc.

PEOPLE NEED PEOPLE TOO

HUMAN-CENTRIC TESTING

BEYOND THE REACH OF AUTOMATED TESTSTest frameworks focus on code, API, or GUI,

Not:• Movement: acceleration, rotation, location

• Tactile, Haptic, audio Inputs or Outputs

• Aesthetics: look & feel, transitions

• User Experience (UX)

• Highly dynamic updates: video, audio output

• Complex inputs: Gestures, multi-touch, shaking

• Light: e.g. camera as an input device

So how do we test these?

QOX

Quality Of Experience (QOX)

From the standpoint of a user’s holistic experience of using the mobile app.

Essentially subjective

http://en.wikipedia.org/wiki/Quality_of_experience

UX Research, Evaluation, and Testing: strategies, tips and tools Mark J. Williams

I’VE GOT FEELINGS TOO

6 hats of software testing

• The ‘red’ hat – emotions, intuitions, feelings

I've Got a Feeling: Emotions in Testinghttp://www.stickyminds.com/s.asp?F=S17922_COL_2

Here's an article I wrote a few years ago on the six hats of software testing http://archive.newsweaver.com/qualtech/newsweaver.ie/qualtech/e_article000666359.html and if you really have lots of time on your hands, here's a video of a talk I gave on the same topic at StarWest 2008 http://www.stickyminds.com/Media/Video/Detail.aspx?WebPage=117

“An inconsistency between the product and the document might be surprising, frustrating, or annoying to that person, but what really matters is inconsistency between the product and the desire.”

WHAT DO WE NEED TO DO?

SO…

FAST, EFFICIENT INTERACTIVE TESTING

Bugfests

Move and rotate the phone

Be mobile

Bugfest: A group of people get together to test in parallel and in common. Developers on-hand for direct feedback.

LUNCH OUTTESTING A VIDEO APP• Around 2 hours

• Around 10 testers

• Test the app while going out for lunch

Examples of what the testers did

• Tourist trip on the top of a double-decker bus

• London Underground

• Out for a walk with sandwiches

ESTABLISH STRONG FOUNDATIONS

Learn:

• The platforms, the norms, what users expect

• How the platform was designed to be used

• The application lifecycle

• The devices

• Common, and unusual problems

INSIGHTFUL TESTINGRAPID software

• Heuristics

• Mnemonics to guide the testing

Session-Based Test Management techniques

• Charter

• Time-boxed

• Simple, lightweight test reporting

• Debrief

http://moolya.com/blog/2012/04/18/test-mobile-applications-with-cop-who-flung-gun/

EXTENDED TESTING

EXTENDED TESTING

Remote devices

Remote testing

REMOTE TESTERS

Credit: "Ken Banks, kiwanja.net"

http://www.kiwanja.net/gallery/texting/kiwanja_kenya_texting_17.jpghttp://qz.com/49916/indian-smartphone-users-are-gobbling-up-mobile-data/http://www.imedicalapps.com/2012/09/researchers-smartphone-spirometer-mobile-app/

CHALLENGES OF EXTENDED TESTING

Loss of information

• Sensory deprivation

• Missing or incomplete data

Loss of fidelity

• Lags, delays

Loss of control

How do we test rich interactive apps unless you have the device in your hand?

“In theory, theory and practice are the same.

In practice, they are not.” A device in your hand is worth 2 in the cloud, and 100 virtual devices.

MORE CONTEXT

MORE CONTEXT

Additional considerations

• Cultural norms & differences

• Generations

• What’s important to them?

• How can we empathize?

Mobile Apps & Platforms

InternationalC

ultu

re?

Cu

lture

?

Culture?

GSMA REVEALS FEARS OVER MOBILE PRIVACY ARE HOLDING BACK THE GROWTH OF MOBILE APPS IN LATIN AMERICA

http://www.gsma.com/newsroom/gsma-reveals

A South Korean woman using her mobile phone to buy a product at the virtual retail shop in Seoul.www.guardian.co.uk

EXAMPLES OF WHAT CAN GO WRONG“What this means is that unlike the iPhone, which is most certainly the closest competitor on the market to this phone, the Storm's UI is not custom built for touch navigation -- touch navigation is added after the fact. Things which flow naturally on an iPhone -- flicking through lists, scrolling for a contact, moving around in a webpage or looking through photos -- feel inelegant and uncomfortable on the Storm. There's no inertia to movement, no assurance that your finger is the lynchpin to control of the device. The screen is sensitive enough, surely, but how its software reacts to those touches makes all the difference, and here the feeling is that you're never completely in charge of the phone.”

http://www.engadget.com/2008/11/19/blackberry-storm-review/Image: http://worldwide.blackberry.com/gettingstarted/storm-9500.png

! XOR[*]

PRACTICAL OUTCOMES

[*] Not Exclusive OR

COMBINING STRENGTHS

Automated Tests

• Fast feedback, technology and developer centric

Interactive Tests

• Exploratory, rich interactions and dynamics

• Remote testing e.g. in other countries

Combined

• Augmented, Cyborg testing e.g. automated preparation, data collection, etc.

Learn to delegate & learn when to take control

SOME FURTHER READINGS

http://www.wipconnector.com/download/GuideToTheParallelUniverse_3rdEdition.pdf

http://www.enough.de/fileadmin/uploads/dev_guide_pdfs/Guide_12thEdition_WEB.pdf

Testing and Test Automation for Mobile Apps

Julian Harty

Summer 2013CRC Press

Development Marketing Testing(2009) Testing(2013)

Q & A ?

IF YOU WANT THE DRAFT BOOK

Draft book available for review & feedback

Email me [email protected]

• Agree not to share or distribute

• Comment to get the next available draft