The Mobile App Quality Challenge: How To Build Quality Into Your Mobile Apps Doron Reuveni | uTest CEO & Co-Founder | [email protected]
Jul 01, 2015
The Mobile App Quality Challenge:How To Build Quality Into Your Mobile Apps
Doron Reuveni | uTest CEO & Co-Founder | [email protected]
uTest | on-demand testing
Agenda
• The Mobile Boom
• A Strategy For Mobile App Quality
• Mobile Testing Challenges
• Crowdsourcing 101
• How Crowdsourced Testing Works
• Sneak Preview of Mobile Testing Tool
1
uTest | on-demand testing
MOBILE MARKET BOOM
2
uTest | on-demand testing
The Mobile Boom
3
• $6.2 billion to be spent in 2010 at mobile app stores
• Mobile app stores expected to exceed 4.5B downloads in 2010
• Expected to surpass $21.6 billion in 2013
• US subscribers:
204K (1985)
277MM (2009)
uTest | on-demand testing
Why Is Mobile Booming?
4
• Mobile is always on, always accessible
• Takes the web with you
Connectivity
Publishing
Sharing
• It’s smarter than traditional web
Geo-intelligent
Contextual
• Combines the best of traditional web and phones
• Continues to innovate
Laptops smart phones net books tablets
uTest | on-demand testing
Mobile Meets The Enterprise
5
• Not just for kids (or consumers)
• Mobile applications in the enterprise: Business apps: CRM, ERP and HR systems
Productivity apps: docs, spreadsheets and presentations
Collaboration apps: email, IM and publishing
• Clinical, financial and customer data being accessed
• Growing pressure for QA to ensure mobile apps work properly and are private and secure Handsets makers & models
Operating systems
Carriers
Locations
uTest | on-demand testing
STRATEGY FOR MOBILE APP QUALITY
6
uTest | on-demand testing
First Things First
7
• Mobile app quality strategy begins with a decision
uTest | on-demand testing
The Big Decision
8
• Mobile Web vs. Native Apps
uTest | on-demand testing
Mobile Web Path
9
• Argument For Single platform (the web)
One app to build, launch and maintain
All that’s needed for some companies
• Argument Against Less controlled user experience
Slower user experience
No app store distribution
Lack of standards across mobile browsers
uTest | on-demand testing
Native Apps Path
10
• Argument For Rich media functionality
Controlled user experience
Faster experience for users
App store distribution
• Argument Against Multi-platform
Must build, test and maintain multiple apps
Not necessary for some companies
uTest | on-demand testing
Alternative Paths
11
• Wholesale Applications Community (WAC) Consortium of dozens of mobile leaders
– Carriers: AT&T, China Mobile, Orange, Verizon and Sprint
– OEMs: LG, Samsung and Sony Ericsson
Goal: build app once and have it work on all devices, carriers & OS
Good theory and wonderful on paper… but will it work?
– J2EE all over again?
– Competing interests?
– Trying to be all things to all people?
• Hybrid apps Native app “shell” or “wrapper”
Shared back-end with existing web app (data, functionality, etc)
uTest | on-demand testing
Before You Begin
12
• Regardless of which path you choose, back-end work remains API / web services
Architecture
Database
Back-end functionality
• The important lesson: app quality begins before coding or testing
uTest | on-demand testing
Word To The Wise
13
• Start with your users – what do they want? Reach users on their terms (web; mobile web; native apps)
What are your users trying to access?
– Static content vs. rich content
– Commerce vs. functionality
• Key fact Consumer conversion (from surf to purchase) on iPhone or Android native
apps is 3X - 10X times that of mobile web sites
Why?
– User comfort with device
– Usability of the native app
– Page load speed
• Forget about making the mobile web; make the web mobile
uTest | on-demand testing
TOP MOBILE TESTING CHALLENGES
14
uTest | on-demand testing
1. Coverage
• Web Testing Matrix Is Tough
15
OS & Browser
Features
uTest | on-demand testing
1. Coverage
• Mobile Testing Matrix Is Insane
16
OS & Browser
Features
Handset Makers& Models
uTest | on-demand testing
1. Coverage
• Mobile Testing Matrix Is Insane
17
OS & Browser
Features
Handset Makers& Models
WirelessCarriers
uTest | on-demand testing
1. Coverage
• Mobile Testing Matrix Is Insane
18
OS & Browser
Features
Handset Makers& Models
WirelessCarriers
Location
uTest | on-demand testing
2. Managing Resources & Costs
• Managing peaks & valleys; unexpected twists & turns
• Enough to make QA leaders sick
uTest | on-demand testing
3. Time-to-Market
• Never-ending race for each new app and version
• Can exhaust the best test managers and teams
uTest | on-demand testing
4. Mobile App Quality
• Becoming impossible for QA leaders to assure quality
uTest | on-demand testing
What’s At Stake
22
• Billions of dollars and the future of apps
• Mobile app competition hotter than ever Media attention
User adoption
App store approvals
• Pressure to build, test & launch apps never been higher
• Apps device sales
uTest | on-demand testing
CROWDSOURCING DEFINED:
23
“Taking a service traditionally performed by a person or team and sourcing it to a large online group of people as an open project.”
- Jeff Howe Wired journalist & author of Crowdsourcing
uTest | on-demand testing
Crowdsourcing Across Industries
24
uTest | on-demand testing
Mob Mentality
BUT crowds often look (and act) like unruly mobs
25
uTest | on-demand testing
From Mob to Community
And mobs don’t work in every category
Example: delivering a skilled service like software testing or development requires an orderly “community” capable of consistently producing the desired results
26
uTest | on-demand testing
The challenge is to turn this –
27
uTest | on-demand testing
Into this
28
uTest | on-demand testing
HOW CROWDSOURCING WORKS IN TESTING
29
uTest | on-demand testing
Crowdsourced Testing 101
• Traditionally, companies had two options: Option 1: Build robust in-house staff & lab
– Cost prohibitive for many companies
Option 2: Outsource or offshore
– Loss of control, communication challenges and quality risks
• A third, alternative trend has emerged to complement: Option 3: Crowdsourcing
– On-demand extension of in-house testing efforts
– Coverage across location, language, OS, browser, handset or carrier
30
uTest | on-demand testing
Where Crowdsourcing Fits
• In-House vs. Offshoring
31
In-House
Pros:• More control• More visibility• App/biz knowledge
Cons:• Fixed costs• Scalability• Coverage• Higher commitment
Offshoring
Pros:• Perceived cost• Scalability• Lower commitment
Cons:• Adaptability• Diversity of testers• Transparency• Communication• Hidden costs
Crowdsourcing
Pros:• Coverage• Agility• Creativity• Total cost
Cons:• Coordination• Learning curve• Signal-to-noise ratio
uTest | on-demand testing
Where Crowdsourcing Fits
• Manual vs. Automated
32
Manual
Pros:• Agility• Creativity• Explores the edges
Cons:• Scalability• Repetitive tasks• Human error• Exhaustive coverage
Automated
Pros:• Good for mainstream UX• Scalability• Cost• Speed for repetitive tasks
Cons:• In-the-lab vs. In-the-wild• Ignores the unexpected• Limits imagination• Hinders adaptability
Crowdsourcing
uTest | on-demand testing
Where Crowdsourcing Fits
• Exploratory vs. Scripted
33
Exploratory
Pros:• Agility• Creative• Explores the edges
Cons:• Scale• Repetitive• Human Error
Scripted
Pros:• Predictability• Scalability• Control
Cons:• In-the-lab vs. In-the-wild• Ignores the unexpected• Limits testers’ creativity
Crowdsourcing
uTest | on-demand testing
Where Crowdsourcing Fits
• On-Device vs. Off-Device
34
On-Device
Pros:• Agility• Creative• Explores the edges
Cons:• Time-consuming• Cost-prohibitive• Coverage
Off-Device
Pros:• Cost• Speed• Convenience
Cons:• In-the-lab vs. In-the-wild• Limited user expertise• Detached from real world UX
Crowdsourcing
uTest | on-demand testing
How Crowdsourced Testing Works
35
Customers Testing Community
Testers paid
Specify Testing Needs- Handset Maker, Model & OS- Carrier, Location & Language
Customer approves
Selected testers invited to project
Testers report in real-time
uTest | on-demand testing
Crowdsourced Testing Benefits
36
• Crowdsourced testing benefits:
Higher quality apps: – Real-world testing;
– Invite favorite testers back
Get to market faster: – 2-3 day test cycles instead of 2-3 weeks
– 24x7 testing cycles
Contain costs: – Costs a fraction of traditional outsourced/offshore testing
– No long-term contracts or fixed costs
uTest | on-demand testing
REAL-WORLD MOBILE EXAMPLES
37
uTest | on-demand testing
Mobile Firms Who Use The Crowd
38
uTest | on-demand testing
Case Study: 1Cast
• Seattle-based mobile company
• iPhone & Android apps for news & videos Challenge:
– Launch leading apps in a multi-platform world -- with a startup team
– Between Wi-Fi, 3G, Edge, and others, 1Cast needed more hands-on testing resources than they could afford to hire
Strategy:– Using only in-house QA was unrealistic financially
– Unwilling to commit or relinquish control to an offshore testing firm
– Researched creative alternatives and discovered crowdsourced testing
Results:– Global testing coverage at a fraction of the cost of alternatives
– Specified testing requirements and built a virtual testing team
– Integrated crowdsourced testing into product development methodology to support testing of weekly builds
39
uTest | on-demand testing
Case Study: Tapulous
• Palo Alto-based mobile company
• Gaming apps for iPhone and Android Challenge:
– Ensure functionality for Tap Tap Revenge across OS, carrier, device and location ahead of a major upcoming launch
– Testing specific features, including multi-player online mode, post-game screens, offline usage and 3G connectivity
Strategy:– In-house team was too familiar with app; needed a fresh set of eyes
– Outsourcing firms were prohibitively expensive and excessive
– Crowdsourcing was an ideal blend of beta users, in-house and outsourcing
Results:– Testing coverage across all of North America; usability and functionality
– Several high-priority bugs discovered before high-profile launch
– Achieved comprehensive testing coverage in weeks, not months over a blend of devices, locations and users
40
uTest | on-demand testing
IN CONCLUSION
41
uTest | on-demand testing
Summary
• Market growing exponentially – including B2B & enterprise
• Testing mobile is more complex than web or desktop
• Top 4 mobile testing challenges Coverage
Peaks & valleys
Time-to-market
App quality
• Crowdsourcing enables coverage across complex testing matrices Geographic Locations & Languages
Handset Makers & Models
Wireless Carriers
OS & Browsers
42
uTest | on-demand testing
SNEAK PREVIEW: REMOTE MOBILE ACCESS
43
uTest | on-demand testing
Remote Mobile Access
• A story… A tester – let’s say Roy from Boston – reports a critical bug on your
Blackberry app
Your QA manager and developer review it, reproduce it and ultimately, the bug is fixed
The developer or tester wants to see the bug fix in the wild – ON the actual device that first encountered the bug
Now, you can “borrow” that tester’s device remotely
• Can also be used for Reproducing bugs on the device
Running automated diagnostics scripts on the device
44
uTest | on-demand testing
QUESTIONS?
45
Doron Reuveni | CEO | uTest, Inc
w: www.utest.com e: [email protected]
p: 1.800.445.3914