< Name > DSTB Workshop - < Date > · 4 © 2016 Cognizant Introduction Thomas Klein, born 1965 • IT Professional since 1990 • QA Specialist since 1996 • Focus on

© 2016 Cognizant

QA in the age of mobile

< Name > - < Date > DSTB Workshop

© 2016 Cognizant 2

Outline

Introduction

What‘s the fuzz with mobile?

Mobile challenges for QA - and how to face them

Q&A and discussion

© 2016 Cognizant 3

Outline

Introduction



Q&A and discussion

© 2016 Cognizant 4

Introduction

Thomas Klein, born 1965

• IT Professional since 1990

• QA Specialist since 1996

• Focus on Mobility QA since 2012

QA Experience

• from Mainframe to Smartphone

• from DB to Web to Services and CMS

• from Dev to Ops

• from Plan to Execution

Mobility QA Experience

• Commerce, Retail, Wholesale

• Travel, Hospitality

• Energy, Telco, IoT

© 2016 Cognizant

Cognizant QE&A Overview

Passionate QA Professionals

30,000+ Professionals with well defined career lattice 6000+ Insurance testing professionals

QA partner to Industry leaders

500+ Clients From Fortune 1000 75% Independent QA Engagements

50+ Global Delivery Centers

Including Valladolid, Grenoble, Munich,

Glasgow, Budapest, Phoenix, Manila,

China, , Brazil, Mexico, UK, India

(multiple cities)

Joint Industry Recognition

Research & Innovation

Structured Innovation Ecosystem with

2172 Best Practices listed

Products & IP’s- 4 Trademarks, 2 Patents

Assurance | Engineering |

Consulting | Test Data |

Mobility | Analytics |

Product & IP’s

Cognizant QAHub | fastest

Smart QA

QA Services

LAUREATE - 2008, 2009, 2010,2011, 2012,2013 The Computerworld Honors Program

FSTech Compliance Project of the year Outsourcing Partnership of the year

Global QE&A Summit

15

Banking & Fin.

Services Insurance Healthcare

Travel &

Hospitality Manufacturing

Technology

Services

Information

Services Retail Communication

Media &

Entertainment

Life

Sciences

Transport

&

Logistics

Energy &

Utilities Education

Consumer

Goods

“Market Leader”

~ Decision Matrix, 2014-15

“Leaders”

~ NEAT Report, 2014

“Leaders”

~ PEAK Matrix, 2014

“Leader”

~ Magic Quadrant, 2014

Analyst Recognitions

QA Transformation

Test factory models | TMMi

Assessment | Agile

transformation | DevOps

5

© 2016 Cognizant

Cognizant Mobility QA – Snapshot

Highlights Alliances Cognizant ’s IP

Device Lab Services

500+ Clients across industries

2000+ Mobile testing experts

State of the art Mobile Lab

High Security Standards

Remote testing on real devices

Multiple carrier options and accelerators

30+ client using on Demand services

6+ Dedicated Private Clouds

Mobile Devices in lab 250+

Mobile Functional Testing

Mobile Automation Solutions

Mobile Performance Solutions

Customer Experience Testing

App Certification

Connected/IoT/Smartwatch Devices

Cognizant’s platform for E2E mobility QA

Global Footprint

* London,UK and Singapore Q1 2016

Phoenix, Arizona Grenoble France

Munich

Chennai

Sidney

2,000+ Mobile Testing Experts

Shanghai

6

http://www.google.co.in/url?sa=i&rct=j&q=&esrc=s&frm=1&source=images&cd=&cad=rja&docid=t_58HiDL2WczTM&tbnid=KfG1Hsn3vr_qpM:&ved=0CAUQjRw&url=http://www.tamo.co/&ei=y0MAU--6LciErAfCi4DACA&bvm=bv.61535280,d.bmk&psig=AFQjCNH_8dUdzf8T0AQBKaWXa-KmD2JKTA&ust=1392612677228481

© 2016 Cognizant 7

Up to the audience: Who’s attending?

© 2016 Cognizant 8

Outline

Introduction



Q&A and discussion

© 2016 Cognizant 9

Your audience is mobile. And they won’t stop.

© 2016 Cognizant 10

Hey, that‘s just for Android.

Mobile is much more work. (At least for the testers.)

© OpenSignal


Mobile landscape complexity constantly increases

© PerfectoMobile


Traditional QA struggles with Mobile.

A classic testing pyramid

© James Crisp

Note:

There‘s also Mike Cohn‘s test automation

pyramid, which looks pretty similar to this.

For more, see:

http://blog.goneopen.com/2010/08/test-

automation-pyramid-review/

http://blog.goneopen.com/2010/08/test-automation-pyramid-review/









Mobile: Flipped pyramid.

The mobile testing pyramid

© Daniel Knott


© James Crisp

More from Daniel Knott:

http://adventuresinqa.com/

But why is the flipped pyramid a problem? See next slide




The problem of the flipped pyramid: Cost.

The mobile testing pyramid

© Daniel Knott


© James Crisp

com

ple

xity,

cost

recommended volume

This means

we‘ll do fewer of the cheap tests

and more of the expensive tests?


Let‘s put the automation „icing“ on that pyramid

Now for automation,

with the manual testing on top:


Traditional QA struggles with Mobile

Now for automation,


And for mobile,


© Alister Scott


Traditional QA struggles with Mobile

Automation advocacy calls this

flipped mobile testing pyramid a

„software testing anti-pattern“

The bottom line:

- Pyramids don‘t match

- But: Both are valid!

Things aren‘t that easy sometimes.


What QA needs to keep in mind for Mobile


Mobile brought along its own ecosystem

• AppStore, GooglePlay, …(oh these ratings)

• Online Services (e.g. GoogleDrive, iCloud)

• Identification and Payment (TouchID, NFC, HCE, …)

• Wearables, Social, IoT

This ecosystem comes with its own release cycles.

So, who is in control of your mobile app/web SDLC?


And that‘s not all.

• Waterfall -> Agile

• Continuous Delivery, ~ Integration, ~Testing

• DevOps, Cloud, SaaS, Service Virtualization

• „Startup mentality“ (start early, scale fast @ low cost)

By coincidence, „Mobile“ came at a bad point in time.

Right when the IT world was changing:


Endless Testing ?

Endless possibilities.


Outline

Introduction



Q&A and discussion

© 2016 Cognizant

Quality Assurance is not enough

23

Mobile needs Digital Assurance as defined by the “CTO Triangle”

© 2016 Cognizant

Our experience from Customer’s Digital Journey - Top 6 concerns

Customer voice Cognizant Solution Approach Tools/frameworks

App Rating / User adoption Analytics Driven Testing mint Insight 1

Omni channel Customer experience (RWD) Customer Experience Testing Fastest 2

Device Scarcity Mobile cloud device lab Cognizant LaaS 3

Application Stability and NFR Testing Early NFR Test strategy 3-tier Perf/Security testing 4

On-Going App maintenance Automated Regression Test mCRAFT framework 5

Internet of Things Getting ready for next tech wave BlueLink, ART 6

24



#1 - Devices and Technologies

#2 - Patterns & Paradigms / Strategies & Approaches

#3 - Automation

#4 - Customer Experience

#5 - That omnichannel thing

#6 – Internet of Things (IoT)

© 2016 Cognizant

Test Device Selection Strategy

Overall Market Share WRT OS Version

Screen Resolution/ Form Factor

User /Demographical adoption

Manufacturer/OEM

Current Customer User Base

Secondary Device Primary Device

Multiple Platform OS Version

( E.g. iOS 7.x till 9.2 and Android 4.0

till 6.0)

Multiple Form factors

(<720px , 720px- 1024px ,>1024px )

Multiple OEM

( Manufactures/Device Models/HW

Processor )

“Market trends” to reflect user

adoption of OS/Platform and

Upcoming & Future models

#1 - Devices and Technologies What YOU can do

26

© 2016 Cognizant

Factors for Mobile Test Lab

Cognizant Recommends a Combinational Approach

Key Considerations for Infrastructure evaluation:

No. Of

Concurrent users

Team Location

( Distributed/

Co-located )

Usage of Physical

Hardware such as

Camera, Touch ID

,etc.

Testing types

(Manual testing or

automation, etc.)

Project duration

( one-off, short

term / long term )

Physical

Devices

Device Cloud

Emulator/

Simulator

#1 - Devices and Technologies What Cognizant does

27


Mobile Infrastructure Options

Infra Options

Emulator/

Simulator Physical Device Mobile Cloud

Public : On-

Demand

Private:

Cognizant

Premise

Dedicated –

Customer

Premise (OR )

Cognizant ODC

#1 - Devices and Technologies What Cognizant does

© 2016 Cognizant

#2 – Paradigms, Strategies, etc. What YOU can do

The Ground Rules:

1. Don‘t start by looking at tools !

2. Figure the „big picture“ around you (CI? Agile? App/Web? Arch/Infra, …)

3. Seek development support

Painfully learned lessons:

• If you „do agile“ : Do it correctly (get help) or stop it

• Don‘t test everything via the UI

• Don‘t make „ E2E automation on real devices“ your only testing!

• UX: Users are not inside, sitting at a desk and using WiFi

• Security is overrated (…when testing functional, of course )

• Automate early: Use stubs, mocks. („Divide and Conquer“)

• Testing is like walking on water: It works best when frozen

29

© 2016 Cognizant

#2 – Paradigms, Strategies, etc. What Cognizant does

UI Testing

Primary Devices

Secondary Devices

UI Testing Page Layout as per break point

definition Page Navigation Test Overlap/Clipping Stretched images

Business Flow Validation – Critical business Flows

Regression Testing

Business critical flows Page Navigation Performance Testing

Not Mandatory

Functional & Integration Testing

Business Flow Validation Transaction Validation Mobile Browser

Compatibility Testing Exception /Error Handling

Business Flow Validation – Critical business Flows

30

http://www.google.co.in/url?sa=i&rct=j&q=&esrc=s&frm=1&source=images&cd=&cad=rja&docid=0MKGQOZedDDF7M&tbnid=mPcat9TWjBHbhM:&ved=0CAUQjRw&url=http://brooksbell.com/blog/iterative-testing-clickipedia&ei=JP1oUuakBIaKrQfxn4HgBw&bvm=bv.55123115,d.dGI&psig=AFQjCNEHCxtYi3WwkVJWMHZyvz4xz9JTDw&ust=1382698643622379

http://www.google.co.in/url?sa=i&rct=j&q=&esrc=s&frm=1&source=images&cd=&cad=rja&docid=0MKGQOZedDDF7M&tbnid=mPcat9TWjBHbhM:&ved=0CAUQjRw&url=http://brooksbell.com/blog/iterative-testing-clickipedia&ei=JP1oUuakBIaKrQfxn4HgBw&bvm=bv.55123115,d.dGI&psig=AFQjCNEHCxtYi3WwkVJWMHZyvz4xz9JTDw&ust=1382698643622379


#3 - Automation What YOU can do

Read these for a start:

https://www.mountaingoatsoftware.com/blog/the-forgotten-layer-of-the-test-automation-pyramid

https://www.mountaingoatsoftware.com/blog/reduce-manual-test-technical-debt

Did you know? Automation can save time without even testing your SUT:

Test data, config, infrastructure, reporting, … are often low-hanging fruits.

Before picking a tool for evaluation, take all your

requirements (dev, test, delivery, infra, operations)

into a list and check how the candidates cope

Before kicking-off a pilot / evaluation phase, read:

http://developer.telerik.com/featured/unexpected-truth-ui-test-automation-pilot-projects/

Be aware that you will probably need more than 1 tool evaluation pilots











































© 2016 Cognizant

Object Based

Image Based

Open Source Licensed Cloud

#3 - Automation Tool classification

32


Tool Selection Criteria (excerpt)

Support Application built on both

Native codebase & MADP platform

Web, Native And Hybrid (HTML5)

Application Support iOS and Android Support

Object Based Automation Data Parameterization Test Data Externalization

Non Jail Broken Device Support For Latest OS Versions Cross Platform Reusability

Gestures Reporting Integration To test management

Tools

Toolset for your environment

#3 - Automation What Cognizant does

© 2016 Cognizant

#4 – Customer experience What YOU can do

CX testing will need understanding of the underlying business processes, customers’ intent across the entire customer journey across the

product life cycle, as understanding the customer is key to achieving a successful Quality Assurance strategy.

Customer Experience (CX) Management encompasses the practices & processes of tracking and managing all customer interactions with the digital product across multiple channels in this customer life cycle from product discovery to fulfillment to post sale support.

Discovery Support Feedback Action

Search, Email Marketing Web Chat, Free Trials

Post Purchase Behavior Social Channels, Web Analytics, Call Centre

Issue Resolution Self Service Apps Call Centre

Financial Transaction Retail Purchase Information access…

34

© 2016 Cognizant

CX Testing – Service Portfolio

• CX Test Strategy Definition

• Social led Requirements validation

• Current state Maturity Assessment

• Last Mile Experience

• Client Side Optimization

• End User Experience Validation

• Resilience Testing

• Crowd Testing in the Wild

• Social Listening analytics

• Sentiment Analysis

• Omni Channel Experience

• Social based App Analysis

Value of

Perception

• Crowd Testing

• Omni Channel Testing

• Responsive Web Design

• A/B testing

• Social Led UX design

• Usability testing

• Accessibility testing

#4 – Customer experience What Cognizant does

35

© 2016 Cognizant

Crowd Testing – Key Tenets

Fastest is Cognizant’s platform based model capable of delivering crowd sourced

test services & tools, cloud based device labs & test infrastructure flexibly packaged

and priced.

Pick from a variety of on-demand test services ranging from black-boxed, crowd-based customer experince testing down to performance testing your mobile offerings.


36

© 2016 Cognizant


Value

• Move Beyond Development based Functional

Testing and Transcend into Perceived Quality

through the Experience of real Customers

Address Post Launch QA

• Run App Store Analytics against current Mobile

App Presence and those of your Industry Peer

Group

Cognizant’s Value

• Synthesize CXT into Functional Validation creating

unified report to development saving time and

money

• Leverage Cognizant’s best practices for UAT plus

Functional Validation and Automation

Portfolio Addressing CXT

• Demographic Specific “In-the-Wild” Crowd Testing

• Google Play & Apple iTunes User Analytic Scouting

for Potential Problem Areas

• Integrated App SDK for Quick Issue Reporting and

Resolution

5-Star Functional

Validation

37

© 2016 Cognizant


• Functional Testing,

• Compatibility Testing,

• Mobile Specific Testing,

• Regulatory and Compliance testing ( such as w3c ,508 , etc.)

• Performance and Security Testing

• Multilingual Support Testing and

• App Pre-certification

• UAT Support

• Review App store rating & comments/Sentiment Analysis

• Continuous validation of target devices

• Beta Test Strategy ( such as OS/ Device/ Fitness trackers )

• Regression Testing ( Both Manual and Automation )

• App feature enhancement and Maintenance testing

• New device compatibility

QA Phase

• Market Analytics - Adoption Rate and Market share of previous OS versions

• Feature Categorization -User Interface, New/Modified/Obsolete Features & Functionalities, New UI themes and Layouts, Access Modes,

Performance, Security & Accessibility, Sync and Connectivity, Device/ OS specific enhancements

Factors Considered for Beta Testing

38

© 2016 Cognizant

#5 – That Omnichannel Thing What‘s it about

Perception of „channels“ needs a refresh. Also in QA.

39

© 2016 Cognizant

#5 – That Omnichannel Thing What YOU can do

„Make omni-channel a part of your test planning, design and execution“

Sounds lame, right? Right.

Reason being: QA is treated reactive and usually gets involved when things

are already too late to take an impacting influence.

Omni-Channel understanding needs to start at C-level and involves

strategists, analytics and brand management before it even reaches IT.

To prepare for it:

Ask yourself if your QA department is ready to test on laptops, desktops,

tablets, smartphones, TVs, smartwatches and wearables.

And also across more than one of the above, within a single test scenario.

40

© 2016 Cognizant

#5 – That Omnichannel Thing What Cognizant does

mCRAFT - Cognizant Reusable Automation Framework

Test Suite Repository Driver Script

Config

Handler

Page object

Handler

Data

Handler

Component Libraries Utility Libraries

Language specific Property Files

CI Based Execution

Reporting Libraries

Mobile Automation Tools Execution platforms

Application under test

Physical Devices

Error & Exception Handler

Embedded

Tool Reports Reports

41

© 2016 Cognizant

INDUSTRY USE CASES

Transact Kiosk

Biometric and

Facial Recognition

Experience Center

Beacon

iWatch, Google Wear

Beacon

iWatch, Google Wear

Health Tracker

Health Motivator Field Workforce App

Telematics

Health Kiosk Ecosystem

• Automated shelves and

coolers

• Perishable tracking

• Digital signage

• Automated store lighting Field Workforce

Assistance Preventive

Maintenance

Home Appliances

• Smart fulfillment

centers

• Fleet operations

monitoring

• Sustainability

monitoring

Usage Based Insurance

Home

Automation

Medical Devices

Remote Patient

Monitoring

Preventive

Maintenance

Digital TV/ STB

LIFE SCIENCE

INSURANCE BANKING

MANF. AND LOGISTICS OTHERS

RETAIL

#6 – Internet of Things What it‘s about

42

© 2016 Cognizant

26

Billion Connections

By 2020 – ABI

Research

83% 300 Billion Revenue

generated by IoT

product and service

supplier by 2020 –

Gartner

C-Level executives

agree that they

gained competitive

advantage from

IOT – Analysis

Mason

37 % 32 % 32 % 29 % 28 % 19 % 17 %

Energy and utilities

Healthcare and pharma

Retail

Manufacturing

Transport and logistics

Automotive

Consumer electronics

THE IOT ADOPTION 2015

Source: Gartner, Vodafone M2M barometer report 2015

#6 – Internet of Things What it‘s about

43

© 2016 Cognizant

Security Integration – Multiple

Touch Points

Customer

Experience

No

Standardization

Interoperability

Difficult to

Simulate

#6 – Internet of Things QA Challenges

44

© 2016 Cognizant

#6 – Internet of Things What YOU can do

Seek help from software developers. Whether they do Apps or backend,

they usually were provided with means to test their code by the hardware

folks. And being code writers, they usually try to use some code instead of

manual actions.

The same is valid for the hardware folks of course. They test devices too,

albeit on a firmware level maybe. They usually have backdoors to test

around the Thing‘s behaviour.

Finally, again: Devide and Conquer.

Strip off the physical interaction from the use cases and see what‘s left:

You can perhaps not simulate an entire „Thing“ and its sensors – but you

CAN mimick what the T-of-„IoT“ communicates to the I-of-„IoT“.

Very often, it‘s only about sending HTTP(s) traffic to some endpoint to have

automated tests done. If in trouble, talk to your software developers.

45

© 2016 Cognizant

Cognizant IoT QA Capabilities

Functional QA

Security

Performance

User Experience

IoT QA

Embedded QA

• API Testing

• System Integration

• Discovery/ Connectivity

Testing

• Compatibility/

Interoperability Testing

• Cloud Integration &

Validation

• Data Analytics and

Business Intelligence

• Firmware Upgrade

• Unit/ Link Test

• FAT

• Operational Acceptance Test

• Performance/ Battery

• Multi Layered Security Validation

• Device Security

• Web Service Security (API)

• Infrastructure Penetration

Testing

• Application security

(DAST/SAST)

• Capacity Planning and

Assessment

• Performance and scalability

• Performance Monitoring

• Crowd Testing

• Story board validation

• Social Listening

#6 – Internet of Things What Cognizant does

46

© 2016 Cognizant

Early Crowd

Testing

Real Time

Environment

Simulation

Business use

case and touch

point matrix

IoT Rapid QA Approach

System Integration

Testing

Automated API

Testing

User Experience

Testing

E2E Automated

Testing

PARALLEL DEV &

TESTING

EARLY

AUTOMATION EARLY

INTEGRATED TEST

REAL TIME

SIMULATION

IoT LAB SETUP CERTIFICATION

SUITES

fastest – CROWD

PLATFORM

CO

GN

IZ

AN

T

EN

AB

LER

S

Shift Left & Right & Up

COMPREHENSIVE QA APPROACH ENABLING RAPID PRODUCT DEVELOPMENT


47

© 2016 Cognizant

Cognizant

IoT QA Solutions

Extreme

Virtualization/ Simulation

Rapid shift from sheer

testing of applications in a

defined environment to

testing the experience in a

dynamic environment

#1

ART

#2

Automated Robotic

Test Platform

Automated Robotic Test

(ART) platform help

accelerate testing and

automate various smart

devices (wearable, home

automation kits, etc).

For Wearable testing

For Home automation/Security testing


48

© 2016 Cognizant

Bluelink – IoT Test Work Bench

WiFi, LAN, BLE and ZigBee Support MQTT, RESTful API, HTTP Support

12+ Sensor types including Temp, Gas, Acoustic,

RFID, BLE Smart Device

Real Time Sensor Simulation Based on Data

Sheet FEATURES

Pre-configured

Templates of

Popular Sensors

Industry Ready –

Compatible with

IBM BlueMix,

Things Works, MS

Azure

Enable Parallel

Development

and Testing

Reduces

Dependency on

Hardware

E2E Automation VA

LU

E A

DD


49

© 2016 Cognizant

RoboWear – Wearable Testing Solution

Simulation up to 8 KMPH API Driven Implementation

Local and Cloud Deployment Fore Arm and Lower Arm Simulation

FEATURES

Pre-configured

Physical Activity –

Walking, Jogging,

Running, etc

Compatible with

Apple Watch, FitBit,

Nike Fuel Band,

Moto 360 and the

likes

Pre-Configured

Human Profiles –

Athlete, Teen,

Adult, Senior

Citizen, etc

Reduces

Dependency on

Human Activity

E2E Automation VA

LU

E A

DD


50

© 2016 Cognizant

Open Source Automation Leveraging RoboWear

USER INTERFACE EXECUTION NODE OBSERVATION NODE

Automation Framework

Test Case Management

Data Parameterization

Reusable Libraries

Reporting

Configure and Start Physical Activity

Verify App Behavior 1 2 Fitness Trackers

like Nike Fuel band, Fitbit, iwatch OR

Smartphones etc..


51

© 2016 Cognizant

Cognizant IoT QA Service Offering

PREVENTIVE MAINTENANCE

MEDICAL DEVICES/REMOTE PATIENT MONITORING

HOME AUTOMATION

TELEMATICS/ FLEET MANAGEMENT

SMART APPLIANCES

DIGITAL BANKING/INSURANCE

WEARABLE/ DIGITAL HEALTH

DIGITAL WAREHOUSE/STORES

Biometric/facial recognition, touch centers, digital wallet, customer experience testing, system integration testing

Verification and validation, connectivity testing, FDA Title 21 CFR Part 3, 4 and 11 compliance

Functional testing, system integration testing, compatibility testing, customer experience testing

Discovery/Connectivity testing, compatibility testing, customer experience testing, API testing

Test work bench, field testing, CRM/backend integration, GPS test, API testing, performance, UBI testing

System integration, external interface testing, cloud testing, pattern simulation, performance

Discovery/Connectivity testing, compatibility testing, customer experience testing, performance

System integration, cloud testing, pattern simulation, performance, iBeacon, wearable, digital wallet


52

© 2016 Cognizant

[email protected]

Thank You

< Name > DSTB Workshop - < Date > · 4 © 2016 Cognizant Introduction Thomas Klein, born 1965 • IT Professional since 1990 • QA Specialist since 1996 • Focus on

Documents