Learn Key Insights from The State of Web Application Testing Research Report

© 2017 Sencha Inc. • CONFIDENTIAL •

Learn Key Insights from The State

of Web Application Testing

Research ReportDiane Hagglund, Senior Research Analyst at Dimensional Research

Kristin Brennan, Vice President of Marketing at Sencha

Jon Jarboe, Technical Marketing Manager at Sencha

The State of Web Application TestingA Survey of QA and Development Professionals

Research Goal The goal of this survey was to capture data on current

trends in web application quality.

Goals and Methodology

Methodology An online survey was sent to global databases of web

technology professionals. Questions were asked on a

wide range of subjects including current quality, impact of

quality issues, quality processes, future of quality

initiatives, and more. The incentive for participant was a

copy of the final report.

Participants 1,011 individuals with responsibility for QA or

development participated in the survey, representing a

wide range of roles, geographies, and company size.

Participant Demographics

Less than 10 employees

23%

10 - 100 employees

30%

100 - 1,000 employees

26%

More than 1,000 employees

21%

Company Size

North or South America

41%

Europe, Middle East, or Africa

36%

Asia Pacific 23%

Region

46%

47%

68%

0% 10% 20% 30% 40% 50% 60% 70% 80%

B2C (Consumer)

Internal (employee or partner use)

B2B (Business)

Type of Application End User

14%

34%

36%

40%

85%

0% 20% 40% 60% 80% 100%

Business stakeholder

Support

Vision and strategy

Quality assurance

Development

Responsibilities

Web application quality is critical

How important is the

quality of the apps you

are responsible for?

Critically important 34%

Very important 50%

Important 15%

Not important 1%

Quality problems can have dire consequences

What are the potential

business impacts of quality

problems with your apps?

1%

3%

30%

43%

51%

56%

59%

79%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90%

Quality problems do not have a business impact

Other

Miss time to market advantage

Lost revenue

Impact employee efficiency

Incur costs to fix problems

Damage reputation

Impact customer satisfaction

Quality issues are the norm – not an exception

In general, how frequently do

business-impacting quality

problems occur with the apps

you have responsibility for?

13% 15% 25% 34% 4% 7%2%

0% 20% 40% 60% 80% 100%

It is an ongoing problem

Weekly

Monthly

A few times a year

Yearly

Less frequently than yearly

Never

Good QA is hard

What challenges does your

team face in doing adequate

quality assurance to prevent

critical issues?

6%

7%

21%

22%

22%

40%

45%

55%

0% 10% 20% 30% 40% 50% 60%

We don’t face any QA challenges

Other

Too easy to ignore defects found in testing

Development and QA teams are siloed and don’t communicate well

Quality is not taken seriously

Testing is not integrated into the workflow

Aren’t able to sufficiently test across all browsers and devices needed

Schedule pressures don’t permit adequate testing

When development schedules slip, testing suffers

Are planned testing

efforts ever compressed

because development

schedules slip?

Yes, always 30%

Sometimes, but not always

56%

No, never 14%

Test automation practices lack maturity

How would you

characterize your team’s

current test automation

capabilities?

8% 17% 30% 25% 21%

0% 20% 40% 60% 80% 100%

We have a fully mature testautomation practice that hasautomated everything possible

We do a great job with testautomation, but we could do abit more

We have automated some ofthe most important tests buthave significant room to domore automationWe do minimal automation

We do not do any testautomation

Hiring good test automation engineers is difficult

In your experience, which

type of professional is

harder to hire? Test automation engineer

61%

Software developer

39%

Cultural issues may impact quality

If a frontline development

or QA professional

discovered a potentially

business-impacting issue

the day before a major

release, is there a process

to flag an urgent issue

and stop the launch?

Yes, we have a clear process

52%In theory we

have a process, but it’s not clear how it

actually works 30%

No, there is no process

18%

61%56%

48%

22%

30%34%

0%

10%

20%

30%

40%

50%

60%

70%

Executive Manager Individualcontributor

By Job Level

Yes, we have a clearprocess

In theory we have a process, but it’s not clear how it actually works

Employees are rarely held responsible for issues

In the past two years,

what has been the result

of business-impacting

quality issues on the

careers of team members?

44%

6%

15%

15%

18%

32%

0% 10% 20% 30% 40% 50%

No team members have had their careersimpacted because of quality issues

Other

Bonuses were not paid

Individuals responsible lost their jobs

Individuals responsible were demoted or notadvanced

Negative performance reviews were given

Compensation rarely tied to quality

Is your compensation

impacted based on the

quality of the applications

your team delivers (i.e.

bonus is paid or held

back)?

Yes 34%

No 66%

46%

33%

31%

54%

67%

69%

0% 20% 40% 60% 80% 100%

Executive

Manager

Individual contributor

By Job Level

Yes

No

Organizations are investing in web app quality

How do you expect your

team’s testing investment

will change in the coming

year?

29% 44% 23% 4%

0% 20% 40% 60% 80% 100%

Increase significantly

Increase somewhat

No change

Decrease

Many factors drive testing investment

Why will your team

increase investment in

testing in the coming

year?

n = will increase testing investment

29%

30%

35%

36%

38%

40%

41%

42%

0% 5% 10% 15% 20% 25% 30% 35% 40% 45%

Create competitive differentiation

Business stakeholders are emphasizing thevalue of higher quality

Reduce overall costs

Response to a business-impacting quality issue

Need to address customer satisfaction issues

Just part of our overall development investmentincrease

Need to improve time-to-market and bettermeet release schedules

End users are demanding higher quality

Addressing Today’s Web Application Quality

Challenges


Where Are You on the Test Automation Maturity Model?

CONFIDENTIAL • Sencha Inc. ©2017

Test Automation Benefits

19


Automated Testing Savings


Automated Regression Testing Savings

Invest in Tools to Help Deliver High-Quality Web Apps

Sencha Test: Unit and Functional Testing Ext JS Apps

• Complete, cohesive, integrated testing

platform for Ext JS apps – incorporates

years of experience

• Deepest integration into Ext JS for

faster test creation & highest accuracy

• Execute tests on multiple browsers

simultaneously

• Works seamlessly with existing tests

• WebDriver integration enables faster

testing of multi-page apps – mimics

user behavior

22

Sencha Test Components

23

Test Studio

Test Authoring

Test Runner

Test Execution

Test CLI

Test Automation

Test Archiver

Results Reporting

And Archiving

Test Authoring: Sencha Test Studio

• Write unit and end-to-end functional tests in

JavaScript (Jasmine)

• Create tests with built-in test editor or in a

separate IDE – store them in SCM

• Quickly write & execute tests while coding

• Leverage pre-authored tests for faster

results

• Advanced APIs improve testing efficiency

by minimizing the lines of code in a test

24

Test Authoring: Targeting Elements

• Any testing tool must allow testers to select elements upon which to act

• Sencha Test helps you avoid brittle selectors and dynamic IDs

• These ”locator strategies” improve effectiveness and reduce maintenance

• Consistently target the right element through:

• Ext JS Component query

• DOM query (XPATH or element ID)

• A combination of the above

• Create powerful page objects that maintain locators in a central location for

consistency and easier maintenance

25

Test Authoring: Event Recorder and Inspect

• Automatically records events while

interacting with the system under

test

• Even as you navigate multi-page

apps

• Generate tests in Chrome and

Microsoft Edge, execute on any

browser

Event Recorder

26

• Helps you easily locate components

on the app under test

• Choose the best locator from a list

of all possible options

• Automatically insert API code into

your test or page object, using that

locator

Inspect

Test Authoring: External Libraries

• Leverage external libraries in test code

• Connect to databases

• Communicate with web services

• Consolidate common code

• Use like you would in any other JavaScript

code (e.g. require)

• Works for local and remote (WebDriver)

scenarios

27

NoSQL

Web

Services

Test Execution: Test Runner

28

• Run selected tests on any or all browsers,

local machine or browser farm

• Dramatically reduce testing time by

executing tests simultaneously on

multiple browsers through integration with

leading browser farms such as Sauce

Labs and BrowserStack

Application Lifecycle Management: JIRA

• Reduce the effort of tracking test failures

and defects in JIRA

• Manage tests, results and defects in Sencha

Studio

• Failed tests may be marked as defects,

easily exported to specified JIRA project

• Sencha Studio tracks status of bugs to

minimize duplicated reports

29

Sencha Test

Test Automation: Sencha Test CLI (Command Line Interface)

• Maximize the efficiency of your testing

through automated test runs

• Launch tests within the CI system as soon as

application changes and tests are checked

into the source control repository

• Out of the box integration with TeamCity and

Jenkins

30

code

testreview

Test Execution: Test Archiver

31

• Review results from automated and

manual test runs

• Summary level results & drill into the

detail of failed tests

• Out of the box integration with Istanbul

enables users to identify and rectify

code coverage gaps

• View % coverage at the Statements,

Branches, Functions and Lines levels

Visual Screen Comparison: Test Archiver

32

• Minimize the time spent

verifying images on hundreds

of screens

• Review images that are

captured from previous test

runs and compare to each

subsequent test run

• Specify screen sizes to

improve accuracy of results

Sencha Test System Overview

33

Sencha Test Studio

Sencha Test Archiver

Sencha Test CLI

Browser Farm

CI Server

Local Workspace

Local Browsers

App Server

PROXY

PROXY

MANAGES

USES

USES

USES

STORES

RETRIEVES

Demo


Long-term Sustainment

• 1+ Months

Test Development

• 1-6 Months

Test Infrastructure

Setup

• 1-3 Weeks

Initial POC

• 1-3 Weeks

Assessment

• 1-2 Weeks

35

Sencha Test Automation


Benefits of Sencha Test Services

• Reduction in cycle time (dev or maintenance) for release to production

• Increase in code quality that can correlate to increased customer satisfaction,

competitive differentiation, reduction in support costs

• Implementing a Test Strategy that supports the business goals that aids in the

management team running the business. Supporting their business metrics

• Increased use of automated testing for reducing cost and cycle time

• Ability to support the range of device and OS combinations necessary in today’s

world

• Extend the useful life of the application – protect the large investment in developing

the app in the first place

36

Final Recommendation: Drive Accountability

• What gets measured/rewarded gets done

• Ensure entire team is held accountable for app quality

• Quality is not just the job of the QA team

37

Q&A

• Try Sencha Test for Free: https://www.sencha.com/products/test/evaluate/

• Contact us for help with your test strategy and automation needs:

https://www.sencha.com/company/contact/

38

Thank You!

Learn Key Insights from The State of Web Application Testing Research Report

Technology