Sgin2013 scrumfromthe trenches-loadandperformancetestsinagilescrumframework-subrahmaninamravindra

Load and Performance tests in Agile Scrum

framework

S Ravindra

S R V Subrahmaniam

Siemens Limited

An approach for complex product development context

Scrum Gathering India Regional 2013

Slide 2

Siemens

Siemens a MNC with over 400000 people in more than 150 countries

Siemens in India over 18000 people across locations

Over 5000 people in software development in India

Agile journey started in 2005

Today more than 45% of the projects following agile scrum

Basic Agile practices include: Time-boxed iterations, Continuous integration, test early & Self-organized, cross-functional teams

Slide 3

Agenda

Context

Challenges

Discussion

Solution

Further actions

Slide 4

New Product development – project context for our presentation

Enterprise grade

Not a typical web

application

Niche product

(domain / target

users / use cases)

Critical Non-

Functional

requirements

Multiple deployment

environments

Centralized communication platform for use by emergency

intervention forces

Operators in the Command center will use it for

communication with public and on-field intervention forces

Safety critical application - used in Public safety and fire

emergency situations

Platform integrates Analog fixed, ISDN, Analog radio,

digital radio (Tetra)

Integration of heterogenous technologies

Slide 5

The Product : Communication platform

Touchscreen –

Thick client

application

Linux server Web server

IP PBX Digital & Analog Radio

Database

Gateways for different types of incoming /

outgoing calls

Voice traffic carried over IP

Slide 6

Product Functionality

Client operator stations

Conference

Call forward

Make / Take calls

Send messages

Listen/talk in Radio

Hold/Unhold

Slide 7

Agenda

Context

Challenges

Discussion

Solution

Further actions

Slide 8

Uncertainty

over the release dates

Traditional waterfall methodology

Post code freeze, critical Load and Performance test defects are uncovered – which sometime require fixes

with high impact

The calendar duration for Load and Perfomance testing took 200% more time, compared to initial estimate

Technical challenges come to the fore much later in the project, when it is tested in a close to real environment

Challenging environment

Complex interplay of different technologies including Java, .NET, messaging platform over VoIP /SIP.

Dependencies with telecom

devices and equipments

Mixing audio from various streams and verifying for voice quality

Limited scope for test automation (using conventional tools) for this kind of a product

Multiple deployment environments

Technology Process

Uncertainty over Product quality. Management wanted to strengthen the tests given the criticality of the application

Slide 9

Traditional delivery model

System test

Design

Coding

Unit tests

Functional tests

Requirements

Code freeze

Critical bugs identified

Related to: Long run stability

Load and Performance Environment Heterogenous end-to-end operations

Prolongs

Slide 10

Nature of Load and Performance defects pose a challenge

Defects with high turn-around time

Occurs during long run

May require architecture fixes

Bug verification – again takes time

Fixes may have functional impact

Hard to reproduce

Defect occurrance does not follow a

pattern

Slide 11

Agenda

Context

Challenges

Discussion

Solution

Further actions

Slide 12

Agile Scrum is the right choice

Complex technology

New product & need for early

customer feedback

Positive Organizational mandate

Late testing needs to be avoided

Internal customer – less of contracting

hassles

Slide 13

Agile Scrum is fit for purpose

Niche

Generic

Simple /

Homegenous

Complex /

Heterogenous

Do

main

Technical complexity

Slide 14

Key considerations in defining a methodology within Sprint framework

Thick client

application, more susceptible to

stability issues

Used in emergency call centers. Zero tolerance for failure

Given nature of application, limited automation options

Device integrations are involved

Defect turnaround time high

Slide 15

Framing Agile testing for a Product in Safety critical environment

Sprint Test

Limited feasibility

to do within Sprint

Testing for NFRs take

time

Accomodating the factors in the Agile test framework which is to be carried out within a Sprint is the critical decision

NFRs need to be

tested at earliest

Slide 16

Sprint testing

IN Scope for Sprint Testing

User story working at a functional level

User story working at an end-to-end level

Horses for the courses

Load and Performance tests

Stability tests run over long duration in multiple environments

Complex concurrent workflows in a full load scenario

Exploratory tests

Negative tests

Production-like environment

To be handled out of Sprint

Slide 17

Testing model practiced in the project

Sprint 1 Sprint 2

Functional tested package

Sprint 3 Dedicated

test phases

NFR test

The Sprint build is taken up for long running stability, load and performance tests,

multiple enviornment tests, exploratory tests and negative tests.

Defects

NFR test

.....

Four weeks

Product

Backlog

Slide 18

Testing methodology adopted

Sprint Testing

Enable test

Monitor resource utilization patterns; Focus on tooling

Do tests where feasible – even if it is partial; Use test stubs where needed

Plan the tests which need to be run; actively collaborate with Product Owner

Bugs / tasks for forthcoming sprints

Early detection will enable certainty in product release schedules

Gather data

Perform test

Act on results

Slide 19

Test Organization

Within Scrum teams

- Testers part of the Scrum teams

- Operationally reporting to Scrum Master; Officially reporting to Test Manager

- Apprisal, Competency development taken care by Test Manager

- Testers need to be good in inter-personal skills

- Testers need to be aware of white box techniques, interpretation of logs, configuring and setting up parameters etc.,

System test team

- Testers do not belong to Scrum teams

- Testers report to Test Manager

- Work on testing the software at System level and for non-functional requirements

- Diversity in terms of technical skills is key. Each tester has to be a stand-alone expert in a chosen technical area as the breadth of testing would typically involve multiple areas

Slide 20

Other measures to foster Agile way of working

Management focus for moving towards Agile 1

Visibility of the project status and key metrics for all stakeholders 2

A possible dip in productivity initially and a rebound afterwards 3

Facilitating the transition with suitable trainings and coaching 4

Transition with new processes and tools, in tandem 5

Co-location and office layout to reflect the new work culture 6

Slide 21

Benefits accrued..1

Team

Product

Methodology

Enhanced collaboration between the development and test teams

Less stress compared to fixing critical issues, late in the project lifecycle

Formulating Test organization for different test types

Re-skilling and Competency development

Riskier aspects of the Product tested upfront

Design refactoring for performance and stability is a continuous process

NFRs are given due focus - Enhanced visibility of the NFR related test cases to development and product teams

Product is as close to being production ready

Clear separation between the different levels of testing

NFR Tests planned and executed with due rigour without getting influenced by Sprint dynamics

Will support a move towards TDD

Compliance to schedule and lesser critical bugs after code freeze

Slide 22

Defect turnaround rate improved by 20%

50% defects found early – i.e. During the Sprint testing

NFR test cycles reduced by 25%

15% less of critical and major defects

Benefits accrued..2

Slide 23

Agenda

Context

Challenges

Discussion

Solution

Further actions

Slide 24

Next improvements – to our project

Continous review

and

improvement of

tools &

techniques

Invest in developing an

automation framework at

end-to-end and

component Integration

levels

Use of White-box tools and techniques for Load and

Performance tests

Have specific user stories to improve performance

Enhance

Automation

Sprint development

Slide 25

Agenda

Context

Challenges

Discussion

Solution

Further actions

S. Ravindra ([email protected]) S R V Subrahmaniam ([email protected])

Thanks for your attention