Sogeti and HP Breakfast Briefing - Does Size Matter? How Technology Supports Your Process Improvement Initiative

6th May 2010

Does Size Matter?

How Technology Supports Your Process Improvement Initiative

Agenda

� Welcome & Introductions

� Who are Sogeti?

� Process Assessment Overview

� Aligning the assessment outcomes with your key business objectives

� How test tools can accelerate and support your improvements

Welcome & Introductions

� Barry Weston, Sogeti UK

� Sogeti Solutions Director

� Over 20 years of experience in software testing as a test engineer, test consultant and programme test manager

� Specialising in process assessment and process improvement

� Gary Voller, HP

� HP Software ALM Solution Consultant

� Came into HP as part of the Mercury acquisition

� Specialising in Application Software Quality and its use in the real world

Page 3

Who are Sogeti?

Page 4

USA 2,000Chicago, Cincinatti, Houston, New York, Seattle, Washington… India 1,000

Mumbai, Bangalore

Europe 17,300

France 10,000Paris, Toulouse, Lyon, Marseille…Benelux 4,500Brussels, Antwerpen, BertrangeSpain 1,100Madrid, Barcelona, Valencia…Scandinavia 1,100Stockholm, Oslo, Copenhagen, Esbo…UK & Ireland 300London, Dublin, GalwayGermany 230Hamburg, Munich, Frankfurt…Switzerland 110Geneva, Basel, Zürich

20,000+ staff spread over 200 locations in 15 countries

Page 5

Process Assessment Overview

1. Interviews performed against key areas

2. Evaluate against standardised Assessment model

3. Compare with industry benchmarks

4. Identify areas for improvement

Page 5

Stakeholder Relations

Test

Profession

Test

Management

Evaluation Against Assessment Model

# Area Managed Efficient Optimizing1 Stakeholder

Commitment1 2 3 4 1 2 3 1 2 3

2 Degree of Involvement

1 2 3 4 1 2 3 1 2

3 Test Strategy 1 2 3 4 1 2 3 1 2

4 Test Organiz. 1 2 3 4 1 2 3 4 1 2 3

5 Communication 1 2 3 4 1 2 3 1 2

6 Reporting 1 2 3 1 2 3 1 2

7 Test Process Management

1 2 3 4 1 2 3 1 2

8 Estimating and Planning

1 2 3 4 1 2 3 4 1 2 3

9 Metrics 1 2 3 1 2 3 4 1 2

10 Defect Management

1 2 3 4 1 2 3 4 1 2 3

11 TestwareManagement

1 2 3 4 1 2 3 1 2 3

12 Methodology 1 2 3 1 2 3 4 1 2

13 Tester Prof. 1 2 3 4 1 2 3 4 1 2 3

14 Test Design 1 2 3 1 2 3 4 1 2 3

15 Test Tools 1 2 3 1 2 3 4 1 2 3

16 Test Environment

1 2 3 4 1 2 3 4 1 2 3

Each test case is related to a test basis document in a transparent way

The test basis, the test object and all testware are identified in a transparent way

The test team has access to all items under testwaremanagement

The procedure by which testware is managed is explicitly laid down and known

Evaluation Against Assessment Model

# Area Managed Efficient Optimizing1 Stakeholder

Commitment1 2 3 4 1 2 3 1 2 3

2 Degree of Involvement

1 2 3 4 1 2 3 1 2

3 Test Strategy 1 2 3 4 1 2 3 1 2

4 Test Organiz. 1 2 3 4 1 2 3 4 1 2 3

5 Communication 1 2 3 4 1 2 3 1 2

6 Reporting 1 2 3 1 2 3 1 2

7 Test Process Management

1 2 3 4 1 2 3 1 2

8 Estimating and Planning

1 2 3 4 1 2 3 4 1 2 3

9 Metrics 1 2 3 1 2 3 4 1 2

10 Defect Management

1 2 3 4 1 2 3 4 1 2 3

11 TestwareManagement

1 2 3 4 1 2 3 1 2 3

12 Methodology 1 2 3 1 2 3 4 1 2

13 Tester Prof. 1 2 3 4 1 2 3 4 1 2 3

14 Test Design 1 2 3 1 2 3 4 1 2 3

15 Test Tools 1 2 3 1 2 3 4 1 2 3

16 Test Environment

1 2 3 4 1 2 3 4 1 2 3

Page 8

Compare with Industry Benchmarks

Page 8

Identify Areas for Improvement

� Primary

� Test strategy

� Secondary

� Reporting

� Test process management

� Defect management

� Testware management

� Test case design

# Area Managed Efficient Optimizing1 Stakeholder

Commitment1 2 3 4 1 2 3 1 2 3

2 Degree of Involvement

1 2 3 4 1 2 3 1 2

3 Test Strategy 1 2 3 4 1 2 3 1 2

4 Test Organiz. 1 2 3 4 1 2 3 4 1 2 3

5 Communication 1 2 3 4 1 2 3 1 2

6 Reporting 1 2 3 1 2 3 1 2

7 Test Process Management

1 2 3 4 1 2 3 1 2

8 Estimating and Planning

1 2 3 4 1 2 3 4 1 2 3

9 Metrics 1 2 3 1 2 3 4 1 2

10 Defect Management

1 2 3 4 1 2 3 4 1 2 3

11 TestwareManagement

1 2 3 4 1 2 3 1 2 3

12 Methodology 1 2 3 1 2 3 4 1 2

13 Tester Prof. 1 2 3 4 1 2 3 4 1 2 3

14 Test Design 1 2 3 1 2 3 4 1 2 3

15 Test Tools 1 2 3 1 2 3 4 1 2 3

16 Test Environment

1 2 3 4 1 2 3 4 1 2 3

Real World: Test Process Improvement

The Client• One of the first, and remains one of the largest, online gaming companies in the

world• Website spans over 30 different markets, across Europe, Australia, Canada, South

America and South Africa

The Problem

• To deliver an additional 14 projects to the business in the financial year • To enable up to an extra 30 – 40% project delivery capacity* through more

effective use of existing resources

Real World: Test Process Improvement

The Solution• TPI identified the following first phase improvements based on their assessed

current level of maturity• Implementation of integrated test toolset• Define the approach to testing third party software• Implement effective test data management• Implement specialist interest groups for test management and test design

The Business Benefit• Cost of missing deadlines

• Lost revenue• Delaying follow-on projects

• Cost of poor quality• Reputation • Lost revenue• Delays to follow-on projects

Page 12

Tool Support for Best Practice Test Processes

Primary Secondary Tertiary

� Test Management and coverage

� Requirements and defects

� Quality Center

� Functional Automation

� QTP

� SOA – Service Test

� Performance Test

� LoadRunner

� Performance Test

� Performance Center

� Security

� ASC

� QA Inspect

� Security

� ASC

� Web Inspect

� Developer

� Diagnostics

Page 13

40% of unplanned downtime is caused by application failures, costing

an average of $100k per hour for mission-critical apps

Gartner, 2008

Gartner, From Concept to Production, Software Changes and Configuration Management, April 2008

Gartner, From Concept to Production, Software Changes and Configuration Management, April 2008

Did You Know…

Page 14

100x100x Cost to repair a defect in production vs. requirements

The Cost of Poor Quality

30%30% Typical cost of testing in a development project

56%56% Number of defects introduced at the requirements phase

82%82% Amount of effort required to fix poor requirements

#1#1 The #1 leading cause of IT waste is poor defect management and rework

Page 15

How to Change the Game

Establish a quality approachand Process

Build and manage

A COE

Establish business Contracts

Automate where

appropriateInvolve keystakeholders

Page 16

Key Components for Success

• Accelerate time to value

• Reduce risk

• Lower cost

Best Practices

Implemented, configured

and integrated

Defined,repeatable and optimisedprocesses

Skilled, knowledgeable and experienced

People ProcessProduct

Identify Areas for Improvement

� Primary

� Test strategy

� Secondary

� Reporting

� Test process management

� Defect management

� Testware management

� Test case design

# Area Managed Efficient Optimizing1 Stakeholder

Commitment1 2 3 4 1 2 3 1 2 3

2 Degree of Involvement

1 2 3 4 1 2 3 1 2

3 Test Strategy 1 2 3 4 1 2 3 1 2

4 Test Organiz. 1 2 3 4 1 2 3 4 1 2 3

5 Communication 1 2 3 4 1 2 3 1 2

6 Reporting 1 2 3 1 2 3 1 2

7 Test Process Management

1 2 3 4 1 2 3 1 2

8 Estimating and Planning

1 2 3 4 1 2 3 4 1 2 3

9 Metrics 1 2 3 1 2 3 4 1 2

10 Defect Management

1 2 3 4 1 2 3 4 1 2 3

11 TestwareManagement

1 2 3 4 1 2 3 1 2 3

12 Methodology 1 2 3 1 2 3 4 1 2

13 Tester Prof. 1 2 3 4 1 2 3 4 1 2 3

14 Test Design 1 2 3 1 2 3 4 1 2 3

15 Test Tools 1 2 3 1 2 3 4 1 2 3

16 Test Environment

1 2 3 4 1 2 3 4 1 2 3

Page 18

Technology supporting different Quality Maturity Levels

Flexibility, Agility and Reduced Cost

Test Management and coverage

• Requirements• Tests• Defects• Services• Reports and traceability

Automated Testing

• Functional Testing• Service Testing

Application Security

• QAInspectPerformance Validation

• Performance Center • Diagnostics• Automated Testing • Business Process Testing

Application Security Center• WebInspect• Application Security Platform

Performance Validation

• Application Virtualisation• Test Management• Center Manager for Quality Center

• CM for Performance Center• Quality Center – Premier

Performance Validation

• LoadRunner

Page 19

Technology supporting different Quality Maturity Levels

Flexibility, Agility and Reduced Cost

• More Process Control• Higher Visibility • Higher Cost• High Maintenance• Inadequate Process Methodology

• High Availability/Throughput• Inadequate Process Methodology• Low Maintenance

• Faster Time to Value• Faster ROI• Lower Cost of Entry• Lower Risks • Modular Service Packages• Flexible Pricing Options• Leveraged Cloud Services• Leveraged Resources• Full Quality Management• Accelerated Testing

• Lower Cost • High Availability/Throughput• Inadequate Process Methodology• High Maintenance

Page 20

Excellence Drives Benefits and Value

Source: vokeStream Market Snapshot Report on Performance Center of Excellence, Feb 2009

Benefits realized since establishing a performance CoE Percent

Benefits realized since establishing a performance CoE 74%

Centralization of performance best practices 74%

Increased performance analysis skills 74%

Increased performance test coverage 71%

Increased effectiveness of tool use 66%

Increased availability of performance resources – people 60%

Identification of architectural defects 51%

Increased availability of performance resources – labs 46%

Reduction in hardware 31%

Other (see next slide) 23%

Benefit Description Typical Benefit/ROI

Cost of Testing Decrease by > 30%

Testing Cycle Time Decrease by 30%

Overall Cost of Quality Savings Reduce by at least 50%

Page 21

Excellence Drives Benefits and Value

… participants reported a substantial quantitative ROI

through the efficiencies listed above…they saw a 400%

ROI, proven multiple times; for every dollar spent …

Source: vokeStream Market Snapshot Report on Performance Center of Excellence, Feb 2009

Page 22

Build for Quality: Ensure Functionality, Performance and Security

ApplicationsStrategy

Improvement in application and business process time to market

10%

Fewer outages due to production defects

31%

Reduction in the number and duration of performance degradations

28%

Reduction in the number and duration of security-induced outages

95%

Operations

*Average customer (probable) level of improvement based on HP ROI model data from IDC

Accelerate time toservice release

Improve business continuity and availability

Ensure ongoing application security

Improve customer experience

Page 23

Real World: How to optimise your process

Situation• Developers complain that the Defects raised are not explicit enough to find the issue and fix

• Tester say that Developers were looking for excuses not to fix defects

But• How big is the problem, is it really real?• How can it be Managed?

Quantify the issue• Configure QC to count defect bounces between Dev and Test• Exception report to identify problem defects real time

Fix the issue• Improved Defect description quality

Page 24

Developers with Return Count Exceptions this Week

Developer Responsible

Page 25

Real World: How to gain visibility of your process

Situation• Mature, visible and enforced Quality Goals• With measures used at Quality Gates incl. amount of ‘open’ Defects at P1 and P2

But• How come all our Projects sail through the Quality Gates? We know there were issues …

Quantify the issue• Use a QC Trend graph to understand when a defect Priority changes

Fix the issue• Measuring changes behaviour• Trigger alerts on Priority changes of Defects• Only allow agreed changes to priority to happen at agreed windows around a Quality Gate

Page 26

Stage Gate Review – Changes to Priority

Page 27

Real World: How to enforce your process

Situation• Customer with Quality Teams and assets under strict FDA regulations

But • We can only run Tests which have been Reviewed and Agreed after that they and the results cannot be changed

Fix the issue• Using QC 10 Baseline and Version Control features• QC workflow which ensures:

• Only tests which are part of a baseline can be ran• Only the latest version of a tests which is Reviews and Agreed can be in a baseline

• When a Test is Review and Agreed all data is locked after that point, if a change is to be made a new version must be produced which will not be Reviewed and Agreed

Page 28

Real World: How to standardise your process

28

Ensure teams across organization follow best practices

• Consistent workflows enable resources to be easily re-allocated across projects based on business demand

Report at enterprise release level with common set of metrics

• Take proactive action on delayed projects

Lower administrative costs to maintain best practices• Point and click interface eliminates administration work and maintenance

SAP Phase 1

New initiative

Web 2.0

Manage quality processes across multiple projects

StandardProcess

Premier Edition

Page 29

So… Does Size Matter?

Yes!** But it depends on what you are measuring

Receive a FREE ½ day of TPI® NEXT Consultancy if you book before 30th June 2010!(Value: £750)*

*Offer valid for the first 15 organisations that register an interest

Register your interest with Harreet Khara

Page 30

Q&A

Page 31

And finally…

� Thank you

� Questions? For further information, please contact:

Barry WestonSogeti UK | Software Testing+44 (0)7976 877 113 [email protected]

Gary VollerHP Software | Quality Management

+44 (0)7917 093 [email protected]