End to end testing (E2E): a unique test method! Rob Smit Senior Test Manager Consultant E2E Sogetis Second Testing Academy 29 April 2009.

End to end testing (E2E): a unique test method!

Rob SmitSenior Test Manager

Consultant E2E

Sogeti’s Second Testing Academy

29 April 2009

Agenda

1. Complex stuff, these chains

2. Facts from the research in 2008

3. What is E2E? (the theoretical picture)

4. What’s so special about E2E?

5. The E2E project model

6. Let’s make an E2E test case!

7. Some highlights from the E2E test method

8. Reference material

9. If you need us, just call us!

Complexity in chains

2006/11/06

How a storm in Germany caused

darkness in Morocco

Facts from the research in 2008

• E2E tests exceeds planning and budget by 200 – 600%

• No special test strategy for E2E

• No risk analysis for E2E

• No ownership of the chain of processes

• No knowledge how to set up an E2E test environment

• No special techniques available for E2E testing

Business processes keep growing, and as they grow, they get more complex

The theoretical picture of E2E testing

X

AB

Y

Z

A B

Interface test

E2E test

E2E test:Test a process from start to finish. Test purpose: system and process integration

Interface test:Sending data from A to B, sending data from B to A

E2E testTest type where one or more processes are tested on a uninterrupted chainof systems and platforms.

Interface testTesting connectivitybetween two systems

E2E testing is complex because the processes are complex

x

data

x

321 4

7

6

5

8

10

9

11

xbr

y

x

xu

x

z

x p

xqCAR

insurance policy

TRAVEL

insurance policy

LIVING

insurance policy

SYSTEM

STANDALONE

Life insurance

insurance policy

DAMAGE

insurance policy

STOCK

insurance policy

FIOD

CONTROL

BANK

BANKACCOUNT

FIN. ACCOUNTING

ACCOUNTING

CURRENTACCOUNT

FIN. ADM

DWH

MISx

x

FIN. ADM

GENERALLEDGER

MyWeb

INTERNET DOMAIN

Testing software was easy in those good old days…Testing a change was simply a matter of testing the system and maybe some regression in parts that didn’t change…

No more…All systems are connected, sending, receiving, processing data! Where do we begin our E2E test and where do we stop? What and where are the risks???

Risk

Risk

Risk

Risk

Risk

RiskRisk

RiskRisk

Risk

RiskRisk

Risk

Risk

RiskRisk

Risk

Risk

Risk

Risk

RiskRisk

Risk

Risk

Risk

Risk

Risk

Risk

Risk

Risk

Risk

Risk

Risk

Risk

Risk

Risk

Risk

Risk

Risk

Risk

Risk

Risk

Risk

Risk

Risk

Risk

Risk

Risk

Risk

Risk

Risk

Risk

Risk

Risk

Risk

Risk

Risk

Risk

Risk

Risk

Risk

Risk

Risk

Risk

Risk

Risk

Risk

Risk Risk

Risk

Risk

Risk

Risk

Risk

Risk

Risk

Risk

Risk

Risk

Risk

Risk

Risk

RiskRisk

Risk

RiskRisk

Risk

RiskRisk

Risk

RiskRisk

Risk

RiskRisk

Risk

RiskRisk

Risk

RiskRisk

Risk

RiskRisk

Risk

RiskRisk

Risk

Risk RiskRisk

Risk

Risk

Risk

Risk RiskRisk

Risk

Risk

Risk

Risk RiskRisk

Risk

RiskRisk

Risk RiskRisk

Risk

Risk

Risk

Risk RiskRisk

Risk

Risk

Risk

Risk RiskRisk

Risk

Risk

Risk

Risk RiskRisk

Risk

Risk

Risk

Risk RiskRisk

Risk

Risk

Risk

Risk RiskRisk

Risk

RiskRisk

Risk RiskRisk

Risk

Risk

Risk

Risk RiskRisk

Risk

Risk

Risk

Risk RiskRisk

Risk

Risk

Risk

Risk RiskRisk

Risk

Risk

Finding risks, failures and dependencies is sometimes very difficult

V

A

12

XY

W

B

3

5

Z

4

C

Finding E2E risks is sometimes very difficult. What’s happening further up the process? And do we need the impact, or is the E2E chance sufficient?

Should the risks in the secondary process be considered ? If so, and suppose risks are found, what do we do? Is the secondary process in scope? Can it be tested anyhow (is the knowledge accessible, do we have the budget and mandate?)

Special attention for the three GO/ NO GO’s! In every phase of the E2E proces, full commitment of ALL stakeholders is mandatory. Remember the Chain principle (the weakest link…)

End to end testing according TMap

First level: E2E director, responsible for the integral E2E test plan, which also contains the business case for the E2E test

Adaptive

GONO GO

GONO GO

GONO GO

Make aninventory ofprocesses

and systems

Analyze E2E- risks and determine feasibility

Determine-Scope

- Clustering-Strategy

-Infra

DesignIntegral E2E

test plan

Planning& Budgeting

Determine limitations

and measures

- Determine strategy-Planning-Budget

DesignPartial E2E test plan

Detailing data flows

Second level: E2E test manager, responsible for the partial E2E test, with it’s own test strategy, planning and budget in a E2E test plan

Setting up Infra-

structure

Build up E2E test

environment

Intake Infra-

structure

Third level: E2E test environment coordinator, responsible for the setting up and building the E2E test environment.Fourth level: E2E test specialist, responsible for the E2E test script and the test execution

Round upExecuteSpecify

logical test cases

Specify physical test

cases and initial

situations

DesignE2E script

Preparation

Example: an online store

Order in ordering system

Invoice in financial system

Letter of contents in logistics system

Work backwards (against the process flow)

Determine the IT-products that supports the business processA proces is triggered by an actor and finishes at an actor

Actor(Customer)

Actor(Customer)

Actor(Sales emp)

Actor(Shipping agent)

Example: designing a E2E test case

For example: a business process with a customer and an online store. The process starts when the customer (= actor) places his order.

Initial situation

Predicted result

Action

Input

Processing

Output

This is the first test case. This test case describes the initial situation (in which circumstances can the customer places his order?)

The next thing you must describe are the actions (the process steps) that must be executed to get the predicted result (a received order).

Finally a test case has a predicted result. In this example, the result would be a received order (in a order management system or some kind of database. Notice: predicted result = Order

Example: designing E2E test cases

An E2E test case must test the process from start to finish, using data flows, products and actors. The next product is the invoice, followed by the shipment bill.

So all test cases (new designed and found in testware) must be connected like this example. The predicted result of test case 1 + primary data (payment) is the initial situation of test case 2.

End to end test case

The E2E test case matrix

Major findings in E2E: design errors. These kind of errors are very expensive, the sooner found, the better! So start your E2E partial tests. This would be your advise ofcourse…

Using partial E2E tests and connecting the tests

Systems, processes or data flows

A B C D E F G H I J K L M N O PX O O O O O X X O O O O O O O X

Logical starting point Logical finishing point

Partial E2E test 1

Partial E2E test 3

Partial E2E test 2

Proces flow

Availability for E2E testX: not availableO: available

E2E test Start with? End with?

Partial E2E test 1 Result A Result prediction F

Partial E2E test 3Initial situation IResult prediction H

Result P

Partial E2E test 3Result prediction E Initial situation F

Result prediction I

Imagine, systems G + H are not available (in building phase for another 6 months). What to do with the E2E test? Wait until G and H are ready, or start with partial E2E testing?

E2E is almost impossible to execute as an integral test. Max. 5-8 systems are manageable, otherwise use partial E2E and connect!

00

11

1/11/1

00

22332/12/11/11/122 111/11/11/11/1

1/11/1

1/11/1

22

2/22/2

Projectenvironment

Testenvironment

Knowledge & knowledge accessibility

Organizing & Management

In project

Within reach of the department

Within reach of the organization

Out of reach of the organization

11= number of processes/ systems with complexity increasing factors

66

22

11

22= number of processes/ systems per circle

11

Feasibility model

2/22/2= number of processes/ systems with number of blocking issues

Reference material

Fortis Insurance-ASR (launching customer in 2008)

•Elapse time User Acceptance Test: 40% less

•No more incidents caused by design errors

•E2E in critical path: not happened sinceKPN Telecom (pilot started in march 2009)

•Still in design phase of the E2E test cases, but already many possible risks and design issues are foundAchmea (pilot started in march 2009)

•Clear view on risks in the chains

•Project quality already much better

Conclusion

• The business processes are increasingly complex

• Supporting IT-processes evidently grow in complexity

• The E2E test method provides the method, activities, tools and examples to handle that complexity

Thank you for your attention

• E2E test method has many new techniques

• White paper release May 2009

• Book July 2009

• English version August 2009

• Course materials available May 2009

• Commercial materials available May 2009

• Support for any E2E issue: any time!

• Contacts

• Rob Smit ([email protected])

• Rob Baarda ([email protected])

Sogeti’s 2nd Testing Academy29 April 2009www.sogeti.com