EXTENT-2016: Key Challenges and Lessons Learned from Testing a New Trading System

Key Challenges and Lessons Learned from Testing a New Trading System

The Xitaro Project

Dr Matthias Burghardt

Head of Business Development

Boerse Stuttgart

London, 22 June 2016

• About Boerse Stuttgart

• Overview of the Xitaro Project

• Ten Lessons Learned

2

Agenda

© Börse Stuttgart, 22 June 2016 Key Challenges and Lessons Learned from Testing a New Trading System

3

Where exactly is Stuttgart?

About Boerse Stuttgart

We are here!

Stuttgart is the capital and largest city of the state of Baden-Württemberg in southwest Germany.


4

What is Stuttgart known for?



5

Boerse Stuttgart is Europe‘s leading market for securitised derivatives


Source: Respective Stock Exchange. Figures based on volume of securitised derivative orders. Last updated: April 2016.

33,6%

9,7% 16,2%

9,4%

23,7%

1,8% 1,8% 0,4%

Stuttgart incl. Nordic Derivatives Exchange

Börse Frankfurt Zertifikate AG

SIX Swiss Exchange

Euronext

Borsa Italiana

BME

NASDAQ Nordic & Baltic

Others


1,500,000

500,000

1,000,000

0 2,382 20,305

1995

1,388,600

2015

1,292,471

2010 2005

535,794

70,010

2000

6

Currently more than 1.4m instruments are listed at Boerse Stuttgart This number is driven by the large amount of securitised derivatives


# of instruments of other asset classes listed at Boerse Stuttgart

Debt instruments: 12,186

Equities: 7,973

Investment fund units:

3,410

Number of securitised derivatives listed at Boerse Stuttgart


7

Boerse Stuttgart ranks among Europe’s Top 10 Stock Exchanges in terms of turnover


Source: Federation of European Securities Exchanges, London Stock Exchange, Borsa Italiana Basis: Order book turnover for whole of 2015 in all asset classes, electronic order book orderbook trades

Rank Stock Exchange Turnover (2015)

in billion €*

1 London Stock Exchange Group 2,871.5

2 Euronext 2,017.0

3 Deutsche Börse 1,628.7

4 BME (Spanish Exchanges) 1,020.1

5 SIX Swiss Exchange 999.1

6 NASDAQ Nordic & Baltic 711.1

7 Borsa Istanbul 430.1

8 Oslo Børs 135.8

9 Boerse Stuttgart Group 106.6

10 Warsaw Stock Exchange 45.6

11 CEESEG - Vienna 29.6

12 Irish Stock Exchange 19.3

13 Athens Stock Exchange 17.1

14 CEESEG – Budapest 7.2





8

Agenda


9

The Xitaro Project – replacing Boerse Stuttgart’s exchange system in a complex environment to gain flexibility and independence

About the Xitaro Project – Mission

Xitaro

Boerse Stuttgart‘s

new trading system

Powered by

Nasdaq Genium INET

XONTRO

• Almost 25 years old

• Shared host based system

• High degree of dependency

Flexible and independent

Competitive costs

State-of-the-Art

Faster innovations

Future proof

Project Xitaro


10

Five dedicated work streams took the challenge

About the Xitaro Project – Project Organisation

Joint work stream with Nasdaq BSG work stream

(Joint) Steering Committee Risk-Controlling

• Management of all stakeholder activities, e.g. trading participants, employees, boards, etc.

• Specification of all Xitaro functionalities

• interfaces, internal systems, network setup

• Solving technical problems and issues

• Preparation for the operational phase

• setup of processes, resources, infrastructure

• Cutover Preparation • Member onboarding

• Ensure software and system quality by intensive testing (automated and manual)

IT Security & Data protection PMO & Quality Assurance

Internal Audit

Project Management

Legal Advisory

Stakeholder Management

Functional Design

Techn. Design & Customizing

Setup Operations & Cutover

Management

Test and Acceptance


4 %6 %

33 %36 %

21 %

11

Accounting for one third of the total effort, Test & Acceptance was Xitaro‘s largest work stream

About the Xitaro Project – Test Effort

Joint work stream with Nasdaq BSG work stream

Stakeholder Management

Functional Design

Techn. Design & Customizing

Setup Operations & Cutover

Management

Test and Acceptance


1: System Integration Test

After a 2 years implementation project, Xitaro went live in September 2015

About the Xitaro Project – Time Line

Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2

SIT1

Detailed Design

Implementation

Q3

Member Tests

Operations

Beta Tests

Project Xitaro

2013 2014 2015

Dress Rehearsals

Hardware / Network Setup for SIT and UAT

Hardware / Network Setup For Production

Member Onboarding Member

Certification Cutover Concept

Test Preparation

Regression Tests

User Acceptance Tests

© Börse Stuttgart, 22 June 2016 Key Challenges and Lessons Learned from Testing a New Trading System 12




13

Agenda


14

Key challenges in testing cover a wide range of areas

Key Challenges

Test automation

Outsourcing

Regression Testing

Test Case Design

Beta Testing

Early Member Testing

Test environments Defect Management

Non-functional Testing


Acceptance Test Preparation

Testing / Contingency

15

You cannot separate functional design and test case design

Lesson #1 – Test Case Design

Functional Descriptions #1

Functional Descriptions # 2



Test Case Design

Test Case Design #1

Test Case Design #2

Test Case Design #3

Test Case Design #4

Functional Descriptions

Testing

Testing

Test case design requires close alignment between functional and

test experts


Day 3 Marc Tom Andi

Env 1 TC 1 TC 3

Env 2 TC 2 TC 5

Env 3 TC 7 TC 6

…

Day 2 Marc Tom Andi

Env 1 TC 1 TC 3

Env 2 TC 2 TC 5

Env 3 TC 7 TC 6

…

Day 2

16

Take one step at a time – from acceptance test specifications to the detailed acceptance test plan

Lesson #2 – Acceptance Test Preparation

Acceptance Test Specifications

Acceptance Test Procedures

Acceptance Test Plan

Acceptance Test Plan Detailed

Acceptance Test

Draft Review Update Approve

Day 1 Marc Tom Andi

Env 1 TC 1 TC 3

Env 2 TC 2 TC 5

Env 3 TC 7 TC 6

…

ATS: •TC1: •TC2: •TC3: •TC4 …

ATP: •TC1 • Step 1 • Step 2 • Step 3 • …

AT Plan: •Day 1 • TC 1 • TC 2 • TC 3 • …




Review Cycle helps aligning customer and supplier expectations

1 2 3 4


17

Choose wisely which functionality outsiders could test Outsourcing may prove difficult when testing complex functionality

Lesson #3 - Outsourcing

36% 64%

43% 57%

43% 57%

44% 56%

46% 54%

52% 48%

70% 30%

91% 9%

100%

100%

100%

Functionality 1 100%

93% 7% Functionality 15

17% 83%

22% 78%

Functional Experts Test Experts

decreasing complexity

Test cases executed by Functionalities with higher complexity e.g. Trading Halts

Functionalities with lower complexity e.g. GUI User Configuration

Overall percentage of executed test cases

53% 47%

Test Experts

Functional Experts


18

Early beta deliveries are good but you need to test them properly

Lesson #4 – Beta Testing

1: User Acceptance Test

Q1 2014 Q2 2014 Q3 2014 Q4 2014

Partly included functionalities

Completely included functionalities

21.02. 25.03. 09.05. 16.06. 08.09. 13.10.

Beta 1.1

Beta 1.2

Beta 1.3

Beta 2.1

Beta 2.2

Beta 2.3

UAT1

52 12

2

14

8

7

27

1

61

76

Test Phases

Number of functionalities

delivered

Six Beta Releases were delivered during the project covering 76 functions of varying complexity


What we should have done

Beta test effort

What we actually did

• Creating automated test cases is more time-consuming than manual test cases

• Not easy to combine several interfaces and reports into one test scenario

• Test scripts can have defects and require debugging

• Lack of flexibility when functional requirements change

• Quick and efficient test case execution

• Good for interface tests

• Allows for cost-effective frequent test execution during regression tests

• More interesting for testers

19

Automating tests frees up valuable experts – if you do it right

Lesson #5 – Test automation

Used to regression test 8 Releases before Go-Live

+ -

Test automation during the project – the pros & cons

+

+

+

-

-

- Project Xitaro test cases

Q1 2015 Q2 2015 Q3 2015

AT 1

1,000

4,000

AT 2

1,700

3,900

Regression Tests

1,800

2,500

Automated Test Cases

Manual Test Cases


20

Create different regression test sets for different purposes

Lesson #6 – Regression Testing

1 Evaluation

6,500 Test Cases

Functional Frequency

Frequency of using the functionality in operations

Damage Potential

Potential damage / impact in the case of defects in the functionality

Defect Frequency

General frequency of defect occurrence in this functionality / test case

2,200 Test Cases

4,300 Test Cases

Possibility for Automation?

2 Selection

3 Categorisation

Small regression test set

1,800

Medium regression test set

2,300

Large regression test set

4,300


Start early with member testing to avoid surprises

Lesson #7 – Early Member Testing

Q4 2014 Q1 2015 Q2 2015 Q3 2015

Early Member Tests Dress

Rehearsals

Member Onboarding

Member Readiness Checks

Simulation Period

• Pilot & Early Member connection • Participant setup • Contract management

• Free and assisted testing • End-to-end Tests • Non-functional tests

• Connection and interface tests • Tests of participant configuration • Debugging activities, both in Xitaro

and customer systems

• 4 Waves of assisted tests to validate Go-Live readiness of all connected members

• 4 weekends of Go-Live rehearsals

• One participation mandatory


Xitaro Member testing activities started about 12 months before Go-Live – early commitment was rewarded

Defect reports by members are important to help your test team

detect all errors

Structured dress rehearsals avoid

unpleasant surprises at Go-Live

Early member connectivity will give you

certainty and remove risks of delays

22

Separating test environments by purpose allows for flexibility during the project and afterwards

Lesson #8 – Test Environments

Non-Production Systems

Xitaro Simulation

(production-like

hardware)

Xitaro External Test

(virtualised

environment)

Xitaro Internal Test

(virtualised

environment)

Xitaro Configuration

(virtualised

environment)

• User Acceptance Test

• Performance testing and operational acceptance test

• Member testing activities

• Automated tests

• Regression tests

• Planned to be made available to members

• Internal test environment for internal software development and integration tests

• Temporary test environment for user configuration before Go-Live


23

Plan your non-functional tests from end to end and include all relevant business day activities

Lesson #9 – Non-Functional Tests

1 How we planned it

SLA

Non-Functional Area # Test Cases

Security 2

High availability and disaster recovery 79

Performance 34

Audit 3

Operations 20

Integration with external systems 9

2 How we should have planned it

• Analyse business day activities according to daily cycle jointly with supplier

• Detect all business relevant performance parameters

• Plan joint activities for the System Integration Test


3 What we did anyway

Xitaro

Close defect management and tracking make sure the project survives the User Acceptance Test

Lesson #10 – Defect Management

527

149

0

200

400

600

SIT Beta 2.3

Resolved incidents

+378

AT1

480

208

0

100

200

300

400

500

Created incidents

+272

AT1 SIT Beta 2.3

0

20

40

60

80

100

120

140

160

180

200

220

240

1.2. 1.3.

Trend reported vs. resolved incidents

~13.5 incidents resolved per day

~9.7 incidents created per day

Reported incidents Resolved incidents Trend reported vs. resolved incidents

15.1.

AT2 release

Average net reduction of ~4 incidents per

day during AT1

~40–50 open incidents not yet retested

End AT1

Average reduction of ~8 incidents

estimated after AT1

Resolved but not retested yet


Questions & Answers


Thank you for your attention!

Dr. Matthias Burghardt

Head of Business Development

Tel.: +49 711 222 985-593

Fax.: +49 711 222 985-91-593

Mail: [email protected]

Börse Stuttgart

Börsenstraße 4

D-70174 Stuttgart

EXTENT-2016: Key Challenges and Lessons Learned from Testing a New Trading System

Technology