Trading Clearing Systems Test Automation

Trading & Clearing SystemsTest Automation

Exactpro Systems

21st December 2013

STEP 1: Test Server Functionality by FIX or other standard protocol

STEP 2: ‘GUI Bypass’ Testing

STEP 5: Fully Automated GUI Testing

STEP 4: Semi Automated GUI Testing

2

Approach to Test Automation Recommended by Exactpro

Given the relative difficulty of achieving efficiencies through automation of GUI testing, Exactpro has evolved a progressive strategy towards test automation that we recommend be followed:

• This builds upon the other levels or automation, now it’s about testing the GUI itself not the underlying system

• Wherever possible exercise functionality by message injection into FIX or other standard interfaces first - it’s far easier to automate & debug

• Connect to the GUI using the best technical interface possible – the technology available makes a big difference in the payback on the automation investment

• Partial automation can also be effective, e.g. enter data with automation script and verify manually on screen or script checks database

• If the system under test has documented interfaces between the GUI and the server emulate the GUI with a test harness to exercise more of the server functionality that is available through external interfaces

STEP 6: ‘Big Button’ Framework • Progressively link all of the automated tests into a framework that allows

highly productive control over test execution & reporting across of the entire test set

STEP 3: Connect to the GUI

GUI Object API

Windows ControlsScreen Scraping

Web Tools

3

STEP 1: Test Server Functionality by FIX or other standard protocol

Sample of Test Automation Achieved Using the Approach

Exactpro’s Bespoke Test Automation Tools Suite

4

ClearTH:• Post-Trade testing tool• Verifies each stage of the DLC• Integrated schedule• Automated matrices• Can create multiple days test scenarios• Concurrent multiple tests• Integrated simulators• SWIFT ISO protocol support

MiniRobots:• Executes multithreaded java code• Complexity of test algorithms is defined by the test developer• Supports multiple client fix connections, order entry and market data via FIX• Can use GUI to iterate through sent and received messages

Dolphin:• Model-based testing of market surveillance systems

Shsha:• Post-transactional tool• Analyzes clients' activity and forecasts system response • Parses and displays logs in a user-friendly way• Parses messages and then puts each to a data base table where each column corresponds to each message field• Allows making summarized reports, etc• Easy to understand GUI

Load Injector:• Simulates multiple client connections with a specified load shape for each connection or a group of connections• Up to 75K messages / second from a single CPU core• Measures latencies in microsecond range• Performance test reports

Sailfish:• Can test Order Entry, Market Data and Post Trade connections in one test scenario• Each test scenario is independent• Allows running test scripts in any sequence• Simulation of multiple user connections• Server simulators• All messages are stored into a data base• Generates test reports

Explanation of Sailfish

5

Item Description

Exactpro Test Strategy Step(s)

STEP 1: Test Server Functionality by FIX or other standard protocol; STEP 2: ‘GUI Bypass’ Testing; STEP 4: Semi Automated GUI Testing

Testing Type Active Real-Time

Target SUT Trading Platforms and Market Data Delivery Systems

SUT Interface Back-end (typically connected to message gateways / APIs, and DBs); GUI Testing Capabilities supported via plug-ins to other tools (e.g., Selenium)

SUT Interaction Method

Message injection and capture for testing of real-time low-latency bi-directional message flows; DB queries for data verification

Protocols Extant plug-ins for Industry-standard (FIX and dialects, FAST, SWIFT, ITCH, HTTP, SOAP, etc.) and Proprietary (MIT, SAIL, HSVF, RTF, RV, Reuters, Fidessa OA, Quant House, etc.) protocols. New plug-ins for additional protocols developed by request (codecs are shared between Sail fish and Shsha)

Test Scripts Human-readable CSV files; scripts generated manually by test analysts or automatically by test script generator using results of passive testing performed by other tool (e.g., Shsha)

Test Management, Execution and Reporting

Integrated (Web front-end), allows for multiple simultaneous heterogeneous connections, consecutive execution of multiple planned scripts, test results summary and detailed test reports. REST API supports remote control of Sailfish instances. Optional Big Button framework supported

Platform requirements Low footprint cross-platform application, MySQL or other RDBMS

Primary Competitor FIX Testing Tools

Explanation of Shsha

6

Item Description



Testing Type Passive Real-Time/Batch


SUT Interface Back-end (typically connected to message gateways / APIs, and DBs); GUI Testing Capabilities not supported


Inputs and outputs monitored by means of message capture and log parsing to analyze client activity and forecast system response; DB queries for data verification; files transfer, upload, export and comparison. Captured messages can be viewed and analyzed in real-time or post-factum

Protocols Extant plug-ins for Industry-standard (FIX and dialects, FAST, SWIFT, ITCH, HTTP, SOAP, etc.) and Proprietary (MIT, SAIL, HSVF, RTF, RV, Reuters, Fidessa OA, Quant House, etc.) protocols. New plug-ins for additional protocols developed by request (codecs are shared between Sail fish and Shsha)

Test Scripts Certification tests and data reconciliation may be performed by using ordinary SQL queries. Test message traffic generated in real-time or replayed from log files by other tool (e.g., Sailfish)


Integrated (Desktop front-end), allows for multiple simultaneous heterogeneous connections, consecutive execution of multiple planned scripts, test results summary and detailed test reports. Optional Big Button framework supported

Platform requirements Low footprint cross-platform application, MySQL

Primary Competitor FIX Onboarding Tools

Explanation of ClearTH

7

Item Description



Testing Type Active Batch

Target SUT Post-Trade Platforms and Middle Office

SUT Interface Back-end (typically connected to message gateways / APIs, and DBs); GUI Testing Capabilities supported via plug-ins to other tools (e.g., Selenium)


Schedule-driven batch processing (often spanning several business days). Internal scheduler aligned with the SUT scheduler. DB queries for data verification

Protocols Extant plug-ins for Industry-standard (FIX and dialects, FAST, SWIFT, ITCH, HTTP, SOAP, FTP, etc.) and proprietary protocols. New plug-ins for additional protocols developed by request

Test Scripts Human-readable CSV files; scripts generated manually by test analysts or automatically by test script generator using transactional and reference databases


Integrated (Web front-end), allows for multiple simultaneous heterogeneous connections, concurrent execution of multiple planned scripts linked to particular global steps, test results summary and detailed test reports. Based on Big Button framework principles


Primary Competitor Custom BPM solutions

Explanation of MiniRobots

8

Item Description



Testing Type Active Multi-Participants (applicable for testing at the confluence of functional and non-functional testing)


SUT Interface Back-end (typically connected to message gateways / APIs); GUI Testing Capabilities not supported


Message injection and capture to emulate multiple participants’ activity in electronic markets (essential when there is a need to reproduce complex scenarios that can be created by trading algorithms)

Protocols Extant plug-ins for Industry-standard (FIX and dialects, etc.) and proprietary protocols. New plug-ins for additional protocols developed by request

Test Scripts Multi-threaded Java code specifying different liquidity profiles


Integrated (Web front-end), allows for multiple simultaneous heterogeneous connections, concurrent emulation of multiple participants, detailed test reports. Optional Big Button framework supported


Primary Competitor Custom market and algo trading simulation solutions

Test Automation Essential Ingredients

9

Essential Ingredients

Bespoke automation tools built up as an investment over several projects, in combination with open source tools

Highest calibre development staff on the team

Control of test environment back-end by QA and the ability to emulate external systems in order to set the required pre-conditions and increase test automation coverage

Pragmatism and determination

Experience

Principles

Low Start

Passive & Active Testing

Codecs, Drivers and Stubs

The “Big Button” Framework

High Volume Automated Testing (HiVAT)

Type of Tests

Transactional

Scheduling

Reconciliation

Participant

10

Design of the Automation Will Embody These Principles

Principle Description1. Low Start • Don’t wait for the full comprehensive test harness to be available; use simple tools as soon as they are ready

• Start with getting tools to execute with partial validation & progressively add more validations• Capture logs and executed files. Even semi-manual test execution provides better evidence and serves as the

foundation for future scripting

2. Passive & Active Testing

• Passive testing is a good way to collect test evidence from other tests and can be used to perform client certification

• Messages stored during passive testing can be used to generate active test cases and repeat previously executed activities

3. Codecs, Drivers and Stubs

• Codecs are test code capable of parsing a particular protocol or file format; using these allows increasing testing depth, localizing defects and validating the components earlier

• A Driver is a test program that can initiate the injection of messages into the system• A Stub is a test program that simulates an absent component called by the system under test• Drivers and Stubs both rely on Codecs• A Test Harness is a set of drivers and stubs for every interface link in the system• An automated test library relies on the Test Harness to execute the test scenarios

4. The “Big Button” Framework

• “Big Button” organizes large automated test libraries, so that they can be executed in an unattended mode• It is necessary to carefully design reference/test data sets and plan operational schedule of the test environment • Reduces effort required for regression test execution, but also allows having a better test library structure• The Big Button approach requires more stability from the test tools and test scripts

5. High Volume Automated Testing (HiVAT)

• Like other complex distributed real-time systems, Trading and Clearing platforms have intermittent defects that manifest themselves only if load is applied

• They can be attributed to race conditions or to rare internal integrity failures• Such defects might not be spotted in the course of ordinary functional test execution due to their low statistical

probability under low volumes• HiVAT techniques are based on automated test case generation

11

Automated Tests can Address Four Areas

Areas Description Sample of Exactpro Tool1. Transactional(Active Real-Time)

• Processes incoming requests in real-time • Set up controlled transaction data set• Execute volume of independent test rapidly• Independent test cases and is easily supportable

2. Scheduling(Active Batch)

• Tests aggregation and batches functionality • Execute scenarios lasting multiple business days• Execute many test scenarios in parallel• Internal scheduler aligned with scheduler of the system

under the test

3. Reconciliation(Passive Real-Time / Batch)

• Verify system consistency by monitoring inputs and outputs

• Agnostic to test input source• Complements active test approaches

4. Participants(Active Multi-Participants)

• Emulation of participants activity in electronic markets• Reproduce complex scenarios • Testing at the confluence of functional and non

functional testing• Verify system from participant's interfaces

Trading Clearing Systems Test Automation

Technology

test reportsshsha

test developer

test harness

test server functionality

sample of test automation

automation of gui testing

running test scripts

test execution reporting