Tech Talks_25.04.15_Session 1_Balazs Kollar FIX_QFJ

1CONFIDENTIAL

FIX with QuickFIX/J and

EPAM FIX

Balázs Kollár

APRIL 25, 2015

2CONFIDENTIAL 2

About EPAM Systems, Inc

EPAM

Systems

1993 founded, US-based

Public (NYSE: EPAM)

B2BITS© EPAM's Capital Markets

Competency Center delivering a

broad range of solutions and

consulting services in Capital

Markets

Development

Centers

Belarus

Ukraine

Russia

Hungary

Kazakhstan

Poland, Bulgaria

Headcount >10000 (engineers)

Products:

FIX engines

FIX servers

Solutions for Direct Exchange Access

OMS & Matching engines

Market data feeds adaptors

FIX Testing & Operational Tools

Services:

Consulting

Integration

FIX on-boarding services

Cloud services

3CONFIDENTIAL

Table of contents

What is FIX protocol1

Systems utilizing FIX2

FIX messages, types3

Transports, threading4

Advantages & drawbacks of QFJ5

Benefits of EPAM FIX5

4CONFIDENTIAL

• Financial Information Exchange, in use since 1992

• Covers pre-trade, trade, post-trade communication

• Much more than another markup language:

• It’s types, value sets give a comprehensive business dictionary

• Standard names make understanding between parties easier

What is FIX protocol

5CONFIDENTIAL

SYSTEMS UTILIZING FIX

6CONFIDENTIAL

1. Enter, change, cancel orders

2. Order event processing

3. Provide order state and history

4. Get a list of orders (open, done)

OMS: Order Lifecycle Management

7CONFIDENTIAL

1. Clients subscribe to Market Data updates

– Clients need to know prices to submit orders

– Machines vs humans

2. Quote engines

– Clients request a “price offer” before submitting quoted orders

MarketData Streaming

8CONFIDENTIAL

1. Reconciliation modules get a notification of each trade. Later they check if all trades

arrived properly to the middle office.

2. Post-trade enrichment modules

– Client Identifying Data

– fees

3. Booking, allocation: splitting trades

4. Position and risk management

– How much USD I owe to the market? What amount I need to hedge?

Post-trade

9CONFIDENTIAL

Questions?

10CONFIDENTIAL

FIX MESSAGE TYPES

11CONFIDENTIAL

• A client would like to see the actual prices

• Market Data Request ->

– initiate subscription for stream of market data of an instrument

• <- Market Data (Snapshot/Incremental)

– Bid/Ask best price and amount: top-of-the book

– May be a full order book: best 5-10 prices and amounts

Pre-trade: Market Data

12CONFIDENTIAL

„A quote is an offer with validity. The sell side is obliged to hold this offer.”

1. Quote request: before entering an order clients can request an „offer”.

2. Quote: an offer containing price and amount, that’s valid until a certain time.

– Example: give me buy price for 2m$. Quote time is shown with progress bar.

Pre-trade: Quoting Orders

13CONFIDENTIAL

• New Order Single

– Side, Instrument, quantity, quantity unit: USD/EUR BUY 1m$

– OrdType: market, limit, stop: Buy @1.11 or better

– Price, StopPx

– TimeInForce

• IOC

• resting: day, good-till-cancel, good-till-date

• Detailed explanation: @Investopedia.com

Order entry: Single-leg (35=D)

14CONFIDENTIAL

• New Order Single

– Effective Time

– Expire Time

– Settlement Type/Date: T+2, T+3 days

– Parties

• Originating Firm

• Trader ID

• Account Nr

Order entry: Single-leg (35=D)

15CONFIDENTIAL

One message type for many purposes

1. Confirmation / Rejection

2. Replaced

3. Cancelled (may be unsolicited)

4. Expired

5. Filled: partial fill or last fill

6. DoneForDay: daily stats at end of the trading day

Order event notifications: ExecutionReport (35=8)

16CONFIDENTIAL

Notification attributes

Filled-, Open-, Cumulative quantity

Filled price

Avarage Price

Order event notifications: ExecutionReport

17CONFIDENTIAL

1. OrderCancelReplaceRequest (35=G)

– Not amendable: instrument, side, quantity unit

2. OrderCancelRequest (35=F)

– Only the open quantity can be canceled, the executed quantity can not

3. Both can be rejected: OrderCancelReject (35=9)

– Nothing happened: the original order remains active

Order amendment and cancellation

19CONFIDENTIAL

Questions?

20CONFIDENTIAL

SOCKET TRANSPORT,

REPLAYING

21CONFIDENTIAL

• Traditional Server-Client communication over sockets

• The server is called Acceptor

• The client is called Initiator

• The Sessions are preconfigured on the server:

• SenderCompID, TargetCompID

• Sequence numbers: both sides has a counter, always increasing

Socket transport

22CONFIDENTIAL

Socket transport

•ASCII, separator 0x00 or SOH

23CONFIDENTIAL

• Session acts like a queue on both sides: store and forward technique

• While offline both parties may produce messages, especially the one producing Executions

• Orders can be rejected while offline or delivered later

• After reconnection the offline messages will be delivered

Offline messages

24CONFIDENTIAL

1. When session was lost, the last SeqNr seen from the Acceptor side was 1000

2. While offline 500 trades are executed on the Acceptor

3. Initiator logs in to the Acceptor -> gets a login confirmation with SeqNr=1501

4. Now the FIX library knows that it missed 500 messages!

5. Initiator sends ResendRequest from 1001 to 1500

6. Messages >1500 will be queued in the FIX stack until the replay is finished.

• You may want to reject any new, replace, cancel requests until eveything is back in sync!

How message replaying works

25CONFIDENTIAL

Questions?

26CONFIDENTIAL

OTHER TRANSPORTS

27CONFIDENTIAL

1. Not plain text anymore, don’t need socket connection

2. Parsing, binding is for you to do: SAX, JAXB, JibX

3. You can send it via any transport: message queues, HTTP

4. QFJ transports it via 212/213 tags (XmlData) but still over Socket transport

5. No other support in QFJ, won’t bind it to it’s model

FIXML

28CONFIDENTIAL

1. Not supported by QuickFIX/J

2. You can use Apache Camel’s QuickFIX/J endpoints and route the messages to any other

transport: JMS, HTTP, file, etc.

3. Or you can use FIXML and generate the java model from XSD

FIX over tunnels

29CONFIDENTIAL

Questions?

30CONFIDENTIAL

THREADING

31CONFIDENTIAL

• QuickFIX/J does transport with async networking

• The thread you’re sending from (Session.send()) won’t deal with networking

• Always check the return value of send(), it may return false and won’t throw an exception

if session is not logged in

Threading: sending

32CONFIDENTIAL

1. You’ll receive messages on the QuickFIX/J listener thread

2. Dispatch all work to another thread if possible

– Throwing an exception on the listener thread you can break QFJ, trigger resending, etc.

3. Single thread or

4. Sticky thread assignment: Given a thread pool for processing. Make sure events of an order

are always processed on the same thread.

Threading: receiving

33CONFIDENTIAL

Questions?

34CONFIDENTIAL

MEMORY

35CONFIDENTIAL

1. Lots of wrappers, even around primitives

2. Produces lots of garbage

QFJ model

36CONFIDENTIAL

1. Replace FIXMessageEncoder

– Write an adaptor to read your internal model directly

2. Replace FIXMessageDecoder

– Bind to your model while parsing

Advanced: skipping QFJ model

37CONFIDENTIAL

Questions?

38CONFIDENTIAL

PROS/CONS FOR QFJ

39CONFIDENTIAL

1. Free and open source.

2. You can extend it.

3. Well tested. Not too many bugs faced.

4. We use it in prod for years now.

PROs

40CONFIDENTIAL

1. Memory footprint, GC overhead

2. Over-a-day scheduling not supported

– Early start on Sunday

3. HA limitations: only file, jdbc and sleepycat

– no cache-based message store, but easy to write one

4. Parser: no annotation-based binding to custom model

– But extensible: you can implement your own parser

CONs

41CONFIDENTIAL

1. Defalt parser creates lots of objects

2. Replay handling is hidden

– Hard to wait until replay is finished. You can add a listener.

3. No monotoring admin UI

– Just a simple JMX interface

4. No synchronuous sending.

CONs

42CONFIDENTIAL

EPAM FIX

43CONFIDENTIAL

EPAM FIX

44CONFIDENTIAL

FIX ANTENNA JAVA ENGINE API WORKING WITH MESSAGES

• Generic model to work with the abstract FIXMessage

class via fields and groups getters and setters

• Provides the highest performance

• Each FIX message type is a class with FIX fields as

members

• Use of intelligence to make work with FIX business

object more pleasant

• Message template to increase speed of sending

messages of the same structure but with different

values

• Allows working with values as with primitive type to

reduce garbage production

CREATE MESSAGE SINGLETON BY FIX

PROTOCOL VERSION &

MESSAGE TYPE

CREATE MESSAGE FROM RAW FIX STRING (PARSE)

SERIALIZE FIX OBJECT MESSAGE

TO RAW FIX STRING

ADD, REMOVE, MODIFY FIX FIELDS AND REPEATING

GROUPS

VALIDATE MESSAGE

API TO WORK

WITH FIX

MESSAGES

FIX FLAT MESSAGE MODEL

FIX OBJECT MODEL

PREPARED MESSAGES

EFFICIENT GETTERS AND SETTERS

45CONFIDENTIAL

FIX Antenna Java Low-Latency Design

1. Zero-GC design for FIX message parse/send/receive paths

2. Avoid the use of garbage producing calls in standard java library

3. Code instrumented GC-free regression testing of new releases

4. Use of pre-allocated object pools

5. Use of Java off-heap memory for data buffers

6. Custom data structures packed in byte / integer arrays (no standard java collections)

7. Use of memory mapped files in persistence layer

46CONFIDENTIAL

FIX ANTENNA JAVA INTEGRATION

• Fully functional applicationsupporting all features ofnew MDP3.0 platform

• Functional core to support FAST

• Set of libraries for FIX FIXML conversion

•Links FIX Antenna with the Java Message Service

• Contains a server for routing FIX messages to and from JMS server

JMS ADAPTOR

FIXML CONVERTOR

CME MDP 3.0

ADAPTOR

FAST ENGINE

Set of adapters extending FIX Antenna Java functionality

47CONFIDENTIAL

FIX ANTENNA JAVA THREADING MODEL

• Send in the same thread where FIXSession.sendMessagewas called

• No additional delays for threads communication (better latency)

• No blocking on sending (till queue has free space)

• FIX Antenna can send messages in batches (better throughput)

• Current queue size is accessible for analysis

SYNCHRONOUS SENDING

USER

THREADSERIALIZE FIX

MESSAGE

OUTGOING MESSAGE

STORAGE

MESSAGE

QUEUE

INCOMING MESSAGE

STORAGE

PER TYPE

HANDLERS

USER

DEFINED

HANDLERS

SYSTEM

HANDLERS

READ &

PARSE FIX

MESSAGE

READING

THREAD

FIX ANTENNA

SYSTEM

THREAD POOL

INCOMING MESSAGE

STORAGE

USER

DEFINED

HANDLERS

SYSTEM

HANDLERS

READ &

PARSE FIX

MESSAGE

READING

THREAD

FIX ANTENNA

SYSTEM

THREAD POOL

USER THREAD

MESSAGE

QUEUE

SERIALIZE

FIX

MESSAGE

OUTGOING

MESSAGE

STORAGESENDING

THREAD

ASYNCHRONOUS SENDING

48CONFIDENTIAL

FIX ANTENNA JAVA THREADING MODEL

FIX ANTENNA

FIX SESSION

FIX MESSAGE

SENDER

FIX MESSAGE

READER

Input

Disruptor

SENDER

OUTPUT

DISRUPTOR

RECEIVER SENDER

FIX

FIX

FIX

• Input and output disruptors

• Asynchronous sending – stay not affected by slow FIX clients

COOPERATION WITH DISRUPTOR

FIX SESSION

FIX MESSAGE

SENDER

FIX MESSAGE

READER

FIX SESSION

FIX MESSAGE

SENDER

FIX MESSAGE

READER

49CONFIDENTIAL

FIX Antenna Java “Hello World!”

SessionParameters details = new SessionParameters();

details.setFixVersion(FIXVersion.FIX42);

details.setSenderCompId("BLP");

details.setTargetCompId("SCHB");

FIXSession fixSession =

connectionDetails.createNewFIXSession();

message.set(11, clordid);

message.set(44, 100); // Price

message.set(38, 10); // OrderQty

fixSession.sendMessage(message);