REGULATORY REPORTING OF ASSET TRADING USING SPARK Sudipto Shankar Dasgupta & Mayoor Rao – Infosys Limited
Aug 14, 2015
REGULATORY REPORTING OF ASSET TRADING USING SPARK
Sudipto Shankar Dasgupta & Mayoor Rao – Infosys Limited
Regulatory Reporting – Current Challenges
2
Financial Institutions need to report asset trade based on defined regulations
Reporting has to comply within a time window from the trade execution
Rules are subject to change and adjustments based on the change in regulations
The transaction volume is high Transactional systems are diverse
Case Study 3
Leading global Financial Institution engaged in around 6 million trades a day in the financial markets
Regulatory requirements necessitates reporting in a specified reporting format within a 15 minute window
The trades are identified for reporting based on steps • Transformation • Enrichment • De-duplication checks • Business Rules
Report Generation Process Flow Asset
Transactions
Transformation Rules
Transformed Asset Data
Master Data
Enriched Asset Data
Business Rules
Transformation Enrichment Report Generation
4
Architecture Diagram
SQOOP
HDFS Data Lake
Tem
pora
ry S
tagi
ng
Pro
cess
ed O
utpu
t
In Memory Analytic and Processing Zone
Processing Logic
Apps Portal
5
Ingestion throughput >130,000
records/second or 18.22 MB/second (per node)
Report execution time -> 35 seconds for 30,000 trade records
Overall Data Volume -> 596 million trade transaction records
Hardware Configuration -> Spark Installed in 10 node 32 GB RAM , 16 Core m/c
Statistics 6
01
02
03
04
DEMO
7
IIP - Leveraging Power of Spark 8
Open Source / Spark Components
Spark ML
Spark Streaming
HIVE / Spark SQL
Hadoop / FS Storage / Infra Mgmt
Infosys & Partner IP Components Tools | Data Extractors | Algorithms | Packaging &
Support
Customization, Integration & Implementation Services
Data Modeling & Cleansing | Agile App Development Data Science & Analytics | Security & Governance
Custom Data Extractors
Infosys Information Platform - Architecture IIP Data Lake!
Query Interface
BI Applications
Down Stream Apps / Portal
IIP Data Lake!
1"
Rea
l Tim
e –
IIP In
gest
ion
Unstructured Data
RDBMS Data
NO SQL Data
Data Streams
JMS Queue
Social data
Machine data
Machine data
Web URL
Resource Manager
Governance
Security
Data Store
OS /LDAP/Kerberos Authentication Authorization Role Based Access
Metadata Management Data Lineage Data Audit
Cluster Resource Manager Cluster Monitor Disaster Recovery
Spark In Memory Analytic Zone
HDFS / Hive
Natural Language Processing
Machine Learning
Data Views
Data Mining and Models
Data Joins
Data Transformation
Single Click Installer Cluster Maintenance
Deployment Manager
IIP Data Explorer
2"
3"
4"
5"
9
DEMO
10
© 2015 Infosys Limited, Bangalore, India. All Rights Reserved. Infosys believes the information in this document is accurate as of its publication date; such information is subject to change without notice. Infosys acknowledges the proprietary rights of other companies to the trademarks, product names and such other intellectual property rights mentioned in this document. Except as expressly permitted, neither this documentation nor
any part of it may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, printing, photocopying, recording or otherwise, without the prior permission of Infosys Limited and/ or any named intellectual property rights holders under this document.
Thank You