Ivan Black, Director, Big Data Platforms Journey to the Cloud FINRA, Starburst, and Presto
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Ivan Black, Director, Big Data Platforms
Journey to the CloudFINRA, Starburst, and Presto
AGENDA
Confidential | Copyright 2019 FINRA
FINRA Mission01
Journey to the Cloud02On-Prem History03Cloud-based Future04
Starburst Partnership05Move to Presto06
1
Growth of Presto07Future Growth08
Confidential | Copyright 2019 FINRA
FINRA Mission
2
09 | FINRA @ Scale
Confidential | Copyright 2019 FINRA 3
Up to
135 Billion
Events Per DayInvestor
ProtectionMarket Integrity
Monitor
100% Equities &
45% Options
in the US
Run Hundreds of
surveillance patterns
plus a wide swath of
data science, ad-hoc &
interactive analytic
query tools
Reconstruct Trillions of
Market Nodes & Edges
Confidential | Copyright 2019 FINRA
Journey to the Cloud
4
03 | On-Prem History
Confidential | Copyright 2019 FINRA 5
Growing 20 to 30 percent YoY
Data growth
Costly to build for peak; constant EOL cycles. Spend more
on infrastructure or core mission?
Infrastructure cost
What do we have? Source? Versions? Retention?
Tracking 40M+ tables is not easy
Data governance questions
How do we manage data at scale?
How can we run analytics despite fragmentation?
Data management problems
04 | Cloud-based Future
Confidential | Copyright 2019 FINRA 6
One location, scalable, durable, performant, cost-effective, cross-region replicated
All data is in Amazon S3 - source of truth
Separation of storage from computeElastic compute, engine-agnostic, rapidly evolving ecosystem of open source software
Confidential | Copyright 2019 FINRA
Starburst Partnership
7
06 | Move to Presto - 2015
Confidential | Copyright 2019 FINRA 8
Better than classic Big Data engines, but behind dedicated
analytic appliances
PerformanceLacked LDAP integration and
internode encryption
Security
Only rudimentary resource management
Resource Management
07 | Growth of Presto - Present
Confidential | Copyright 2019 FINRA 9Confidential | Copyright 2019 FINRA 9
Performance of some queries has increased 20x
over past 4 years, resulting in improved price
performance and new use case development
PerformanceInternode encryption, extensible
authentication plugin architecture
Security
Sophisticated, group-based
resource management
Resource Management
Confidential | Copyright 2019 FINRA
Future Growth
10
New Use Cases
Confidential | Copyright 2019 FINRA 11
Improved performance and security allows for additional Presto use cases
Replacement of classic big-data engine to create DataMarts for thousands of end-users
Replacement of common key/value query system for trillionsof events
Consolidated Audit Trail
Confidential | Copyright 2019 FINRA 12
In excess of135 BillionEvents Per Day Investor
ProtectionMarket Integrity
MonitorAll Equities & Options Exchanges and Broker Dealers in the US
Run Hundreds of surveillance patterns, plus a wide swath of data science, ad-hoc & interactive analytic query tools
Reconstruct Trillions of Market Nodes & Edges
Confidential | Copyright 2019 FINRA
Q A
13
Related Documents
