1
Finance and Manufacturing Big Data Use Cases and Solutions
Sanjay SharmaPrincipal Architect
August 2013
Impetus Big Data Services
2
Big Data Platform Implementation Operations and Visualization
Business Analyticsand Data Science
Solution Architecture, POC and Production planning
Technology strategy, Use Case development & Validation
BUSINESS PROCESS
MANAGEMENT
Assessment
Solution Modeling
Solution Analysis
People, Process,
Technology Impact
Analyze & Optimize
Objectives & Strategy Model
© 2013 Impetus Technologies
Big Data : Value Drivers
© 2013 Impetus Technologies
Financial World: Key Challenges
Business Challenges
• Fraud• Regulatory
Compliances• Customer Insights• Risk Handling
Technical Challenges
• Reduce IT costs• Do more with
LESS• Unstructured data
© 2013 Impetus Technologies
Financial World: Some Use Cases
Fraud Detection and
Analysis
Risk Management
Customer Insights/Mana
gement
Micro targeting
Trade/Payments Analytics
Long Term ‐Storage & Analytics
New Opportunities
Reputational Risk
Marketing Campaigns
© 2013 Impetus Technologies
Financial World: Technical Solution Challenges
• Regulatory Compliances• Machine Learning based supervised and
unsupervised analytics
Large Data Storage and Advanced
Analytics
• Mash up structured and unstructured data• Enrich TX with NLP/Text analyticsUnstructured Data
• Customer level Profiling/ Recommendations
• Transaction/Trade Ticker level analytics
Individual Level Analytical Processing
© 2013 Impetus Technologies
Big Data Solution: Building Blocks
© 2013 Impetus Technologies
Big Data Use Case Solution: Example
• Major Credit Card Company• Traditionally Oracle/ DB2 + SAS +
Informatica• Huge data– Existing solutions/ infrastructure
inadequate to meet new business requirements and data growth cost effectively
– Successfully revamping to Big Data Platform
© 2013 Impetus Technologies
Big Data Use Case Solution: Example
Step 1: Hadoop POC with Sentiment
Analysis
Step 2: Hadoop Distribution Selection
Step 3: Production-ized Recommendation
Engine
Step 4: Production-ized Negative Fraud
Detection ($100+mlln)Step 5: …
Hive
Hadoop Platform
HBase
Oozie
Mahout + NLP
Transactions
Unstructured Sources
(Email, Surveys, CRM notes, Social Feeds etc.)
Sqoop
Flume
MapRed
BI/ Visualization
Analytics- SAS/R
Real Time DeliveryRDBMS/API
RDBMS/MPP
Informatica
ETL/ELTL
Advantages= PB scale= ETL and Analytical Offloading= Prescriptive Analytics than Predictive
© 2013 Impetus Technologies
Manufacturing Domain: Challenges
Business Challenges
• Early preventive maintenance/repair• Real time/ near real time
actionable response• Improve productivity/Margin
Recovery• Reduce wastes, improve
efficiency• Improve Yield
• Supply Chain• Optimize supply chain
Technical Challenges
• High Ingestion Rates• Sensor/ tool data with sub
second ingestion requirements
• Millions of read/writes per second
• Complex log formats• Semi-structured data
• Huge amount of data• TB/PB of data storage for
deeper analytics
© 2013 Impetus Technologies
Manufacturing Domain: Architectural Building Blocks
Real Time AnalyticsStreaming Analytics + CEP
Hadoop
Advanced Analytics
Rules and Alerts Machine Learning
Loopback for RuleEnhancement/ Enrichment
NoSQL + Search
Machine Data
© 2013 Impetus Technologies
Machine Data
Ingestion Engine
(Real time + Batch
components)
Real Time Processing
Engine(CEP/Analytics/
Rule Engine)
Real Time Data
Storage Engine
(Store + Indexing/Searc
h)
Business Process Engine(Business Process+ Rule management)
Kafka/ Storm Storm + Esper Cassandra + Solr
JBoss Drools/jBPM
Manufacturing Domain: Data Ingestion/ Streaming – Customer Example
© 2013 Impetus Technologies
HBase + Elastic SearchSpark Streaming
Hadoop
Hive/PIG/Shark
Rules and Alerts Machine Learning
Loopback for RuleEnhancement/ Enrichment
NoSQL + Search
Machine Data
Manufacturing Domain: Reference Architecture-Cloud– Customer Example
© 2013 Impetus Technologies
• Software Solutions and Services Company• Leader in Innovation led Technology services• 17 years of customer success, 1500 people
across US/ India• Big Data, Enterprise Mobility, Test and
Performance Engineering, Carrier Grade Large Systems
• Vendor neutral, open source contributor with Big data accelerator products
Impetus Technologies
