10-Step Methodology to Building a Single View Mat Keep, Director of Product & Market Analysis. [email protected] @matkeep Jon Rangel, Director of Professional Services, EMEA. [email protected]
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
10-Step Methodology to Building a Single View
Mat Keep, Director of Product & Market Analysis. [email protected] @matkeep
Jon Rangel, Director of Professional Services, EMEA. [email protected]
What You Will Learn
1. Single View: Opportunities & Challenges
2. Repeatable 10-Step Methodology
3. Required Technical Capabilities
Why Single View
Single View Defined• What
– Single, real-time representation of a business entity or domain
– Customer, product, supply chain, financial asset class, & more
• How– Gathers and organizes data from multiple,
disconnected sources;– Aggregates information into a standardized format
and joint information model
• Why– Improves business visibility– Serve operational applications– Foundation for analytics
Single View Use Cases
• Comparative view of traders or products
• Firm-wide view of asset exposure
• Aggregated transactions for fraud models
• Omni-channel view of customers for personalized marketing
• Inventory control & management
• Single view of product across channels & demographics
• Management of patient medical records for treatment plans
• Macro-analysis view for public health
• Medical history to identify insurance risk
Finance Retail Healthcare
Challenges• Current State
– Data dispersed across multitude of systems– Different structures, different attributes– Apps built to meet specific business requirements, not
integrated– New data sources from new apps, M&A
• Governance Processes– How to deliver & maintain single view in face of
constant business change
• Technology Limitations– Traditional databases not well suited to single view
required capabilities
10-Step Methodology
ETL
or M
essa
ge Q
ueue
Web
Mobile
CRM
MainframeSingle View
Call Center
Analytics
Technical Support
Billing
Source Systems Consuming Systems
Load Reads
High Level Architecture
10-Step Methodology
Step 1:Define Scope
Step 4:Appoint
Data Stewards
Step 5:Develop
Data Model
Step 6:Load &
Standardize
Step 7:Merge,
Test & Reconcile
Step 8:Infrastructure
Design
Step 3:Identify
Data Producers
Step 2:Identify
Data Consumers
Step 9:Modify Consuming
Systems
Step 10:Maintenance Processes
Discover
Develop
Deploy
Step 1: Define Scope & Sponsorship• Scope needs to be realistic, defined by specific success metric
– Long term: aggregate all customer data into a single view, serving all business functions
– Initial phase: collecting all customer interactions on digital channels over past 3-months to improve call center MTTR
• Appoint executive sponsors– Senior: allocate resources and command credibility– Combination of senior title from the business, and from the technology
group
Discover
Web
Mobile
CRM
Mainframe
Source Systems
Steps 2 & 3: Identify Data Consumers & Producers
• Single View Consumers Define – Typical queries and SLAs – Required data attributes– Current data sources
• Identify apps generating the source data– Identify application owners + associated databases– Profile apps: operational, analytical
Step 2: Data Consumers
Step 3: Data Producers
Discover
Step 4: Appoint Data Stewards
• Data steward appointed for each data source.
• Deep knowledge of:– Source system schema– Which tables store required attributes, what format– Clients and apps that generate & consume the
source data
• Advise on data loading strategies
Develop
Step 5: Develop Single View Data Model• Key inputs
– Required data attributes– Query patterns
• Define common fields & data types– Create rules to validate common data
• Define primary & secondary indexes• Identify dynamic fields
– No need to pre-declare when using a document database
• Localize data into a single document (where appropriate)
{_id : “[email protected]”,first_name : "Mark",last_name : "Smith",city : "San Francisco",phones: [ {
number : “1-212-777-1212”,dnc : true,type : “home”
},{
number : “1-212-777-1213”, type : “cell”
}]}
Single View
Develop
Resources to Support Schema Design
MongoDB Documentation
MongoDBDevelopment Rapid Start
Develop
Step 6: Load2 phases: Initial Load & Delta LoadEmit JSON to preserve data types. Use Extended JSON
Load
ETL
or M
essa
ge Q
ueue
Single View
Develop
Initial Load• ETL Tools• Custom Loaders
Delta Load• Batch loads: use tools above• Real-time loads: Message queue
Step 6 (cont’d): Standardize
DataSourceA DataSourceB DataSourceC
14 7726
cust_id:14f_name:Jamesl_name:Bonddob:07/14/1968eMail:[email protected]
fno:77first:Jimlast:Bondborn:1968-07-14email:[email protected]
xc_id:26name:JamesBindbdate:July14,68Email:[email protected]
Develop
Step 7: Match, Merge & Reconcile Develop
cust_id:14f_name:Jamesl_name:Bonddob:07/14/1968eMail:[email protected]
xc_id:26name:JamesBindbdate:July14,68Email:[email protected]
source_id:A_14first_name:Jameslast_name:Bonddob:1968-07-14eMail:[email protected]
source_id:B_77first_name:Jimlast_name:Bonddob:1968-07-14eMail:[email protected]
source_id:C_26first_name:Jameslast_name:Binddob:1968-07-14eMail:[email protected]
_id:[email protected]_name:Jameslast_name:Bonddob:1968-07-14
SourceData
StandardizedDataFieldnames&datatypes
SingleViewDatamerged,tested&reconciled
fno:77first:Jimlast:Bondborn:1968-07-14email:[email protected]
Step 7 (cont’d): Match, Merge & Reconcile• Use iterative grouping functions to cluster records with similar
attributes1. Match against unique, authoritative attributes (email address, credit card #)2. Match by combining attributes (last name, DoB, zip code)3. Use fuzzy matching to catch errors in source data (i.e. different spellings of customer
name)
• Apply confidence factor to dictate merging– Automatically merge records with 95%+ confidence– Manually inspect records with lower confidence
Develop
Step 7 (cont’d): MongoDB Tools• Workers framework to parallelize document comparisons• Grouping tool to cluster documents based on attribute similarity
– Levenshtein to calculate distances, single-linkage clustering for matching
Develop
Step 8: Architecture DesignDeploy
• Deployment infrastructure• MongoDB Production Readiness Consulting
Package provides recommendations: – Hardware sizing– HA/DR strategies– Scaling– Security for corporate and regulatory compliance
• Follow-on services for implementation
Step 9: Modify Consuming SystemsDeploy
• Modify the apps that consume the single view
– Create an API that exposes the single view (i.e. RESTful web service)
– Re-point apps to the web service (reads initially)
• Modify one consuming application at time
Call Center
Analytics
Technical Support
Billing
Consuming Systems
Reads
Single View
Step 10: Implement Maintenance ProcessesDeploy
• Frequency of application launch & evolution is accelerating
• Impacts to single view– Adding new attributes from source systems– Onboarding new data sources or digital channels– Creating new apps that consume the single view
• Single view team needs to institutionalize governance around on-going maintenance– Repeat the 10-step process– Dynamic schema is HUGE!
Single View Maturity Model
Scope
Bus
ines
s B
enef
its
Transactions are written first to the single view, whichpropagates the data back to the source system of record.
Writes are performed concurrently to the source systems aswell as the single view
The single view data model is enriched with additionalsources to serve more applications, including real-timeanalytics. The single view becomes a platform servingmultiple applications
Single ViewPlatform
Records are copied via ETL or message queuemechanisms from the source systems into the single view,serving read queries. The single view serves one specificapplication
Single ViewApplication
Single View First
Dual Writes
Read Centric
Transforming the role of the single view
Reads & Writes
Single View Maturity Model
• Advantages of writing to the single view– Fresher data– Reduced app complexity– Improved application agility
Architecture for Writes to the Single View
ETL
or M
essa
ge Q
ueue
Web
Mobile
CRM
Mainframe
Single View Call Center
Analytics
Technical Support
BillingUpdate Queue
Reads
Writes
Source Systems Consuming Systems
Load
Required Capabilities for Single View
Single View with a Relational Database
Required Database Capabilities
• Data model flexibility with a dynamic schema• Real-time analytics• Performance, scale & always-on• Enterprise deployment model
MongoDB Compass MongoDB Connector for BI
MongoDB Enterprise Server
Enterprise Deployment Model24
x 7
Sup
port
(1 h
our S
LA)
Com
mercial License
(No AG
PL Copyleft R
estrictions)
Platform Certifications
MongoDB Ops Manager
Monitoring&Alerting
QueryOptimization
Backup&Recovery
Automation&Configuration
SchemaVisualization
DataExploration
Ad-HocQueries
Visualization
Analysis
Reporting
Authorization Auditing Encryption(InFlight&atRest)Authentication
RESTAPIEmergency Patches
Customer Success Program
On-Demand Online Training
Warranty
Limitation of Liability
Indemnification
Single View In Action
Single View of CustomerInsurance leader generates coveted single view of customers in 90 days – “The Wall”
Problem WhyMongoDB ResultsProblem Solution Results
No single view of customer, leading to poor customer experience and churn
145 years of policy data, 70+ systems, 24 800 numbers, 15+ front-end apps that are not integrated
Spent 2 years, $25M trying build single view with RDBMS – failed
Built “The Wall,” pulling in disparate data and serving single view to customer service reps in real time
Flexible data model to aggregate disparate data into single data store
Expressive query language and secondary indexes to serve any field in real time
Prototyped in 2 weeks
Deployed to production in 90 days
Decreased churn and improved ability to upsell/cross-sell
Single View of LHC AnalyticsData aggregation system to accelerate scientific research & discovery
Problem WhyMongoDB ResultsProblem Solution Results
Raw data from LHC & experiments distributed across multitude of source systems
Scientists don’t know location of source data, or how to extract it
Relational databases rigid data model prevented aggregation of data from different sources
Data Aggregation System built on MongoDB, consolidating analytics into a single view
Dynamic schema represents data of any structure
MongoDB query language supports simple lookups to complex search, traversals & analytics
A single query to MongoDB can return 10,000 documents from different data sources for real time analytics
Accelerates scientific time to insight
Accessed by 3,000 physicists from 200 research institutions across the globe
Wrap Up
Where to Go from Here?• Single view projects are challenging
– Partner with a vendor offering proven methodology, tools & technologies
• Learn More– Download the whitepaper– 10-Step Methodology to Building a Single View
• Engage– MongoDB Global Consulting Services can help you
scope the project and get started– Book a workshop
10-Step Methodology to Building a Single View
Single View of the Customer360° view of the customer increases customer satisfaction, cross-sell & up-sell with MongoDB, Spark, & Hadoop
Problem WhyMongoDB ResultsProblem Solution Results
Customer data scattered across 100+ different systems
Poor customer experience: no personalization, no consistent experience across brands or devices
No way to analyze customer behavior to deliver targeted offers
Single View application on MongoDB flexible data model, expressive query language, secondary indexes, & horizontal scalability
Data from old relational systems fed into Spark for analysis and then stored in MongoDB to support real-time CRM
Customer data synced from MongoDB to Hadoop for nightly batch jobs, then fed back to MongoDB for personalized recommendations
Single view serves customers from any channel
Stores 10s of TBs of customer data across multiple data centers
Increased revenues from improved customer intimacy, driving cross-sell and upsell
Global Airline
Data Model Flexibility
…Mobile App
Web
Call Centre CRM Social
Feed
COMMON FIELDSCustomerID | eMail |
DYNAMIC FIELDSCan vary from record to record: location, action
Single View
Customer Service Application
MongoDBPrimary ReplicaSingleView
BI & Reporting REST Data Services
Real-time Data Services for Regulators & Partners
VisualisationsQueries & Updates
Aggregates
Predictive Analytics
MongoDBSecondaryReplicaSingleViewMongoDBSecondaryReplicaSingleViewMongoDBSecondaryReplicaSingleViewMongoDBSecondaryReplicaSingleViewMongoDBSecondaryReplica
MongoDBSecondaryReplica
Data AnalyticsPipeline
Real-Time Analytics
Predictable Scale & Always-On
Shard 1
Horizontally Scalable
Shard 2 Shard 3 Shard n
Related Documents
