Transcript
The Role of Data in Your Digital TransformationLes KleinField CTO DataPivotal
@LesKlein #PivotalForum #DigitalTransformation
Great software companies leverage Big Data
to fundamentally change the consumer experience and pioneer entirely new business models
4© 2016 Pivotal Software, Inc. All rights reserved.
$4BN
Financial Services
$26BN
Hospitality
$50BN
Transportation
$54BN
Entertainment
$30BN
Automotive
$3.2BN
Industrial Products
CLOUD NATIVE SOFTWARE IS CHANGING INDUSTRIES
Data is Fueling Software
5© Copyright 2015 Pivotal. All rights reserved.
Hundreds of thousands of “trip” events each day
400+ billion of viewing-related events per day
Five billion training data
points for Price Tip feature
Disruptors Use a LOT of Data
6© Copyright 2015 Pivotal. All rights reserved.
“We’ve found that when a host selects a price that’s
within 5% of their tip, they’re nearly 4 times
more likely to get booked”
“The importance of accuracy and efficiency […], will continue to
rise as we expand and improve products like uberPOOL and
beyond.”
“Over 75% of what people watch come
from our recommendations”
Data manifests as features in an app
7© Copyright 2015 Pivotal. All rights reserved.
(Data)Microservices
Loosely coupled services architecture, bounded by context
Cloud-NativePlatforms
Enabling continuous delivery & automated
operations
Open SourceDatabase
Innovation
Extreme scale & performance advantages,
built for the cloud
Machine Learning
Use of predictive analytics to build
smart apps
How are they accomplishing this?
8© Copyright 2015 Pivotal. All rights reserved.
These companies…
Release new features in minutes, multiple times a day
Support a micro-services architecture
Consume a wide range of data sources and protocols
Store and Analyze all their data
Update algorithms and predictive models daily
Continuously ask lots of questions of their data
Modify data pipelines and add processing steps daily
They are Data Driven, are you?
9© Copyright 2015 Pivotal. All rights reserved.
“Companies need to learn how to catch
people or things in the act of doingsomething and affect the outcome“
PAUL MARITZExecutive Chairman, Pivotal
Real-time andPersonalised Informationin Context is what Wins!
10© Copyright 2015 Pivotal. All rights reserved.
A lot of reports do not make you data
drivenA lot of
dashboards do not make you data
driven A lot of alerts do not make you data
driven
11© Copyright 2015 Pivotal. All rights reserved.
Converging Trends
Innovation
New Data New Processes New Insights
The Journey to the Data-Driven Enterprise
Data Scienceand Machine
LearningBig Data
Internet of Things &
Social Media
12© Copyright 2015 Pivotal. All rights reserved.
HDFS
Data Lake
Ingest Store Analytics
Hard to changeLabor intensive
Inefficient
Coding basedNo real-time informationBased on expensive ETL
Migrating from a Reactive, Static and Constrained Model…
13© Copyright 2015 Pivotal. All rights reserved.
HDFSData LakeExpert System /
Machine Learning
In-Memory Real-Time Data
Continuous LearningContinuous Improvement
Continuous Adapting
Data Stream Pipeline
Multiple Data SourcesReal-Time ProcessingStore Everything
To Pro-Active, Self-Improving, Machine Learning Systems
15© 2016 Pivotal Software, Inc. All rights reserved.
The new normal: “an audience of one”DATA DEVICES
Media
Banks
DeliveryServices
Marketers
Government
Individuals Employers
AnalyticServices
Advertising
CatalogCo-ops
ListBrokers
Websites
InformationBrokers
CreditBureausMedia
Archives
GOVERNMENTGOVERNMENT
PHONE/TV
PHONE/TV
INTERNETINTERNETAD
AGENCYAD
AGENCY
RETAILRETAIL
20© 2016 Pivotal Software, Inc. All rights reserved.
“A data native is someone who expects their world to not just be digital,
but to be smart and to adjust immediately to their taste and habits.”
Beyond tech savvy …..
22© 2016 Pivotal Software, Inc. All rights reserved.
Data natives expect the thermostat to program itself
23© 2016 Pivotal Software, Inc. All rights reserved.
Do you know the full value of your data and how to leverage it?
24© 2016 Pivotal Software, Inc. All rights reserved.
µs ms s hour day month yr+
“Fast Data” “Big Data”
year
TraditionalSystems
Pivotal Data Science Labs
Value ofData ($)
Time
Value of data over time…
25© 2016 Pivotal Software, Inc. All rights reserved.
Analytics - the journey so far (BI)
Complexity
Value ofAnalytics ($)
Descriptive Analytics
DiagnosticAnalytics
Whathappened?
Why did ithappen?
Hindsight
Insight
26© 2016 Pivotal Software, Inc. All rights reserved.
Analytics – where you need to be today
Value ofAnalytics ($)
Descriptive Analytics
DiagnosticAnalytics
PredictiveAnalytics
PrescriptiveAnalytics
Whathappened?
Why did ithappen?
What willhappen?
How can wemake it happen?
Hindsight
Insight
Foresight
Complexity
27© 2016 Pivotal Software, Inc. All rights reserved.
Interesting questions are found here
Value ofAnalytics ($)
Descriptive Analytics
DiagnosticAnalytics
PredictiveAnalytics
PrescriptiveAnalytics
Whathappened?
Why did ithappen?
What willhappen?
How can wemake it happen?
Hindsight
Insight
Foresight
Complexity
Data science
28© 2016 Pivotal Software, Inc. All rights reserved.Data Science at Pivotal
Theory-driven approach
‘start with the problemand work towards the data’
‘start with the data and work towards the problem’
Data-driven approach
29© 2016 Pivotal Software, Inc. All rights reserved.Data Science at Pivotal
‘start with the data and work towards the problem’
Data-driven approach
30© 2016 Pivotal Software, Inc. All rights reserved.
User Feedback
Scoping Data Review
Model Building
Feature Review
Feature Engineering
Model Evaluation
Operationalization
Our tried and tested approach helps extract value from data
“THE MAGIC HAPPENS WHEN YOU MARRY THE
TRADITIONAL ENGINEERING APPROACH WITH THE DATA SCIENCE ENABLED BY THE DATA LAKE. IT OPENS UP A WHOLE NEW WORLD OF POSSIBLE
‘WHAT IF’ QUESTIONS.
”DAVE BARTLETT, GE AVIATION
32© 2016 Pivotal Software, Inc. All rights reserved.
IT’S ALL ABOUTHOW YOU
OPERATIONALISE YOUR INSIGHTS
33© 2016 Pivotal Software, Inc. All rights reserved.
Scale-out analytic
database
Model API Cloud Native Application
Platform
Data Sources
0 5
Smart Apps: Models Manifesting as Microservices
34© 2016 Pivotal Software, Inc. All rights reserved.
OSSSmart/Data-driven App Technology Needs
Fast Ingest/Pipelining
• Pipelines to consume streaming and batch data from various endpoints
Speed/Serving Layer
• In-memory data grids for real-time and aggregated data sets
• Low latency
Big Data Analytics Platform
• Scale-out Storage• Ability to process large structured and unstructured data• Advanced SQL query capabilities• Ability to run machine learning at scale
Platform for ScalableWeb/Mobile Apps
• Infrastructure Automation
• Develop, run and manage Web applications without the complexity of building and maintaining the infrastructure
35© 2016 Pivotal Software, Inc. All rights reserved.
Fast Ingest/Pipelining Speed/Serving Layer
Big Data Analytics Platform
Platform for ScalableWeb/Mobile Apps
Pivotal’s Answer for Technology Needs
Spring Cloud Data Flow GemFire
GreenplumDatabase
Pivotal HDB
Pivotal Cloud Foundry
Spring
OSS
36© 2016 Pivotal Software, Inc. All rights reserved.
● Silo’d and aging database systems
● Spaghetti data pipelines
● Expensive, proprietary data management systems
● Lack of structured platforms for continuous software delivery
● Monolithic application architectures
● Batch-oriented data integration
● Limited operationalization of analytics
● Proprietary systems
Today’s Enterprise Challenge
37© 2016 Pivotal Software, Inc. All rights reserved.
Data Programming Model
Cloud-Native Platform
Microservices FrameworkPlatform Runtime
Stream Data Platform
Hadoop Spark DW
Apps & Microservices
DBMS
IMDG
K/V Store
Relational DB
Big Data & Machine Learning
Modern Cloud-Native Data Architecture
38© 2016 Pivotal Software, Inc. All rights reserved.
1. Modernize your data platforms:
• Improve performance and scalability for mission-critical data workloads
• Lower overall TCO by adopting industry-leading open source database technologies on modern cloud platforms
2. Prove out a specific business use-case:
• Build a smart app end-to-end
• Focus on a specific analytical model / data-microservice
• Automate ingestion and data pipelines
Pivotal can help you to:
39© 2016 Pivotal Software, Inc. All rights reserved.
Pivotal Offerings
Big Data Suite:
• Best-in-class open source data management software
• GemFire: In-Memory Data Grid
• Greenplum: Data Warehouse
• Pivotal HDB (Apache HAWQ): Hadoop-Native SQL
Pivotal Cloud Foundry
• Industry’s Leading Cloud-Native Platform
Spring Boot, Cloud and Data Flow
• Modern-Java micro-services framework
Pivotal Labs & Data Science
• Agile software development and machine learning
42© 2016 Pivotal Software, Inc. All rights reserved.
AppDevelopment
Data analytics
Cloud-nativeApp platform
Data Science & Model building
DataMicro-service
AP
PA Real-World Example
43© 2016 Pivotal Software, Inc. All rights reserved.
AppDevelopment
Data analytics
Cloud-nativeApp platform
Data Science & Model building
DataMicro-service
AP
P Must support scale-out query processing
Must support scale-out query processing
Must deliver as an API Must deliver as an API
Must embrace agile development, focus on outcomes
Must embrace agile development, focus on outcomes
Must support microservices, agile dev, and connect to big data analytics
Must support microservices, agile dev, and connect to big data analytics
A Real-World Example
44© 2016 Pivotal Software, Inc. All rights reserved.
Major Credit Card Issuer Builds Infrastructure Determining Fraud Markers Across 30 Disparate Sources
Challenge:• Desire to identify potential fraudulent activity within Fortune
100 credit card issuer with 100K employees• Identifying markers across 30 internal data sources caused
lags in reporting• Establish Infrastructure to support Palantir, Platfora and RSA
reporting frameworks
Solution:• Provided ability to store 3 months of data at 60-70TB, interest
in moving to 3-year retention period• Build large-scale security analytics platform to scale with data
needs, analytics and reporting requirements• Support organizational desire to evolve from pure BI to
advanced, predictive analytics
Pivotal Solution includes: GPDB, Hadoop, HAWQ
45© 2016 Pivotal Software, Inc. All rights reserved.
Leading Swiss Bank Re-shapes Business to Better Monetize and Regulate Fraudulent Activity with Big Data
Challenge:• Needed to monetize data around privilege user access and
assure security around fraudulent activity • Reduce the cost of monitoring privileged user access for
regulatory requirements. (currently taking 45 people to do this)• Extend across the enterprise to solve business pains around
faster data as opposed to just longer looks in “Hadoop”
Solution:• Built on top of Big Data Suite to operationalize, automate and
extend across the enterprise • What took 6 hours should now take less than 1 hour with real
time decision-making• Capability of monitoring relationship manager’s activity with
analytics to ensure regulatory compliance
Pivotal Solution includes: Big Data Suite and Data Labs
46© 2016 Pivotal Software, Inc. All rights reserved.
Pivotal Data Innovation Across Industries
RETAIL / TRANSPORT
MEDIA / SERVICES
TELCO / ISP
FINANCIAL SERVICES
HIGH TECH / ELECTRONICS
EDUCATION
GOVERNMENT
OTHER
INSURANCE
AUTO / INDUSTRIAL
HEALTHCARE
MANUFACTURING
top related