Qlik Data Integration Data Revolution Tour 2019 Ian Crosland
Qlik Data IntegrationData Revolution Tour 2019
Ian Crosland
2
Qlik’s 3rd Generation BI Vision
Democratization
of Data
All data, governed, and
universally accessible
Augmented
Intelligence
Raising Data Literacy
through technology
Embedded
Everywhere
From the edge
to the C-Suite
Enterprise-Class Technology
Hybrid, multi-cloud, and edge continuum
3
Qlik’s 3rd Generation BI Vision
Democratization
of Data
All data, governed, and
universally accessibleQlik Data Catalyst®
Attunity®
A Division of Qlik
4
Qlik’s 3rd Generation BI Vision
Democratization
of Data
All data, governed, and
universally accessibleQlik Data Catalyst®
Attunity®
A Division of Qlik
• From raw to analytics-ready, faster
• Accessible and actionable
• Reuse & collaborate, securely
Qlik Sense®
5
ANALYTICS
Qlik Data and Analytics Supply Chain
Speed and efficiency from raw data to shared insights
6
DataOps CatalogProfiling, Lineage,
Governance, Collaboration,
and Provisioning
Change Data CaptureReal-time streaming, replication,
Efficient Cloud Delivery
Data Warehouse AutomationETL generation, Self-Service Marts,
Cloud Optimization Data PreparationEnrichment, transformation,
protection of derivative
datasets
Data Lake PipelinesOrchestration and automation
Analytics-Ready Datasets
Qlik Data Integration Platform
• DataOps for Analytics
Qlik Data
Catalyst
Attunity(Qlik)
7
Shifting Data Architectures
$205B by 2020Cloud & Multi-cloudOn Premises
2X DATA Every 2 YearsCloud Warehouses
+ Data LakesData Warehouse
82%Adopting Real-TimeStreamingBatch
Sources: WW Public IT Cloud Services Revenue in 2020, IDC. The Digital Universe of Opportunities, IDC. Cloudview Survey, IDC.
8
Growing Demands
Automated, Agile
Data Delivery
Governed,
Enterprise-Ready
Data
Real-Time Data
SPEED SKILLS TRUST
9
CODE DATAINFRASTRUCTURETOOLS
PEOPLE TECHNOLOGYPROCESS
• Emerging discipline to build and
manage efficient data pipelines
• Applies principles of DevOps to
data integration
• Improves collaboration between
data managers and consumers
Source: Gartner Innovation Insight for DataOps, December 2018
Long cycles
Batch processing
Brittle ETL scripts
DATA MGRSDATA USERS
Fast cycles
Real-time data
Resilience
DataOps: Accelerating Democratization of Data
10
DataOps for Analytics
17
AUTOMATE
REFINEMENTCONTINUOUS
DELIVERY
CATALOG
& GOVERN
1 Real Time Data for
Faster, Better Insights 2 Agile Data Delivery 3 Trusted, Enterprise
Ready Data
Qlik Data Catalyst®Attunity Replicate® Attunity Compose®
11
Continuous Delivery & Refinement
MAINFRAME
SAP
Other…
Databases
PaaS DB
DatabaseReplication
Data Warehouse Automation
SAAS
APPS
FILES
DATA WAREHOUSE
RDBMS
StreamingData PipelineAutomation
Design & Manage
Generate Deliver Refine
change stream
to cloud, lakes
for analyticuse
Data Lake Automation
Other…
Data Warehouses
AzureSQL DW
Redshift
Other…
Cloud & Data Lakes
12
Database Replication
Enterprise-wide Data
Replication &
Distribution for
Multiple Analytics Use
Cases
Mainframe to Oracle
Data Consolidation for
Customer 360 Initiative
Oracle Transaction
Streaming and for
Multi-Channel
Customer
Engagement
Features
• Agentless, real-time data capture and
delivery
• Automated target table creation,
instantiation and mappings
• Supports RDBMS, file systems and
streaming targets, on-premise & Cloud
Benefits
• Ensures consistency and ease of use
across all major platforms (sources and
targets)
• No impact to production systems
• Easy to manage, automated, low TCO
13
Data Warehouse Automation
SQL Server Data
Warehouse
Automation Cut ETL
scripting time 96%,
accelerated change
process 6x
High-Scale, Multi-
sourced Data
Consolidation into
Azure Data Lake and
Azure SQL DW
Real-Time
Consolidation from
SAP, Oracle to
Snowflake for DW
Modernization
Features
• Automates Data Model/Data
Warehouse design, build, maintenance
and documentation
• Automated table creation, instantiation
and mappings
• Continuous, real-time data replication
Benefits
• Reduce risk, save time and money
– no scripting or coding required
• DW now built in hours, changed in
minutes
• Future-proof for new requirements and
new platforms
14
Data Lake Automation
Mainframe Data
Streaming and
Refinement for
Microservices
Enablement
Data Lake
Consolidation for
Marketing/Sales
Analytics and
Engagement
Multi-sourced Data
Integration and
Refinement for
Operational Analytics
Features
• Automates creation of tables, organizes
data structures and tracks lineage to
deliver a Managed Data Lake
• Instantiates data, maps source and
target, keeps schemas in sync
• Continuous, real-time data replication
Benefits
• Quickly and easily create high-scale
data pipelines
• Eliminate risky, expensive and complex
custom coding
• Close the “last mile” by provisioning
analytics-ready data in real-time
15
Catalog and GovernQlik Data Catalyst
MainframeFilesRDBMS NoSQLBig Data Cloud
BI/Reporting
Data Science
Applications
Cloud Platforms
Business
Processes
• Available
• Searchable
• Reliable
• Relatable
• Secure
• Role Based
Qlik Data Catalyst
16
Qlik Data Catalyst Benefits
DATA ENGINEER
Brings new assets into the
data catalog
• Quickly add new datasets
• Understand the exact
content and condition of
onboarded data
• Easily filter, transform or
combine incoming data
DATA STEWARD
Makes sure naming conventions
and security standards are enforced
• Improve consumer‘s ability to
find the right dataset
• Systematically enforce and
monitor data policies and usage
• Ensure sensitive or personal
data is automatically masked
DATA CONSUMER
Find and utilize the data they
need to gain new insights
• Eliminate reliance on data
experts or IT
• Reuse and build on previously
created assets
• Easily export, share or
automatically publish data sets
17
Qlik Data Catalyst
Features
• Smart, Integrated Data Catalog of all
technical, operational & business metadata
• Data access and obfuscation capabilities to
guarantee that data is always protected
and secure
• Shop for Data with role-based security
• Quickly publish data to Qlik Sense, other BI
tools, and cloud storage platforms
Benefits
• Improved data literacy as users can find,
understand and generate insights faster
• Removes potential failure points, tightens
security, and reduces risk
• Removes the IT bottleneck through a
governed, self-service approach
• Reduced data onboarding from 8 weeks to as little as 2 hours
• Cost to onboard data 90% less vs. conventional ETL
• Cost of Environment 30x cheaper than least expensive RDBMS alternative
• Able to utilize cloud storage by carefully tracking and labeling PII data
18
Qlik Data Catalyst now supports QVD filesSee, Profile, Shop For, and Share QVDs
Manage and Re-use Your QVD Investments
• Full, searchable catalog of QVDs created in Qlik Sense and QlikView
• Full profile, data tagging of data elements in QVDs with ability to secure and obfuscate data when sending out of the Qlik environment
• Ability to prepare and integrate QVDs with non-Qlik assets
• Shopping and Publication of QVDs to Qlik, data science tools, and competitive BI tools
Publish
Data Science Tools
Within Qlik
Competitors
19
DataOps for Analytics
17
AUTOMATE
REFINEMENTCONTINUOUS
DELIVERY
CATALOG
& GOVERN
1 Real Time Data for
Faster, Better Insights 2 Agile Data Delivery 3 Trusted, Enterprise
Ready Data
Qlik Data Catalyst®Attunity Replicate® Attunity Compose®
20
By 2021, organizations that offer
a curated catalog of internal and
external data to diverse users
will realize twice the business
value from their data and
analytics investments than
those that do not.
Source: Gartner - Augmented Data Catalogs: Now an Enterprise Must-Have for Data and Analytics Leaders - September 2019
2X
21
DBaaS
STORAGE
HADOOP
STREAMING
DWaaS
OTHER DWaaS
SPARK
Multi-Cloud & Partner Alignment
All
ADLS, BLOB
HDInsight
Event Hubs
Azure SQL DW
Snowflake
Databricks
DB All
GCS
DataProc
Cloud Pub/Sub
BigQuery
RDS (All)
S3
EMR
Kinesis
Redshift
Snowflake
Databricks
* Planned
for 2020
*
Snowflake*
This document and Qlik‘s strategy and possible future developments are subject to change and may be changed by Qlik at any time for any reason without notice. This document is provided without a warranty of any kind. The document may not be copied, distributed, or otherwise shared with any third party.
22
Production
Systems:
Fast Growing
Platforms: (SPARK)(KAFKA)
Major Cloud
Platforms:
Hadoop
Evolution:
Comprehensive Partner Ecosystem
23
What’s Next – Roadmap
• Continue to lead in universal access – new sources, targets
- CDC for Salesforce and MongoDB now available! Additional NoSQL and SaaS apps coming soon.
• Dynamic, hybrid and multi-cloud scaling via Kubernetes
• Continued integration of Attunity with Qlik Data Catalyst and the rest of the Qlik portfolio
This document and Qlik‘s strategy and possible future developments are subject to change and may be changed by Qlik at any time for any reason without notice. This document is provided without a warranty of any kind. The document may not be copied, distributed, or otherwise shared with any third party.
24
Qlik Data Integration – Guiding Principles
Independent Real-TimeUniversal
Agile &
AutomatedSelf-ServiceGoverned
25
Next Step
Read the Attunity and
Gartner report:
“Enabling DataOps
for Analytics”
@ www.qlik.com