1 Trusted, Enterprise QlikView- reporting with Informatica data Integration and data Quality (It’s all about data) Arjan Hijstek senior sales consultant Informatica Nederland bv [email protected] 06-22.454.327
1
Trusted, Enterprise QlikView-
reporting with Informatica
data Integration and
data Quality
(It’s all about data)
Arjan Hijstek
senior sales consultant
Informatica Nederland bv
06-22.454.327
2
Agenda
• Introduction Informatica
• Qlikview and reporting : data warehouse?
- What data do I get? Relationships? data profile!
- (bad) data quality? data quality!
- Extraction, Transform and Load of any data ETL!
• Qlikview and Informatica
• Data Virtualization
- faster access to new data
• Quick demo
3
Do you know Informatica? (PowerCenter, DataQuality, other)
Do you use Informatica?
(Data Warehouse, Migration)
4
Informatica The #1 Independent Leader in Data Integration
• Founded: 1993
• 2011 Revenue: $784 million
• 6-year Average Growth Rate:
20% per year
• Employees: 2,554
• Partners: 400+
• Major SI, ISV, OEM and
On-Demand Leaders
• Customers: 4,630
• > 70% of the Global 500
• Customers in 82 Countries
• Direct Presence in 26 Countries
• # 1 in Customer Loyalty Rankings
(6 Years in a Row)
$0
$100
$200
$300
$400
$500
$600
$700
$800
2005 2006 2007 2008 2009 2010 2011
5
Improve
Decisions
Business &
Operational
Intelligence
Data
Warehouse
Beyond Data Warehousing Empowering the Data-Centric Enterprise
Improve
Business
Processes
Improve
Efficiency &
Reduce Costs
Mergers
Acquisitions &
Divestitures
Acquire &
Retain
Customers
Outsource
Non-core
Functions
Governance
Risk
Compliance
Increase
Partner
Network
Efficiency
Increase
Business
Agility
Business Imperatives
Application
Portfolio
Optimization
Application
Retirement
Application
Consolidation
Customer,
Supplier,
Product Hubs
BPO
SaaS
Risk
Mitigation &
Regulatory
Reporting
B2B
Integration
Zero Latency
Operations
IT Initiatives
Data
Services
Data Migration
& Archiving
Master Data
Management
Data
Synchronization
B2B Data
Exchange
Data
Consolidation
Complex
Event
Processing
Ultra
Messaging
Data Integration Projects
6
The Tradition Approach 87% of Enterprises Use Hand-Coding for Data Integration
75% of enterprises reported
increased maintenance costs
1 Forrester Research, The State Of Enterprise IT Budgets: 2008, March 27, 2008 2 Forrester Research, “Addressing Data Integration Challenges with SOA”, 2007
Data
Warehousing
Data
Services
Data Migration
& Archiving
Master Data
Management
Data
Synchronization
B2B Data
Exchange
Data
Consolidation
Complex
Event
Processing
Ultra
Messaging
7
The Informatica Approach Comprehensive, Unified, Open and Economical platform
Data
Warehousing
Data
Services
Data Migration
& Archiving
Master Data
Management
Data
Synchronization
B2B Data
Exchange
Data
Consolidation
Complex
Event
Processing
Ultra
Messaging
8
9
Agenda
• Introduction Informatica
• Qlikview and reporting : data warehouse?
- What data do I get? Relationships? data profile!
- (bad) data quality? data quality!
- Extraction, Transform and Load of any data ETL!
• Qlikview and Informatica
• Data Virtualization
- faster access to new data
• Quick demo
10
Reporting: do you see unexpected results/values? (e.g. gender: M/F/1/0/missing values etc.)
11
Data Analysis
& Discovery
Using Informatica Analyst Tools to Profile your Data
Increase productivity and efficiency by enabling the business to
proactively take responsibility for data quality and reduce their
reliance on IT.
Data
Steward
12
Agenda
• Introduction Informatica
• Qlikview and reporting : data warehouse?
- What data do I get? Relationships? data profile!
- (bad) data quality? data quality!
- Extraction, Transform and Load of any data ETL!
• Qlikview and Informatica
• Data Virtualization
- faster access to new data
• Quick demo
13
Data Quality
Problem:
- many systems and data-sources having information about
customers/products/suppliers etc.
- need to profile, standardize, cleanse, de-dup: automatically!
- otherwise wrong/strange results in Data Warehouse or in new systems after
migration/conversion
- where ‘to do Data Quality’: at source? In Data Warehouse?
- when is ‘data quality’ finished?
Think about Qlikview-reports/dashboards!
14
Reporting: do you have/see Data Quality issues?
How is it solved?
What about Data Ownership?
15
What data is missing?
How do you measure data quality?
What data gives
conflicting information?
What data does not reflect
reality or is out of date?
What data is not
referenced or missing? What data or attributes
are repeated?
Completeness
What data is stored in non-
standard formats?
Conformity Consistency
Accuracy Duplications Integrity
Data Quality Dimensions
16
Data Quality issues, examples
COMPLEETHEID STANDAARDEN CONSISTENTIE DUPLICATEN INTEGRITEIT ACCURATESSE
17
Frequent Requirements
Data Analysis
& Discovery
Parsing
and
Standardization
Address
Validation
Matching &
De-duplication
Monitoring
&
Reporting
And do this for all domains & data
types…
18
Using Informatica Analyst Tools to Profile your Data
Data Analysis
& Discovery
Increase productivity and efficiency by enabling the business to
proactively take responsibility for data quality and reduce their
reliance on IT.
Data
Steward
19
Agenda
• Introduction Informatica
• Qlikview and reporting : data warehouse?
- What data do I get? Relationships? data profile!
- (bad) data quality? data quality!
- Extraction, Transform and Load of any data ETL!
• Qlikview and Informatica
• Data Virtualization
- faster access to new data
• Quick demo
20
Data Integration / ETL
- Extraction, Transformation and Load
- For Data Warehousing, conversions, migrations, testdata-management, MDM, …
- For Informatica this means:
- integration of Profiling, Data Quality and ETL
- independent vendor, so connectivity to almost all databases, incl. Teradata,
Netezza, Greenplum etc.
- low or very high data volumes, batch and/or realtime
- simple or complex environments
- no coding! Infrastructure-independent (database , OS)
- multi-user, multi-project
21
Data Integration / ETL: data mapping
• Source, target & transformation blocks
• connectors
• expressions
22
Transformation types “Lego blocks”
Joiner Expression Sorter
Lookup Aggregator Normalizer
Router Sequence
generator
Filter
Union Update strategy Transaction
control
Stored procedure External
transformation
Java
transformation
Source qualifier Association
transformation
Consolidation
transformation
Ranker SQL
transformation
Web services
… and more
23
Data Integration / ETL: workflow
24
Data Integration / ETL: monitoring
25
Data Integration / ETL: lineage
26
Agenda
• Introduction Informatica
• Qlikview and reporting : data warehouse?
- What data do I get? Relationships? data profile!
- (bad) data quality? data quality!
- Extraction, Transform and Load of any data ETL!
• Qlikview and Informatica
• Data Virtualization
- faster access to new data
• Quick demo
27
Qlikview connector for Informatica Powercenter
28
Qlikview connector for Informatica Powercenter
QVX
29
Agenda
• Introduction Informatica
• Qlikview and reporting : data warehouse?
- What data do I get? Relationships? data profile!
- (bad) data quality? data quality!
- Extraction, Transform and Load of any data ETL!
• Qlikview and Informatica
• Data Virtualization
- faster access to new data
• Quick demo
30
Data Virtualization
Problem:
- Enterprise Data Warehouse takes 3-9 months (modeling, extract, reports/dashboards)
- what if new data from a new source needs to be added to a report?
- another 3+ months? Or just 1 day?
- (complex) operational reporting?
Data Virtualization
31
Virtual
View
DATA CONSUMERS
(operational) DATA SOURCES
Portals
Messages Cloud Semi-structured Data Unstructured Data Application Database Mainframe Flat Files Database
Solution with Data Services (Federation)
Daily extract
(PowerCenter)
DWH / DM
SA/ODS
Daily update to
DWH / DM
‘operational extracts’
from DWH and/or
even directly from
Data Sources, accessible
via standard SQL
(Reporting Tool)
32
Quick demo!
Questions?