Chapter 2-data-warehousingppt2517 vero

Chapter 2Chapter 2

Data WarehousingData Warehousing

Learning ObjectivesLearning Objectives

Understand the basic definitions and concepts of Understand the basic definitions and concepts of data warehousesdata warehouses

Describe data warehouse architectures (high Describe data warehouse architectures (high level). level).

Describe the processes used in developing and Describe the processes used in developing and managing data warehousesmanaging data warehouses

Explain data warehousing operationsExplain data warehousing operations Explain the role of data warehouses in decision Explain the role of data warehouses in decision

supportsupport

Learning ObjectivesLearning Objectives

Explain data integration and the extraction, Explain data integration and the extraction, transformation, and load (ETL) processestransformation, and load (ETL) processes

Describe real-time (active) data Describe real-time (active) data warehousingwarehousing

Understand data warehouse Understand data warehouse administration and security issues administration and security issues

Data Warehousing Data Warehousing Definitions and ConceptsDefinitions and Concepts

Data warehouseData warehouse

A physical repository where relational data A physical repository where relational data are specially organized to provide are specially organized to provide enterprise-wide, cleansed data in a enterprise-wide, cleansed data in a standardized format standardized format


Characteristics of data warehousing Characteristics of data warehousing Subject oriented Subject oriented Integrated Integrated Time variant (time series)Time variant (time series) Nonvolatile Nonvolatile Web based Web based Relational/multidimensional Relational/multidimensional Client/server Client/server Real-time Real-time Include metadata Include metadata


Data martData mart A departmental data warehouse that stores A departmental data warehouse that stores only relevant data only relevant data

Dependent data martDependent data mart A subset that is created directly from a data A subset that is created directly from a data warehouse warehouse

Independent data martIndependent data martA small data warehouse designed for a A small data warehouse designed for a strategic business unit or a department strategic business unit or a department


Operational data stores (ODS)Operational data stores (ODS)

A type of database often used as an interim A type of database often used as an interim area for a data warehouse, especially for area for a data warehouse, especially for customer information files customer information files


Enterprise data warehouse (EDWEnterprise data warehouse (EDW))

A technology thatA technology that provides a vehicle for provides a vehicle for pushing data from source systems into a pushing data from source systems into a data warehouse data warehouse

Metadata Metadata

Data about data. In a data warehouse, Data about data. In a data warehouse, metadata describe the contents of a data metadata describe the contents of a data warehouse and the manner of its use warehouse and the manner of its use

Data Warehousing Data Warehousing Process OverviewProcess Overview

Organizations continuously collect data, Organizations continuously collect data, information, and knowledge at an information, and knowledge at an increasingly accelerated rate and store increasingly accelerated rate and store them in computerized systemsthem in computerized systems

The number of users needing to access the The number of users needing to access the information continues to increase as a information continues to increase as a result of improved reliability and availability result of improved reliability and availability of network access, especially the Internet of network access, especially the Internet



The major components of a data The major components of a data warehousing process warehousing process Data sources Data sources Data extraction Data extraction Data loading Data loading Comprehensive database Comprehensive database Metadata Metadata Middleware tools Middleware tools

Data Warehousing ArchitecturesData Warehousing Architectures

Three parts of the data warehouseThree parts of the data warehouse The data warehouse that contains the data and The data warehouse that contains the data and

associated softwareassociated software Data acquisition (back-end) software that Data acquisition (back-end) software that

extracts data from legacy systems and external extracts data from legacy systems and external sources, consolidates and summarizes them, sources, consolidates and summarizes them, and loads them into the data warehouseand loads them into the data warehouse

Client (front-end) software that allows users to Client (front-end) software that allows users to access and analyze data from the warehouseaccess and analyze data from the warehouse







1.1. Information Information interdependence interdependence between organizational between organizational unitsunits

2.2. Upper management’s Upper management’s information needsinformation needs

3.3. Urgency of need for a Urgency of need for a data warehousedata warehouse

4.4. Nature of end-user tasksNature of end-user tasks

5.5. Constraints on resources Constraints on resources

6.6. Strategic view of the data Strategic view of the data warehouse prior to warehouse prior to implementationimplementation

7.7. Compatibility with existing Compatibility with existing systemssystems

8.8. Perceived ability of the in-Perceived ability of the in-house IT staffhouse IT staff

9.9. Technical issuesTechnical issues

10.10. Social/political factorsSocial/political factors

Ten factors that potentially affect the architecture selection decision:

Data Integration and the Data Integration and the Extraction, Transformation, Extraction, Transformation, and Load (ETL) Processand Load (ETL) Process

Data integrationData integration

Integration that comprises three major Integration that comprises three major processes: data access, data federation, processes: data access, data federation, and change capture. When these three and change capture. When these three processes are correctly implemented, data processes are correctly implemented, data can be accessed and made accessible to can be accessed and made accessible to an array of ETL and analysis tools and data an array of ETL and analysis tools and data warehousing environmentswarehousing environments


Extraction, transformation, and load (ETL)Extraction, transformation, and load (ETL)

A data warehousing process that consists of A data warehousing process that consists of extraction (i.e., reading data from a extraction (i.e., reading data from a database), transformation (i.e., converting database), transformation (i.e., converting the extracted data from its previous form into the extracted data from its previous form into the form in which it needs to be so that it can the form in which it needs to be so that it can be placed into a data warehouse or simply be placed into a data warehouse or simply another database), and load (i.e., putting the another database), and load (i.e., putting the data into the data warehouse)data into the data warehouse)



Issues affect whether an organization will Issues affect whether an organization will purchase data transformation tools or build purchase data transformation tools or build the transformation process itself the transformation process itself Data transformation tools are expensiveData transformation tools are expensive Data transformation tools may have a long Data transformation tools may have a long

learning curvelearning curve It is difficult to measure how the IT organization It is difficult to measure how the IT organization

is doing until it has learned to use the data is doing until it has learned to use the data transformation tools transformation tools


Important criteria in selecting an ETL toolImportant criteria in selecting an ETL tool Ability to read from and write to an unlimited Ability to read from and write to an unlimited

number of data source architecturesnumber of data source architectures Automatic capturing and delivery of metadataAutomatic capturing and delivery of metadata A history of conforming to open standardsA history of conforming to open standards An easy-to-use interface for the developer and An easy-to-use interface for the developer and

the functional user the functional user

Data Warehouse DevelopmentData Warehouse Development Direct benefits of a data warehouseDirect benefits of a data warehouse

Allows end users to perform extensive analysis Allows end users to perform extensive analysis Allows a consolidated view of corporate data Allows a consolidated view of corporate data Better and more timely information A Better and more timely information A Enhanced system performance Enhanced system performance Simplification of data access Simplification of data access

Data Warehouse DevelopmentData Warehouse Development

Indirect benefits result from end users using Indirect benefits result from end users using these direct benefits these direct benefits Enhance business knowledgeEnhance business knowledge Present competitive advantagePresent competitive advantage Enhance customer service and satisfactionEnhance customer service and satisfaction Facilitate decision makingFacilitate decision making Help in reforming business processesHelp in reforming business processes

Data Warehouse DevelopmentData Warehouse Development Data warehouse vendors Data warehouse vendors

Six guidelines to considered when developing a Six guidelines to considered when developing a vendor list:vendor list:1.1. Financial strengthFinancial strength

2.2. ERP linkagesERP linkages

3.3. Qualified consultantsQualified consultants

4.4. Market shareMarket share

5.5. Industry experienceIndustry experience

6.6. Established partnerships Established partnerships

Data Warehouse DevelopmentData Warehouse Development Data warehouse development approachesData warehouse development approaches

Inmon Model: EDW approach Inmon Model: EDW approach Kimball Model: Data mart approach Kimball Model: Data mart approach

Which model is best?Which model is best? There is no one-size-fits-all strategy to data There is no one-size-fits-all strategy to data

warehousing warehousing One alternative is the hosted warehouseOne alternative is the hosted warehouse

Data Warehouse DevelopmentData Warehouse Development Data warehouse structure: The Star Data warehouse structure: The Star

Schema Schema Dimensional modelingDimensional modeling

A retrieval-based system that supports high-A retrieval-based system that supports high-volume query access volume query access

Dimension tablesDimension tables

A table that address A table that address howhow data will be analyzed data will be analyzed


Data Warehouse DevelopmentData Warehouse Development Grain Grain

A definition of the highest level of detail that A definition of the highest level of detail that is supported in a data warehouse is supported in a data warehouse

Drill-downDrill-down

The process of probing beyond a The process of probing beyond a summarized value to investigate each of summarized value to investigate each of the detail transactions that comprise the the detail transactions that comprise the summary summary

Data Warehouse DevelopmentData Warehouse Development Data warehousing implementation issues Data warehousing implementation issues

Implementing a data warehouse is generally a Implementing a data warehouse is generally a massive effort that must be planned and massive effort that must be planned and executed according to established methodsexecuted according to established methods

There are many facets to the project lifecycle, There are many facets to the project lifecycle, and no single person can be an expert in each and no single person can be an expert in each area area


1.1. Establishment of Establishment of service-level service-level agreements and data-agreements and data-refresh requirementsrefresh requirements

2.2. Identification of data Identification of data sources and their sources and their governance policiesgovernance policies

3.3. Data quality planningData quality planning4.4. Data model designData model design5.5. ETL tool selectionETL tool selection

6.6. Relational database Relational database software and platform software and platform selectionselection

7.7. Data transportData transport8.8. Data conversionData conversion9.9. Reconciliation processReconciliation process10.10. Purge and archive Purge and archive

planningplanning11.11. End-user supportEnd-user support

Eleven major tasks that could be performed in parallel for successful implementation of a data warehouse (Solomon, 2005) :

Data Warehouse DevelopmentData Warehouse Development Some best practices for implementing a Some best practices for implementing a

data warehouse data warehouse (Weir, 2002):(Weir, 2002): Project must fit with corporate strategy and Project must fit with corporate strategy and

business objectivesbusiness objectives There must be complete buy-in to the project There must be complete buy-in to the project

by executives, managers, and usersby executives, managers, and users It is important to manage user expectations It is important to manage user expectations

about the completed projectabout the completed project The data warehouse must be built The data warehouse must be built

incrementallyincrementally Build in adaptability Build in adaptability

Data Warehouse DevelopmentData Warehouse Development Some best practices for implementing a Some best practices for implementing a

data warehouse data warehouse (Weir, 2002):(Weir, 2002): The project must be managed by both IT and The project must be managed by both IT and

business professionalsbusiness professionals Develop a business/supplier relationshipDevelop a business/supplier relationship Only load data that have been cleansed and Only load data that have been cleansed and

are of a quality understood by the organizationare of a quality understood by the organization Do not overlook training requirementsDo not overlook training requirements Be politically aware Be politically aware

Data Warehouse DevelopmentData Warehouse Development Failure factors in data warehouse projects:Failure factors in data warehouse projects:

Cultural issues being ignoredCultural issues being ignored Inappropriate architectureInappropriate architecture Unclear business objectivesUnclear business objectives Missing informationMissing information Unrealistic expectationsUnrealistic expectations Low levels of data summarizationLow levels of data summarization Low data quality Low data quality

Data Warehouse DevelopmentData Warehouse Development Issues to consider to build a successful Issues to consider to build a successful

data warehouse:data warehouse: Starting with the wrong sponsorship chainStarting with the wrong sponsorship chain Setting expectations that you cannot meet and Setting expectations that you cannot meet and

frustrating executives at the moment of truthfrustrating executives at the moment of truth Engaging in politically naive behaviorEngaging in politically naive behavior Loading the warehouse with information just Loading the warehouse with information just

because it is availablebecause it is available


data warehouse:data warehouse: Believing that data warehousing database Believing that data warehousing database

design is the same as transactional database design is the same as transactional database designdesign

Choosing a data warehouse manager who is Choosing a data warehouse manager who is technology oriented rather than user orientedtechnology oriented rather than user oriented

Focusing on traditional internal record-oriented Focusing on traditional internal record-oriented data and ignoring the value of external data data and ignoring the value of external data and of text, images, and, perhaps, sound and and of text, images, and, perhaps, sound and videovideo


data warehouse:data warehouse: Delivering data with overlapping and confusing Delivering data with overlapping and confusing

definitionsdefinitions Believing promises of performance, capacity, Believing promises of performance, capacity,

and scalabilityand scalability Believing that your problems are over when the Believing that your problems are over when the

data warehouse is up and runningdata warehouse is up and running Focusing on ad hoc data mining and periodic Focusing on ad hoc data mining and periodic

reporting instead of alertsreporting instead of alerts

Data Warehouse DevelopmentData Warehouse Development Implementation factors that can be Implementation factors that can be

categorized into three criteriacategorized into three criteria Organizational issuesOrganizational issues Project issuesProject issues Technical issues Technical issues

User participation in the development of User participation in the development of data and access modeling is a critical data and access modeling is a critical success factor in data warehouse success factor in data warehouse development development

Data Warehouse DevelopmentData Warehouse Development Massive data warehouses and scalability Massive data warehouses and scalability

The main issues pertaining to scalability:The main issues pertaining to scalability:• The amount of data in the warehouseThe amount of data in the warehouse• How quickly the warehouse is expected to growHow quickly the warehouse is expected to grow• The number of concurrent usersThe number of concurrent users• The complexity of user queries The complexity of user queries

Good scalability means that queries and other Good scalability means that queries and other data-access functions will grow linearly with the data-access functions will grow linearly with the size of the warehouse size of the warehouse

Real-Time Data WarehousingReal-Time Data Warehousing Real-time (active) data warehousingReal-time (active) data warehousing

The process of loading and providing data The process of loading and providing data via a data warehouse as they become via a data warehouse as they become available available

Real-Time Data WarehousingReal-Time Data Warehousing Levels of data warehouses:Levels of data warehouses:

1.1. Reports what happenedReports what happened

2.2. Some analysis occursSome analysis occurs

3.3. Provides prediction capabilities,Provides prediction capabilities,

4.4. OperationalizationOperationalization

5.5. Becomes capable of making events happenBecomes capable of making events happen

Real-Time Data WarehousingReal-Time Data Warehousing

Real-Time Data WarehousingReal-Time Data Warehousing

Real-Time Data WarehousingReal-Time Data Warehousing The need for real-time dataThe need for real-time data

A business often cannot afford to wait a whole day for A business often cannot afford to wait a whole day for its operational data to load into the data warehouse for its operational data to load into the data warehouse for analysisanalysis

Provides incremental real-time data showing every Provides incremental real-time data showing every state change and almost analogous patterns over timestate change and almost analogous patterns over time

Maintaining metadata in sync is possibleMaintaining metadata in sync is possible Less costly to develop, maintain, and secure one huge Less costly to develop, maintain, and secure one huge

data warehouse so that data are centralized for BI/BA data warehouse so that data are centralized for BI/BA toolstools

An EAI with real-time data collection can reduce or An EAI with real-time data collection can reduce or eliminate the nightly batch processes eliminate the nightly batch processes

Data Warehouse Data Warehouse Administration and Security IssuesAdministration and Security Issues

Data warehouse administrator (DWA)Data warehouse administrator (DWA)

A person responsible for the administration A person responsible for the administration and management of a data warehouse and management of a data warehouse

Data Warehouse Data Warehouse Administration and Security IssuesAdministration and Security Issues

Effective security in a data warehouse Effective security in a data warehouse should focus on four main areas:should focus on four main areas:

Establishing effective corporate and security Establishing effective corporate and security policies and procedurespolicies and procedures

Implementing logical security procedures and Implementing logical security procedures and techniques to restrict accesstechniques to restrict access

Limiting physical access to the data center Limiting physical access to the data center environmentenvironment

Establishing an effective internal control review Establishing an effective internal control review process with an emphasis on security and process with an emphasis on security and privacy privacy

Chapter 2-data-warehousingppt2517 vero

Technology

data warehouse metadata

data warehousingdefinitions

data access

relational data

cleansed data

data federation

extracted data

concepts data warehouse