BI Lecture Number 9

7/27/2019 BI Lecture Number 9

1/70

Dr. N.P. Singh,Professor (IT)15.10.13


2/70

Subject oriented Integrated

Near current data delivery Current data Detailed


3/70

An ODS is an environment where data fromdifferent operational databases is integrated.

The purpose is to provide the end usercommunity with an integrated view ofenterprise data.

It enables the user to address operationalchallenges that span over more than onebusiness function.


4/70

It is the right place to have a central version ofreference data that can be shared among differentapplication systems.

One way could be that the applications access thedata in the ODS directly. Another way is to replicate data changes from the

ODS into the databases of the legacy systems.

The ODS can help to integrate new and existingsystems. The ODS may shorten the time required to populate a

DW, because a part of the integrated data alreadyresides in the ODS.


5/70

The ODS provides improved accessibility to criticaloperational data.

With an ODS, organizations have a complete view of their

financial metrics and customer transactions. This is useful for better understanding of the customer and

to make well-informed business decisions. The ODS can provide the ability to request product and

service usage data on a real or near real-time basis. Operational reports can be generated with an improved

performance in comparison to the legacy systems.


6/70

Frequency is how often the ODS is updated, quitepossibly from completely different legacy systems,

using distinct population processes, and also takesinto account the volume of updates that areoccurring.

Velocity is the speed with which an update must take

place

from the point in time a legacy systemchange occurs, to the point in time that it must bereflected in the ODS.


7/70


8/70


9/70

How to position the ODS within the BIarchitecture


10/70


11/70


12/70


13/70


14/70


15/70

ODS in DSS Environment -Corporate Information Factory


16/70


17/70

class I where transactions were moved to the ODS in an immediate manner

from applications - in a range of 1 to 2 seconds from the moment thetransaction was executed in the operational environment until the

transaction arrived at the ODS. In this case, the end user could hardlytell the difference between an activity that had occurred in theoperational environment and the same activity as it was transmittedin the ODS environment.

class II where activities that occurred in the operational environment were

stored and forwarded to the ODS every four hours or so. In this case,there was a noticeable lag between the original execution of thetransaction and the reflection of that transaction in the ODSenvironment. However this class of ODS was much easier to build andto operate than a class I ODS.


18/70

class III in this case the time lag between execution in the operational

environment and reflection in the ODS is not four hours or so, but isovernight. In a class III ODS there is a noticeable time lag between the

execution of the transaction in the operational environment and thereflection of the transaction in the ODS environment. This type ofODS is relatively very easy to build.

class IV a class IV ODS is one that is fed from the data warehouse from

analysis created by the DSS analyst in the data warehouseenvironment and condensed down to a point where the results of theanalytical processing fit comfortably in the ODS. The input to the ODScan be either regular or irregular. This class of ODS is very easy to buildas long as the data warehouse has already been constructed.


19/70

Insurance Retail

Banking Telecommunications


20/70

How can I provide an up-to-date view of insuranceproducts owned by each customer for ourCustomer Relationship Management (CRM)system?

How can I consolidate all the information requiredto solve customer problems?

How can I decrease the turn-around time forquotes?

How can I reduce the time it takes to produce

claim reports?


21/70

How can we give suppliers the ability to co-manage our inventory?

What inventory items should I be adjusting

throughout the day? How can my customers track their own orders

through the Web?

What are my customers ordering across allsubsidiaries? What is the buying potential of my customer at

the point of sale?


22/70

What is the complete credit picture of mycustomer, so I can grant an immediate increase?

How can we provide customer service have a

consolidated view of all products and transactions? How can we detect credit card fraud while the

transaction is in progress? What is the current consolidated profitability

status of a customer?


23/70

Can we identify what our Web customers arelooking for in real-time?

Which calling cards are being used forfraudulent calls? What are the current results of my campaign? How can we quickly monitor calling patterns

after the merger?


24/70

How do I provide an up-to-date view of crossfunctional information for a particular

business process when the data is spreadacross several disparate sources?


25/70

To maximize customer satisfaction andprofitability, the ultimate data store would

contain all of the organizations operationaldata. This, of course, is not economical or

technically feasible at one place with onetechnology solution.


26/70

It seems to have the characteristics of an On-Line Transactional Processing (OLTP) system

while at the same time accommodating someof the attributes of a data warehouse (forexample, integrating and transforming datafrom multiple sources).


27/70


28/70


29/70


30/70

Transferring the data Data characteristics

The ODS environment ODS administration and maintenance


31/70


32/70


33/70


34/70


35/70


36/70

Analyzing the business requirements Defining the ODS type needed

Data modeling Defining and describing the different ODS

layers


37/70

The business scenarios were created from thefollowing three business questions:

Banking/finance: What is my customersentire product portfolio? Retail: How can my customers track their

own orders through the Web? Telecommunications: Which calling cards are

being used for fraudulent calls?


38/70


39/70


40/70

Fig describes the data flow for the order maintenancebusiness scenario.

Data is integrated and transformed from multipleheterogeneous data sources and used to populate the order

maintenance ODS. An order maintenance application will be used by both

customers and the customer service department to accessand update the ODS.

Changes made to the ODS through the order maintenanceapplication will flow back to the source systems using atrigger and apply mechanism. Regularly scheduled reports

will be created for the inventory management department.


41/70


42/70

Figure represents the data flow for the consolidated callinformation scenario.

Data is integrated and transformed from multiplehomogeneous data sources and used to populate the calling

transaction ODS. This data flow into the ODS is real-time. A custom-built fraud application will be used to verify calls

and trigger customer service when a suspect call is identified. The existing customer service and billing applications will be

migrated to the ODS, eliminating their data stores.

A follow-on phase will eliminate the customer data store.


43/70


44/70

An ODS type A includes real-time (or near-real-time) legacydata access andlocalized updates (data modifications are notfed back to the legacy systems). The localized updates wouldtypically include new data not currently captured in the

operational systems. An ODS type B includes the characteristics of an ODS type A

along with a triggerand apply mechanism to feed data backto the operational systems. Typically these feedback

requirements would be very specific to minimize conflicts. An ODS type C is either fully integrated with the legacy

applications or uses real-time update and access.


45/70


46/70

The ODS can be directly updated by front-end applications (such asCampaign Management, Customer Service, Call Center) or by the userdirectly through an application interface (such as a new Web application).

The ODS can be a source of data for the warehouse. Batch processes will

be used to populate the data warehouse. The ODS complements or extends the operational systems. It is not

intended to replace them. Although most sources will be used to populate both the ODS and the

data warehouse, two data acquisition streams will probably exist due to

the temporal differences in the data required. For example, the data warehouse may require a monthly inventory snapshot whereas

the ODS may require an up to the minute inventory status.


47/70

Data flows from the operational systems to the ODSthrough the data acquisition layer.

Updates to the ODS can be real-time, store andforward, and/or batch.

In a real-time environment changes are applied to theODS immediately, for example, using the sameoperational application.

A store and forward scheme may use tools such as

replication or messaging to populate the ODS. Changes which are only required daily, for example,could use a normal batch process.

Operational systems are not updated from an ODStype A.


48/70


49/70

The ODS type B includes the characteristicsof an ODS type A plus the additional feature

of an asynchronous triggering mechanism. This triggering mechanism is used to send

ODS changes back to the operationalsystems.


50/70


51/70


52/70

Data flows back and forth between the datasources and the ODS through the data

acquisition layer on a real-time basis. The ODS becomes the single source for much

of the corporations key operational data.


53/70


54/70

ODS DW

ORGN. SUBJECT SUBJECT

USERS LARGE NUMBER FEW

SIZE SMALL VERY LARGE

GROWTH 20 - 30 % Pa 50 - 180 % Pa

STRUCT. NORMALIZED DeNORMALIZED

UPDATE SEVERAL NONE

VOLATILE YES NO

METADATA YES YES

DESIGN PROCESS DRIVEN DATA DRIVEN


55/70

Issue Operational Warehouse

How Built One application at a time in

the legacy environment or one

subject area at a time in the

ODS

One or more subject

areas at a time

Requireme

nts

Known Vague

Data

Access

Smaller number of rows

retrieved in a single call

Large set of data is

scanned to retrieve

results

Critical to Daily Business operation Management

Decisions that may

affect profitability


56/70

Issue Operational Warehouse

Tuning Highly tuned for frequent

access to small amounts of

data

Tuned for infrequent access to

larger quantities of data

Data

volume

Volume needed for daily

operation

Larger volumes needed to support

statistical analysis, forecasting, adhoc reporting, and querying

Data

Retention

Data retrieved to meet

daily requirements

Data retained longer to support

historical reporting, comparison ,

analysis etc.

Data

currency

Must be up to the minute Usually does not require as high

availability as the production

environment unless world wide

access is necessary


57/70

Data Warehouse OLTP

Designed for analysis of business

measures by categories and

attributes

Designed for real-time business

operations

Optimized for bulk loads and large,complex, unpredictable queries

that access many rows per table

Optimized for a common set oftransactions, usually adding or

retrieving a single row at a time

per table

Loaded with consistent, valid data;

requires no real time validation

Optimized for validation of

incoming data during transactions;

uses validation data tables

Supports few concurrent users

relative to OLTPSupports thousands of concurrent

users


58/70

At one hand, ODS is decidedly operational.It provides high response time and highavailability and is certainly qualified to actas the basis of Mission Critical Systems.

On the other hand, ODS has some veryclear DSS features.

The ODS is integrated, subject orientedand supports some important kinds ofdecision support system.


59/70

ODS sits between the legacy applications & the DW. It is fed by integration & transformation programs. These program may be the same that feed to DW or different ODS

feeds data in to data warehouse. Some operational data traverse directly to DW through I/T layers.

Some data passes from the operational foundations in to I/T layers,then to ODS and on to DW. ODS is enablement of integrated, collective online processing It support online updates. Integrated many applications.

It provide view of the enterprise. It provide decision support processing

Complex Structure Underlying technology Design Monitoring & maintaining


60/70

Two types of users Farmers (same task repetitively, look for small amount of data, always

get what they are looking for, work in structured world-

Structured data

Structured processing

Structured procedures and so forth)

Explorers (antithesis of farmer, operate in random manner, does notknow what he/she is looking for, operate in heuristic mode, very largeset of data. Look for

Associations,

Patterns Relationship

Not yet discovered)

Nothing

Huge gold mines

Unstructured manner


61/70

Satisfy the need of both Classical Design:

DSS environment with a data model, which reflects theinformational needs of the corporation.

From the data model are generated normalized tables. Tables are known as logical model Tables are combined in to a form of physical design that

can be termed as lightly normalized design. Tables are combined on the basis of containing common

keys and general common usage. There is a fly in the ointment of this approach

Performance where many tables must be joined Performance where many occurrences of the data User may find it unnatural to join many tables.


62/70

Second Approach:

Volume & usage

Volume & usage of the data are factored in to design,a mutant form of normalization is achieved.

The normalization turn in to heavy normalization

A structure star join is created.


63/70

Star Join:

Two parts

Fact Tables (represent the structure that holds the

majority of the occurrence of the data, it combine dataand cross reference keys from a variety of other tables)&

Dimension tables ( contain data which is not terriblyvoluminous, related to fact tables by foreign key)

Fact tables are efficient to access because data has beenpre-joined in to table at the moment of loading


64/70

Star Join:

Usage of the data must be known in advance.

With out knowing pattern of access & usage of the

data it is difficult to design the fact tables.

One department may look differently for the

same of data in comparison to other.

Star join for finance may be different from join forproduction.


65/70

Normalized

Structure

Star Structure

Inefficient to access Efficient to access

Holds modest amountsof data

Holds large amount ofdata

Applicable to a wide

audience

Applicable to a restricted

audience

Handles updates Does not handle

updates


66/70

ODS environment serves both operational & DSSenvironment, the ODS is built with both a waterfalloperational & a spiral DSS methodology

Water fall methodology

Requirements gathering & assimilation

Analysis & systemization

Design

Programming

Testing

implementation


67/70

Legacy systems

ETL Tools

Operational Data store

Access Tools


68/70

Legacy systems: ERP, CRM, Web or any legacysystem, where in operations data is recorded.

ETL Tools: These tools are used to extract,

transform and load data from legacy systems tooperational data stores.

Operational data store

BI Tools: for analyzing the data & generating

reports


69/70

Legacy systems : data is extracted from e-mails,direct mails, telemarketing, kiosk, stores, call

centers, web using ETL tools and stored in

operational data stores. Operational data store.

Data warehouse.

BI tools or Analytical tools.


70/70

Gartner introduced the concept of zero latencystrategy which means any strategy that exploits theimmediate exchange of information across

technical and organizational boundaries to achievebusiness benefit. Organizations that can make decisions based on up-

to-the-second information and apply thosedecisions to operational systems and businessprocesses are known as ZLE.

Pull & Push Process

BI Lecture Number 9

Documents