CS 345: Topics in Data Warehousing Thursday, October 14, 2004.

CS 345:Topics in Data Warehousing

Thursday, October 14, 2004

Review of Tuesday’s Class• Customer Relationship Management (CRM)• Dimension-focused queries

– Drill-across– Conformed dimensions

• Customer dimension– Behavioral attributes– Auxiliary tables

• Techniques for very large dimensions– Outriggers– Mini-dimensions

Outline of Today’s Class

• Bridge tables– Hierarchies– Multi-Valued Dimensions

• Extraction-Transformation-Load– Data staging area vs. data warehouse– Assigning surrogate keys– Detecting changed rows– Detecting duplicate dimension rows

More Outriggers / Mini-Dims• Lots of information about some customers, little info about others

– A common scenario• Example: web site browsing behavior

– Web User dimension (= Customer dimension)– Unregistered users

• User identity tracked over time via cookies• Limited information available

– First active date, Latest active date, Behavioral attributes– Possibly ZIP code through IP lookup

– Registered users• Lots of data provided by user during registration

– Many more unregistered users than registered users– Most attribute values are unknown for unregistered users

• Split registered user attributes into a separate table– Either an outrigger or a mini-dimension– For unregistered users, point to special “Unregistered” row

Handling Hierarchies• Hierarchical relationships among dimension attributes are common• There are various ways to handle hierarchies

– Store all levels of hierarchy in denormalized dimension table• The preferred solution in almost all cases!

– Create “snowflake” schema with hierarchy captured in separate outrigger table

• Only recommended for huge dimension tables• Storage savings have negligible impact in most cases

• What about variable-depth hierarchies?– Examples:

• Corporate organization chart• Parts composed of subparts

– Previous two solutions assumed fixed-depth– Creating recursive foreign key to parent row is a possibility

• Employee dimension has “boss” attribute which is FK to Employee• The CEO has NULL value for boss• This approach is not recommended• Cannot be queried effectively using SQL

– Alternative approach: bridge table

Bridge Tables• Customer dimension has one row

for each customer entity at any level of the hierarchy

• Separate bridge table has schema:

– Parent customer key– Subsidiary customer key– Depth of subsidiary– Bottom flag– Top flag

• One row in bridge table for every (ancestor, descendant) pair

– Customer counts as its own Depth-0 ancestor

– 16 rows for the hierarchy at right• Fact table can join:

– Directly to customer dimension– Through bridge table to customer

dimension

Customer 1

Customer 2

Customer 3

Customer 5

Customer 6

Customer 4

Customer 7

Fact Bridge Customer

cust_id

parent_id

child_id

cust_id

Bridge Table Example

parent_id child_id depth top_flag bottom_flag

1 1 0 Y N

1 2 1 Y N

2 2 0 N N

1 3 1 Y Y

3 3 0 N Y

1 4 1 Y N

4 4 0 N N

1 5 2 Y Y

2 5 1 N Y

5 5 0 N Y

1 6 2 Y Y

2 6 1 N Y

… … … … …

Using Bridge Tables in Queries• Two join directions

– Navigate up the hierarchy• Fact joins to subsidiary customer key• Dimension joins to parent customer key

– Navigate down the hierarchy• Fact joins to parent customer key• Dimension joins to subsidiary customer key

• Safe uses of the bridge table:– Filter on customer dimension restricts query to a single customer

• Use bridge table to combine data about that customer’s subsidiaries or parents

– Filter on bridge table restricts query to a single level• Require Top Flag = Y• Require Depth = 1

– For immediate parent / child organizations• Require (Depth = 1 OR (Depth < 1 AND Top Flag = Y))

– Generalizes the previous example to properly treat top-level customers

• Other uses of the bridge table risk over-counting– Bridge table is many-to-many between fact and dimension

Restricting to One Customer

parent_id child_id depth top_flag bottom_flag

1 1 0 Y N

1 2 1 Y N

2 2 0 N N

1 3 1 Y Y

3 3 0 N Y

1 4 1 Y N

4 4 0 N N

1 5 2 Y Y

2 5 1 N Y

5 5 0 N Y

1 6 2 Y Y

2 6 1 N Y

… … … … …

Restricting to One Depth

parent_id child_id depth top_flag bottom_flag

1 1 0 Y N

1 2 1 Y N

2 2 0 N N

1 3 1 Y Y

3 3 0 N Y

1 4 1 Y N

4 4 0 N N

1 5 2 Y Y

2 5 1 N Y

5 5 0 N Y

1 6 2 Y Y

2 6 1 N Y

… … … … …

Multi-Valued Dimensions

• Occasionally a dimension takes on a variable number of multiple values– Example: Bank accounts may be owned by one, two,

or even more customers (individual vs. joint accounts)

• Can be modeled using a bridge table• Bank transaction fact table

– Grain: one row per transaction– Dimensions: Date, Branch, TransType, Account,

Customer– Including Customer dimension would violate the grain

Multi-Valued Dimensions

account_id account_id

customer_id

account_id

customer_id

weight

FactTable

BridgeTable

AccountDimension

CustomerDimension

Account-relatedattributes

Customer-relatedattributes

• Weights for each account sum to 1• Allows for proper allocation of facts when using Customer dimension

Weighted Report vs. Impact Report

• Two formulations for customer queries• Weighted report

– Multiply all facts by weight before aggregating– SUM(DollarAmt * weight)– Subtotals and totals are meaningful

• Impact report– Don’t use the weight column– SUM(DollarAmt)– Some facts are double-counted in totals– Each customer is fully credited for his/her activity– Most useful when grouping by customer

Loading the Data Warehouse

Source Systems Data Staging Area Data Warehouse

(OLTP)

Data is periodically extracted

Data is cleansed and transformed

Users query the data warehouse

Staging Area vs. Warehouse

• Data warehouse– Cleansed, transformed data– User-friendly logical design– Optimized physical design

• Indexes, Pre-computed aggregates

• Staging area– Intermediate representations of data– “Work area” for data transformations

• Same server or different?• Separate staging server and warehouse server

– Run extraction in parallel with queries• Staging area and warehouse both part of same database

– Less copying of data is required

Alternating Server Approach

Warehouse

Staging

Warehouse

Staging

LoadNewData

Backup

Make a Copy

Staging

Warehouse

Swap Roles

Load the Copy

Surrogate Key Assignment• Maintain natural key → surrogate key mapping

– Separate mapping for each dimension table– Can be stored in a relational database table

• One or more columns for natural key• One column for surrogate key

• Need a separate mapping table for each data source– Unless data sources already use unified natural key scheme

• Handling multiple dimension rows that preserve history– 1st approach: Mapping table contains surrogate key for most

current dimension row– 2nd approach: Mapping table lists all surrogate keys that were

ever used for each natural key• Add additional columns to mapping table:

– Begin_date, End_date, Is_current_flag• “Late-arriving” fact rows can use historically correct key• Necessary for hybrid slowly changing dimension schemes

Detecting Changed Rows

• Some source systems make things easy– All changes timestamped → nothing to do!– Usually the case for fact tables

• Except for Accumulating Snapshot facts– For each source system, record latest timestamp

covered during previous extraction cycle• Some source systems just hold snapshot

– Need to detect new vs. changed vs. unchanged rows– New vs. old rows: Use surrogate key mapping table– Detecting changed vs. unchanged rows

• Approach 1: Use a hash function– Faster but less reliable

• Approach 2: Column-by-column comparison– Slower but more reliable

Handling Changed Rows

• Using a hash function– Compute a small summary of the data row– Store previous hash value in mapping table– Compare with hash value of current attribute values– If they’re equal, assume no change

• Hash table collisions are possible– Cyclic redundancy checksum (CRC)

• Commonly used hash function family• No collisions under local changes and byte reorderings

• Determine which attributes have changed– Requires column-by-column comparison– Store untransformed attribute values in mapping table– Choose slowly changing dimension approach based on changed

attributes

Dimension Loading Workflow

Natural key in mapping table?

Has rowchanged?

Which SCDtype?

Do nothing

Update row indimension table/Mark aggregates

Update row inmapping table

Insert row in dimension table

Insert row in mapping table

Insert row in dimension table

Update row inmapping table

Yes

No

Yes

NoType 1

Type 2

Duplicates from Multiple Sources

• Information about the same logical entity found in multiple source systems

• Combine info into single dimension row• Problems:

– Determine which rows reference same entity• Sometimes it’s hard to tell!• Referred to as the merge/purge problem• Active area of research

– Resolve conflicts between source systems• Matching records with different values for the same field• Approach #1: Believe the “more reliable” system• Approach #2: Include both values as separate attributes

Merge/Purge

• How can we determine that these are the same person?

• Fuzzy matching based on textual similarity• Transformation rules• Comparison to known good source

– NCOA: National Change Of Address database• Related application: Householding

– Can we determine when two individuals/accounts belong to the same household?

– Send one mailing instead of two

FName LName Address City Zip

B Babcock 3135 Campus Dr #112 San Mateo 94403

Brian Bobcock 3135 Compass Drive Apt 112 San Mateo 94403-3205

Next Week: Query Processing

• We’re done with logical database design• Next topic: how can the database answer

queries efficiently?• No textbook from here on

– Optional readings for each topic will be posted on the course web page

• Mostly research papers• Some readings from books on reserve in Math/CS library

• Tuesday’s topic: Query processing basics– Will be review if you’ve taken CS 245– Haven’t taken CS 245 → may want to read DSCB

Chapter 15.1-15.5, 15.8