When Facts and Dimensions Alone Aren't the Answer: Logically Reversing the Star Schema

When Facts and Dimensions Alone Aren't the Answer:

Logically Reversing the Star Schema

Kathryn Watson

Solution Architect at Perficient

Economist & Financial Analyst by education,

BI Professional by passion

Email: [email protected]

LinkedIn: www.linkedin.com/pub/kathryn-watson/a/431/354/

Kathryn Watson

What is a Reverse Star Schema?

Why and when would I use a Reverse Star Schema?

How would I implement a Reverse Star Schema?

What about data integrity?

While addressing these questions, we’ll walk through a

real life implementation within the healthcare industry.

The Questions We’ll Answer Today:

The Star Schema has long been the foundation of

effective BI applications● Optimized for the speedy return of business facts and their

corresponding descriptors

● Supports the manner in which businesses ask typical business

questions

Let’s Start With the Basics

Show me A, filtered by B and aggregated by C & D

Let’s Start With the Basics (cntd.)

A

B C

D

Show me Z, filtered and aggregated by W, X & Y

Let’s Start With the Basics (cntd.)

Z

W X

Y

Data warehouses commonly include multiple facts

These multiple facts often share dimensions, called

conformed dimensions

Let’s Start With the Basics (cntd.)

Conformed dimensions facilitate answering multi-

faceted business questions.

Show me A & Z, aggregated and/or filtered by N

Let’s Start With the Basics (cntd.)

A

B C

D

Z

W X

Y

N

As BI implementations mature, the questions your

users are asking often do as well.

Show me Z, filtered by B & C and aggregated by Y

And Continue Along the BI Maturity Model

A

B C

D

Z

W X

Y

N

In this scenario:● B and C are not dimensions of Z

● B and C are not subsets of Y

● B and C are not directly related to the Fact (Z) they are required to

affect

And Continue Along the BI Maturity Model (cntd.)

A

B C

D

Z

W X

Y

N

We’ve now encountered a very practical use case for

the Reverse Star Schema.

And Continue Along the BI Maturity Model (cntd.)

A typical Reverse Star Schema flips the concept of

what information is being returned to the user. Rather

than factual information being the star of the show, the

dimensional information is the main event.

In all Reverse Star Schemas, the relationships between

facts and dimensions are flipped; rather than a one-to-

many dimension to fact relationship, the Reverse Star

Schema introduces a one-to-many or many-to-many

fact to dimension relationship.

So, What is a Reverse Star Schema?

Sometimes, the return of the intersection of

dimensional data is truly all the user needs

More often, the Reverse Star Schema is the avenue

to establish cause-and-effect relationships amongst

indirectly related data concepts

When and Why Would I Use This?

Let’s return to the scenario we showcased previously:

show me Z, filtered by B & C and aggregated by Y

When and Why Would I Use This? (cntd.)

A

B C

D

Z

W X

Y

N

B & C’s relationship to Z is established through fact A

and conformed dimension N

When and Why Would I Use This? (cntd.)

A

B C

D

Z

W X

Y

N

The values of A & N aren’t of interest to the user

A & N are simply avenues through which the needed

attributes are accessed

When and Why Would I Use This? (cntd.)

A

B C

D

Z

W X

Y

N

Let’s apply this concept to the real world:

When and Why Would I Use This? (cntd.)

One of the nation’s top cancer care centers was seeking an on-

demand means to identify patient cohorts.

Physicians should be able to refine cohorts by basic tumor

information and various disconnected factors of cancer care:

Tumors – initial presentation date and type

Histologies – cancer type and structure over time

Stagings – cancer severity

Statuses – active, in remission, etc.

Treatments – chemotherapy, radiation, other

A filter applied to any of these concepts should affect the results

of all concepts, even though the concepts are independent of

each other.

Let’s apply this concept to the real world (cntd.):During the analysis phase, it was confirmed that everything was

directly related to the patient’s instance of cancer (Tumor).

All other concepts, though factual concepts in their own right, for

this purpose were informational.

It’s all about the Tumor. Everything else is informational.

When and Why Would I Use This? (cntd.)

Much like our use case scenario, a multi-fact Star Schema

emerged.

● 5 Facts with Independent Dimensions● Tumors/Documents (F_CLNCL_DOC)

● Histologies (F_TMR_HISTOL)

● Stagings (F_TMR_STG)

● Statuses (F_TMR_STAT)

● Treatments (F_TMR_TRTMT)

● 1 Conformed Dimension shared by all facts● Tumors (D_PT_TMR)

Let’s apply this concept to the real world (cntd.):

When and Why Would I Use This? (cntd.)

Let’s apply this concept to the real world (cntd.):

When and Why Would I Use This? (cntd.)

Implemented in OBIEE, the Dimensional Model = OBIEE Physical

Layer, except for:

● EIW Dimensions● Date dimension aliased for each of its purposes (17 aliases)

● Provider dimension aliased for each of its purposes (3 aliases)

● 4 Additional Facts● Users stated the requirement to analyze treatments as separate entities for some use-

cases, and as consolidated activities in other use-cases

● This is achieved by placing database views on top of F_TMR_TRTMT, with one view

for each treatment type

Let’s apply this concept to the real world (cntd.):

When and Why Would I Use This? (cntd.)

Let’s apply this concept to the real world (cntd.):

When and Why Would I Use This? (cntd.)

Let’s apply this concept to the real world (cntd.):

When and Why Would I Use This? (cntd.)

But the classic Star Schema did not support our main

requirement:

A filter applied to any of these concepts should affect the results of all

concepts, even though the concepts are independent of each other.

It’s all about the Tumor. Everything else is informational.

In the classic Star Schema design, each fact and its dimensions

are independent of the other facts.

Filters applied to the dimensions of one fact do not affect the

results returned in another fact.

Ok, How Would I implement This?

To fill our requirement that filters applied to the dimension of one

fact should affect the results returned in another fact:

● “Sub-Facts” (Statuses, Stagings, Histologies, Treatments) should be

treated as dimensions of the central fact (Tumors/Documents) through

the conformed dimension (Tumors).

● This would create a reverse snowflake schema● One fact: F_CLNCL_DOC (Tumors/Documents)

● Central Dimension: D_PT_TMR (Tumors)

● All other tables are dimensions that flake off of D_PT_TMR and their “Sub-Facts”.

(One to many join between D_PT_TMR and other facts reversed, becoming many to

one)

Continuing with our real world example:

Ok, How Would I implement This? (cntd.)

So, should our Dimensional Model/RPD Physical Layer be re-

designed to support this?

No.

Continuing with our real world example:

Ok, How Would I implement This? (cntd.)

The Dimensional Model/RPD Physical Layer should remain As Is

(a classic Star Schema)

● The “Sub-Facts” truly are facts/transactions. They can be analyzed

independently

● Current requirements dictate that the “Sub-Facts” are simply avenues

to descriptive attributes

● However, future requirements may dictate the need to analyze these

“Sub-Facts” individually; we don’t want to lose this capability through a

redesign

Continuing with our real world example:

Ok, How Would I implement This? (cntd.)

The Reverse Star Schema is built logically in the RPD BMM layer:

● Individual Dimension Tables are brought over from the Physical Layer

● The Logical Table Source (LTS) for each Dimension Table is edited to

include its “Sub-Fact” table as well as D_PT_TMR

Continuing with our real world example:

Ok, How Would I implement This? (cntd.)

The Reverse Star Schema is built logically in the RPD BMM layer

(cntd.):

● The dimension table is then logically joined directly to the central fact

(FACT_ALS_F_CLNCL_DOC)

Continuing with our real world example:

Ok, How Would I implement This? (cntd.)

The Reverse Star Schema is built logically in the RPD BMM layer

(cntd.):

● The joins in the Physical Layer remain the same.

Continuing with our real world example:

Dimension DimensionFact Fact

Ok, How Would I implement This? (cntd.)

This process is completed until all dimension tables join directly

to the central fact (FACT_ALS_F_CLNCL_DOC) in the BMM

“Sub-Facts” were not brought in as logical tables in the BMM for

this implementation, as there were no requirements for them.

● These “Sub-Facts” can be brought in as independent facts later if

needed, creating a multi-facted Star Schema

● If a “Sub-Fact” is brought in at a later time, it need simply be joined to

its relevant Dimensions; OBIEE will recognize that only one table in the

LTS need be used for the Join

Continuing with our real world example:

Ok, How Would I implement This? (cntd.)

Continuing with our real world example:

What about data integrity?

Continuing with our real world example:Aggregation is fine so long as your aggregations are all count

distinct.

But what about when your aggregation is something other than a

count distinct?

Ay, there’s the rub! Luckily, OBIEE has a quick and easy

solution: level based measures and/or measures based upon

dimensions

What about data integrity?

Continuing with our real world example:We had to set our non-count distinct aggregations to be Based on

Dimensions in order to achieve correct aggregation of our

measures

● For the conformed dimension (DIM_ALS_D_PT_TMR or Tumors), take

the Distinct Average of the measure

● For all Other Dimensions, simply Average the Measure

What about data integrity?

Continuing with our real world example:Why does this work?

● OBIEE will first average distinct the measure at the Tumor level.

Since the Tumor level is the lowest granularity, this will equate to the

original measure

● OBIEE will then average the population of these distinct averages by

whatever other dimensions are brought into the request

What about data integrity?

Continuing with our real world example:Why is this necessary?

● In our data set, our “Sub-Facts” can have multiple instances per tumor.

● We have transitioned our “Sub-Facts” to act as avenues between their

own dimensions and the central fact (FACT_F_CLNCL_DOC) for the

formation of the Star Schema in the BMM

● As a result, we can (and do) experience “ballooning”

What about data integrity?

Continuing with our real world example:Why is this necessary (cntd.)?

Two Tumors, each with one row of data:

vs.

Same two Tumors, with Staging Date added to the query. The 1st Tumor now has two rows

of data, one for each of its Staging Dates. The 2nd Tumor still has only one row of data, as it

has only one Staging Date.

What about data integrity?

Continuing with our real world example:Why is this necessary? (cntd.)

● With the inflated data, we can’t simply average: tumors that have

ballooned out would be weighted more heavily in our average

calculation than those that have not

349.67

If the Aggregation Rule in our RPD was simply set to “Average,”

OBIEE would calculate the average of the above as 349.67

[ (493 + 493 + 63) / 3 ]

What about data integrity?

Continuing with our real world example:Why is this necessary? (cntd.)

● To remove the “weight” given to a measure when its tumor appears in

the results more than once, we have to tell OBIEE to base the

aggregation on Dimensions

What about data integrity?

Continuing with our real world example:Why is this necessary? (cntd.)

● OBIEE would first calculate the Average at the Tumor Level AVG(DISTINCT "SCD-Tumor Management"."FACT_ALS_F_CLNCL_DOC"."Follow Up Time")

What about data integrity?

Continuing with our real world example:Why is this necessary? (cntd.)

● OBIEE would then average this averageAVG("SCD-Tumor Management"."FACT_ALS_F_CLNCL_DOC"."Follow Up Time")

● Our average calculates correctly, at 278[ (493 + 63) / 2 ]

Questions?

Additional Implementation Notes

Non Star Schema Structures:● Non-Star Schema structures can work and do not necessarily

cause incorrect results

● However, they are not ideal and can greatly complicate

implementation, expansion, and maintenance

Additional Implementation Notes (cntd.)

Non Star Schema Structures (cntd.):● To demonstrate, let’s start with a simple snowflake structure

Additional Implementation Notes (cntd.)

Non Star Schema Structures (cntd.):● And then create a Dimensional Hierarchy on

DIM_PROVIDERS

Additional Implementation Notes (cntd.)

Non Star Schema Structures (cntd.):● OBIEE names your hierarchy DIM_PATIENTSDim, but you

created it on DIM_PROVIDERS. Why?

● OBIEE considers the dimension that is joined directly to the fact to be

the “true”, as it’s the lowest level dimension

● DIM_PROVIDERS is connected to DIM_PATIENTS at a higher level,

and OBIEE has consolidating both tables into one dimensional

hierarchy, with DIM_PROVIDERS the parent level of DIM_PATIENTS

Additional Implementation Notes (cntd.)

Non Star Schema Structures (cntd.):● What would have happened if DIM_PATIENTS had 2 flakes?

Dimension

Dimension Dimension

Dimension

Fact

Additional Implementation Notes (cntd.)

Non Star Schema Structures (cntd.):● You’d end up with 1 dimensional hierarchy with 2 separate

trees

Additional Implementation Notes (cntd.)

Non Star Schema Structures (cntd.):● So why does this complicate things?

● Level-based measures are based upon Dimensional Hierarchies

● In a Snowflake Schema, constructing many types of level-based

measures becomes impossible, or at the least very complex.

Additional Implementation Notes (cntd.)

Non Star Schema Structures (cntd.):● Let’s calculate the average Follow-Up Days for a Provider by

Age Bucket

● Because Providers and Age Buckets are in the same

Dimensional Hierarchy, we can’t do this calculation.

● OBIEE limits you to only one level per Dimensional Hierarchy

Additional Implementation Notes (cntd.)

RPD Consistency Check Warnings:● Because you have mapped multiple tables in the LTS’s of

your Dimensions, OBIEE expects you to use those tables in

the Column Mappings

● If you do not make use of a mapped table, the Consistency

Checker will give you a warning:

Additional Implementation Notes (cntd.)

RPD Consistency Check Warnings (cntd.):● To resolve this warning, create a new Logical Column in your

table and map it to a column in the unused table

Additional Implementation Notes (cntd.)

Fabricated Data: Missing Data● In our data set we were faced with the fact that a Tumor may not

be present in all of the major subject areas (Statuses, Stagings,

Histologies, Treatments)

● In the instances where a Tumor was not present in a subject

area, that Tumor would fall out of the data population if a

dimension from that subject area was brought in

Users did not want this fallout to occur and they emphasized

that it was just as important to see where a Tumor did not

have information in a given subject area as it was to see what

information a tumor did have in a given subject area

Additional Implementation Notes (cntd.)

Fabricated Data: Missing Data (cntd.)● RPD Resolution Possibility

● This fallout could have been prevented through Outer Joins in the OBIEE

RPD

● The Logical Mappings of all dimension attributes could be changed to

include an IFNULL clause [It was required that a NULL/missing object state

its status (Missing)]

● With hundreds of attributes this is neither sustainable nor realistic

Additional Implementation Notes (cntd.)

Fabricated Data: Missing Data (cntd.)● Data Creation in Database Tables Resolution Possibility

● Each and every Dimension Table in our Dimensional Model has an entry for

a key of -2 and attribute codes, names, descriptions, etc. of ‘!Record Does

Not Exist in Source’

● At the end of the ETL process, the central conformed dimension

(D_PT_TMR) was LEFT OUTER JOINed to each of the dimensional

database “Sub-Facts”

● F_TMR_STAT

● F_TMR_STG

● F_TMR_HISTOL

● F_TMR_TRTMT

Additional Implementation Notes (cntd.)

Fabricated Data: Missing Data (cntd.)● Data Creation in Database Tables Resolution Possibility (cntd.)

● Where a match was not found in a Fact table, a row was inserted in that

Fact table for the missing Tumor ID. All Dimensional Keys were set to -2,

!Record Does Not Exist in Source

● Users then gained the ability to pull tumors with missing information.