A Metadata Architecture For Enterprise-Wide Data Sharing

1

A Metadata ArchitectureFor

Enterprise-Wide Data Sharing

2

Same Data RequirementsDifferent Functional Needs, Same Descriptions, Different Names

Department of Defense (DoD)Data Interoperability Challenge

Logistics

Components/Services

Medical

Procurement

Finance

Transportation

PersonnelCommand& Control

3

MIMMS

ATLAS II

CAMS

AMMRL

IMRL

NAOMISATAC

NSIPS

NALDA II

U2

SAUDI ARABIA

QATAR

KUWAIT

UAEMEXICO

JAPANFRANCE

UKCANADA

DVD Prime Vendors

(personnel?)

ARAMIS

CMIS

SUPPLY MGMT

SCS

CRIM

CAIMS

GATES

GDSS

WPS

IBS

CFM

DITTS

RFT-E

RFT-K

COP CSE

GCCS - C2

MTMCAMS

CMOSTCACCIS

G081BROKER

GOPAX

ADANSJALIS

MC-TFS

LIF

DSS

CC SS

AWRDS APS

SAILS

SAMS

TAMMISMEDSUP

ALP

JL ACTD

DOD CAV

TAMMISMEDASM

MEDSILS

UDR

VMI

NAV

NAC

MEDLOG

DBSS

MUFFIN

SNAP/SUADPS/FIMAR

FUELS - NEURS

CAIMS

ATAC

MANPERS

SBSS

CAS A

GO81

AFEMSDO35

MAARS II

MPS (BIC)

SCS

SASSY

FIMAR

CASEMIS

ISRAEL

OAS

ASIAN

SEATO

NATO

TCAIMS II

SARSS

SAAS-MOD

ROAMS

TAPDB

TAMMIS

ATAV (LIDB)

WARS

AMMIS

ARMS

PMIS

ATEMS

SPS/SDW

SAMMS

FLIS

JECPO

URD

DAAS LOTS

IC3

DoD JointApplicationsGCCS & GCSS

(IDEs)

CLASPENTAGON

CLASPACOM

CLASUSFKJTAV

CLASJFCOM

CLASEUCOM

CLASCENTCOM

UNCLASPENTAGON

UNCLASPACOMServer

UNCLASUSFK JTAV

Server

UNCLASJFCOMServers

UNCLASEUCOMServer

UNCLASCENTCOM(SOUTHCOM

SOCOM)Server

DISA

TRANSCOMGTN

DLA

USA

COALITION

USN USMC

USAF

MILSEALIFT

Commercial

JMAR

DARPA

GSA

TAPETAPETAPEISSE GUARDISSE GUARD

Freight Links

SCCR

CAIMS

USCG

DoD TARGET DATA SHARING ENVIRONMENTDoD TARGET DATA SHARING ENVIRONMENTA Logistics PerspectiveA Logistics Perspective

ATAC

MRP II

GCCS = Global Command & Control System

GCSS = Global Combat Support System

Essential AIS

Contributing AIS

Legend:

4

METADATAREPOSITORY

Defense DataDictionary System

(DoD Standardized Data Elements {SDE})

17000+ SDEs

The DDDS: The Current DoD Repository of DoD Standardized Data Elements

Intended to be the DoD Repository of Data Elements toSupport DoD Enterprise-wide Interoperable Data Sharing

5

PART I:The Problem

6

Current Problems

1. Incorrect data architecture abstraction level for representing Enterprise Level Data Elements for interoperable data sharing

2. Numerous redundant representations of Standardized Data Elements (SDEs) (DIFFERENT NAMES – SAME DEFINITION)

3. Incomplete, non-existent and/or, non-current SDE metadata

4. Inadequate categories of SDE metadata

5. Inadequate support / enforcement of data administration processes for data management

7

A Data Element Name and Data Element Definition Refresher:Two Data Element Metadata Attributes

• Data Element Name is a label given to establish Data Element identity

• Data Element Definition is a description providing complete, unambiguous meaning represented by a Data Element

• Name and Definition together provide the semantic context for the data item values represented by a Data Element

• Some key concepts/principles/facts about Data Element Naming and Definition:o NAME and DEFINITION are inseparableo NAME is a unique identifier for DEFINITION

o A NAME can be viewed as a kind of very short DEFINITION

o In order of precedence, DEFINITION creation should always precede NAME creationo Impossible to correctly NAME a data element with precision without a DEFINITION

o NAME and DEFINITION are at the core of the data integration / sharing process o Many data sharing issues arise from “bad” data element NAMING and DEFINITION practices

8

A Proposed Metadata Architecturefor Shared Enterprise Data Elements

• Focuses on solutions for:o Problem 1 – Incorrect data architecture abstraction levelo Problem 2 – Differently named data elements for the same data element concepto Problem 4 - Inadequate categories of standardized data element metadata

• Not a solution for every kind of impediment to interoperable data sharing

• Problems 3 and 5 require quality improvements in process execution and in data management governance and will be addressed in Part II.

Will begin by looking more closely at the fundamental metadata architecture levels…….

9

Design Layers of a Business Information System Database Architecture

• Specified Context Data Model Layer Roughly analogous to high level conceptual Entity-Relationship (E-R) data models of functional area/business domain, or Community of Interest (COI) data depicting the structure and relationships of entities and their attributes.

• Implemented Technology Data Model Layer Database schemas represented in a particular technology (SQL, COBOL, etc) based on fully attributed 3rd normal form logical data models

• Operational (Vendor) DBMS Data Model Layer Roughly analogous to a physical data model representing a particular vendor’s version of a technology based schema, i.e., Oracle SQL DBMS vs IBM DB2 SQL DBMS vs Sybase SQL DBMS, etc, etc

• Business Application View Data Model Layer Represents the application system access interface to a DBMS which preserves the separation and integrity of the database data from systems that operate on and manipulate the data

10

ATTRIBUTEENTITYSUBJECT

“PERSONNEL” FUNCTIONAL AREA DEPENDENT DATA MODEL TEMPLATES


“LOGISTICS” FUNCTIONAL AREA DEPENDENT DATA MODEL TEMPLATES

Design Layers for a Business Information System Database Architecture


“FINANCIAL” FUNCTIONAL AREA DEPENDENT DATA MODEL TEMPLATES

“SPECIFIED” DATA MODEL LAYER

(DoD FDAd Domain)

• May be represented in one, a combination of, or all of the following views:

o Entity w/attributes; no key designations; un-normalized; unresolved many – to – many relationshipso Key based entities; un-normalized; un-resolved many – to – many relationshipso Fully attributed; un-normalized; resolved or un-resolved many – to – many relationshipso Fully attributed; 3rd normal form; developed sub-typing; resolved many – to manys

• Current DDDS / DDA functional / subject area domains map to domains represented by designated DoD Functional Data Administrators (FDAds):

o Logistics DUSD(L)o Personnel USD(P&R)o Comptroller USD(C)o Health Affairs ASD(HA)o Etc, etc

• Each DoD “Subject Area” DAd should have a “specified” data model that represents the data element structures and relationships of functional area data element requirements for their respective functional area or community of interest.

Layer 1

Specified Context (Community of Interest) Data Model Layer

11

Layer 1

Implemented Technology Data Model Layer

TECHNOLGY DEPENDENT (e.g.,IDMS,)FULLY ATTRIBUTED, LOGICAL DATA MODEL

COLUMNTABLESCHEMATECHNOLGY DEPENDENT (e.g.,COBOL,)

FULLY ATTRIBUTED, LOGICAL DATA MODELCOLUMNTABLESCHEMA

TECHNOLGY DEPENDENT (e.g.,SQL,)FULLY ATTRIBUTED, LOGICAL DATA MODEL

“IMPLEMENTED” DATA MODEL LAYER(Data Architect / Modelers Domain)

COLUMNTABLESCHEMALayer 2

• 3RD Normal form ERD logical data model

• Represented in a technology dependent data architecture schema

• Technology driven / constrained data element naming

• Subject area entity and attribute templates are deployed into schema tables and columns that must conform to a particular chosen technology such as COBOL or SQL.

• Layer 1 attribute metadata is inherited by Layer 2 columns

• The relationship between Layer 1 and Layer 2 is one – to – many. That is to say that any attribute from Layer 1 may be deployed as a column into many Layer 2 schemas.



FUNCTIONAL AREA / SUBJECT MATTER DEPENDENT DATA MODEL TEMPLATES


(DoD COI DAd Domain)

12





Layer 1

Operational Vendor DBMS Data Model Layer

TECHNOLGY DEPENDENT (SQL, COBOL, ETC),FULLY ATTRIBUTED, LOGICAL DATA MODEL


COLUMNTABLESCHEMA Layer 2

DBMS COLUMN

DBMS TABLE

DBMS SCHEMA

DBMS DEPENDENT (e.g., Sybase)DBMS DATA MODEL

DBMS COLUMN

DBMS TABLE

DBMS SCHEMA

DBMS DEPENDENT (e.g., DB2)DBMS DATA MODEL

DBMS COLUMN

DBMS TABLE

DBMS SCHEMA

DBMS DEPENDENT (e.g., Oracle)DBMS DATA MODEL

“OPERATIONAL” DATA MODEL LAYER

(Domain of Database Administrators (DBAs))

Layer 3

• Roughly analogous to a physical data model

• Vendor’s versions of particular technology based schema such as SQL, i.e., Oracle SQL DBMS vs Informix SQL DBMS, vs Sybase SQL DBMS, etc, etc.

• Data element naming is bound by vendor’s implemented DBMS business rules for a particular technology based schema.

• Again, the relationship between Layer 2 and Layer 3 is one – to – many. That is to say that any column from a Layer 2 schema may be deployed as a DBMS column in many Layer 3 DBMSs.


13

Business Application View Data Model Layer



“SPECIFIED” DATA DATA MODEL LAYER


Layer 1




DBMS COLUMN

DBMS TABLE

DBMS SCHEMA

DBMS DEPENDENT (Oracle, DB2, Sybase, etc)DBMS DATA MODEL



Layer 3

APPLICATION VIEW COLUMN

APPLICATIONVIEW TABLE

BUSINESS INFORMATION

SYSTEM

BUSINESS APPLICATION SYSTEMVIEW DATA MODEL (Command & Control)




SYSTEM

BUSINESS APPLICATION SYSTEM“VIEW” DATA MODEL (Personnel App)




SYSTEM

BUSINESS APPLICATION SYSTEM“VIEW” DATA MODEL (Logistics App)

(Domain of Application System

Managers (SMs and/or PMs)

Layer 4


• Data element naming in conformance with functional area common business language terms

• Finally, with respect to a single DBMS, the relationship between Layers 3 and 4 is also one– to – many from 3 to 4. That is, a DBMS column may be deployed as view columns in many applications that may interface with a particular DBMS.

14

Layer 1 Person Given Name

Layer 2 Salesman First Name

Layer 4 Name

Layer 3 EFN

Standalone Database #1

Enterprise CommonData Element Concept “A”

Layer 1 Sailor Given Name

Layer 2 Sailor First Name

Layer 4 First Name

Layer 3 Sail_Frst_Nm



Enterprise Registry ofStandardized Data Elements (SDE)To Represent Common Enterprise

Data Element Concepts

Enterprise Data Element Concept “A”

Layer 1 Employee Given Name

Layer 2 Employee First Name

Layer 4 Given Name

Layer 3 Emp_Gv_Nam



SDE “A”: Person Given Name

SDE “A”: Employee First Name

SDE “A”: Sail_Frst_Nm

SDE “A”: Legal First Name

Effective Result: Four differently named versions, or, representations of the Enterprise common Data Element Concept, “A”, that will exist in the Registry as Standardized Data Elements.Redundancy and ambiguity is the consequence.

The Problem: Sourcing Enterprise “Context Independent” Data Element Standards from Enterprise “Context Dependent”

Data and Information Systems and Databases

Enterprise Registry ofStandardized Shared

Data Elements:The DDDS



Layer 1 Legal Given Name

Layer 2 Authoritative First Name

Layer 4 Legal First Name

Layer 3 Lg_Auth_Frst_Nm

15





Layer 1

Database System Architecture Design Steps

• Specified Context Data Model Layer

• Implemented Technology Data Model Layer

• Operational Vendor DBMS Data Model Layer

• Business Application View Data Model Layer


“IMPLEMENTED” DATA MODELLAYER(Data Architect / Modelers Domain)


DBMS COLUMN

DBMS TABLE

DBMS SCHEMA




Layer 3




SYSTEM

BUSINESS APPLICATION SYSTEM“VIEW” DATA MODEL



Layer 4

METADATAREPOSITORY


(DoD Standardized Data Elements)

17000+ SDEs

The DDDS RepositoryIntended to represent DoD globally shared enterprisestandard data elements. Thus, the repository should contain only one named data element standard for each unique enterprise level data element concept.


16





Layer 1

Database System Architecture Design Steps

• Specified Context Data Model Layer

• Implemented Technology Data Model Layer

• Operational Vendor DBMS Data Model Layer

• Application View Data Model Layer



COLUMNTABLESCHEMALayer 2

DBMS COLUMN

DBMS TABLE

DBMS SCHEMA




Layer 3




SYSTEM

BUSINESS APPLICATION SYSTEM“VIEW” DATA MODEL



Layer 4

METADATAREPOSITORY


The DDDS RepositoryIntended to represent DoD globally shared enterprisestandard data elements. Thus, the repository should contain only one named data element standard for each unique enterprise level data element concept. In reality, the repository contains many cases of differently named data elements that represent the same data element concept. The result is uncontrolled redundancy and ambiguity incapable of supporting seamless and interoperable data sharing.


A source for DDDS SDEs




One GIGANTICsemantic mess

17000+ SDEs

17

….But, Where’s the Beef ??

.…the Enterprise Data Element “Beef” ??

TheDDDS

“Big Bun”

18

Design Layers for a Business Information System Data Architecture

DATA ELEMENT CONCEPT STRUCTURE TYPE

DATA ELEMENT CONCEPT STRUCTURE

DATA ELEMENT CONCEPT

DATAELEMENT

CONCEPT CONCEPT STRUCTURE

CONCEPT STRUCTURE TYPE

CONCEPTUAL VALUE DOMAIN

CONCEPTUAL VALUE DOMAIN STRUCTURE

CONCEPTUAL VALUE DOMAIN STRUCTURE TYPE

VALUEDOMAIN

VALUE DOMAINSTRUCTURE

VALUE DOMAINSTRUCTURE TYPE

ISO 11179BUSINESS CONTEXT INDEPENDENTDATA ELEMENT REPRESENTATION

Functionally Independent Business Fact Semantic Templates (Globally Shared Data Elements)

(Domain of DoD Data Administration)

Layer 0

The Enterprise Data Element Layer

• ISO 11179 Naming and Definition

• Context Independent Data Elements

o Uniform Naming

o Uniform Semantics

o Uniform Value Domains

19




(DoD FDAd Domain)

Layer 1


“IMPLEMENTED” DATA MODEL LAYER(Data Architects / Modelers Domain)

COLUMNTABLESCHEMA

Layer 2

DBMS COLUMN

DBMS TABLE

DBMS SCHEMA



(Domain of Database Administrators (DBAs)) Layer 3




SYSTEM

BUSINESS APPLICATION SYSTEM“VIEW” DATA MODEL LAYER


Managers (SMs and/or PMs) Layer 4

The DDDS:

17000+ SDEs


One giganticsemantic mess-redundancies &

ambiguities

DDDS Source

DDDS Source

DDDS Source

DDDS Source




DATAELEMENT






VALUEDOMAIN






Layer 0

20



VALUEDOMAIN






DATAELEMENT

ATTRIBUTE

COLUMN

DBMS COLUMN

ENTITYSUBJECT

TABLESCHEMA

DBMS TABLE DBMS SCHEMA


FUNCTIONALLY DEPENDENT & TECHNOLOGYINDEPENDENT DATA MODEL TEMPLATESATTRIBUTE INHERITS DATA ELEMENT

“SPECIFIED” DATA MODEL (DoD FDAd Domain)

TECHNOLGY DEPENDENT &DBMS INDEPENDENT MODEL / SCHEMA

COLUMN INHERITS ATTRIBUTE“IMPLEMENTED” DATA MODEL)

(Data Architects / Modelers Domain)

DBMS DEPENDENT &APPLICATION VIEW INDEPENDENT

DBMS COLUMN (Oracle, DB2, etc) INHERITS COLUMN“OPERATIONAL” DATA MODEL


VIEW

VIEW COLUMN

BUSINESS INFORMATION APPLICATION

SYSTEM

VIEWCOLUMN

STRUCTURE

VIEW COLUMNSTRUCTURE

PROCESS

VIEWCOLUMN

STRUCTURETYPE

APPLICATION VIEWS OF DBMS TABLES & COLUMNS

“VIEW” DATA MODEL(Domain of Application System




META MODEL ARCHITECTURE SUPPORTING ENTERPRISE WIDE SHARED DATA

Data sharing occurs at the “operational andapplication” view layers. Made possible throughthe relationships between all layers represented by metadata in a repository that enables relating syntax,structure, and semantics from any layer to a common ISO 11179 standard representation.

METADATAREPOSITORY

Implemented Model

Operational DBMS

Application View

Specified Model ISO 11179




21




(DoD FDAd Domain)

Layer 1


“IMPLEMENTED” DATA MODEL LAYER(Data Modelers Domain)

COLUMNTABLESCHEMA

Layer 2

DBMS COLUMN

DBMS TABLE

DBMS SCHEMA




Layer 3




SYSTEM

BUSINESS APPLICATION SYSTEM“VIEW” DATA MODEL LAYER



Layer 4





DATAELEMENT






VALUEDOMAIN



ISO 11179 BUSINESS CONTEXT INDEPENDENT DATA ELEMENT

REPRESENTATION



Layer 0

DoD CORE DATA ELEMENT METADATA

REPOSITORY

Implemented Model Layer

Operational DBMS Layer

Application View Layer

Specified Model Layer

ISO 11179 Model Layer

22



VALUEDOMAIN






DATAELEMENT

ATTRIBUTE

COLUMN

DBMS COLUMN

ENTITYSUBJECT

TABLESCHEMA











VIEW

VIEW COLUMN


SYSTEM

VIEWCOLUMN

STRUCTURE


PROCESS

VIEWCOLUMN

STRUCTURETYPE






META MODEL ARCHITECTURE SUPPORTING ENTERPRISE WIDE SHARED DATA

Data sharing occurs at the “operational and application” view layers. Made possible throughthe relationships between all layers represented by metadata in a repository that enables relating syntax,structure, and semantics from any layer to a common ISO 11179 standard representation.

METADATAREPOSITORY

Implemented Model

Operational DBMS

Application View





23

Supply ItemResourceQuantity




Supply ItemResourceQuantity Supply Item

ResourceQuantity

Functional/OrganizationalContext Dependent “Specified” Model

Army LogisticsManagement

Navy LogisticsManagement

Technology Dependent“Implemented” Model

ANSI SQL

ANSI SQL

Vendor DependentSQL DBMS

“Operational” Model

“Oracle” DBMS

“Sybase” DBMS

Business ApplicationInformation System (AIS)

“View” Model

Navy UADPS (AIS)

Army SAMS (AIS)

Concepts

Data Element Concept

ValueDomain

Data Element

View ColumnNames

DBMS ColumnNames

SQL ColumnNames

AttributeNames

Business Fact SemanticTemplate Name

Metadata Repository Architecture of RelatedRepresentations of DoD Enterprise Shared Data Elements

in Support of Data and Information Sharing

The quantity of each type ofFederal Supply System materielitem contained in an identifiableinventory of materiel objects.

Data Element Definition:

Additional Data Element Structural Metadata:


MaterielResource

Physical ItemBalance

Quantity

PhysicalMeasure

ConceptualValue Domain

Data type characteristics,local definition, enumerated values ( if specific ), etc.

ISO 11179 Context Inde pendentData Element Representation Meta Model

An Optimal Application of an ISO 11179 Based Data Element Architecture for Resolving Disparate Representations of Shared Enterprise Data Elements



METADATAREPOSITORY

Defense DataDictionary System(DoD Standardized Data Elements)

17000+ SDEs

24

The “Optimal” Solution

25

Mat_Itm_Inv_Qt

MaterielInventoryQuantity

Materiel ItemInventoryQuantity

Mat_Inv_Qty

Materiel UnitInventoryQuantity

Supply UnitQuantity

Stocked MaterielQuantity Ships Stores

Quantity





ANSI SQL

ANSI SQL



“Oracle” DBMS

“Sybase” DBMS


“View” Model

Navy UADPS (AIS)

Army SAMS (AIS)

Concepts


ValueDomain

Data Element

CORE METADATAREPOSITORY

Implemented Model

Operational DBMS

Application View

Specified Model

ISO 11179 Model

View ColumnNames

DBMS ColumnNames

SQL ColumnNames

AttributeNames




The quantity of each type ofFederal Supply System materielitem contained in an identifiableinventory of materiel objects.




MaterielResource

Physical ItemBalance

Quantity

PhysicalMeasure


Data type characteristics,local definition, enumerated values ( if specific ), etc.


Example Logistics Application of an ISO 11179 Based Data Element Architecture for Relating Disparate Representations of Shared Enterprise Data Elements

26

Sail_Rat_Cde

SoldierRank Code

SailorRating Code

Sold_Rnk_Cd

Unit MemberRank Code

Squad MemberRank Code

Crew MemberRating Code Launch Team

MemberRating Code


Army PersonnelManagement

Navy PersonnelManagement


ANSI SQL

ANSI SQL



“Oracle” DBMS

“Sybase” DBMS


“View” Model

Navy (AIS)

Army (AIS)

Concepts


ValueDomain

Data Element


Implemented Model

Operational DBMS

Application View

Specified Model

ISO 11179 Model

View ColumnNames

DBMS ColumnNames

SQL ColumnNames

AttributeNames




The code that represents the level ofauthority and responsibility occupied by Person in a hierarchy of levels ranging from most superior to most subordinate in which each level issubordinate to levels above andsuperior to levels below.



Person Grade Code

HumanResource

Personnel Classifi- cation

Grade Code

PersonnelRankingMeasure


Data type characteristics, etc.


Example Personnel Application of an ISO 11179 Based Data Element Architecture for Relating Disparate Representations of Shared Enterprise Data Elements

27

Part II:

Implementing theMetadata Architecture

For Enterprise-wide DataSharing in a Legacy System

Environment

28

Table of Contents

1. A Realistic and Practical Approach to Achieve the Ideal Solution

2. How Do We Get to the Ideal?

3. Find Our Metadata

4. Perform Smart Meta-Data Mining

5. Find the Right Starting Layer

6. Reverse Engineer to Build the Upper Layers

7. Overall Forward Engineering Process

8. Process Statistics

9. Lessons Learned

29



VALUEDOMAIN






DATAELEMENT

ATTRIBUTE

COLUMN

DBMS COLUMN

ENTITYSUBJECT

TABLESCHEMA











VIEW

VIEW COLUMN


SYSTEM

VIEWCOLUMN

STRUCTURE


PROCESS

VIEWCOLUMN

STRUCTURETYPE






1.0 META MODEL ARCHITECTURE SUPPORTING ENTERPRISE WIDE SHARED DATA

Data sharing occurs at the “operational and application” view layers. Made possible throughthe relationships between all layers represented by metadata in a repository that enables relating syntax,structure, and semantics from any layer to a common ISO 11179 standard representation.

METADATAREPOSITORY

Implemented Model

Operational DBMS

Application View





30

Sail_Rat_Cde

SoldierRank Code

SailorRating Code

Sold_Rnk_Cd

Unit MemberRank Code

Squad MemberRank Code

Crew MemberRating Code Launch Team

MemberRating Code


Army PersonnelManagement

Navy PersonnelManagement


ANSI SQL

ANSI SQL



“Oracle” DBMS

“Sybase” DBMS


“View” Model

Navy (AIS)

Army (AIS)

Concepts


ValueDomain

Data Element


Implemented Model

Operational DBMS

Application View

Specified Model

ISO 11179 Model

View ColumnNames

DBMS ColumnNames

SQL ColumnNames

AttributeNames




The code that represents the level ofauthority and responsibility occupied by Person in a hierarchy of levels ranging from most superior to most subordinate in which each level issubordinate to levels above andsuperior to levels below.



Person Grade Code

HumanResource

Personnel Classifi- cation

Grade Code

PersonnelRankingMeasure


Data type characteristics, etc.


1.1 Example Personnel Application of an ISO 11179 Based Data Element Architecture for Relating Disparate Representations of Shared Enterprise Data Elements

31

2. How Do We Get to the Ideal?(or the least un-ideal)

• Find our metadata

• Perform smart metadata mining

• Pick the right starting layer

• Reverse engineer to build the upper layers

• Forward engineer to build standard-data based applications

32

3. Find Our Metadata

• Existing schemas within running applications as that’s the only place where data-truth resides

• Extract Cobol FDs within running applications for the same truth reason

• Finally, research metadata libraries like ERwin models

33

3.1 Where We Started

• DoD had 493 (Erwin) data models that were developed in the 1990s. There were 5709 tables and 16921 columns in these tables.

• We did not inventory each DoD Agency, but the key investigator (Hank Lavender) is very much aware of what, where, and how much all the schemas overlapped.

• This effort was to “prove the process”. We will soon start real Enterprise-wide data sharing projects.

34

4. Perform Smart Meta-Data Mining

• Pick backbone and rib-cage (HR, Finance, Inventory Customer Management, Sales) Applications

• Pick the most commonly used schemas across the enterprise that support the backbone and rib-cage applications

• Pick the subset of schemas that have the most commonly used tables (note: commonly used is different from exactly the same as…)

• Make Where-Used and Frequency-Used Matrices

35

4.1 Where Used & Frequency Matrices

IDM Data Model Counts Data Model Description Tables Columns Relationships

C-03 Budgets & Currency 56 178 53

C3-12 Command & Control 28 276 65

ES-07 Environmental Hazards 41 464 46

ES-08 Environmental Projects 28 185 40

LG-06 Transportation Operations 19 91 26

LG-23 Materiel Documentation 36 272 51

LG-28 Materiel Characteristics 45 225 48

PR-22 Training & Instruction 20 135 23

PR-31 Person Characteristics 36 118 41

Totals 309 1944* 393*542 Unique Data Element Concepts

Basic Types and Populations

36

4.1 (cont) Subject Areas Use Across IDM Schemas

SDM IDM SchemasSubject Areas C-03 C3-12 ES-07 ES-08 LG-06 LG-23 LG-28 PR-22 PR-31

EnvironmentalManagement X X X

HealthManagement X X X X

LogisticsManagement X X X XLogisticsOperations X X X X X

LogisticsPlanning X X X

MaterielMaintenance X X

MaterielManagement X X X X X X X

TransportationOperations X X X

PropertyManagement X X X

PersonnelManagement X X X X X X

ManagementAdministration X X X

37

4.1 (cont) Where Used & Frequency Matrices

Organization X X X X X X X

Person X X X X X X

Country X X X X X

Location X X

Task X X

Facility X X

Plan X X

Guidance X X

Geolocation X X X

SDM Subject Area Entity C-03 C3-12 ES-07 ES-08 LG-06 LG-23 LG-28 PR-22 PR-31

LogisticsManagementPersonnelManagementLogisticsOperations

LogisticsOperationsManagementAdministrationPropertyManagementLogisticsPlanningEnvironmentalManagement

PropertyManagement

IDM Schemas & Tables

Frequency of Use Matrix

38

5. Find the Right Starting Layer

Data Modeling Layer Description Start Here ?

Data Elements Context independent business factsemantic templates

NO, these are not databasemodels and have no context

Specified Data Model Technology independent datamodel templates

NO, as these are just templates and not database models

Implemented Data Model

DBMS independent databasedata models and hosts fordatabase object classes

OK- if you have “Erwin” like data models that can beresearched, tabulated, andextracted via Excel orSQL DDL

Operational DataModel

DBMS dependent and OperatingSystem specific

YES! This is the best as itmatches the reality ofoperating databases andapplications

View Data ModelDatabase application specificSQL views

No, as this is tooapplication-use specific, and not data model centric.

39

6. Reverse Engineer to Build The Upper Layers

• Import to appropriate layer

• Promote to higher data modeling layer

• Re-engineer the Specified Data Model layer

• Analyze to discover the Data Elements

• Build Data Element Model Metadata layer

40

6.1 Import to Appropriate Layer

IDM

Importing SQL DDL

41

6.1 Import to Appropriate Layer

IDM Tables

IDM

42

6.2 Promote to Higher DataModeling Layer

Promote IDM to SDM

43

6.2 Key Promotion Issues

Key Difference Between Subject-Entity-Attribute vs Schema-Table-ColumnModel of a subject area. Intellectual boundaries, notdata processing boundaries. Not a conceptual versionof a logical database. It’s a subject based data modeltemplate. Define once, use many times, differently inIDM models.

Subject-Entity-Attribute (SDM)

Model of a database schema that may involve attributesfrom multiple entities in one table, or attributes ofentities across multiple tables. Intended to be implemented within a DBMS as an operational database.

Schema-Table-Column (IDM)

Subject-SchemaNot related. This would then mean TransformationalRelationship.

Entity-Table Not related. This would mean TransformationalRelationship

Attribute-ColumnYes, Related. This allows define once, use many timesmodeling.

44

• Assign Entities to different Subjects• Reassign Entities to within Entities (sub-typing)• Reassign Attribute’s Semantics• Conform Attribute Names to Subject Area Scope• Reassign Attributes to different Entities• Reassign Attributes to different Data Elements

6.3 Re-engineer the SpecifiedData Model

Reassign Entity to Subject

SDM

45

6.3 Re-engineering the SpecifiedData Model

Reassign Entity to Subject Assign Attribute Meta Category Values

Reassign Attribute to Data Element Reassign Attribute to Entity

BASIC PROCESSES SDM

46

6.4 Reallocate Foreign Keys toEncapsulate Subject’s Entities

For Each Subject Area:

• Make a List of Entities

• Make a Subject Area Based E-R Model Diagram

• Delete Unnecessary Foreign Keys from Existing Entities

• Make New Foreign Keys Where Needed

• Export to E-R Diagrammer to Verify Result

• Recycle if Necessary

SDM

47

SDM

6.4 (cont) Reallocate Foreign Keys toEncapsulate Subject’s Entities

Validate/Create SDM Foreign Keys

48

SDM

6.4 (cont) Reallocate Foreign Keys toEncapsulate Subject’s Entities

Modify SDM Foreign Keys

49

6.5 Discover the Data Element

Promote SDM Attributes to Data Elements

50

6.6 Build Data Element ModelLevel Metadata

51

6.7 Data Element, Attribute,Column Differences

Key Differences Among Data Element, Attribute, Column

Characteristic Data Element Attribute Column

Context

Reason forexistence

Frequency ofuse

Example

Stand-alone independentbusiness fact template.

A characteristic of anentity that exists withinthe context of a subject.

A characteristic of atable that exists withinthe context of a schema.

Source of semantics andcommon general meaningfor classes of attributesand columns.

Source of value basedrefinement of the intentof the entity. The set ofall attributes fully definean instance of an entity.

Source of value basedrefinement of the intent of the table. Theset of all columns fullydefine an instance ofa table.

Defined once within thecontext of an entity.source of business factsacross one or more columns within one ormore tables.

Define once, use manytimes to provide semanticsto attributes or columns.

Defined once withinthe context of a table.

Identifier Asset IdentifierPerson IdentifierCustomer Identifier

Invoice Line Item• Part Number• Salesman Identifier• Customer Identifier

52

7.0 Overall Forward Engineering Process

• Import from higher level to lower level

• Map IDM to ODM legacy schemas to preserve existing systems environment and/or Generate new ODM schemas to replace legacy systems

• SQL Views can support legacy names or new names

• Generate Application

53

7.1 Import From Higher LevelTo Lower Level

Subject Area Data Model to Implemented Data Model

• Start Metabase IDM• Make the Target Schema• Pick an SDM Subject• Select the Root Entity• Create the Data Model Entity Tree• Perform Import (from SDM to IDM)• “Prune” Schema-Table Set to Just Those Needed• “Prune” Table-Column Set to Just Those Needed• Move Columns Among Tables as Needed• Import Next SDM Model and Perform “Pruning” Steps

• Mapping to New IDM from SDM Preserved-----Of Course!

54

7.1 (cont) Import From Higher LevelTo Lower Level

Subject Area Data Model to Implemented Data Model

Import SDM Entities to IDM Tables

55

Implemented Data Model to Operational Data Model

• Start Metabase ODM• Make the Target DBMS Schema• Pick an IDM Schema• Select the Root Table• Create the Data Model Table Tree• Perform Import (from IDM to ODM)• “Prune” DBMS Schema-DBMS Table Set to Just Those Needed• “Prune” DBMS Table-DBMS Column Set to Just Those Needed• Move DBMS Columns Among DBMS Tables as Needed• Import Next IDM Model and Perform “Pruning” Steps

• Mapping to New ODM from IDM Preserved-----Of Course!


56

Implemented Data Model to Operational Data Model


Import IDM Schema Tables to ODM DBMS Tables

57

ODM

7.2 Generate SQL

Generate DBMS SQL DDL

58

7.3 Generate Application

ODM

59

Task Name Notes Effort Metric

1 Find the Right Starting Point

With MS/Access based repository of data models. Close to about 100 models

2 days

2 Import to appropriate layer

Had to fix a number of data modeling errors in source CASE tool

40 hours for 10+ data models

3 Promote to Higher Data Modeling Layer

Required several cycles of distilling subjects

40 hours for 10+ models

4 Re-Engineer Had to re-engineer Fkeys, rename entities and some attributes. Also had to reconnect new attributes to “old columns.”

240 hours for 10+ models

8. Process Statistics

60

Task Name Notes Effort Metric

5 Abstract to Data Element

Required review of each attribute, and creation of MCVs, etc.

0.25 hours per attribute for 1000 attributes. So, 250 hours.

6 Build Data Element Model Level Metadata

Required generation of higher level concepts, value domains, etc.

80 hours

7 Import from higher level to lower level

Required design of new data models for new databases from data model templates, and/or just re-mapping to existing models

8 hours per model for 10 existing models and for 2 new models. Thus, 100 hours

8 Generate SQL Required specification of data types and lengths

20 columns per hour for 80 hours.

9 Input to and thenGenerate Application

Export of one Model from IDM 1 hour to export,1 hour to generate1st cut application

8. (cont) Process Statistics

61

9. Lessons Learned

• It can be done. However, it is not a walk in the park!

• It requires clear understanding of separation of Data Models. Data Element from Specified DMs, from Implemented DMs, and from Operational DMs. These are NOT transformations (conceptual to logical to physical). These are different data models.

• Subject Matter Experts are Essential, Critical, and Absolutely Necessary.

• It’s not top down. It’s bottom-up. But once built, use it top-down.

• You must have a metadata repository and data modeling tool that works at the enterprise level, and not just at the database or data model level.

• We made changes to the metadata repository system along the way. So, being able to change the meta model, entry and update and reports, is essential.

• Given that Entity reuse for just these ODS models was about 4x, the value for the data model template reuse in data warehouses and data marts is incalculable.

62

THANK YOU

Michael M. GormanFounder and President

2008 Althea LaneBowie, Maryland 20716-1518Phone: +1.301.249.1142Fax: +1.301.249.8955Email: [email protected]: <http://www.wiscorp.com>

Hank LavenderSenior Information Engineer

1310 Braddock PlaceAlexandria, Virginia 22314-1648Phone: +1.703.836.5900Fax: +1.703.836.8691Email: [email protected]: <http://www.amerind.com>

nI c

63

Back-ups

64

DoD CORE DATA ELEMENT METADATA

REPOSITORY

Implemented Model Layer

Operational DBMS Layer

Application View Layer

Specified Model Layer

ISO 11179 Model Layer

DoDEnterprise

InteroperabilityMetadata Repository

Core DoD Enterprise

Data ElementMetadata

Repository

Proposed DoD Metadata Repository

Complete Representationsof Data Element Metadata

Data Element Metadata Relationshipsto Multiple Categories of Metadata

Today

The Future

The Payoff

Seamless & TransparentInformation Interoperability

65

Mat_Itm_Inv_Qt

MaterielInventoryQuantity

Materiel ItemInventoryQuantity

Mat_Inv_Qty

Materiel UnitInventoryQuantity

Supply UnitQuantity

Stocked MaterielQuantity Ships Stores

Quantity





ANSI SQL

ANSI SQL



“Oracle” DBMS

“Sybase” DBMS


“View” Model

Navy UADPS (AIS)

Army SAMS (AIS)

View ColumnNames

DBMS ColumnNames

SQL ColumnNames

AttributeNames

The quantity of each type ofFederal Supply System materiel item contained in an identifiable inventory of materiel objects.



Data type characteristics,local definition, numerated values ( if specific), etc.

The Current DoD Architecture for Defining Standard Data Element Representations of Shared Enterprise Data Elements

METADATAREPOSITORY

Defense DataDictionary System(DoD Standardized Data Elements)

16000+ SDEs Business Rule: Only one named representation is permitted to exist in the repository as an Enterprise SDE.

(SDE Access Name)

66

METABASEREPOSITORY

Implemented Data Model (IDM)

Specified Data Model (SDM)

ISO 11179 Data Element Templates

Operational DBMS Model (ODM) SQL DBMS

ODMLAYER

XML Schema Info Tables

OUTPUT ODM LAYER

HUMAN RESOURCESDATABASE

Output – XML Wrapped Metadata

Output Schema Information

Feedback

DoDGlobal Information

Grid (GIG)

A Metadata Architecture For Enterprise-Wide Data Sharing

Documents

database data

data integration

coi data

data element naming

data sharing issues

physical data model

data management governance

data item values