Top Banner
Apache Atlas & Open Metadata Dataworks Sydney 2017 Nigel Jones, Software Architect IBM Ferd Scheepers Chief Information Architect ING
30

Open Metadata and Governance with Apache Atlas

Jan 22, 2018

Download

Technology

Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Open Metadata and Governance with Apache Atlas

Apache Atlas & Open Metadata

Dataworks Sydney 2017

Nigel Jones,

Software Architect

IBM

Ferd Scheepers

Chief Information Architect

ING

Page 2: Open Metadata and Governance with Apache Atlas

2

Open Metadata and Governance will allow…

… metadata to be captured when the data is created, moved with the data and

be augmented and processed by any of the vendor tools.

Page 3: Open Metadata and Governance with Apache Atlas

Open Metadata and Governance consists of:

1. Standardized, extensible set of metadata types

2. Metadata exchange APIs and notifications

3. Frameworks for automated governance

Open Metadata and Governance will allow you to have:

1. An enterprise data catalogue that lists all of your data, where it is located, its origin (lineage),

owner, structure, meaning, classification and quality

2. New data tools (from any vendor) connect to your data catalogue out of the box

3. Metadata being added automatically to the catalogue as new data is created and analysed

4. Subject matter experts collaborating around the data

5. Automated governance processes protect and manage your data

3

What is Open Metadata and Governance?

Page 4: Open Metadata and Governance with Apache Atlas

4

Positioning of Apache Atlas for Open Metadata

Open andUnified Metadata

Metadata repository

Apache Atlas

Metadata repository

IBM

Metadata repository

SAS

Open Metadata Repository ServiceOMRS

Open Metadata Access ServiceOMAS

Components defined

and being developed

by Open Metadata &

Governance project

Metadata

highway

Page 5: Open Metadata and Governance with Apache Atlas

• Apache Atlas provides an open community for developing the reference implementation

for open metadata and governance. In essence Apache Atlas delivers 2 main

capabilities:

• it plays a role of a metadata repository (Graph Database) for a metadata end-user tool

• and, it plays the important role of delivering the federated/unified metadata layer

across the entire landscape of an enterprise

• The software development governance from the Apache Software Foundation (ASF)

creates confidence that the technology will be maintained and enhanced as appropriate

in an equitable manner.

Role of Apache Atlas

5

Page 6: Open Metadata and Governance with Apache Atlas

… because Apache is mostly focused on development and we are missing a governance

body for managing the adoption of and compliance to the Open Metadata and Governance

standards. We envision the following roles for ODPI:

1. Be an advocate of the Open Metadata and Governance standards, make them visible

and their value understood.

2. Facilitate discussions around the Open Metadata and Governance standards evolution,

maintenance and development.

3. Test and sign-off compliance of vendor offerings to the Open Metadata and Governance

standards.

6

Doing all of this under Apache Atlas flag is not enough…

Page 7: Open Metadata and Governance with Apache Atlas

1. Hands-on Community members:

• ING

• IBM

• HortonWorks

2. Companies we have had conversations with:

• CIBC

• SAS

• Microsoft

• Oracle

• Informatica

• Waterline

• RBC

• DBS

7

Who is in ?

Page 8: Open Metadata and Governance with Apache Atlas

1. Ambition level:

• End of September 2017: Open Metadata working demo.

• Mid-December November 2017: first version of user access.

• Google for Data

2. Next steps:

• End of Q2 2018: production ready version of Virtual Data

Connector.

8

Timeline and next steps

Page 9: Open Metadata and Governance with Apache Atlas

About Me

https://www.linkedin.com/in/nigelljones

https://www.twitter.com/planetf1

[email protected]@

Page 10: Open Metadata and Governance with Apache Atlas

Objective

Why

How

Excite & Engage

Apache Atlas

Open

Metadata

Page 11: Open Metadata and Governance with Apache Atlas

Atlas has graduated!

DOB: 2015-05-05R: 0.8.1

Page 12: Open Metadata and Governance with Apache Atlas

Atlas Architecture

Storage Repository

Graph

Type System

REST API

Models

UI & Apps

Hooks &

Bridges

https://cwiki.apache.org/confluence/display/ATLAS/Open+Metadata+and+Governance

Page 13: Open Metadata and Governance with Apache Atlas

A reminder of our problem.. And solution

Open andUnified Metadata

Page 14: Open Metadata and Governance with Apache Atlas

Extend beyond Hadoop

++

Page 15: Open Metadata and Governance with Apache Atlas

Common Core Data model

Data Assets Governance Lineage

Glossary CollaborationModels & Reference

Data

Base Types, Systems &

Infrastructure

Metadata Discovery

https://cwiki.apache.org/confluence/display/ATLAS/Building+out+the+Open+Metadata+Typesystem

Page 16: Open Metadata and Governance with Apache Atlas
Page 17: Open Metadata and Governance with Apache Atlas

Open APIs - OMRS

Metadata Highway

Adapter

Plugin

Open Connector

Framework

https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=70258803

Page 18: Open Metadata and Governance with Apache Atlas

Open APIs - OMAS

OMRSGovernance

Engine OMAS

Glossary OMAS

Asset OMAS

Information View OMAS

++......

https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=70258799

Page 19: Open Metadata and Governance with Apache Atlas

OMAS – detail

Project ListMetadata Service

Data/Asset

Community Metadata Service

Landscape Definition Metadata Service

Asset CatalogMetadata Service

Classification and Mapping Metadata Service

Information View Metadata Service

Connector Directory Metadata Service

Governance Definitions Metadata Service

Information Process Metadata Service

Glossary and Taxonomy Metadata Service

AssetMetadata Service

DiscoveryMetadata Service

Governance ActionMetadata Service

Roles and AccessMetadata Service

Models and SchemaMetadata Service

Connector

https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=70258799

Page 20: Open Metadata and Governance with Apache Atlas

Businessmetadata

Structuralmetadata for

a data store

New glossary function for semantic processing

EMPNAME EMPNO JOBCODE SALARY

EMP

LOYE

E

REC

OR

D

Employee

Work Location

Annual Salary

Job Title

Employee IdEmployee Name

Hourly Pay RateManager Compensation Plan

HAS-A

HAS-A

HAS-AHAS-A

HAS-A

HAS-A

IS-A IS-A

SensitiveIS-A

Data00 3809890 6 7 Lemmie Stage 818928 3082 4 New York 4 27 DataStage Expert 1 45324 300 27 Code St Harlem NY 1 3

https://cwiki.apache.org/confluence/display/ATLAS/Area+3+-+Glossary

Page 21: Open Metadata and Governance with Apache Atlas

Replacing v1 Taxonomy (tech preview)

Categories

Terms

hierarchies

Rich Relationships

Classifications

Glossary

https://cwiki.apache.org/confluence/display/ATLAS/Area+3+-+Glossary

Page 22: Open Metadata and Governance with Apache Atlas

Open Discovery Framework

Open Framework

Plugins characterize data & relationships

Updates metadata with results

Initial implementation in master

https://cwiki.apache.org/confluence/display/ATLAS/Automated+metadata+discovery

Page 23: Open Metadata and Governance with Apache Atlas

Governance Action Framework

metadata drives enforcement

Classification (tag) based – scalable, glossary driven

Access, Masking, Filtering

Supports Apache Ranger but open APIs for others

Audit,Rights - Exception management, Rights, Privacy (to look at in future)

https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=70258801

Page 24: Open Metadata and Governance with Apache Atlas

Open ecosystem

https://cwiki.apache.org/confluence/display/ATLAS/Open+Metadata+and+Governance

Page 25: Open Metadata and Governance with Apache Atlas

Summary

Open Metadata

Enterprise Catalog

Discovery

Multi Vendor

Open, Layered

APIs

Metadata store

integration

Open Source &

Governance

ubiquitous

Standard Models

Page 26: Open Metadata and Governance with Apache Atlas

How can I get involved?

Discuss: Mailing List

Document, Explain: Wiki

Report, Design: Jira

Face to face

Code

Vendors!

https://cwiki.apache.org/confluence/display/ATLAS/Getting+Involved

Page 27: Open Metadata and Governance with Apache Atlas

Governance & Security BOF

Thursday 18:00

C4.7

Page 28: Open Metadata and Governance with Apache Atlas
Page 29: Open Metadata and Governance with Apache Atlas

Backup

Page 30: Open Metadata and Governance with Apache Atlas

VDC End to End

https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=69407333