DH03 Educating for the Future: Data Engineering Techniques · Agenda • Introduction • Master Data Management - Principle • ETL Overview –“Transforming”

DH03 – Educating for the Future:

Data Engineering Techniques

https://education.phuse.eu/eftf/data-engineering/

Contribution:

Andy Richardson, Amy Gillespie, Beate Hientzsch, Berber Snoeijer,Beverly Hayes, JatinPatel, Karnika Dalal, Mark Bynens, Mike Carniello, Parag Shiralkar, Paul Slagle, Ralf Goetzelmann, Rohit Banga, Sagar Jain, Susan Olson, Vijay Pasapula, Vince Marinelli, Xiaohui Wang

Special thanks:Ian Fleming, James McDermott, Wendy Dobson

Presentation:Guy Garrett, Achieve IntelligenceRenu Shukla, Janssen Research & DevelopmentMohit Juneja, LyfeScience

Agenda

• Introduction

• Master Data Management - Principle

• ETL Overview – “Transforming”

• Data Lake – “Trawling”

• Data Marketplace – “Targeting”

• Conclusion

Agenda

• Introduction

• Conclusion

Data Engineering Landscape

Bio-markers

eDiaries

Wearables

Profiles

CrowdSource

Data Pipeline

Data Lake

Centralised

Data Marketplace

Agenda

• Introduction

• Conclusion

StudyStudyStudy

Data Management & Quality Control

Analysis & Reporting

Data Capture & Site Management

Master Data Management

Data Management & Quality Control

Analysis & Reporting

Data Capture & Site Management

DM Scripts

A&R Scripts

Others

AutomatedData Ingestion

Routines

* F.A.I.R. = Findability, Accessibility, Interoperability, Reusability

Centralised

• Principle

Approach Investment Return on Investment

Master Data Management More up-front investment Longer term savings

Siloed Data Processing Less up-front investment Most costly long-term

• Principle

Approach Investment Return on Investment

Master Data Management More up-front investment Longer term savings

Siloed Data Processing Less up-front investment Most costly long-term

Data Engineering Project is looking for Use Cases to evidence this principle.

1) Measure current processes2) Adopt MDM3) Measure MDM processes4) Analyse ROI

Agenda

• Introduction

• Conclusion

Agenda

• Introduction

• ETL Overview – “Transforming”– Definition

– Staging & Loading

– Change Data Capture

– Data Validation

– Scheduling & Batch Processing

• Conclusion

ETL Overview – “Transforming”

Bio-markers

eDiaries

Wearables

Profiles

CrowdSource

Data Pipeline

Data Lake

Centralised

Data Marketplace

• Wikipedia definition

“In computing, extract, transform, load (ETL) is the general procedure of copying data from one or more sources into a destination system which represents the data differently from the source(s) or in a different context than the source(s).”

• Wikipedia definition

“In computing, extract, transform, load (ETL) is the general procedure of copying data from one or more sources into a destination system which represents the data differently from the source(s) or in a different context than the source(s).”

“…data engineering, is a software engineering approach to designing and developing information systems. “

i.e. It’s about building data pipelines, rather than datasets.

Data PipelineData Pipeline

SourceDataSourceData

SourceData

Data Pipelines StagingArea

CentralDataHub

Data Pipeline

ControlTables

Data Validation(Simple)

Data Validation(Complex)

DV Rules(Simple)

DV Rules(Complex)

Scheduling Controls

Staging

SourceData

CentralDataHub

Data Pipeline

ControlTables

DV Rules(Simple)

DV Rules(Complex)

Scheduling Controls

Loading

SourceData

CentralDataHub

Data Pipeline

ControlTables

DV Rules(Simple)

DV Rules(Complex)

Scheduling Controls

Change Data Capture

SourceData

CentralDataHub

Data Pipeline

ControlTables

DV Rules(Simple)

DV Rules(Complex)

Scheduling Controls

Changes: Additions, Updates, Deletions

Methods:•Database Timestamping (Control Tables)•Delta Identification (Comparison)

Change Data Capture

SourceData

CentralDataHub

Data Pipeline

ControlTables

DV Rules(Simple)

DV Rules(Complex)

Scheduling Controls

Data Validation (At Source)

SourceData

CentralDataHub

Data Pipeline

ControlTables

DV Rules(Simple)

DV Rules(Complex)

Scheduling Controls

e.g.: EDC Patient DOB must be a valid date.

Data Validation (Simple)

SourceData

CentralDataHub

Data Pipeline

ControlTables

DV Rules(Simple)

Scheduling Controls

e.g.: Lab Data Value unlikely to be > nnn

Failed records get diverted for cleansing processes

Data Validation (Complex)

SourceData

CentralDataHub

Data Pipeline

ControlTables

DV Rules(Simple)

DV Rules(Complex)

Scheduling Controls

• Across different sourcesThis lab value is not consistent with that EDC value(e.g. Males can’t be pregnant)

• Summarisation Checks

Data Validation (Complex)

SourceData

CentralDataHub

Data Pipeline

ControlTables

DV Rules(Simple)

DV Rules(Complex)

Scheduling Controls

Scheduling & Batch Processing

SourceData

CentralDataHub

Data Pipeline

ControlTables

DV Rules(Simple)

DV Rules(Complex)

Scheduling Controls

• Frequencies• Dependencies• Alerts

SourceData

CentralDataHub

Data Pipeline

ControlTables

DV Rules(Simple)

DV Rules(Complex)

Scheduling Controls

Agenda

• Introduction

• Conclusion

Data Lake – “Trawling”

Bio-markers

eDiaries

Wearables

Profiles

CrowdSource

Data Pipeline

Data Lake

Centralised

Data Marketplace

• Big-Data• Unstructured/Semi-Structured• Just-in-case - “Disk space is cheap”

Agenda

• Introduction

• Conclusion

Data Marketplace – “Targeting”

Bio-markers

eDiaries

Wearables

Profiles

CrowdSource

Data Pipeline

Data Lake

Centralised

Data Marketplace

• Small-Data• Specific Information• “As required” via APIs

Agenda

• Introduction

• Conclusion

Conclusion

Data Engineering Project(Educating for the Future PHUSE Working Group)

BHayes2@its.jnj.com

Guy.Garrett@AchieveIntelligence.com

No Copyright infringement intended.https://education.phuse.eu/eftf/data-engineering/

DH03 Educating for the Future: Data Engineering Techniques · Agenda • Introduction • Master Data Management - Principle • ETL Overview –“Transforming”

Documents

Defensive ETL

Manual Etl

etl testing training | etl testing course | best etl testing...

-Training Business Intelligence - BI - ETL - Developer ·.....

Sybase ETL

Educating the without educating the is no education at...

DH03 - SDTM in Business Intelligence (PPT)

Understanding ETL

ETL 프로젝트 Best Practice -...

ETL Matter

How to extract transform and load - OHDSI...ETL ETL ETL ETL....

Jeroen Hogeboom Introductie - FOSS4GNL 2019 · Introductie....

ETL Evolution

Etl Report

· tl-127 cap net a disc inserted to help replace mr16...

ETL on Hadoop –What is Required - Meetup ETL on... · ETL...