DH03 Educating for the Future: Data Engineering Techniques · Agenda • Introduction • Master Data Management - Principle • ETL Overview –“Transforming”

DH03 – Educating for the Future:

Data Engineering Techniques

https://education.phuse.eu/eftf/data-engineering/

Contribution:

Andy Richardson, Amy Gillespie, Beate Hientzsch, Berber Snoeijer,Beverly Hayes, JatinPatel, Karnika Dalal, Mark Bynens, Mike Carniello, Parag Shiralkar, Paul Slagle, Ralf Goetzelmann, Rohit Banga, Sagar Jain, Susan Olson, Vijay Pasapula, Vince Marinelli, Xiaohui Wang

Special thanks:Ian Fleming, James McDermott, Wendy Dobson

Presentation:Guy Garrett, Achieve IntelligenceRenu Shukla, Janssen Research & DevelopmentMohit Juneja, LyfeScience



Agenda

• Introduction

• Master Data Management - Principle

• ETL Overview – “Transforming”

• Data Lake – “Trawling”

• Data Marketplace – “Targeting”

• Conclusion



Agenda

• Introduction





• Conclusion



Data Engineering Landscape

ePro

RWE

IoT

Bio-markers

eDiaries

HEcon

Wearables

Omics

RWD

Profiles

EHR

CrowdSource

eCRF

Lab

Data Pipeline

Data Lake

Centralised

Data

Hub

Data Marketplace



Agenda

• Introduction





• Conclusion


StudyStudyStudy

Data Management & Quality Control

Analysis & Reporting

Data Capture & Site Management

Master Data Management




Data Management & Quality Control

Analysis & Reporting

Data Capture & Site Management

eCRF

DM Scripts

A&R Scripts



eCRF

eCRF

Lab

Others

AutomatedData Ingestion

Routines


* F.A.I.R. = Findability, Accessibility, Interoperability, Reusability


Centralised

Data

Hub




• Principle

Approach Investment Return on Investment

Master Data Management More up-front investment Longer term savings

Siloed Data Processing Less up-front investment Most costly long-term




• Principle

Approach Investment Return on Investment

Master Data Management More up-front investment Longer term savings

Siloed Data Processing Less up-front investment Most costly long-term

Data Engineering Project is looking for Use Cases to evidence this principle.

1) Measure current processes2) Adopt MDM3) Measure MDM processes4) Analyse ROI



Agenda

• Introduction





• Conclusion



Agenda

• Introduction


• ETL Overview – “Transforming”– Definition

– Staging & Loading

– Change Data Capture

– Data Validation

– Scheduling & Batch Processing



• Conclusion



ETL Overview – “Transforming”

ePro

RWE

IoT

Bio-markers

eDiaries

HEcon

Wearables

Omics

RWD

Profiles

EHR

CrowdSource

eCRF

Lab

Data Pipeline

Data Lake

Centralised

Data

Hub

Data Marketplace




• Wikipedia definition

“In computing, extract, transform, load (ETL) is the general procedure of copying data from one or more sources into a destination system which represents the data differently from the source(s) or in a different context than the source(s).”


https://en.wikipedia.org/wiki/Computing



• Wikipedia definition

“In computing, extract, transform, load (ETL) is the general procedure of copying data from one or more sources into a destination system which represents the data differently from the source(s) or in a different context than the source(s).”

“…data engineering, is a software engineering approach to designing and developing information systems. “

i.e. It’s about building data pipelines, rather than datasets.


https://en.wikipedia.org/wiki/Computing

https://en.wikipedia.org/wiki/Software_engineering

https://en.wikipedia.org/wiki/Information_systems

Data PipelineData Pipeline


SourceDataSourceData


SourceData

Data Pipelines StagingArea

CentralDataHub

Data Pipeline

ControlTables

Data Validation(Simple)

Data Validation(Complex)

DV Rules(Simple)

DV Rules(Complex)

Scheduling Controls



Staging



SourceData


CentralDataHub

Data Pipeline

ControlTables



DV Rules(Simple)

DV Rules(Complex)

Scheduling Controls



Loading



SourceData


CentralDataHub

Data Pipeline

ControlTables



DV Rules(Simple)

DV Rules(Complex)

Scheduling Controls



Change Data Capture



SourceData


CentralDataHub

Data Pipeline

ControlTables



DV Rules(Simple)

DV Rules(Complex)

Scheduling Controls

Changes: Additions, Updates, Deletions

Methods:•Database Timestamping (Control Tables)•Delta Identification (Comparison)



Change Data Capture



SourceData


CentralDataHub

Data Pipeline

ControlTables



DV Rules(Simple)

DV Rules(Complex)

Scheduling Controls



Data Validation (At Source)



SourceData


CentralDataHub

Data Pipeline

ControlTables



DV Rules(Simple)

DV Rules(Complex)

Scheduling Controls

e.g.: EDC Patient DOB must be a valid date.



Data Validation (Simple)



SourceData


CentralDataHub

Data Pipeline

ControlTables



DV Rules(Simple)

DV Rules(Simple)

Scheduling Controls

e.g.: Lab Data Value unlikely to be > nnn

Failed records get diverted for cleansing processes



Data Validation (Complex)



SourceData


CentralDataHub

Data Pipeline

ControlTables



DV Rules(Simple)

DV Rules(Complex)

Scheduling Controls

• Across different sourcesThis lab value is not consistent with that EDC value(e.g. Males can’t be pregnant)

• Summarisation Checks



Data Validation (Complex)



SourceData


CentralDataHub

Data Pipeline

ControlTables



DV Rules(Simple)

DV Rules(Complex)

Scheduling Controls



Scheduling & Batch Processing



SourceData


CentralDataHub

Data Pipeline

ControlTables



DV Rules(Simple)

DV Rules(Complex)

Scheduling Controls

• Frequencies• Dependencies• Alerts






SourceData


CentralDataHub

Data Pipeline

ControlTables



DV Rules(Simple)

DV Rules(Complex)

Scheduling Controls



Agenda

• Introduction





• Conclusion



Data Lake – “Trawling”

ePro

RWE

IoT

Bio-markers

eDiaries

HEcon

Wearables

Omics

RWD

Profiles

EHR

CrowdSource

eCRF

Lab

Data Pipeline

Data Lake

Centralised

Data

Hub

Data Marketplace

• Big-Data• Unstructured/Semi-Structured• Just-in-case - “Disk space is cheap”



Agenda

• Introduction





• Conclusion



Data Marketplace – “Targeting”

ePro

RWE

IoT

Bio-markers

eDiaries

HEcon

Wearables

Omics

RWD

Profiles

EHR

CrowdSource

eCRF

Lab

Data Pipeline

Data Lake

Centralised

Data

Hub

Data Marketplace

• Small-Data• Specific Information• “As required” via APIs



Agenda

• Introduction





• Conclusion



Conclusion


Data Engineering Project(Educating for the Future PHUSE Working Group)

[email protected]

[email protected]

No Copyright infringement intended.https://education.phuse.eu/eftf/data-engineering/


DH03 Educating for the Future: Data Engineering Techniques · Agenda • Introduction • Master Data Management - Principle • ETL Overview –“Transforming”

Documents