Optimizing the Data Supply Chain for Data Science

Optimizing theData Supply Chainfor Data Science

61 Broadway Suite 1105 New York, NY 10006

info@vital.ai http://www.vital.ai

Marc HadfieldCEO, Vital A.I.

about: vital ai

Software Applications:Artificial Intelligence, Machine Learning, Data Science.

Software Vendor & Consulting Services

agenda

• Data Models • How A.I., Data Science, & Data Governance relate • Data Supply Chain & the Data Product • Problem: the “Telephone Game” across the DSC • Architecture Transition from Data Warehouse to DSC • Data Models and DSC; a Framework for Solutions • Examples • Collaboration & Visualization

note: general methodology, with some specific examples from Vital AI implementations.

takeaways:

• The Data Supply Chain is a supply chain to deliver Data Products

• Data Models can capture the implicit meaning of data (and that is the goal!)

• Data Models can help negotiate the implicit differences across the DSC

• Data Models offer a means to collaborate on data standards (meaning) across the DSC partners

about data models:

Semantic Models

big data:

volume, velocity, variety, veracity

variety: data models“Product”: different meaning in Manufacturing vs Retail context

Healthcare, same entity: “Patient”, “InsuredPerson”, “BillableEntity”

example:

Class: PersonProperty: birthday

Standardized Unique Global Identifier (URI) data type: date relationship with property: age allowed range of values (can’t be born in the future) typical (average/expected) value…(Birthdays in Wikipedia vs Customer Database)

about: vital ai tech

Vital AI Development Kit (VDK)VitalSigns — Data Modeling & Code Generation

VitalService — Common API for Databases, Machine Learning, Apache Spark, Data Transforms

about: vital ai tech

VitalServiceQuery

ExecutableQuery

Query Generator

Common Query API:Relational DB (SQL) Graph DB (Sparql) Key/Value Store NOSQL DBDocument DBApache SparkHive (Hadoop) Predictive Models (a query for an unknown value)

Goal: Build A.I. applications across variety of infrastructure with consistent API & Models.

example data:

Person:Recipient

Person:Sender Message

hasRecipient

hasSender

example “MetaQL” query:

GRAPH { value segments: ["mydata"] ARC { node_constraint { Message.class } constraint { "?person1 != ?person2" } ARC_AND { ARC { edge_constraint { Edge_hasSender.class } node_constraint {

Person.props().emailAddress.equalTo(“john@example.org") }

node_constraint { Person.class } node_provides { "person1 = URI" } } ARC { edge_constraint { Edge_hasRecipient.class } node_constraint { Person.class } node_provides { "person2 = URI" } } } } }

“Person” may have subtypes, like Student or Employee.

a.i. and data quality

data models & machine learning:

using the meaning of classes and properties, automatically generate predictive models.

predictive models features:birthday, zip code, …

data governance =defining the meaning of data = feature (pre)engineering

critical aspect of data science

Progression of Analytics:

where a.i. happens

Progression of Analytics:

Garbage In = Garbage Out

= Bad A.I.

data governance required for Good A.I.

one more point ondata governance…

think outside the box(data warehouse)

data governance: data in motion

vs.inside data warehouse

outside data warehouse

supply chain

supply chain:

product

data supply chain:

data product

Retail Recommendations… Shipping/Logistics Optimization… Compliance, Auditing, Security, Fraud Detection…

data product:

why data supply chain?

Partner DW Your DW

"No matter who you are, most of the smartest people work for someone else.” — Bill Joy.

why data supply chain?

Partner DW Your DW

"No matter who you are, most of the smartest people data works for someone else.” — Bill Joy. (revised)

data supply chain

Partner DW

Your DW

why not ETL?

Partner DW

Extract…

not quite as expected…

Transform…

a bit extreme…

Load…

a bit messy…

Clean…

a lot of manual effort…

… your imported data61 Broadway Suite 1105

New York, NY 10006info@vital.ai

http://www.vital.ai

Your DW61 Broadway Suite 1105

http://www.vital.ai

Partner DW

what goes wrong?

telephone game…

You61 Broadway Suite 1105

http://www.vital.ai

Partner

Model “A”

Model “B”

Implicit Model

Resolution: Make explicit the implicit. Align Data Models.

Reason:Implicit assumptions in the data. ETL can’t see the forest for the trees. (or it’s very difficult with missing assumptions)

Example: Internet of Things

Predictive Analytics

“Nest for Office Buildings” Office Tower with Building Management System (BMS) containing 100,000 monitored points (temperature, energy usage of chiller, fan speed, etc.) with significant missing data, errors, and noise. Reconciliation of data to produce predictive models to minimize energy usage. Rules for data correctness.

Sensor Data Validation:

Source data had temperature values of “0” (zero) which meant either the temperature was 0 degrees or that the sensor had an error.Data Model “knows” that it’s rarely 0 degrees in July (far from the standard deviation), and that the temperature can be compared to weather data on a day in December for reasonableness. If Data Model also knows the maintenance schedule for the sensors, then it “knows” when to expect 0 error values and exclude them.

Missing Maintenance Assumptions. Fill in secondary (weather) data for validation.

how did we get here?61 Broadway Suite 1105

http://www.vital.ai

Architecture Review:a quick step back…

What is a Data Supply Chain architecture?

“traditional” data warehouse:

ETL within the organization.Data Governance across the organization.

tech co. “agile” data warehouse:

storage

compute

DataSetsJobs

Batch/StreamingBuild Predictive Models Realtime: Spark/Storm

hadoop cluster

enterprise: data lake

storage

compute

X(save $)

“Data Swamp”

aside: Data Lake

better analogy: Scriptorium

library,manuscript copying, & book distribution.

but not as Pithy as “Lake”…

tech co. microservices (micro-SOA):

storage

compute

service

“Composed” App

external: social data, weather API

independent clusters,local data expertise

optimize development processes, scale up.

microservices example:

Amazon: product search uses 170 independent microservices

including services for predicting customer characteristics, getting product images, etc.

http://www.infoworld.com/article/2903144/application-development/how-to-succeed-with-microservices-architecture.html

Netflix similar architecture

Data Supply Chain:

storage

compute

service

Data Product

“ETL”

Owner “A” Owner “B”

optimize development processes, scale up.

independent clusters,local data, ownership

Interaction Points:

Data Product

service

compute ETL

Data Lineage: Cloudera Navigator

…within a Data Warehouse

trace back jobs that produced every data field.

Data Supply Chain with Provenance:

include provenance data directly in imported dataset. use in rules to interpret the data.

entity-123 | hasSource | datasource-A entity-123 | name | “John Doe”

Data Warehouse B

Interaction Points: Data Models

Data Product

service

computeETL

Data Models: Gatekeepers & Transform

Data Supply Chain using Models:

storage

compute

service

Data Product

Owner “A” Owner “B”Model Server

Data Models: focus of data governance

Semantic Data Models:

Make explicit the meaning of data

Transformation and Validation Rules leverage the Model and Meaning.Such Rules may be packaged with the Model, and managed together.

Protect against implicit assumptions

Example: Financial Services

Service Provider

Reconciliation of Corporate Structure across 1,000’s of organizations. Compliance Rules barring communication between “researchers” and “traders”.Rules to infer if “Mary” is a “researcher” or “trader”.Conflicting concepts of “Branch-Office”, “Direct-Report”, etc. across the Globe.

Example: Hospital Group

Data Analytics

Reconciliation across Patient Records, Insurance, & Billing for Patient Predictive Analytics.Rules for identity: “same person”

Data Models:

OWL: Semantic Ontology Model (W3C Standard, Various Standards for Rules)

VitalSigns: Generate Codevalidation, transformation, …

VitalSigns: Versioning, Dependencies, Exchange, Storage, Change Management (Semantic “Diff”)

Example: Personally Identifiable Information

Data Governance determines that “Profession” and “ZipCode” cannot be used together. (Maybe a single “Dentist” in a small town…)

Within a single Data Warehouse we can bar these data elements from being combined. But:Microservice A provides value of “Profession” Microservice B provides value of “ZipCode” How to enforce that these two microservices cannot be combined?

Example: Personally Identifiable Information

Validation code enforcing data usage:

Person person123 = get_person_details(“entity-123”) // this call works: person123.profession = get-profession(person123)// this call blocks because of data model validation // person123 already has “profession” propertyperson123.zipcode = get-zipcode(person123)

GatekeepersExternally Managed.Active not Passive, more like “code”.Defining what should exist, not cataloguing what exists.Can decide when to be tolerant or strict.

Semantic Data Models:

Collaborative Conversations:

Infrastructure DevOps

Data Scientists

Business +Domain Experts

Developers SemanticModel

Collaborative Conversations:

SemanticModel

Partner A Partner B

Model Alignment

What Concepts to combine, not what Tables to combine (that comes later).

Authoring Tool: OWL IDE Protege

Visualization: Semantic Data

Visualization: WebVOWL

in conclusion, takeaways:

• The Data Supply Chain is a supply chain to deliver Data Products

• Data Models can capture the implicit meaning of data (and that is the goal!)

• Data Models can help negotiate the implicit differences across the DSC

• Data Models offer a means to collaborate on data standards (meaning) across the DSC partners

Questions?

Marc HadfieldCEO, Vital A.I.marc@vital.ai

Optimizing the Data Supply Chain for Data Science

Data & Analytics

Optimizing Supply Chain and Warehousing with Microsoft...

OKLAHOMA UNIVERSITY OPTIMIZING … · optimizing...

The Art of Optimizing Your Supply Chain

Optimizing birth certificate data...

Optimizing Data Requests for the Clinical Research Data...

Optimizing Product Realization Costs Across the Value Chain

TOC - Manufacturing at Warp Speed Optimizing Supply Chain...

Optimizing the Restaurant Industry Supply Chain since 1990SM

Optimizing the Content Supply Chain: What Manufacturing Can....

Optimizing supply chain network design with location...

Optimizing Supply Chain Conﬁguration Considering Supply...

Optimizing the retail bank supply chain Download the report

OPTIMIZING THE ORDERING POLICY IN A SUPPLY CHAIN

Optimizing Supply Chain Performance ADC Solution … ·...

On optimizing production nofed in supply chain systems

OPTIMIZING COLD CHAIN TRANSPORT