Top Banner
v v Are New Modeling Techniques Worth It? TORONTO SAS USER GROUP MAY 2, 2018 Tom Zougas PhD PEng, Manager – Data Science, TransUnion
24

Are New Modeling Techniques Worth It? Group Presentations... · v v Are New Modeling Techniques Worth It? TORONTO SAS USER GROUP MAY 2, 2018 Tom Zougas PhD PEng, Manager – Data

Nov 10, 2018

Download

Documents

dothuan
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Are New Modeling Techniques Worth It? Group Presentations... · v v Are New Modeling Techniques Worth It? TORONTO SAS USER GROUP MAY 2, 2018 Tom Zougas PhD PEng, Manager – Data

v

v

Are New Modeling Techniques Worth It?

TORONTO SAS USER GROUP MAY 2, 2018

Tom Zougas PhD PEng, Manager – Data Science, TransUnion

Page 2: Are New Modeling Techniques Worth It? Group Presentations... · v v Are New Modeling Techniques Worth It? TORONTO SAS USER GROUP MAY 2, 2018 Tom Zougas PhD PEng, Manager – Data

© 2018 TransUnion LLC All Rights Reserved | 2

Are New Modeling Techniques Worth It?

Tom Zougas PhD PEng, Manager – Data Science, TransUnion

As Senior Manager of Data Science at TransUnion Canada, Tom and his team are tasked with delivering advanced analytics projects.He has 20 years of technical consulting experience in data analysis, system design, and software development. Tom has worked with clients in industries as diverse as financial services, insurance, software, utilities, telecom, pharmaceuticals and metals.Prior to joining TransUnion, he was Director of Analytics at Angoss Software, where he managed a team of data scientists, worked as a senior advanced analytics consultant at IBM and at SAS, and managed the advanced analytics team at BlackBerry (RIM).He has also authored and taught courses in data mining. Tom holds a PhD in Engineering from the University of Toronto.

Presenter

Page 3: Are New Modeling Techniques Worth It? Group Presentations... · v v Are New Modeling Techniques Worth It? TORONTO SAS USER GROUP MAY 2, 2018 Tom Zougas PhD PEng, Manager – Data

© 2018 TransUnion LLC All Rights Reserved | 3

Agenda

• Know The Core

• Applying The Core Models

• A Sampling of New Model Types

• Q&A

Page 4: Are New Modeling Techniques Worth It? Group Presentations... · v v Are New Modeling Techniques Worth It? TORONTO SAS USER GROUP MAY 2, 2018 Tom Zougas PhD PEng, Manager – Data

v

v

Know The Core

Page 5: Are New Modeling Techniques Worth It? Group Presentations... · v v Are New Modeling Techniques Worth It? TORONTO SAS USER GROUP MAY 2, 2018 Tom Zougas PhD PEng, Manager – Data

© 2018 TransUnion LLC All Rights Reserved | 5

Top 3 Machine Learning Methods

• Regression

• Clustering

• Decision Trees

Page 6: Are New Modeling Techniques Worth It? Group Presentations... · v v Are New Modeling Techniques Worth It? TORONTO SAS USER GROUP MAY 2, 2018 Tom Zougas PhD PEng, Manager – Data

© 2018 TransUnion LLC All Rights Reserved | 6

KDnuggets 2017 SurveySource: https://www.kdnuggets.com/2017/12/top-data-science-machine-learning-methods.html

Page 7: Are New Modeling Techniques Worth It? Group Presentations... · v v Are New Modeling Techniques Worth It? TORONTO SAS USER GROUP MAY 2, 2018 Tom Zougas PhD PEng, Manager – Data

© 2018 TransUnion LLC All Rights Reserved | 7

What Makes Them the Top 3

• Are they state of the art?

• Are they leading edge?

• Are they exotic?

• What are they?

…No

…No

…No

Simple Interpretable They Work

Page 8: Are New Modeling Techniques Worth It? Group Presentations... · v v Are New Modeling Techniques Worth It? TORONTO SAS USER GROUP MAY 2, 2018 Tom Zougas PhD PEng, Manager – Data

© 2018 TransUnion LLC All Rights Reserved | 8

Categories of Machine Learning Algorithms

• Supervised Learning

• Unsupervised Learning

• Others

InputsAttributesVariables

OutputOutcome

Target

ML AlgorithmModel

InputsAttributesVariables

ML AlgorithmModel

Page 9: Are New Modeling Techniques Worth It? Group Presentations... · v v Are New Modeling Techniques Worth It? TORONTO SAS USER GROUP MAY 2, 2018 Tom Zougas PhD PEng, Manager – Data

© 2018 TransUnion LLC All Rights Reserved | 9

Landscape of Machine Learning Algorithms

Supervised Learning

Support Vector MachinesLinear/Logistic RegressionNaive BayesLinear Discriminant AnalysisDecision TreesK Nearest NeighborNeural Networks

Unsupervised Learning

Clustering (k-means, hierarchical)Anomaly DetectionNeural Networks (autoencoders, SOM)Expectation–MaximizationPrincipal Component AnalysisSingular Value DecompositionAssociation Analysis

Often times, the baseline is sufficient to satisfy the business objective.

Page 10: Are New Modeling Techniques Worth It? Group Presentations... · v v Are New Modeling Techniques Worth It? TORONTO SAS USER GROUP MAY 2, 2018 Tom Zougas PhD PEng, Manager – Data

v

v

Applying The Core Models

Page 11: Are New Modeling Techniques Worth It? Group Presentations... · v v Are New Modeling Techniques Worth It? TORONTO SAS USER GROUP MAY 2, 2018 Tom Zougas PhD PEng, Manager – Data

© 2018 TransUnion LLC All Rights Reserved | 11

What Analytics Methodology Do You Use

CRISP-DM SEMMA

KDDKnowledge Discovery in

Databases

Custom

Page 12: Are New Modeling Techniques Worth It? Group Presentations... · v v Are New Modeling Techniques Worth It? TORONTO SAS USER GROUP MAY 2, 2018 Tom Zougas PhD PEng, Manager – Data

© 2018 TransUnion LLC All Rights Reserved | 12

What Is The Business Problem

• Do you want to categorize a record or observation?• Do you want to predict a numerical quantity?• Do you want to rank order your records based on some outcome of

interest?• Do you want to identify naturally occurring groupings in the data?

By defining the business problem:Ensure you are collecting the right dataDetermine what type of model to applyKnow when you are done and can go to the next phase

Page 13: Are New Modeling Techniques Worth It? Group Presentations... · v v Are New Modeling Techniques Worth It? TORONTO SAS USER GROUP MAY 2, 2018 Tom Zougas PhD PEng, Manager – Data

© 2018 TransUnion LLC All Rights Reserved | 13

The Data Science Workflow

Page 14: Are New Modeling Techniques Worth It? Group Presentations... · v v Are New Modeling Techniques Worth It? TORONTO SAS USER GROUP MAY 2, 2018 Tom Zougas PhD PEng, Manager – Data

© 2018 TransUnion LLC All Rights Reserved | 14

Model Selection

The modeling step is where you apply one of the relevant model types:• If the data contains known outcomes, then you use a

supervised learning algorithm: regression or decision tree (or possibly both).

• If the data does not have known outcomes (or labels), then apply the unsupervised learning algorithm: clustering.

Page 15: Are New Modeling Techniques Worth It? Group Presentations... · v v Are New Modeling Techniques Worth It? TORONTO SAS USER GROUP MAY 2, 2018 Tom Zougas PhD PEng, Manager – Data

© 2018 TransUnion LLC All Rights Reserved | 15

Assessing Model PerformanceInterpreting ROC

Which is better:• Random• Model 1• Model 2• Model 3

Page 16: Are New Modeling Techniques Worth It? Group Presentations... · v v Are New Modeling Techniques Worth It? TORONTO SAS USER GROUP MAY 2, 2018 Tom Zougas PhD PEng, Manager – Data

v

v

A Sampling of New Model Types

Page 17: Are New Modeling Techniques Worth It? Group Presentations... · v v Are New Modeling Techniques Worth It? TORONTO SAS USER GROUP MAY 2, 2018 Tom Zougas PhD PEng, Manager – Data

© 2018 TransUnion LLC All Rights Reserved | 17

Deep LearningA New Generation of Artificial Neural Networks

Source: https://www.quora.com/What-is-the-difference-between-Neural-Networks-and-Deep-Learning

Page 18: Are New Modeling Techniques Worth It? Group Presentations... · v v Are New Modeling Techniques Worth It? TORONTO SAS USER GROUP MAY 2, 2018 Tom Zougas PhD PEng, Manager – Data

© 2018 TransUnion LLC All Rights Reserved | 18

Deep Learning Use Cases

• Fraud Detection• Automatic speech

recognition• Image recognition• Visual art processing• Natural language

processing• Drug discovery and

toxicology

• Customer relationship management

• Recommendation systems

• Bioinformatics• Mobile advertising• Image restoration

Page 19: Are New Modeling Techniques Worth It? Group Presentations... · v v Are New Modeling Techniques Worth It? TORONTO SAS USER GROUP MAY 2, 2018 Tom Zougas PhD PEng, Manager – Data

© 2018 TransUnion LLC All Rights Reserved | 19

EnsembleIf one model is good, many models should be better

Source: SAS® Enterprise Miner™ 14.3: Reference Help

Page 20: Are New Modeling Techniques Worth It? Group Presentations... · v v Are New Modeling Techniques Worth It? TORONTO SAS USER GROUP MAY 2, 2018 Tom Zougas PhD PEng, Manager – Data

© 2018 TransUnion LLC All Rights Reserved | 20

Elastic Net

High Dimensional Data Correlated Variables

Elastic Net

Improve Accuracy Improve Interpretability

Page 21: Are New Modeling Techniques Worth It? Group Presentations... · v v Are New Modeling Techniques Worth It? TORONTO SAS USER GROUP MAY 2, 2018 Tom Zougas PhD PEng, Manager – Data

© 2018 TransUnion LLC All Rights Reserved | 21

Automated Machine Learning

Source: https://www.slideshare.net/SparkSummit/07-sourabh-chaki

Page 22: Are New Modeling Techniques Worth It? Group Presentations... · v v Are New Modeling Techniques Worth It? TORONTO SAS USER GROUP MAY 2, 2018 Tom Zougas PhD PEng, Manager – Data

© 2018 TransUnion LLC All Rights Reserved | 22

Where Do We Go From Here

Has the business problem been identified?• Make sure the focus is on solving the business problem

Which modeling approach to use?• The core models provide a good starting point and may be sufficient for

satisfying the business objective

Are there issues with the data?• A new model will not fix bad data

Is the model usable/deployable?• An overly complex model may be difficult to deploy – make sure it’s worth it

Page 23: Are New Modeling Techniques Worth It? Group Presentations... · v v Are New Modeling Techniques Worth It? TORONTO SAS USER GROUP MAY 2, 2018 Tom Zougas PhD PEng, Manager – Data

v

v

Q&A

Tom Zougas PhD [email protected]

Page 24: Are New Modeling Techniques Worth It? Group Presentations... · v v Are New Modeling Techniques Worth It? TORONTO SAS USER GROUP MAY 2, 2018 Tom Zougas PhD PEng, Manager – Data