QConSF2013-JoshWills-BuildingAProductionMachineLearningInfrastructure

8/12/2019 QConSF2013-JoshWills-BuildingAProductionMachineLearningInfrastructure

http://slidepdf.com/reader/full/qconsf2013-joshwills-buildingaproductionmachinelearninginfrastructure 1/39

1

From The Lab to the Factory

Building A Production Machine Learning InfrastructureJosh Wills, Senior Director of Data Science

Cloudera



About Me

2



3

What Do Data Scientists Do?



What I Think I Do

4



What Other People Think I Do

5



What I Actually Do

6



7

Data Science In the Lab



Data Science as Statistics

8



Investigative Analytics

9



Tools for Investigative Analytics

10



Inputs and Outputs

11



On Actionable Insights

12



13

Data Science in the Factory



Building Data Products

14



A Shift In Perspective

Analytics in the Lab

• Question-driven

• Interactive

• Ad-hoc, post-hoc

• Fixed data

• Focus on speed and

flexibility

• Output is embedded into a

report or in-database

scoring engine

Analytics in the Factory

• Metric-driven

• Automated

•

Systematic• Fluid data

• Focus on transparency andreliability

•

Output is a productionsystem that makescustomer-facing decisions

15



Data Science as Decision Engineering

16



All* Products Become Data Products

17



18

From the Lab to the Factory:First Steps



Step 1: Choose a Good Problem

19





Step 3: Log Everything

21



Step 4: Hire (More) Data Scientists

22



23

Workflow Optimization



The Data Science Workflow

24



Identifying the Bottlenecks

25



Myrrix

26





Generational Thinking

28



Oryx ALS Recommender Demo

29



30

Rolling to Production



The Limits of Our Models

31



Space Exploration

32



Data Science Needs DevOps

33



Introducing Gertrude

• Multivariate Testing

• Define and explore a

space of parameters

•

OverlappingExperiments

• Tang et al. (2010)

• Runs multiple

independentexperiments on every

request

34



Simple Conditional Logic

• Declare experiment

flags in compiled code

• Settings that can varyper request

• Create a config file thatcontains simple rulesfor calculating flagvalues and rules for

experiment diversion

35



Separate Data Push from Code Push

• Validate config files and

push updates to servers

• Zookeeper via Curator

•

File-based• Servers pick up new

configs, load them, and

update experiment

space and flag valuecalculations

36



The Experiments Dashboard

37



A Few Links I Love

• http://research.google.com/pubs/pub36500.html • The original paper on the overlapping experiments

infrastrucure at Google

• http://www.exp-platform.com/

• Collection of all of Microsoft’s papers and presentations on

their experimentation platform

• http://www.deaneckles.com/blog/596_lossy-better-

than-lossless-in-online-bootstrapping/

• Dean Eckles on his paper about bootstrapped confidence

intervals with multiple dependencies

38

http://research.google.com/pubs/pub36500.html

http://www.exp-platform.com/

http://www.deaneckles.com/blog/596_lossy-better-than-lossless-in-online-bootstrapping/
























J h Will Di f D S i Cl d @j h ill

Thank you!

QConSF2013-JoshWills-BuildingAProductionMachineLearningInfrastructure

Documents