Spark tutorial pycon 2016 part 1

DavidTaiebSTSM-IBMCloudDataServicesDeveloperadvocatedavid_taieb@us.ibm.com

HANDS-ONSESSION:DEVELOPINGANALYTICAPPLICATIONSUSINGAPACHESPARK™ANDPYTHONPart1:FlightDelayPredictwithSparkML PyCon2016,Portland

©2016IBMCorpora6on �

Agenda

•  Pre-requisitestepstobecompletedbeforethesession

•  FlightPredictappdescrip6onandarchitecture•  TrainthemodelsintheNotebook•  AccuracyAnalysisandmodelsrefinement•  Deployandrunthemodels


Sign up for Bluemix •  AccessIBMBluemixwebsiteonhMps://console.ng.bluemix.net•  ClickonGetStartedforFree

•  CompletetheformandclickCreateaccount•  Lookforconfirma6onemailandclickonconfirmyouaccountlink

Signupforflightstats


Sign up for a free trial at Flightstats.com

•  SignupathMps://developer.flightstats.com/signup•  Fillouttheformandmonitoremailforconfirma6onlink(accesstoAPIsmay

takeupto24hours)•  Onceaccessisgrantedgoto

hMps://developer.flightstats.com/admin/applica6onstoviewappIdandappKey(youwillneedtheminthesimple-data-pipetooltocreatetrainingsets.

•  Op6onal:getfamiliarwiththevariousflightstatsapis:–  hMps://developer.flightstats.com/api-docs/scheduledFlights/v1–  hMps://developer.flightstats.com/api-docs/airports/v1

Howtofindyourappidandkey


Where to find the FlightStats app id and app key

APPID

APPKey

Prepareyourbluemixspace


Create a new space on Bluemix Inprepara6onforrunningtheproject,wecreateanewspaceonBluemix

CreateaSparkInstance

Op6onal:YoucanskipthisstepifyoualreadyhaveaspacewithSparkinstancethatyouwouldliketoreuse


Create a Spark Instance

Op6onal:YoucanskipthisstepifyoualreadyhaveaspacewithSparkinstancethatyouwouldliketoreuse


Create New Spark Instance Op6onal:YoucanskipthisstepifyoualreadyhaveaspacewithSparkinstancethatyouwouldliketoreuse


Agenda




Flight App Project Description • Usecase

–  Flightdelaysareacommondisturbanceduringbusinesstrips–  Beingabletopredicthowlikelyaflightwillbedelayedcanremoveuncertaintyandenableuserstoplanaroundit.

–  Idea:Weatherdatacanbeagoodexplanatoryvariableforbuildingpredic6vemodels

•  ImplementaSon–  Combineflightsta6s6csfromflightstats.com(Systemofrecords)withweatherdatafromIBMInsightforWeather(Systemofopera6ons)tobuildatraining,testandblindset

–  UseSparkMLLibtotrainpredic6vemodelsandcrossvalidatethem–  CreateacustomcardforGoogleNowthatwillautoma6callyno6fyuserofimpendingflightdelay

–  Proposealterna6ngflightroutes(e.g.Freebird)Get/Build/Analyze


Get/Build/Analyze methodology


Flight Predict App Architecture

Weather

SimpleDataPipes

Airports

FlightSchedules

FlightStatus

MetadataTrainingSet

TestSet

BlindSet

CustomConnectorrunevery24hours

Notebook


Flow Diagram

DataAcquisi6on

DataPrepara6on

DataAnnota6on(GroundTruth)

ModelTraining

•  Cleansing•  Shaping•  Enrichment

ModelTes6ng

TrainingSet

TestSet

BlindSet

Iterative

Cross-Validation

Evaluate Performance and optimize model

Train Model

•  Itera6veinNature:weareneverdone!•  Wewillbeusingthisdiagramasaroadmapthroughoutthiscourse

DeployandRunModel


Get the data and build the training/test/blind sets Inthisstepwe’lluseSimpleDataPipesopensourceprojecttoacquiredatafromFlightstats,combineitwithWeatherdatafromIBMInsightforWeatherandsavethedatasetsintoaNoSQLCloudantDatabase.

DataAcquisi6on

DataPrepara6on


ModelTraining


ModelTes6ng

TrainingSet

TestSet

BlindSet

Iterative

Cross-Validation


Train Model DeployandRunModel


Acquiring the data

•  Inthenextsec6on,weshowhowtoacquirethetrainingdatabyusingthesimple-data-pipetoolandflightpredictconnector.

•  Theflightpredictconnectorcombinehistoricalflightdatafromflightstats.comwithweatherdatafromIBMInsightforWeather

•  Ifyouwanttoskipthesesteps,youcanusethealreadybuiltdatasetbyusingthefollowingcreden6als:–  cloudantHost:dtaieb.cloudant.com–  cloudantUserName:weenesserliffircedinvers–  cloudantPassword:72a5c4f939a9e2578698029d2bb041d775d088b5

Deploysimple-data-pipe


Deploy simple-data-pipe with flightstats connector •  GotohMps://github.com/ibm-cds-labs/simple-data-pipe•  ClickonDeploytoBluemixbuMon

ClickbuMonwilltakeyoutoBluemix


Complete simple-data-pipe deployment

AddWeatherservice


Add an instance of IBM Weather Service on Bluemix •  Returntotheapplica6ondashboard•  Weatherserviceisrequiredbythe

flightpredictconnectorandmustbeinstalledbefore

•  Fromappdashboard,clickonAddaserviceorAPI


Create an instance of IBM Weather Service on Bluemix SearchforWeather

Makesuretoselect“premiumplan”tohaveenoughauthorizedAPIcalls


Checkpoint: simple data pipe app dashboard •  Verifythatyourappiscorrectlyboundtotherightservices

WeatherServiceusedtoenrichflightrecordswithweatherobserva6ons

CloudantServiceusedtostoretraining,testandblinddatasets

You’llneedtoclickonthisbuMonforthesteponthenextpageItisrecommendedtoincrease

theappmemoryto1GB


Install flight predict connector •  ClickEditCodebuMon,editpackage.jsontoaddflightpredictmodule:

– "simple-data-pipe-connector-flightstats":"git://github.com/ibm-cds-labs/simple-data-pipe-connector-flightstats.git"

addflightpredictmoduletodependencies

Saveyourchanges

don’tforgettoaddcommainthelinebeforetokeepjsonvalid


Install flight predict connector •  ClickFile/Savetosaveyourchanges

Redeploysimpledatapipe


Redeploy simple data pipe app •  UseliveeditEditortoredeploytheapp

Verifyyoursdpinstall


Verify connector install •  Inthisstep,weverifythattheflightpredictconnectoriscorrectlyinstalledthroughtheUI

Fightconnectorcorrectlyinstalled

Createnewflightstatspipe


Create a new FlightStats pipe •  Followeachscreentocreateandconfigureanewpipe

Runthepipe


Run the pipe •  Skipoverthescheduletab•  Intheac6vitytab,clickonRunNowtostartthepipe

Explorethedataset

ClickRunNowThenopenthelogtomonitortheac6vity


Explore the data sets •  Inthisstep,wetakeamomenttoexplorethedifferentdatasetsthathavebeencreatedbythesimpledatapipetool

•  Frombluemixdashboard,clickonthecloudantservice6le,thenontheLaunchbuMon•  FromtheCloudantdashboard,openthetrainingdatabase•  Openadocumenttolookatthedatastructure

Buildthetestset


Run the pipe again to build the test set

Trainthemodels


Train the Models •  Intheprevioussec6onwehavecreatedthetrainingdataandwearenowreadytotrainthemodels.•  Stepsinthissec6on:

–  CreateanIPythonNotebook–  LoadthedatasetsfromtheCloudantdatabaseintoaSparkCluster–  Explorethedataandtrainthemachinelearningmodels

DataAcquisi6on

DataPrepara6on


ModelTraining


ModelTes6ng

TrainingSet

TestSet

BlindSet

Iterative

Cross-Validation



CreateIPythonNotebook


Create a new IPython Notebook


Notebook tour


Notebook tour: Notebook Info


Notebook tour: Environment


Notebook tour: Sharing

`


Agenda




Before we start building the app…

•  Youcanop6onallyfollowthistutorialfromGithubbyusingafullybuiltnotebook:– hMps://github.com/ibm-cds-labs/simple-data-pipe-connector-flightstats/blob/master/notebook/Flight%20Predict%20PyCon%202016.ipynb


Optional: use prebuilt notebook

ImportrequiredPythonpackages

• CreatenotebookfromURL• UsehMps://github.com/ibm-cds-labs/simple-data-pipe-connector-flightstats/raw/master/notebook/Flight%20Predict%20PyCon%202016.ipynb


Using Python Packages •  Writecodeinlinewithincells•  EncapsulatehelperAPIswithinPythonpackage•  2waysofusinghelperPythonpackages

–  eggdistribu6onpackage:pipinstallfromPyPiserverorfileserver(e.g.Github)

•  Persistentinstallacrosssessions•  RecommendedinProduc6on

–  SparkContext.addPyFile•  Easyaddi6onofapythonmodulefile•  Supportmul6plemodulefilesviazipformat•  Recommendedduringdevelopmentwherefrequentcodechangesoccur

Manageeggpackages


Flight Predict Python Package on Github

SetupscriptforinstallingPythonPackage

FlightPredictPythonlibrary


Method 1: Install Flight Predict Package •  UsepiptoInstallFlightPredictpackage•  Recommendedalterna6ve:buildeggdistribu6onpackageanddeployinPyPi


Manage Python packages •  Checkstatus•  Uninstallpackage

Installpackagesviasc.addPyFilemethod


Method 2: Install py modules via sc.addPyFile

•  addPyFileinstallindividualpymodulesandmakethemavailabletoallexecutorprocesses

•  Workswithmodulesinzippedfiles

Modulecontainingapisfortrainingthemodels

Modulecontainingapisforrunningthemodels

Configurecreden6alsforvariousservices


Setup credentials and Import required python modules

Inthisstep,weimportpythonmodulesthatwillbeneededthroughoutthenotebookandsetupcreden6alstovariousservices.

Howtogetcreden6alsforCloudantandWeather

Creden6alforCloudantNoSQLService

Creden6alsforWeatherService


Get Credentials for Cloudant Fromtheappdashboard,clickonEnvironmentVariablesfromthelessidebar


Get Credentials for Weather

LoadtrainingsetfromCloudant


Load training set in Spark SQL DataFrame

…

Inthisstep,weusethecloudant-sparkconnector(hMps://github.com/cloudant-labs/spark-cloudant)toloaddataintoSpark

Makesuretochangethedbnametomatchtheonecreatedforyourtrainingsetbyyourac6vity(opentheCloudantdashboardtofindthename)


Loading data: Behind the scene UseSparkSQLconnectortoloaddataintoaDataFrame

connectorid

Op6ons

Cachedataforop6mizedreuse

CreatetempSQLTable

ScaMerPlotVisualiza6on


Scatter plot visualization


Visualization api

CreateanRDDofLabeledPoint


Transform into an RDD of LabeledPoint UseSparkSQLconnectortoloaddataintoaDataFrame


loadLabeledDataRDD api

TrainMachineLearningModels


Machine Learning Algorithms ConSnuousOutput DiscreteOutput

SupervisedLearning(requireGround-Truth)

•  Regression-Linear-Ridge-Lasso-Isotonic•  DecisionTree•  RandomForest•  GradientBoostedTree

• Classifica6on-Logis6cRegression-SVM-NaiveBayes• DecisionTree• RandomForest• GradientBoostedTree• K-NN(availableasadd-onsparkpackage)

UnsupervisedLearning(noGround-Truthdatarequired)

•  Clustering-KMeans-GaussianMixture•  DimensionalityReduc6on-PCA-SVD

•  FP-Growth

TrainLogis6cRegressionModel


Train Logistic Regression Model

TrainNaïveBayesModels


Train NaiveBayes Model

TraindecisionTreeModel


Train Decision Tree Model

TrainRandomForestModel


Train Random Forest Model

AccuracyAnalysis


Naïve Bayes vs Decision Tree •  Probabilis6c:computetheprobability

ofadatainstancetobeinaspecificclass

•  Assumethateachfeature(variable)isindependentfromtheothers

•  Performancedependsonthepredic6venatureofthefeatures(nonpredic6vefeatureswillaffecttheaccuracy)

•  Workswellwithlowamountoftrainingdata.Doesn’tneedallthepossibili6es

•  Doesn’tworkwithcategoricalfeatures.

• Non-Probabilistic: partition the data into subsets that best describe the variable

• The deeper the tree, the better the model fits the data

• Watch out for overfiting: need to prune the tree

• Can handle categorical or continuous features

• No need for input to be scaled or standardized: Set you features and go!

• Requires a lot of data covering all possibilities


Accuracy Analysis of the Machine Learning Models Inthissec6on,wewillperformaccuracyanalysisonthetestdata.Wewillstartbycompu6ngtheaccuracymetricsforeachmodel,includingtheconfusionmatrix.Wewillthenusehistogramcharttounderstandthedatadistribu6onandrefinehowtoclassesarecomputed.

DataAcquisi6on

DataPrepara6on


ModelTraining


ModelTes6ng

TrainingSet

TestSet

BlindSet

Iterative

Cross-Validation




Agenda




Load Test data Makesuretochangethedbnametomatchtheonecreatedforyourtestsetbyyourac6vity(opentheCloudantdashboardtofindthename)


Accuracy Metrics


Confusion Matrix


Confusion Matrix


Confusion Matrix


Confusion Matrix


Accuracy metrics API OutputHTML

DisplayresultsHTMLinNotebookCell

ComputeMetricsfromlabeledandpredic6ondata

Gettheconfusionmatrixandbuildhtmltable


Understand the distribution of your data with Histograms


Training Handler class

•  Provideflexibilityandextensibilitytotheapplica6on

•  Provideafailfastandtrysomethingelsemechanism

•  Enableusertoeasilycustomizeclassesofdatabasedonhowdataisdistributed

•  Enableusertoeasilyaddtrainingfeatures


Default Training Handler class

Returndescrip6onforeachclasses

Returntotalnumberofclasses:Defaultis5

Re-classifyarecord:defaultusess.classifica6onfieldinJsonrecord

ExtrafeaturesNamestobeadded.Nonebydefault

Extrafeaturestobeadded.ArraymustmatchtheonereturnedbycustomTrainingFeaturesNames


Customize Training Handler Providenewclassifica6onandadddayofdepartureasanewfeature

InheritfromdefaultTrainingHandler

Adddayoftheweekusingatechniquecalleddummycoding


Re-train the models


Re-compute accuracy Models1

Models2BeMeraccuracyforNaiveBayesandLogis6cRegressionWorseforDecisionTreeandRandomForest


Agenda




Deploy and Run the models Inthelastsec6on,wewillsimulatedeploymentandrunningofthemodelsthroughthenotebookbycallingAPIsfromtherunpackage.

DataAcquisi6on

DataPrepara6on


ModelTraining


ModelTes6ng

TrainingSet

TestSet

BlindSet

Iterative

Cross-Validation


Train Model DeployandRunModels


Run the predictive model


runModel API


Get Weather Predictions


Show prediction results


Resource

•  hMps://developer.ibm.com/clouddataservices/•  hMps://github.com/ibm-cds-labs/simple-data-pipe•  hMps://github.com/ibm-cds-labs/pipes-connector-flightstats•  hMp://spark.apache.org/docs/latest/mllib-guide.html•  hMps://console.ng.bluemix.net/data/analy6cs/


Thank You

Spark tutorial pycon 2016 part 1

Data & Analytics