Spark tutorial pycon 2016 part 1

DavidTaiebSTSM-IBMCloudDataServicesDeveloperadvocatedavid_taieb@us.ibm.com

HANDS-ONSESSION:DEVELOPINGANALYTICAPPLICATIONSUSINGAPACHESPARK™ANDPYTHONPart1:FlightDelayPredictwithSparkML PyCon2016,Portland

Agenda

•  Pre-requisitestepstobecompletedbeforethesession

•  FlightPredictappdescrip6onandarchitecture•  TrainthemodelsintheNotebook•  AccuracyAnalysisandmodelsrefinement•  Deployandrunthemodels

Sign up for Bluemix •  AccessIBMBluemixwebsiteonhMps://console.ng.bluemix.net•  ClickonGetStartedforFree

•  CompletetheformandclickCreateaccount•  Lookforconfirma6onemailandclickonconfirmyouaccountlink

Signupforflightstats

Sign up for a free trial at Flightstats.com

•  SignupathMps://developer.flightstats.com/signup•  Fillouttheformandmonitoremailforconfirma6onlink(accesstoAPIsmay

takeupto24hours)•  Onceaccessisgrantedgoto

hMps://developer.flightstats.com/admin/applica6onstoviewappIdandappKey(youwillneedtheminthesimple-data-pipetooltocreatetrainingsets.

•  Op6onal:getfamiliarwiththevariousflightstatsapis:–  hMps://developer.flightstats.com/api-docs/scheduledFlights/v1–  hMps://developer.flightstats.com/api-docs/airports/v1

Howtofindyourappidandkey

Where to find the FlightStats app id and app key

APPKey

Prepareyourbluemixspace

Create a new space on Bluemix Inprepara6onforrunningtheproject,wecreateanewspaceonBluemix

CreateaSparkInstance

Op6onal:YoucanskipthisstepifyoualreadyhaveaspacewithSparkinstancethatyouwouldliketoreuse

Create a Spark Instance

Op6onal:YoucanskipthisstepifyoualreadyhaveaspacewithSparkinstancethatyouwouldliketoreuse

Create New Spark Instance Op6onal:YoucanskipthisstepifyoualreadyhaveaspacewithSparkinstancethatyouwouldliketoreuse

Agenda

Flight App Project Description • Usecase

–  Flightdelaysareacommondisturbanceduringbusinesstrips–  Beingabletopredicthowlikelyaflightwillbedelayedcanremoveuncertaintyandenableuserstoplanaroundit.

–  Idea:Weatherdatacanbeagoodexplanatoryvariableforbuildingpredic6vemodels

•  ImplementaSon–  Combineflightsta6s6csfromflightstats.com(Systemofrecords)withweatherdatafromIBMInsightforWeather(Systemofopera6ons)tobuildatraining,testandblindset

–  UseSparkMLLibtotrainpredic6vemodelsandcrossvalidatethem–  CreateacustomcardforGoogleNowthatwillautoma6callyno6fyuserofimpendingflightdelay

–  Proposealterna6ngflightroutes(e.g.Freebird)Get/Build/Analyze

Get/Build/Analyze methodology

Flight Predict App Architecture

Weather

SimpleDataPipes

Airports

FlightSchedules

FlightStatus

MetadataTrainingSet

TestSet

BlindSet

CustomConnectorrunevery24hours

Notebook

Flow Diagram

DataAcquisi6on

DataPrepara6on

DataAnnota6on(GroundTruth)

ModelTraining

•  Cleansing•  Shaping•  Enrichment

ModelTes6ng

TrainingSet

TestSet

BlindSet

Iterative

Cross-Validation

Evaluate Performance and optimize model

Train Model

•  Itera6veinNature:weareneverdone!•  Wewillbeusingthisdiagramasaroadmapthroughoutthiscourse

DeployandRunModel

Get the data and build the training/test/blind sets Inthisstepwe’lluseSimpleDataPipesopensourceprojecttoacquiredatafromFlightstats,combineitwithWeatherdatafromIBMInsightforWeatherandsavethedatasetsintoaNoSQLCloudantDatabase.

DataAcquisi6on

DataPrepara6on

ModelTraining

ModelTes6ng

TrainingSet

TestSet

BlindSet

Iterative

Cross-Validation

Train Model DeployandRunModel

Acquiring the data

•  Inthenextsec6on,weshowhowtoacquirethetrainingdatabyusingthesimple-data-pipetoolandflightpredictconnector.

•  Theflightpredictconnectorcombinehistoricalflightdatafromflightstats.comwithweatherdatafromIBMInsightforWeather

•  Ifyouwanttoskipthesesteps,youcanusethealreadybuiltdatasetbyusingthefollowingcreden6als:–  cloudantHost:dtaieb.cloudant.com–  cloudantUserName:weenesserliffircedinvers–  cloudantPassword:72a5c4f939a9e2578698029d2bb041d775d088b5

Deploysimple-data-pipe

Deploy simple-data-pipe with flightstats connector •  GotohMps://github.com/ibm-cds-labs/simple-data-pipe•  ClickonDeploytoBluemixbuMon

ClickbuMonwilltakeyoutoBluemix

Complete simple-data-pipe deployment

AddWeatherservice

Add an instance of IBM Weather Service on Bluemix •  Returntotheapplica6ondashboard•  Weatherserviceisrequiredbythe

flightpredictconnectorandmustbeinstalledbefore

•  Fromappdashboard,clickonAddaserviceorAPI

Create an instance of IBM Weather Service on Bluemix SearchforWeather

Makesuretoselect“premiumplan”tohaveenoughauthorizedAPIcalls

Checkpoint: simple data pipe app dashboard •  Verifythatyourappiscorrectlyboundtotherightservices

WeatherServiceusedtoenrichflightrecordswithweatherobserva6ons

CloudantServiceusedtostoretraining,testandblinddatasets

You’llneedtoclickonthisbuMonforthesteponthenextpageItisrecommendedtoincrease

theappmemoryto1GB

Install flight predict connector •  ClickEditCodebuMon,editpackage.jsontoaddflightpredictmodule:

– "simple-data-pipe-connector-flightstats":"git://github.com/ibm-cds-labs/simple-data-pipe-connector-flightstats.git"

addflightpredictmoduletodependencies

Saveyourchanges

don’tforgettoaddcommainthelinebeforetokeepjsonvalid

Install flight predict connector •  ClickFile/Savetosaveyourchanges

Redeploysimpledatapipe

Redeploy simple data pipe app •  UseliveeditEditortoredeploytheapp

Verifyyoursdpinstall

Verify connector install •  Inthisstep,weverifythattheflightpredictconnectoriscorrectlyinstalledthroughtheUI

Fightconnectorcorrectlyinstalled

Createnewflightstatspipe

Create a new FlightStats pipe •  Followeachscreentocreateandconfigureanewpipe

Runthepipe

Run the pipe •  Skipoverthescheduletab•  Intheac6vitytab,clickonRunNowtostartthepipe

Explorethedataset

ClickRunNowThenopenthelogtomonitortheac6vity

Explore the data sets •  Inthisstep,wetakeamomenttoexplorethedifferentdatasetsthathavebeencreatedbythesimpledatapipetool

•  Frombluemixdashboard,clickonthecloudantservice6le,thenontheLaunchbuMon•  FromtheCloudantdashboard,openthetrainingdatabase•  Openadocumenttolookatthedatastructure

Buildthetestset

Run the pipe again to build the test set

Trainthemodels

Train the Models •  Intheprevioussec6onwehavecreatedthetrainingdataandwearenowreadytotrainthemodels.•  Stepsinthissec6on:

–  CreateanIPythonNotebook–  LoadthedatasetsfromtheCloudantdatabaseintoaSparkCluster–  Explorethedataandtrainthemachinelearningmodels

DataAcquisi6on

DataPrepara6on

ModelTraining

ModelTes6ng

TrainingSet

TestSet

BlindSet

Iterative

Cross-Validation

CreateIPythonNotebook

Create a new IPython Notebook

Notebook tour

Notebook tour: Notebook Info

Notebook tour: Environment

Notebook tour: Sharing

Agenda

Before we start building the app…

•  Youcanop6onallyfollowthistutorialfromGithubbyusingafullybuiltnotebook:– hMps://github.com/ibm-cds-labs/simple-data-pipe-connector-flightstats/blob/master/notebook/Flight%20Predict%20PyCon%202016.ipynb

Optional: use prebuilt notebook

ImportrequiredPythonpackages

• CreatenotebookfromURL• UsehMps://github.com/ibm-cds-labs/simple-data-pipe-connector-flightstats/raw/master/notebook/Flight%20Predict%20PyCon%202016.ipynb

Using Python Packages •  Writecodeinlinewithincells•  EncapsulatehelperAPIswithinPythonpackage•  2waysofusinghelperPythonpackages

–  eggdistribu6onpackage:pipinstallfromPyPiserverorfileserver(e.g.Github)

•  Persistentinstallacrosssessions•  RecommendedinProduc6on

–  SparkContext.addPyFile•  Easyaddi6onofapythonmodulefile•  Supportmul6plemodulefilesviazipformat•  Recommendedduringdevelopmentwherefrequentcodechangesoccur

Manageeggpackages

Flight Predict Python Package on Github

SetupscriptforinstallingPythonPackage

FlightPredictPythonlibrary

Method 1: Install Flight Predict Package •  UsepiptoInstallFlightPredictpackage•  Recommendedalterna6ve:buildeggdistribu6onpackageanddeployinPyPi

Manage Python packages •  Checkstatus•  Uninstallpackage

Installpackagesviasc.addPyFilemethod

Method 2: Install py modules via sc.addPyFile

•  addPyFileinstallindividualpymodulesandmakethemavailabletoallexecutorprocesses

•  Workswithmodulesinzippedfiles

Modulecontainingapisfortrainingthemodels

Modulecontainingapisforrunningthemodels

Configurecreden6alsforvariousservices

Setup credentials and Import required python modules

Inthisstep,weimportpythonmodulesthatwillbeneededthroughoutthenotebookandsetupcreden6alstovariousservices.

Howtogetcreden6alsforCloudantandWeather

Creden6alforCloudantNoSQLService

Creden6alsforWeatherService

Get Credentials for Cloudant Fromtheappdashboard,clickonEnvironmentVariablesfromthelessidebar

Get Credentials for Weather

LoadtrainingsetfromCloudant

Load training set in Spark SQL DataFrame

Inthisstep,weusethecloudant-sparkconnector(hMps://github.com/cloudant-labs/spark-cloudant)toloaddataintoSpark

Makesuretochangethedbnametomatchtheonecreatedforyourtrainingsetbyyourac6vity(opentheCloudantdashboardtofindthename)

Loading data: Behind the scene UseSparkSQLconnectortoloaddataintoaDataFrame

connectorid

Op6ons

Cachedataforop6mizedreuse

CreatetempSQLTable

ScaMerPlotVisualiza6on

Scatter plot visualization

Visualization api

CreateanRDDofLabeledPoint

Transform into an RDD of LabeledPoint UseSparkSQLconnectortoloaddataintoaDataFrame

loadLabeledDataRDD api

TrainMachineLearningModels

Machine Learning Algorithms ConSnuousOutput DiscreteOutput

SupervisedLearning(requireGround-Truth)

•  Regression-Linear-Ridge-Lasso-Isotonic•  DecisionTree•  RandomForest•  GradientBoostedTree

• Classifica6on-Logis6cRegression-SVM-NaiveBayes• DecisionTree• RandomForest• GradientBoostedTree• K-NN(availableasadd-onsparkpackage)

UnsupervisedLearning(noGround-Truthdatarequired)

•  Clustering-KMeans-GaussianMixture•  DimensionalityReduc6on-PCA-SVD

•  FP-Growth

TrainLogis6cRegressionModel

Train Logistic Regression Model

TrainNaïveBayesModels

Train NaiveBayes Model

TraindecisionTreeModel

Train Decision Tree Model

TrainRandomForestModel

Train Random Forest Model

AccuracyAnalysis

Naïve Bayes vs Decision Tree •  Probabilis6c:computetheprobability

ofadatainstancetobeinaspecificclass

•  Assumethateachfeature(variable)isindependentfromtheothers

•  Performancedependsonthepredic6venatureofthefeatures(nonpredic6vefeatureswillaffecttheaccuracy)

•  Workswellwithlowamountoftrainingdata.Doesn’tneedallthepossibili6es

•  Doesn’tworkwithcategoricalfeatures.

• Non-Probabilistic: partition the data into subsets that best describe the variable

• The deeper the tree, the better the model fits the data

• Watch out for overfiting: need to prune the tree

• Can handle categorical or continuous features

• No need for input to be scaled or standardized: Set you features and go!

• Requires a lot of data covering all possibilities

Accuracy Analysis of the Machine Learning Models Inthissec6on,wewillperformaccuracyanalysisonthetestdata.Wewillstartbycompu6ngtheaccuracymetricsforeachmodel,includingtheconfusionmatrix.Wewillthenusehistogramcharttounderstandthedatadistribu6onandrefinehowtoclassesarecomputed.

DataAcquisi6on

DataPrepara6on

ModelTraining

ModelTes6ng

TrainingSet

TestSet

BlindSet

Iterative

Cross-Validation

Agenda

Load Test data Makesuretochangethedbnametomatchtheonecreatedforyourtestsetbyyourac6vity(opentheCloudantdashboardtofindthename)

Accuracy Metrics

Confusion Matrix

Accuracy metrics API OutputHTML

DisplayresultsHTMLinNotebookCell

ComputeMetricsfromlabeledandpredic6ondata

Gettheconfusionmatrixandbuildhtmltable

Understand the distribution of your data with Histograms

Training Handler class

•  Provideflexibilityandextensibilitytotheapplica6on

•  Provideafailfastandtrysomethingelsemechanism

•  Enableusertoeasilycustomizeclassesofdatabasedonhowdataisdistributed

•  Enableusertoeasilyaddtrainingfeatures

Default Training Handler class

Returndescrip6onforeachclasses

Returntotalnumberofclasses:Defaultis5

Re-classifyarecord:defaultusess.classifica6onfieldinJsonrecord

ExtrafeaturesNamestobeadded.Nonebydefault

Extrafeaturestobeadded.ArraymustmatchtheonereturnedbycustomTrainingFeaturesNames

Customize Training Handler Providenewclassifica6onandadddayofdepartureasanewfeature

InheritfromdefaultTrainingHandler

Adddayoftheweekusingatechniquecalleddummycoding

Re-train the models

Re-compute accuracy Models1

Models2BeMeraccuracyforNaiveBayesandLogis6cRegressionWorseforDecisionTreeandRandomForest

Agenda

Deploy and Run the models Inthelastsec6on,wewillsimulatedeploymentandrunningofthemodelsthroughthenotebookbycallingAPIsfromtherunpackage.

DataAcquisi6on

DataPrepara6on

ModelTraining

ModelTes6ng

TrainingSet

TestSet

BlindSet

Iterative

Cross-Validation

Train Model DeployandRunModels

Run the predictive model

runModel API

Get Weather Predictions

Show prediction results

Resource

•  hMps://developer.ibm.com/clouddataservices/•  hMps://github.com/ibm-cds-labs/simple-data-pipe•  hMps://github.com/ibm-cds-labs/pipes-connector-flightstats•  hMp://spark.apache.org/docs/latest/mllib-guide.html•  hMps://console.ng.bluemix.net/data/analy6cs/

Thank You

Spark tutorial pycon 2016 part 1

Data & Analytics

Pycon 2016

Hadoop and Spark Tutorial for Statisticians · Hadoop and.....

Spark tutorial pycon 2016 part 1

McDonough Spark Tutorial Spark Summit 2013

A Tutorial on Apache Spark - Michael...

Download Apache Spark Tutorial (PDF Version)

TUTORIAL: BIG DATA ANALYTICS USING APACHE SPARK ·...

Plone.app.discussion (Pycon DE)

Tutorial Spark

Design Spark Tutorial

Ashutosh pycon

big data tutorial w2 spark - ee.columbia.edu

Elasticwulf Pycon Talk

Spark Tutorial @ DAO · Spark Tutorial @ DAO ... to get...

PyCon 2015 (Py.15): Python Beginner's Tutorial

Excel tutorial by Sir Spark Microsoft Xcel