Modernizing Business Intelligence and Analyticscdn.govexec.com/media/ctd_analytic_db_final.pdfUse your EDW more efficiently by offloading workloads to Hadoop Fast, flexible ETL over

1©Cloudera,Inc.Allrightsreserved.

ModernizingBusinessIntelligenceandAnalytics

1©Cloudera, Inc.Allrightsreserved.

JustinEricksonSeniorDirector,ProductManagement


•WhatbenefitscanIachievefrommodernizingmyanalyticDB?•WhenandhowdoImigratefromcurrentsystems?• Howdoesitworkinthecloud?

Agenda


EDWOptimization

DataPreparation

Self-ServiceBI&Exploration

UseyourEDWmoreefficientlybyoffloadingworkloadstoHadoop

Fast,flexibleETLoverlargedatavolumes,sodataisalwaysreadyforyourbusiness

Fastesttime-to-insightswithamodernanalyticdatabasedesignedwithHadoop’sflexibilityandagility

KeyApplications


Cloudera’sAnalyticDatabase

Identify,offload,&optimizeworkloadsto

Hadoop

NavigatorOptimizer

IntelligentSQLeditor

Hue

Audit,lineage,encryption,key

management,&policylifecycles

Navigator

IntegrationwiththeleadingBItools

BIPartners

InteractivequeryengineforBI&SQLanalytics

Impala

Large-scaleETL&batchprocessingengine

Hive-on-Spark

Multi-Storage,Multi-Environment

DataStorageforFast&ChangingData

Kudu


KeyBenefitsAnanalyticdatabasedesignedforHadoop

High-PerformanceBIandSQLAnalytics

FlexibilityforDataandUseCaseVariety

Cost-effectiveScaleforTodayandTomorrow

GoBeyondSQLwithanOpenArchitecture


AnalyticDBAnatomyBuiltforself-serviceandhybridcloud


AnatomyofanAnalyticDatabaseCloudera DecoupledbyDesign

QueryEngine

StorageEngine

Catalog

QueryEngine(Impala)

Catalog(HMS)

MonolithicAnalyticDatabase ModernAnalyticDatabase

Storage(Kudu)

Storage(S3)

Storage(HDFS)


LimitedtoSQLonly• Maintaindatacopiesfornon-SQL

RigidDataModel• Tightlycoupledstorageandcompute

StaticSizing• Majormaintenancetoaddcapacity/nodes

PoorlyDesignedforCloud• Noelasticityorintegrationwithobjectstorage

PainPointsTraditionalMonolithicAnalyticDatabases

∞

COMPUTESTORE


Benefits ofCloudera’sModernApproachCloud-Native&On-Premise

GoBeyondSQL• OpenArchitecture:Openformatsandopenstorage

• ShareddataacrossSQLandnon-SQLworkloads

DataFlexibility• Faster,moreagiledataacquisition• Dataportability:Openformatsandopenstorage

Cost-EffectiveScalability• Elasticscaleon-premorinthecloud

• Cloud-nativepay-per-useandtransience

• Provenatbigdatascale

Hybrid• Runsacrossmulti-cloud&on-prem

• Multi-storageoverS3,HDFS,Kudu,Isilon,DSSD,etcSharedData


EDWOptimizationExpandtheValueofYourDataWarehousingLandscape


MotivationsforOptimizingtheEDW

CostcontainmentforexistingworkloadsLimitedbudgetforexpansion

UnabletotakeonnewworkloadsUnabletokeepupwithchangingbusinessneeds

Difficultyhandlingbothfixed-SLAreportsandself-serviceexploration

Growingimportanceofself-serviceBI,advancedanalytics,andcloud

$$


ExistingEDWLandscape

DataSources

ETL/Staging

EDW

Archive

DataMarts

CannedReports

Dashboards/AnalyticApplications

Non-SQLWorkloads

Self-ServiceBI/AdHoc


OptimizingtheEDWwithCloudera

• Cost-EffectiveScale• Sayyestomorewithouttherisk

• GoBeyondSQL• Exploration,advancedanalytics,andmoreallinoneplatform

•ModernizetheDataWarehouseLandscape• MaximizetheEDWwhileenablingiterative,self-serviceaccess/BI• Well-suitedforon-prem,cloud,andhybriddeployments

90%lessperTBvsRDBMSand75%lessvsNetezza

Augmented itsOracleEDWwithmulti-tenantClouderasystemwiththeirBItoolconfiguredtoallowuserstopullreportsfromboth

MediaResearchFirmSavedtensofmillionsbyoffloadingDBMStoClouderainthecloud


ModernDataWarehouseEnvironment

DataSources

EDW

AnalyticDatabase

OperationalDatabase

DataScience&Engineering

SharedDataLayer

ModernDataPlatform

FixedReports

Dashboards/AnalyticApplications

Non-SQLWorkloads

Self-ServiceBI/AdHoc

FlexibleReporting


Plan Offload Optimize

EstimateEffort

RiskAnalysis

SchemaDesign

FineTuningDataModelonHadoop

OptimizeQueriesforPerformance

Test&Validate

Evaluate

IdentifyUseCases

ImpactAnalysis

Objectives PrioritizedPlan

ValidateROI,CostInitialPOC

OffloadeachworkloadEvaluatetheneedforoffload Impactanalysis,prioritizedplan

Optimizeperformance

WorkloadVisibility

NavigatorOptimizerBuilttohelpyouthroughtheoptimizationprocess

OffloadActions


WorkloadVisibilityGetinsightsintowhat’shappeningtoday

EvaluateQueries• Topqueries• Queryduplication• Querycomplexity• Commonaccesspatterns

EvaluateDataAccess• Toptables,topcolumns• Usage-basedERdiagram• Alltables/columnsinuse

EvaluatePOC• IdentifyinitialworkloadpieceforPoC• Getpartitioningkeysuggestions

Evaluate


ImpactAnalysis&PrioritizedPlanUnderstandwhatittakestooffload

ImpactAnalysis• Focuseffortsbyidentifyingduplication• Workloadriskassessmentbasedoncomplexityandbestpractices

• Understandquerycompatibility

PrioritizedPlan• Estimateeffort• Identifyeasiestpiecestostartforfastsuccess• Prioritizeworkloadsforoffload

Plan


PredictableOffloadRemovetheguesswork

Understandoffloadrequirements• Determinemostcommonworkload

patterns• Developdata-/usage-drivenoffload

strategy

Actionablerecommendations• Complexityassessmentforriskierareas• Focuseffortsbyidentifyingduplication• Designrecommendationsforbestresults

Offload


OptimizingwithinHadoopMaintainpeakperformance

Understandusageandkeepupwithdataneeds• Understandmostcommonusagepatterns• Identifyoptimizationopportunities• Proactivelyadjustdatamodels

Performanceoptimizations• BestpracticeguidanceforHiveandImpala• Queryperformanceoptimization• Increaseplatformadoption

Optimize


Builtforhybridcloud


What’sDrivingAnalyticstotheCloud?Bigdatadeploymentsincloudareaccelerating:

● ExecutiveMandate:Minimizeon-premdatacenterfootprint

● IncreasedAgility:End-userself-service

● Elasticity:Optimizeinfrastructureusage

● LowerOverallTCO


MostOrganizationsAreorWillbeHybridCloud

• 76%willembracehybridcloud(Gartner1)• 82%willhaveamulti-cloudstrategy(RightScale2)• 50%will“repatriate”atleastonepubliccloudworkloadbacktoprivatecloudor

on-prem forcostreasons(4513)• 50%ofCloudera’scloudcustomersrunahybridenvironment

1Gartner,MarketTrends:CloudAdoptionTrendsFavorPublicCloudWithaHybridTwist20152RightScale 2016StateoftheCloudReport3451Research:AWSLambda:newandexciting,oldandrehashed,morevendorlock-in(oralltheabove)?,November22,2016

Whyisthisacriticalstrategy?

Portability&Cost Functionality DataGravity


Cost-Efficiencies&FlexibilityintheCloudPrimaryAnalyticDatabasePatterns

Onlypayforwhatyouneed,whenyouneedit

▪ Transientclusters▪ Objectstoragecentric▪ Cloud-nativedeployment

ETL

ReduceOperatingCosts NewInsights,NewRevenue

BI/Analytics

Exploreandanalyzealldata,whereveritlives

▪ Long-runningclusters▪ Objectstorageorlocalstorage▪ Lift-and-shiftdeployment


AddUseCases,Analytics,andDataOn-Demand• AvoidtheITbacklogwithinstantaccesstoalldata

• On-demandclustersquerydirectlyonsharedobjectstorage

PredictableResultsWheneverYouWant• Consistentqueryperformance,evenduringpeaktimes

• Multi-tenancyviaisolatedclustersonshareddata

Just-in-TimeResources• Real-timecapacityforyourneeds,astheychange

• Elasticallygrow/shrinkyourclusterviadecoupledarchitecture

Contention-FreeETL• ETLanytimewithoutimpactingotherworkloadsorriskingSLAs

• SeparateETLclustersas-neededonshareddata

AdditiveBenefitsintheCloudExtendingcoreperformance,flexibility,scalability,andopenarchitecturebenefits


BI/AnalyticsintheCloudThreeArchitecturesOptionstoOptimizePrice/Performance

ObjectStorage

TransientCluster

TransientBI(infrequentusage)Spinupclusterswhenneeded● On-demandinstances● Usage-basedpricing● Grow/shrink● Clusterpertenantoruser

PersistentBI(regularusage)PersistentclustersforBIanytime● Reservedinstances● Node-basedpricing● Grow/shrink● Clusterpertenantgroup

PersistentCluster

PersistentBIwithLocalStorage(fastest)Maxspeedformoreregularworkloads● Reservedinstances● Node-basedpricing● Lessfrequentgrow/shrink● Sharedclusterforsharedlocaldata

PersistentCluster HDFSand/orKudu

PersistentCluster

TransientCluster

DefaultChoice


PersistentBIonObjectStorageBestforelasticity(andspeedvstransient)

● Thisisusuallythebestchoice● Bestwhenworkloadsare:

o Flexibleandchangingo Frequentduringmostworkingdayso Notscheduledforfixedhours

● Benefitsinclude:o Predictableresultsreadilyavailableo Fullmulti-tenantisolationo Commondatainsharedobjectstorageo Grow/shrinkforTCOefficiency

● Tradeoffs:o Pernodeperfofobjectstorage(usemore,

cheapernodes)ObjectStorage

SharedHMSDB

PersistentBI(regularusage)Persistentclustersforreadyavailability● Reservedinstances● Node-basedpricing● Grow/shrink● Clusterpertenantgroup

PersistentCluster

PersistentCluster

DefaultChoice


PersistentBIwithLocally-AttachedStorageBestperformanceforconsistentworkloads

● Bestwhenworkloadsare:o Regularandconsistento Consistentlyqueryingcommondatao TightSLAsforperformanceo Fastchangingdata(thatneedsKudu)o Runningwithoutobjectstorage(eg.Azure,GCE)

● Benefitsinclude:o Fasterperformancepernodeonlocaldatao Abilitytoqueryobjectstorageforrestofdata

● Tradeoffs:o Lesselasticthanobjectstoredbasedclusterso Lessisolationformulti-tenantworkloadsusing

sameHDFSdatao Costifthereareoff-peakhours

ObjectStorage

PersistentBIwithHDFS(fastest)Maxspeedformoreregularworkloads● Reservedinstances● Node-basedpricing● Lessfrequentgrow/shrink● SharedclusterforsharedHDFSdata

PersistentCluster

LocalHMSDB

HDFSand/orKudu


TransientBIonObjectStorageBestTCOforinfrequentusage

ObjectStorage

ClouderaDirector

● Bestwhenworkloadsare:o Infrequentorscheduled

● Benefitsinclude:o LowestTCOwithclustersonlywhenneededo Fullmulti-tenantisolationo Commondatainsharedobjectstorage

● Tradeoffs:o Delaytospin-upclusterswhenneededo CapabilityofBIuserstospinupclusterso Pernodeperfofobjectstorage(usemore,

cheapernodes)SharedHMSDB

TransientCluster

TransientBI(infrequentusage)Spinupclusterswhenneeded.● On-demandinstances● Usage-basedpricing● Grow/shrink● Clusterpertenantoruser

TransientCluster

©Cloudera,Inc.Allrightsreserved. 29

ThankyouThankYouJustinErickson

Modernizing Business Intelligence and Analyticscdn.govexec.com/media/ctd_analytic_db_final.pdfUse your EDW more efficiently by offloading workloads to Hadoop Fast, flexible ETL over

Documents