The Five Dysfunctions of a Data Engineering Team 1 / 18 Copyright © 2016 Smoking Hand LLC. All rights Reserved. Version: ef81f3f
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
TheFiveDysfunctionsofaDataEngineeringTeam
1/18Copyright©2016SmokingHandLLC.AllrightsReserved.Version:ef81f3f
Chapter1
TheFiveDysfunctionsofaDataEngineeringTeam
2/18Copyright©2016SmokingHandLLC.AllrightsReserved.Version:ef81f3f
WhyWorryTheDysfunctionsWhattoDo?
TheFiveDysfunctionsofaDataEngineeringTeam
3/18Copyright©2016SmokingHandLLC.AllrightsReserved.Version:ef81f3f
85%ofBigDataprojectsfailtogetintoproduction
Source:http://tiny.bdi.io/gartnerfail
Failure
4/18Copyright©2016SmokingHandLLC.AllrightsReserved.Version:ef81f3f
I'dtrainatcompaniesandseefailuresattheirmid-point
Ittookawhiletoseethepatterns
Ittookmoretimetofigureoutthemostcommonpatterns
BigdataonlyamplifiesexistingproblemsIfyoubarelygetbywithsmalldata,you'llhavebigproblemswithBigData
Youcanbesuccessfulbyavoidingtheseproblems
Why?
5/18Copyright©2016SmokingHandLLC.AllrightsReserved.Version:ef81f3f
WhyWorryTheDysfunctionsWhattoDo?
TheFiveDysfunctionsofaDataEngineeringTeam
6/18Copyright©2016SmokingHandLLC.AllrightsReserved.Version:ef81f3f
DBADefinition-SomeonewhoseonlyprogramminglanguageisSQL
Thisincludesdatawarehouse,SQLDevelopers,etc
BigDataisnotanextensionorthelogicalextensionofdatawarehousing
It'smuchmuchmorecomplexhttp://tiny.bdi.io/complex
It'snotjustaskillsgap;it'sanabilitygaphttp://tiny.bdi.io/abilitygap
AllDBAs
7/18Copyright©2016SmokingHandLLC.AllrightsReserved.Version:ef81f3f
BigDataiscomplexhttp://tiny.bdi.io/complex
Beginnersneedtobegivethetimeandresourcestolearn
Ittakesatleast6monthsforabeginnertobecomeproficient
Asyoulookatsuccessfulcasestudytalks,theyleaveout
ExpertresourcesprovidedStartingproficiencyoftheteamTotaltimeused
SetUpForfailure
8/18Copyright©2016SmokingHandLLC.AllrightsReserved.Version:ef81f3f
Schemaproblemsdon'tmanifestimmediately
Takes6-12monthstosee
Youcan'tlaydownPBsofdataandchangeit
Datapipelinesneedtochangedataformats
Whichroletypicallyhasthisskill?DBAs(Ididn'tsaynoDBAs-IsaidnotjustDBAs)
NoOneUnderstandsSchema
9/18Copyright©2016SmokingHandLLC.AllrightsReserved.Version:ef81f3f
AprojectveteranissomeonewhohasputaBigDataordistributedsysteminproduction
BeginnerstodistributedsystemsandBigDataarethesourcesoftheworstabominations
Averagetimelostis1-2manmonths
Verydifferenttowhiteboardanderasethancodeandrewrite
OnlyaprojectveterancancritiqueawhiteboardedarchitectureRememberit'sprogrammingandoperations
NoVeterans
10/18Copyright©2016SmokingHandLLC.AllrightsReserved.Version:ef81f3f
Youcan'tgofrom0toBigDataallatonce
Youreallycan'tgofrom0totheholygrail
Yourteamneedsthetimetogofrombeginnerstointermediatetoadvanced
YouneedtobuildmomentumfirstProjectswithoutmomentumgetcanceled
TooAmbitious
11/18Copyright©2016SmokingHandLLC.AllrightsReserved.Version:ef81f3f
WhyWorryTheDysfunctionsWhattoDo?
TheFiveDysfunctionsofaDataEngineeringTeam
12/18Copyright©2016SmokingHandLLC.AllrightsReserved.Version:ef81f3f
TakeanhonestevaluationoftheteamSkillsAbilitiesUsecaseResources
Doestheteamhaveaskillsgap?
Doestheteamhaveanabilitygap?http://tiny.bdi.io/abilitygap
DoesthisSoundLikeYourTeam?
13/18Copyright©2016SmokingHandLLC.AllrightsReserved.Version:ef81f3f
Dataengineeringteamsneedtobemultidisciplinary
http://tiny.bdi.io/detbook
DataEngineeringTeams
14/18Copyright©2016SmokingHandLLC.AllrightsReserved.Version:ef81f3f
Someteamssaytheydon'tneedhelpTechnicalpeoplethinkit'snotneeded(smalldatamentality)Admissionoffailure
Veryimportanttotakeanhonestlookattheteam
Training
ConsultingVeryimportanttogetacompanywithagoodtrackrecord
MentoringOngoinghelpforthetechnicalandbusinessteams
GettingHelp
15/18Copyright©2016SmokingHandLLC.AllrightsReserved.Version:ef81f3f
100
80
60
40
20
0First ReleaseTeam Creation Project Start Second Release Nth Release
Phase In Project
Percent of Blame
Management Development Operations
WhenThingsFail
16/18Copyright©2016SmokingHandLLC.AllrightsReserved.Version:ef81f3f
Early
Nevertoolatetofix,butfixingwillbemuchmore
costly
WhenShouldYouFix?
17/18Copyright©2016SmokingHandLLC.AllrightsReserved.Version:ef81f3f
Current:Instructor,ThoughtLeader,MonkeyTamer
Previously:CurriculumDeveloperandInstructor@ClouderaSeniorSoftwareEngineer@Intuit
Covered,ConferencesandPublishedIn:GigaOM,ArsTecnica,PragmaticProgrammers,Strata,OSCON,WallStreetJournal,CNN,BBC,NPR
SeeMeOn:http://www.jesse-anderson.com@jessetandersonhttp://tiny.bdi.io/linkedinhttp://tiny.bdi.io/youtube
AboutMe
18/18Copyright©2016SmokingHandLLC.AllrightsReserved.Version:ef81f3f
Related Documents
