TIAD - Is Automation Worth My Time?

IS IT WORTH MY TIME?AUTOMATION

@JRHUNTJ. RANDALL HUNT

@JRHUNTJ. RANDALL HUNT

AGENDA

▸Why automate?

▸When to automate?

▸How to automate?

▸Stories from the field

WHY THE HELL AM I DOING THIS?

EXAMPLE!sorry, I like big fonts!

WHY?

FIND LARGEST FILE IN A DIRECTORY

WHY?

FIND LARGEST FILE IN A DIRECTORY

▸ ls -S | head -1

▸ takes about 4 seconds to type

▸ takes about 45 seconds to google

WHY?

FIND LARGEST FILE IN A DIRECTORY

▸ ls -S | head -1

▸ takes about 4 seconds to type

▸ takes about 45 seconds to google

▸ largest () { ls -S | head -1 }

▸ takes about 15 seconds to type...

▸ and then 2 seconds forever after

WHY?

FIND LARGEST FILE IN A DIRECTORY

▸ ls -S | head -1

▸ takes about 4 seconds to type

▸ takes about 45 seconds to google

▸ largest () { ls -S | head -1 }

▸ takes about 15 seconds to type...

▸ and then 2 seconds forever after

WHY?

FIND LARGEST FILE IN A DIRECTORY

WHY?

FIND LARGEST FILE IN A DIRECTORY

▸ find . -type f -print0 | xargs -0 du -h | sort -hr | head -1

WHY?

FIND LARGEST FILE IN A DIRECTORY

▸ find . -type f -print0 | xargs -0 du -h | sort -hr | head -1

WHY?

5 HOURS OF FASCINATED CLICKING LATER

https://xkcd.com/1319/

https://xkcd.com/1319/

https://xkcd.com/974/

AWS SIMPLE SYSTEMS MANAGEMENT

WHEN SHOULD I DO THIS?

https://xkcd.com/1205/

https://xkcd.com/1205/

https://xkcd.com/1205/

PERF

ORMA

NCE

QUALITY

WHEN?

CHESS GAME WINNERS

▸ ~14 million games recorded in ASCII

▸ Read in files, aggregate winners, report stats

▸ Sounds like a job for:

AMAZON EMR

AMAZING PERFORMANCE -- ENTIRE DATASET COMPUTED IN JUST 20 MINUTES FOR <$10!!

HMM....

WHEN?

CHESS GAME WINNERS

▸ ~14 million games == ~10 gigs

▸ Read in files, aggregate winners, report stats == grep / awk

▸ Sounds like a job for: THE COMMAND LINE!

find . -type f -name '*.pgn' -print0 | xargs -0 -n4 -P4 mawk '/Result/ { split($0, a, "-"); res = substr(a[1], length(a[1]), 1); if (res == 1) white++; if (res == 0) black++; if (res == 2) draw++ } END { print white+black+draw, white, black, draw }' | mawk '{games += $1; white += $2; black += $3; draw += $4; } END { print games, white, black, draw }'

Thanks to Adam Drake for figuring this out!

http://aadrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html

30 SECONDS

OK, TELL ME HOW!

HOW?

I HAVE NO IDEA WHAT I'M DOING, BUT THIS IS WHAT I DO

1. Identify constraints

‣ Money, Business Goals, Time, Performance, Beauty

2. Pursue mitigating factors within constraints

3. Relentlessly pursue constraints until the laws of physics tell you to stop

4. Iterate

WHAT ARE THE ACTUAL CONSTRAINTS?

CONSTRAINTS

HARDWARE

▸ AWS EBS gp2 volumes have max ~160MiB/s*

▸ AWS EBS io2 volumes have max ~320 MiB/s*

▸ Realistic max network performance of ~800 MiB/s*

▸ CPU performance varies on instance type but up to 8 terraflops per instance

CONSTRAINTS

BUSINESS CONSTRAINTS (MONEY)

▸ How much money can we spend on this problem?

▸ How much time can we spend on this problem?

▸ Will a partially automated solution allow us to kick this problem down the road?

▸ Is it cheaper/easier/more accurate to hire humans?

CONSTRAINTS

QUALITY CONSTRAINTS (AI VS HUMANS)

▸ How much better are humans at solving this?

CRAZY STUFF WE'VE AUTOMATED

BRINGING IT BACK HOME

CLOSING THOUGHTS

INFRASTRUCTURE AS CODE == AUTOMATION AS CODE

▸ Chef, Puppet, Ansible or something similar is vital (you already know this)

▸ AWS CloudFormation lets you launch entire FLEETS and NETWORKS in moments. This system is free...

▸ Heterogeneity is the death of automation (cattle not pets)

CLOSING THOUGHTS

STOP DOING THE THINGS YOU ALREADY KNOW HOW TO DO

▸ You already know how to run a reverse proxy -- write the code once and be done with it (Amazon API Gateway).

▸ You already know how to maintain your database (AWS RDS) set up protocols and be done with it

▸ You already know how to manage security and permissions, don't set it up yet again

CLOSING THOUGHTS

▸ Identify the biggest automation wins

▸ Pursue your automation by pursuing constraints

▸ Add humanity to the process

▸ Don't overthink it, but definitely think about it

AWS User Groups

LilleParisRennesNantesBordeauxLyonMontpellierToulouse

facebook.com/groups/AWSFrance/

@aws_actus

AWS Enterprise Summit – 27/10/2016, Paris

http://amzn.to/1X2yp0i