Hashicorp at holaluz

BARCELONA HUG - 14 SEP 2017

at

7 YEARS AGO…• Vagrant first commit was in 17 Jan 2010

• Holaluz was founded in 12 Nov 2010

• DevOps starts becoming a concept

• A backend developer in a big e-commerce

WHO AM I?• Currently working as CTO at Holaluz

• Ex Wonga, Hailo, SocialPoint, Ulabox, Privalia…

• Developer for many years, been automating things for a while, DevOps before it was trendy!

• https://devops.barcelona conference co-organiser

HOLALUZ WEBAPPS INFRA

Web servers under ELBS

running Apache+ PHP7.1

Dedicatedcron job boxesusing PHP7.1

RDS PostgreSQLinstances

EC2 Elasticsearch

Some S3buckets tostore files

ElasticacheRedis

for messagededuping

SQS queuesbackground

jobs

RDS MySQLinstances

EFS to storeshared files(moving to S3 & ELK)

HOLALUZ BIGDATA INFRA

Python apps & cron tasks using Django and Celery

R runninginstances

RDSPostgreSQL

instances

EC2 Elasticsearch

EC2 MongoDB

Some S3buckets tostore files

ElasticacheRedis

for Celery

HOLALUZ ERP INFRA

RDSSQLServer

Per env S3 buckets and SQS communicating

with Web Infra

License ServersCluster

Windows 2012R2 Servers Cluster running Java8

PROD

INTEGRATION

DEVELOPMENT

OTHER THINGS AROUND

Shared ELK Stack for logs aggregation and inspection

Middleman static sitesgenerator

Grafana for Time Series inspection and alerting

InfluxDB for Time Series metrics storage

E-Invoice Document Management + approval

flows

Cloudwatch Time Series metrics storage and alerting

HASHICORP TOOLS• Confession: I am a massive fan

• Work everywhere, automate anything

• Excellent open-source community

• Well documented, specially BC changes

Development Environments Made Easy

VAGRANT AT HOLALUZ• Static website generator middleman (Ruby based)

• Local environments for PHP projects

• Local environments for Python Big Data projects

• Coming soon, Windows boxes for people who need to connect to sites needing old IE

Build Automated Machine Images

CONCEPTS• Builders: Platforms you build images in. It is all

about what you start from!

• Provisioners: Installs and configures

• Post-processors: Optional final steps

WHAT I LIKE• Builds for multiple platforms from a single

source configuration

• VERY Easy to understand

• Works (and can provision) in Win, Mac, Linux

• Easy to share provisioning scripts or use Puppet / Ansible recipes

CAVEATS• Need to be very prescriptive or you end up

with multiple very similar templates

• A bit hard to go with a DRY approach

• Some things are hard to destroy / replace with new images

PACKER AT HOLALUZ• All PHP infrastructure with ASGs using

AMIs we recycle periodically

• Kibana, Logstash, Grafana, InfluxDB

• Python BigData infrastructure as well (R soon)

• ERP first steps on top of official Windows AMIs

Write, Plan and Create Infrastructure as Code

CONCEPTS• Provider: Platform we are automating

• Resources: Automatable things in the Provider

• Modules: Reusable set of resources

• State: Used to diff desired state to existing. Can be stored remotely and supports distributed locking

WHAT I LIKE• Can integrate with anything that has an API

• Easy to extend, contribute and really quick to add new features. Excellent Github community

• Existing resources can be imported (PAIN)

• Have used it for 18 months, multiple providers, rarely hit a bug and was always quickly fixed

CAVEATS• Once you go Terraform, STOP using Console

• Some providers don´t have nice update support

• Terraform modules feel a bit hacky

• Sometimes state needs manual edition (getting much better but beware new providers)

TERRAFORM AT HOLALUZ• Most infrastructure rebuilt using Terraform

• VPC / Network for test environments

• Some S3 buckets and SQS configuration

• Most IAM policies

INTERESTING TRICKS (I)

Automatic code deployment via EC2userdata on startup

INTERESTING TRICKS (II)

Always apply latest built AMI using a filter

on tags created by Packer

INTERESTING TRICKS (III)

Environments Modules

State in S3 per envLock via DynamoDB

Use env output as variables in another env

BACK IN JANUARY• Deployments were painful

• Unreliable test and local dev environments

• Hard to integrate new things

• Poor logging and monitoring metrics

• Painful error recovery, no real DR strategy

BAD HABITS AND PROBLEMS• Development teams did not talk much

• Little understanding of network and infra

• Lack of technical guidelines

• No time to rethink processes

• Lots of legacy code with no tests at all

HOLALUZ IN SEPTEMBER• One-click deployments almost everywhere

• Development teams talk and collaborate

• Releases speed MASSIVELY improved

• Decent logging and monitoring

• Better error recovery, decent DR strategy

LONG WAY TO GO• Lots of technical debt still to pay

• Fragility in crucial parts of our business

• Still too many bugs, still too little testing

• Incoming changes in Energy (electric cars, smart batteries, prosumers, blockchain…)

NEAR FUTURE• Complete missing Vagrant environments

• Packer generating Docker containers for Dev envs

• Import all existing resources into Terraform and rebuild the production network (PAIN)

• Increase AWS usage (Redshift, DynamoDB…)

• Full automated EVERYTHING

DEVOPS IS…• Devs and Ops working together to deliver value

• Empower teams, reduce hard dependencies

• Communicaton, Integration, Collaboration

• Boosting productivity, make life easier!

• Automation, CI/CD, Infrastructure as code…

ENABLE THE BUSINESS!That´s what they pay us for!

BOOKS AND REPOS• The Phoenix Project - Gene Kim, Kevin Behr, George Spafford

• The DevOps Handbook - Gene Kim, Patrick Debois

• Terraform: Up and Running - James Turnbull

• https://github.com/ricardclau/geekshubsbcn (Packer + Terraform)

• https://github.com/kaorimatz/packer-templates (Linux) & https://github.com/joefitzgerald/packer-windows (Windows)

• https://github.com/terraform-community-modules (MANY examples)

QUESTIONS? CONTACT?• Email: [email protected]

• Twitter : @ricardclau

• Github: https://github.com/ricardclau

• If you think these techniques help your company, let´s talk!