Habitat Workshop at Velocity London 2017

Mandi Walls

Technical Community Manager for EMEA

@lnxchk

[email protected]

https://www.habitat.sh/

http://slack.habitat.sh/

Ian Henry @Eeyun___ Habitat Community lead

EVERY business is a software business

We’re going to be a software

company with airplanes.– CIO, Alaska Airlines

We Run InfrastructureTo Run Applications

That the Business Requires

How Do We Run Applications?

• On a computer

• With an OS

• And some libraries

• And some configuration

• And some way to start it and stop it

That isn’t managing complexity.

It is Adding to it.Or at least just moving it

around.

So. Habitat.

• Reduce snowflakeness

• Support microservices

• Manage container creep

https://www.bonanza.com/listings/Premier-Food-Storage-Containers-20-Piece-Set-Grey/443972348

Modern Applications Are Trending Toward

• Immutability

• Platform agnosticism

• Complexity reduction

• Scalability

https://amazingmusthaves.com/products/steel-insulated-food-containers/

Ways of Working with Habitat

• Artifacts run themselves via the supervisor

• Export to Docker

• Link your github to habitat builder and let us do the

work

• Build plans from scratch or leverage scaffolding

Habitat Goals

• Defer some decisions to runtime

• Do clean room builds

• Repeatable builds

• Distro agnostic packaging system

• Service runtime and discovery

• Configuration exposed via API

• Packages are signed by the system

Single Artifact Demo - Habitatize

• Ruby app with HTTP interface, ImageMagick

• Built via scaffolding

• Run locally

• Login to your machine

User: velocity

Pass: London2017

Run the Package

•sudo hab start learn-chef/habitatize -

-channel unstable

• Open http://{your machine IP}:8000 in a web

browser

Upload a Pic!

What’s in This Thing?

• https://github.com/burtlo/habitatize

pkg_name=habitatize

pkg_origin=franklinwebber

pkg_scaffolding="core/scaffolding-ruby"

pkg_version="1.0.0"

pkg_deps=(franklinwebber/imagemagick)

Basic Parts of a Simple Habitat App

• Your application code

• A plan file – can be as short as a few lines

Include dependencies or use a scaffolding package

• Habitat installation for running it

(and build, we’ll do that later)

• That’s it!

• Stop with

sudo hab stop learn-chef/habitatize

Dependencies?

Community

• Teams and individuals contribute packages to

builder

• The “core” packages are maintained by the Habitat

team

• You can use any public package on builder for your

dependencies

• Packages are libraries, runtimes, applications

Habitat and BuilderApplication Automation Build Machine

Lab: Habitat and Builder

• The fastest way to get packages out of the Habitat

workflow is with Builder

• SaaS for automated build and publish

• Code -> Github -> Builder -> Docker Hub

• You’ll need Github and Docker Hub accounts

No worries, you can get them as we go along

Our Sample Application

• https://github.com/habitat-sh/sample-node-app

• Fork this to your github account

Sign in to Builder

• https://bldr.habitat.sh/#/sign-in

Origins

• Are like organizations for grouping habitat

applications

• You can publish to multiple origins as long as you

have the keys – we’ll talk about that later

• For now, create a new unique origin to use

“username-velocitylondon”

• Origins can be public or private

For this lab it doesn’t matter

Make the App Our Own

• In github, edit inline in the browser

• https://github.com/YOU/sample-node-

app/blob/master/habitat/plan.sh

• Change

pkg_origin

To your new origin

• Commit changes, add a comment if you want

Add Habitat Builder

• As an app in github

• https://github.com/apps/habitat-builder

• Install

• Choose repositories

All, or limit to just the sample-node-app repo

Add Docker Hub Integration

• We’ll link Docker Hub to our origin on habitat builder

Docker Hub

• https://hub.docker.com/

• Sign up for an account if you don’t have one

• Each time you create a new origin, you’ll need to

hook it to Docker Hub

Automatic Builds of Our Sample App

• Habitat Builder will watch our repo for changes

• New builds of our app will then initialize

• We connect Builder to github from “Packages” on

the Builder page

Turn on “Publish to Docker Hub”

Build It!

Our App Shows Up In Docker Hub

• Builder retrieved our code from github

• Built the latest version

• Exported to a Docker container

• Published to Docker Hub

Run Our App with Docker

sudo docker pull your-docker-org/your-

docker-repo

sudo run -it -p 8000:8000 your-docker-

org/your-docker-repo

Open {your IP}:8000/ in a browser

(Make sure you stopped habitatize)

sudo hab stop learn-chef/habitatize

Make a New Version

• On github, update your plan.sh file to show a new

version of your app – 1.0.2

• Commit your change

• Builder will pick up your change and start a new

build

Habitat StudioBuild Locally

Application Life

Source

Code

RepoArtifact

Bare Metal

Container

Cloud Instance

VM

Artifact

Repo

Habitat Studio

• Provides a busy box clean

room for your app

• Plus a set of tools for

manipulating and running

harts

Why the Studio?

• Declare explicit dependencies

• Ship exactly what you need

• Sign your packages and store artifacts

Habitat Plans

• Plan files are where you put together your builds

• They are bash

• Plans are versioned and can be shared on the

habitat depot

• Plans have multiple sections for your needs

What’s In A Plan?

pkg_origin=firstnamelastname

pkg_name=national-parks

pkg_description="A sample JavaEE Web app deployed in Tomcat8"

pkg_version=0.1.3

pkg_maintainer="First Last <[email protected]>"

pkg_license=('Apache-2.0')

pkg_source=https://github.com/billmeyer/national-parks/archive/v${pkg_version}.tar.gz

pkg_shasum=56a70d7fc432ad275ba256150bd010ce2581346c848e40ddc0fd0f62c5b17a54

pkg_upstream_url=https://github.com/billmeyer/national-parks

pkg_deps=(core/tomcat8 billmeyer/mongodb)

pkg_build_deps=(core/git core/maven)

pkg_expose=(8080)

pkg_svc_user="root"

pkg_svc_group="root"

Plans Have Callbacks

Shell functions executed when building

artifacts:

• do_begin()

• do_download()

• do_verify()

• do_check()

• do_clean()

• do_unpack()

• do_prepare()

• do_build()

• do_install()

• do_strip()

• do_end()

Variables

• HAB_CACHE_SRC_PATHThe default path where source archives are downloaded, extracted, & compiled.

• ${pkg_dirname}Set to ${pkg_name}-${pkg_version} by default. If a .tar file extracts to a directory that's different from the filename, then you would need to override this value to match the directory name created during extraction.

• ${pkg_filename} Set manually in the plan.sh

• ${pkg_version} Set manually in the plan.sh

Habitat Tools for Dependencies

•hab pkg path team/app

• You can have build deps and runtime deps

• Build deps get into your hab studio when you are

building

• Runtime deps come along into your hart

Keep artifacts small and under

control!

Scaffolding, like for habitatize

• Default core-built dependencies for common

runtimes

• Ruby and Node so far

Go, Python, Java on the way!pkg_name=MY_APP

pkg_origin=MY_ORIGIN

pkg_version=MY_VERSION

pkg_scaffolding=core/scaffolding-ruby

What Gets Built?

• Everything. Sort of.

• Build your own apps from source

• Decide if you want upstream binaries or source for

things like runtime

You don’t have to build Tomcat, but you can

• For COTS, use the binaries and skip steps

Habitat Hooks

• Travels with the app

• Tell the app how to start, and if there is something

that needs to be done before start

• Let’s look at a run hook file

Run Hook for Tomcat#!/bin/bash

exec 2>&1

echo "Starting Apache Tomcat"

export JAVA_HOME=$(hab pkg path core/jdk8)

export TOMCAT_HOME="$(hab pkg path core/tomcat8)/tc"

source {{pkg.svc_config_path}}/catalina-opts.conf

echo "\$CATALINA_OPTS=$CATALINA_OPTS"

cp "$(hab pkg path {{pkg.origin}}/{{pkg.name}})/{{pkg.name}}.war" "$TOMCAT_HOME/webapps/"

exec ${TOMCAT_HOME}/bin/catalina.sh run

Configuration

• Can be manipulated at runtime

• Also travels with the app

• Provides variable substitution and templating

A catalina-opts File

{{~#if bind.database}}{{~#each bind.database.members}}

export CATALINA_OPTS="-DMONGODB_SERVICE_HOST={{address}} -DMONGODB_SERVICE_PORT={{port}}"

{{~/each}}{{~/if}}

Builder

• You can share plans with builder, and other hab

users share theirs

• Has team namespacing – our origin

• The core plans are those built by the Habitat team

• https://app.habitat.sh/

Caveat - Internet

• You can build your own stuff inside your own

network, sort of, when it’s all on one machine

• This isn’t the best way to use Habitat

• You’ll want internet connectivity on your build box

and on the hosts running habitat artifacts

• Builder provides private origins, linked to github

users

Build Output

• By default, it’s a hart – a compressed tarball with

some metadata and a signature

• The hart itself it runnable

• You can export to other formats, like Docker

containers like we saw eariler

Runtime

• The hab runtime includes management, service

discovery, other features

• The habs in your application create a mesh so they

can talk to each other

• You can even update your application via the mesh

without restarting every application manually

Running a Hart

sudo hab start learn-chef/habitatize --

channel unstable

sudo hab start nathenharvey/national-parks --

listen-gossip 0.0.0.0:9639 --listen-http

0.0.0.0:9632 --peer 127.0.0.1 –bind

database:mongodb.default

Setup

• On your remote host

•hab setup

• Set up a default origin? Yes

Enter what you used in builder – username-velocity

• Create a key – Yes

• Github access token!

Github Access Token

• Go to your account on github

• Select “settings” from your account on the top right

• Click “Developer Settings” on the left bar

• Click “Personal access tokens” on the left bar

• “Generate new token” button

• Description “habitat” or whatever

Select read:org and user:email

• Generate Token

Hab Setup continued

• Copy the github access token into the hab setup

HAB_AUTH_TOKEN

• Your github auth token

• Export as an environment variable

• Can be added to ~/.bashrc

•export HAB_AUTH_TOKEN=3db7f……yourtoken

Add Keys

• We generated new keys

• We need to add them to Builder

• From your host, run

hab origin key upload –s {YOUR ORIGIN}

A place to work

git clone

https://github.com/YOU/sample-node-

app.git

cd sample-node-app

• Your apps have a habitat/ subdirectory for hab-

related files

Hab Studio

•hab studio enter

•build

•hab svc start ORIGIN/sample-node-app

Habitat Supervisor

• Keeps an eye on your applications

• Also provides and API for viewing what’s going on

with the apps running in the supervisor

•http://{Your IP}:9631/services

• It’s running now for you as part of the studio, but

doesn’t need the studio to run

When you exit the studio, it will shut down

• Also allows for applications to connect to each other

Two-Tier Application: National Parks

• Java on Tomcat with MongoDB

• https://blog.chef.io/2016/09/07/deploy-java-web-

app-on-tomcat-8-with-habitat/

• You’ll want a couple terminals open to your host

App

Decomposing into Habitat

• Microservice architectures lend themselves well to

Habitat

Every service becomes a hab package

• For generic services like databases, the initial

package should be very basic

• Your application layer or a separate package

initializes the data

MongoDB

• Already built – we’re just going to use it

sudo hab start nathenharvey/mongodb --

channel unstable

https://bldr.habitat.sh/#/pkgs/nathenha

rvey/mongodb/latest

Java App

• https://bldr.habitat.sh/#/pkgs/nathenharvey/national-

parks/latest

• Let’s look at the plan.sh a bit

Where Is the Code Coming From

• This app isn’t hooked into the Builder pipeline

• The source code is downloaded from github into the

studio when the package is built

• A checksum can be included in the plan.sh

pkg_deps and pkg_build_deps

• Build deps are present in the studio or build

environment but don’t make it into the package

• In this case, we’re not taking Maven with us!

• Also notice the dependency on mongodb!

Finding Stuff that Lives in Other Packages

• Check out the setting for JAVA_HOME

•export JAVA_HOME=$(hab pkg path

core/jdk8)

• Java will be included in our package via the

dependencies

• It arrives via another habitat package

• The filesystem locations are different, so we have to

query for where java will live

Let’s Start the App

•sudo hab start nathenharvey/national-

parks --channel unstable

• The log output in this case will go to the MongoDB

screen (STDOUT for the supervisor)hab-sup(MR): Unable to start nathenharvey/national-

parks, hab-

sup(SS)[src/manager/service/spec.rs:208:23]: Missing

required bind(s), database

Supervisor Can Help Us

Service

Discovery

Service

Groups

Service Groups

• Applications running in the supervisor belong to

service groupsmongodb.default(SR): Configuration recompiled

mongodb.default(SR): Initializing

mongodb.default(SV): Starting service as user=hab,

group=hab

• When connecting services, use the service group

• Groups can be 1 or N instances of an application

Uh Oh

national-parks.default(SR): Initializing

national-parks.default hook[init]:(HK): Seeding Mongo Collection

national-parks.default hook[init]:(HK): $MONGOIMPORT_OPTS=--host= --

port=27017

national-parks.default hook[init]:(HK): 2017-10-17T17:52:35.106+0000

connected to: localhost:27017

national-parks.default hook[init]:(HK): 2017-10-17T17:52:35.106+0000

dropping: demo.nationalparks

national-parks.default hook[init]:(HK): 2017-10-17T17:52:35.109+0000 Failed:

not authorized on demo to execute command { drop: "nationalparks" }

national-parks.default hook[init]:(HK): 2017-10-17T17:52:35.109+0000

imported 0 documents

national-parks.default(HK): Initialization failed! 'init' exited with status code 1

What is This?

• Our app brings along its seed data

• It needs permissions to load that data into

MongoDB

• The app isn’t authorized to do that!

YOLO!

• We’re going to open up mongo to the wild. It’s fine.

• Let’s see how MongoDB is currently configured

hab sup config nathenharvey/mongodb

• In the middle are some security settings[mongod.security]

cluster_auth_mode = "keyFile"

javascript_enabled = true

key_file = ""

Tunables

• The supervisor allows us to update application

config at runtime

• Those mongod.security settings can be changed on

the fly

Make a new tunable toml

hab sup config nathenharvey/mongodb >

user.toml

Edit user.toml

• Around line 46 are the mongod.security settings

• Update cluster_auth_mode to “”

46 [mongod.security]

47 authorization = "disabled”

48 cluster_auth_mode = "”

49 javascript_enabled = true

50 key_file = ""

hab config apply

• Load this new config into the running mongodb:

sudo hab config apply mongodb.default 2 user.toml

• The serial number just has to increase; it’s not

otherwise meaningful

• Must be valid configuration

• The app will reload

Now try the Java again

sudo hab start nathenharvey/national-

parks --peer 127.0.0.1 --bind

database:mongodb.default

Check the Output

national-parks.default(O): 17-Oct-2017 18:07:58.322 INFO

[main] org.apache.catalina.startup.Catalina.load

Initialization processed in 525 ms

national-parks.default(O): 17-Oct-2017 18:07:58.348 INFO

[main] org.apache.catalina.core.StandardService.startInternal

Starting service Catalina

national-parks.default(O): 17-Oct-2017 18:07:58.348 INFO

[main] org.apache.catalina.core.StandardEngine.startInternal

Starting Servlet Engine: Apache Tomcat/8.5.9

national-parks.default(O): 17-Oct-2017 18:07:58.367 INFO

[localhost-startStop-1]

org.apache.catalina.startup.HostConfig.deployWAR Deploying

web application archive

/hab/pkgs/core/tomcat8/8.5.9/20170215223900/tc/webapps/nation

al-parks.war

Check the Browser!

• http://{Your IP}:8080/national-parks/

Habitat Info

• http://{Your IP}:9631/services

Other Places to Find Hints

• https://github.com/habitat-sh/core-plans/

• The plan files for the core packages

• Good for “how do I do x?”

Join Us!

• On Slack!

http://slack.habitat.sh

• Online! With Tutorials!

https://www.habitat.sh/

• On Github!

https://github.com/habitat-sh

• This talk

Try it out!

Share your story!

Get some swag!!

goo.gl/WrHQTU

Other References

• Summary on The New Stack

https://thenewstack.io/chef-habitat-addresses-issues-

moving-containers-production

• Our YouTube Channel

https://www.youtube.com/user/getchef