ChefConf2015 Chef and Security (keyNote)...Exercise: Download your client .pem •You may reuse your Chef server setup from the morning Analytics workshop •Only do this if you don’t

v0.2

ChefConf 2015Chef & Security Workshop

v0.2

Introductions

LARRY EICHENBAUM

[email protected] larryebaumlarryebaum

@larryebaum

Solutions Engineer

mailto:[email protected]

v 3.0.0

S C O T T F O R D

smford22

smford22

@sford422

Solutions Engineer

v0.2

Goals

• Participants will learn the key access points on the Chef server and know how to secure them

• Participants will learn how to utilize industry standards for security and how the Chef analytics platform can be leveraged to streamline auditing and compliance

6

Goals

• Participants will understand the value of Chef Vault as compared to encrypted data bags

• Participants will successfully use Chef Vault to generate, distribute and encrypt/decrypt secrets both via CLI and the Chef client run

• Participants will learn strategies for managing and rotating keys

7

Goals

v0.2

Setups

Login to HipChat• For the purpose of this class, and in an effort to work

together as a team, we have setup a public HipChat channel for all of us to use. You will need to create an account if you do not have one, but you can find that link here…

• http://larryebaum.hipchat.com/chat

http://larryebaum.hipchat.com/chat

HipChat Quick Tips• @all notifies everyone regardless if they are present or not

• @here notifies everyone currently in the room

• @username to address a specific person in the room

• /code <paste code> allows you to paste code snippets in the room

Get logged in and say hello!

Chef Servershttp://goo.gl/L6CPdz

Find your card…

Chef Server Chef Analytics Server

Record the hostnames…

CloudShare NodeRed Cards: https://use.cloudshare.com/Class/xfjh0 Blue Cards: https://use.cloudshare.com/Class/cegfd Login: <COLOR><CARD><SUIT>@mail.com Password: chef

CloudShare Node

The authenticity of host 'uvo1qrwls0jdgs3blvt.vm.cld.sr (69.195.232.110)' can't be

established.

RSA key fingerprint is d9:95:a3:b9:02:27:e9:cd:74:e4:a2:34:23:f5:a6:8b.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'uvo1qrwls0jdgs3blvt.vm.cld.sr,69.195.232.110' (RSA) to

the list of known hosts.

[email protected]'s password:

Last login: Mon Jan 6 16:26:24 2014 from

host86-145-117-53.range86-145.btcentralplus.com

[chef@CentOS63 ~]$

Lab - Login$ ssh chef@<EXTERNAL_ADDRESS>

CheckpointAt this point you should have •One virtual machine (VM) or server that you’ll use for the lab exercises

•The IP address or public hostname •An application for establishing an ssh connection

• ‘sudo’ or ‘root’ permissions on the VM

Exercise: Set up a working directoryMake a working directory on your laptop under your home directory called ‘~/workshop’, i.e. • Windows:- C:\Users\you\workshop

• Mac/*nix:- /Users/you/workshop

Navigate to this working directory

Exercise: Download the Chef Fundamentals Repo

Download the Chef Fundamentals working repohttps://github.com/learnchef/chef-fundamentals-repo

https://github.com/learnchef/chef-fundamentals-repo

Exercise: Download the Chef Fundamentals Repo

Locally shared atafp://10.35.1.104 or smb://10.35.1.104

Share among yourselves using AirDrop or USB stick (passing around)

afp://10.35.1.104

smb://10.35.1.104

Archive: chef-fundamentals-repo-master.zip

bb06ea2c0cabaa855e4cb1d1c43bbe4d75caf70d

creating: chef-fundamentals-repo-master/

inflating: chef-fundamentals-repo-master/Berksfile

inflating: chef-fundamentals-repo-master/README.md

inflating: chef-fundamentals-repo-master/Vagrantfile

creating: chef-fundamentals-repo-master/cookbooks/

creating: chef-fundamentals-repo-master/cookbooks/apache/

inflating: chef-fundamentals-repo-master/cookbooks/apache/CHANGELOG.md

inflating: chef-fundamentals-repo-master/cookbooks/apache/README.md

creating: chef-fundamentals-repo-master/cookbooks/apache/attributes/

creating: chef-fundamentals-repo-master

...

Exercise: Extract the repo$ cp ~/Downloads/chef-fundamentals-repo-master.zip . $ unzip chef-fundamentals-repo-master.zip

Exercise: Create a New Org•Connect to your assigned Chef Server •Create an account •Create a new organization

Exercise: Download knife.rb; create validator .pem

Download the knife.rb & copy validator .pem text Paste text into file: <ORGNAME-validator.pem>

Exercise: Download your client .pem

•You may reuse your Chef server setup from the morning Analytics workshop

•Only do this if you don’t already have the .pem file available on your laptop • Reset and copy/paste your key to <USER.pem>

$ cd ~/workshop/chef-fundamentals-repo-master $ mkdir .chef $ mv ../<yourname>.pem .chef/ $ mv ../<your-org>-validator.pem .chef/ $ cp ~/Downloads/knife.rb .chef/

Exercise: Create and populate a .chef dir

• knife.rb & .pem files reside in the .chef directory which can be in o <current-directory>/.chef o /etc/.chef o ~/.chef

WARNING: Certificates from ec2-52-11-224-17.us-west-2.compute.amazonaws.com will be fetched and placed in your trusted_cert

directory (/Users/larryeichenbaum/workshop/chef-fundamentals-repo-master/.chef/trusted_certs).

Knife has no means to verify these are the correct certificates. You should

verify the authenticity of these certificates after downloading.

Adding certificate for ec2-52-11-224-17.us-west-2.compute.amazonaws.com in /Users/larryeichenbaum/workshop/chef-fundamentals-repo-master/.chef/trusted_certs/ec2-52-11-224-17_us-west-2_compute_amazonaws_com.crt

Exercise: Setup Trust with Chef Server$ knife ssl fetch

<your-org>-validator.pem

Exercise: Test your workstation$ knife client list

Uploading logrotate [1.4.0] Uploading cron [1.2.8] Uploading chef-client [3.4.0] Uploaded 3 cookbooks.

Exercise: Upload Cookbook$ knife cookbook upload logrotate cron chef-client

OPEN IN EDITOR:

SAVE FILE!

Exercise: Edit base role

name "base"

description "Base Server Role"

run_list "recipe[chef-client::delete_validation]", "recipe[chef-client]"

roles/base.rb

Updated Role base!

Exercise: Upload role$ knife role from file base.rb

uvo164727i3mvh1jup2.vm.cld.sr --2014-05-13 04:31:10-- https://www.opscode.com/chef/install.sh

uvo164727i3mvh1jup2.vm.cld.sr Resolving www.opscode.com... 184.106.28.90

uvo164727i3mvh1jup2.vm.cld.sr Connecting to www.opscode.com|184.106.28.90|:443... connected.

uvo164727i3mvh1jup2.vm.cld.sr HTTP request sent, awaiting response... 200 OK

uvo164727i3mvh1jup2.vm.cld.sr Length: 15934 (16K) [application/x-sh]

uvo164727i3mvh1jup2.vm.cld.sr Saving to: `STDOUT'

uvo164727i3mvh1jup2.vm.cld.sr

100%[======================================>] 15,934 --.-K/s in 0s


uvo164727i3mvh1jup2.vm.cld.sr 2014-05-13 04:31:10 (538 MB/s) - written to stdout [15934/15934]


uvo164727i3mvh1jup2.vm.cld.sr Downloading Chef 11.8.2 for el...

uvo164727i3mvh1jup2.vm.cld.sr downloading https://www.opscode.com/chef/metadata?v=11.8.2&prerelease=false&nightlies=false&p=el&pv=6&m=x86_64

uvo164727i3mvh1jup2.vm.cld.sr to file /tmp/install.sh.41533/metadata.txt

uvo164727i3mvh1jup2.vm.cld.sr trying wget...

uvo164727i3mvh1jup2.vm.cld.sr url https://opscode-omnibus-packages.s3.amazonaws.com/el/6/x86_64/chef-11.8.2-1.el6.x86_64.rpm

...

Exercise: Bootstrap the target node$ knife bootstrap <external address> \ --sudo –x chef –P chef –N "node1" -–bootstrap-version 12.1.1

node1:

run_list:

role[base]

Exercise: Set run list for managed node$ knife node run_list set node1 'role[base]'

[email protected]'s password:

Last login: Sat Mar 28 11:13:08 2015 from ool-43570b90.dyn.optonline.net

_______________

/ ___/ __/_ __/

/ (_ / _/ / /

\___/___/ /_/

_______ ___________

/ __/ _ \/ __/ __/ _ \

_\ \/ ___/ _// _// // /

/___/_/ /___/___/____/

___ ___ _____

/ __| __|_ _|

| (_ | _| | |

\___|___| |_|

___ _____ ___ ___ __ __ ___

/_\ \ / / __/ __|/ _ \| \/ | __|

/ _ \ \/\/ /| _|\__ \ (_) | |\/| | _|

/_/ \_\_/\_/ |___|___/\___/|_| |_|

Exercise: SSH to Node$ ssh chef@<external name>

Starting Chef Client, version 11.12.8

resolving cookbooks for run list: ["chef-client::delete_validation", "chef-client"]

Synchronizing Cookbooks:

- chef-client

Compiling Cookbooks...

...

Exercise: Run Chef Clientchef@node1$ sudo chef-client

root 8933 0.3 2.2 130400 37816 ? Sl 03:19 0:01 /opt/chef/embedded/bin/ruby /usr/bin/chef-client -d -c /etc/chef/client.rb -L /var/log/chef/client.log -P /var/run/chef/client.pid -i 1800 -s 300

Exercise: Verify chef-client is runningchef@node1$ ps awux | grep chef-client

v0.2

Securing the Chef Server

Securing Chef• Secure Chef

http://commons.wikimedia.org/wiki/File:Trakošćan_2007.JPG

Securing Chef• Secure Chef • Security is two fold –

• Inner => Chef • Outer =>OS, environment, other tools, etc.

http://cadw.wales.gov.uk/docs/cadw/publications/WHS_additional_maps_plans_EN.pdf

Chef-Specific SecurityInstall… Omnibus version on a secure internal network • Avoid using insecure public networks unless you have valid reason to do so.

Chef-Specific SecurityLock down… Only open needed ports

• iptables • EC2 security groups

http://www.geograph.org.uk/photo/1613103

Chef-Specific SecurityLock down… Only open required ports • Standalone Chef server 80 & 443

Chef-Specific SecurityLock down… Only open required ports • Chef Server HA Front end: 80 & 443 Back end: 4321, 80, 443, 9683, 9090, 8000, 5432, 5672, 16379

Chef-Specific SecurityLock down… Only open required ports • With Chef analytics 80, 443, 5672

UPDATE PIC

Chef’s Secure… What’s Next?

http://images.mocpages.com/user_images/26934/12878311242_DISPLAY.jpg

Keeping Chef SecureImportant Chef Updates

https://lh6.googleusercontent.com/-9GWoh0a_t7E/AAAAAAAAAAI/AAAAAAAAADg/K7JOQKVPRtA/

photo.jpg

Keeping Chef SecureImportant Chef updates • Chef blogs

• Installation guides

Keeping Chef SecureImportant Chef updates • Chef blogs • Twitter feeds, email distro

Keeping Chef SecurePerform updates

http://www.maclife.com/files/imagecache/futureus_imagegallery_fullsize/gallery/02_1.jpg

Keeping Chef SecurePerform updates •docs.chef.io • Step-by-step instruction

Keeping Chef SecureSecurity incident alerts

http://commons.wikimedia.org/wiki/File:Red_Emergency_Light.jpg

Keeping Chef SecureSecurity incident alerts • Chef blogs

• Important security updates • Common security solutions

Keeping Chef SecureSecurity incident alerts •Chef blog •Status feeds • status.chef.io • twitter feed • tumblr feed

There's More

But Wait…

Non-Chef-Specific Security

http://www.securitygeneration.com/wp-content/uploads/2011/02/chain-padlock-security-fail.jpg

Non-Chef-Specific SecurityRestrict… who can access the Chef server: • Chef server stores all! • If you have access, you have access to database, cookbooks, etc.

http://midstatelockandsafe.com/wp-content/uploads/2010/10/Keys-To-The-Castle.jpg

Non-Chef-Specific SecurityRestrict further… Two-factor authentication • Chef controls/manages all your infrastructure.

• Be protective… • Or explain later. http://vignette3.wikia.nocookie.net/koc/images/7/75/

Defend_victory.jpg/revision/latest?cb=20101126001303

Non-Chef-Specific SecurityRestrict further… Two-factor authentication • Google Authenticator

• iOS, Android, Blackberry • Supports multiple accounts • Backup codes and devices

https://itunes.apple.com/us/app/google-authenticator/id388497605?mt=8

Non-Chef-Specific SecurityRestrict further… two-factor authentication • Duo Security

• iOS, Android, Blackberry • Palm, Windows Phone 7, etc. • Supports multiple accounts • Backup codes and devices

https://itunes.apple.com/us/app/duo-mobile/id422663827?mt=8

Non-Chef-Specific SecurityRestrict further… admins only via SSH keys • Passwords not THAT secure • Avoid duplicate passwords on multiple hosts

• More convenient • Reduces likelihood of brute-force attacks

http://www.correderajorge.es/wp-content/uploads/2013/09/public_key_cryptography_sm.png

http://commons.wikimedia.org/wiki/File:Crypto-Nerd.png

Non-Chef-Specific SecuritySecure Chef cookbooks • Can be a point of entry 1000011001

0BACKDOOR10011100010010000101000100101000001000110

Non-Chef-Specific SecuritySecure Chef cookbooks • Can be a point of entry • Contains the code that defines your servers

https://www.flickr.com/photos/bradhigham/14760496352/in/photolist-ispVZE-mNbLn-6AXqjt-89yaQo-oukpjq-prBQef-jqtAUm-8fQsXR-cBahob-anqncf-8qtez7-639A8i-4x7zyB-oVXXQ4-8hUGrR-gYkwn3-nGjyXe-61PBPr-86n2rp-89uVET-Ci1ba-2616FK-7ZiXZa-ct1ddE-

cZtAH9-5n3EuN-6KDsVR-8Szby-q5U5FY-2cHYb4-fjHU2S-cCxp4E-nac2Mw-5oh1AN-uWzN6-3aRRbX-85ZcvN-6jPyC1-oudwgE-8xYTVr-6B13jx-a6RZTN-5sn8t4-7smmD-992dQL-7ZZHTn-oupy5F-dP8sHj-bE8t-7YRahJ

Non-Chef-Specific SecuritySecure Chef cookbooks • Can be a point of entry • Contains the code that defines your servers

• Contains the code that runs on your servers

http://cdn.arstechnica.net/wp-content/uploads/2014/01/falken.jpg

Yea,

I'm

old

whi

te g

uy…

but

hav

e yo

u re

ally

not

see

n W

ar G

ames

?

Non-Chef-Specific SecuritySecure code repo • Be cautious with public GitHub or equivalent

https://www.flickr.com/photos/gsimmonsonca/3285952133/in/photolist-61nn68-6W6HKX-9tkjXz-787xod-9Gxpiz-9iQjBN-qNyuAp-au8kjx-8ApgTa-49iUwy-8e9adZ-

gGdDuw-9uu261-7wDeBV-5TaCkm-B5xwM-59vc3E-72sPTp-ad5jTc-oXiowS-7Bu8hZ-JgGsp-3cQcoc-aggJUx-b3inBe-asbtAs-5bBJMD-6rch3G-6aDzQR-dX8M7i-24L3Yq-6XCAQH-3H5HE-Dcfrn-dAPegg-8vJGry-8dfP18-dcLWaq-7MyZPH-nPFmX9-h8YsAC-Brr18-cCczjL-898gbm-cXUrLq-nvyPib-3XiC5w-8v6sC-8cc475-6oKosU

Non-Chef-Specific SecuritySecure code repo • Be cautious with public GitHub or equivalent • grants insight to your infra code

• GitHub Enterprise or equivalent • internal network • secure • control access

https://connect.decknetwork.net/i/github_enterprise2.png

Non-Chef-Specific SecuritySupermarket content • Publicly shared cookbooks

Premier Malt Products Co.; Printing Not Stated edition (1928)

Public Domain Cooking

How to Share Your Very Own Cook Book

Non-Chef-Specific SecuritySupermarket content • Publicly shared cookbooks • Can be a point of entry

https://www.flickr.com/photos/innoxiuss/3306092527/in/photolist-639A8i-4x7zyB-oVXXQ4-8hUGrR-gYkwn3-nGjyXe-61PBPr-Ci1ba-2616FK-7ZiXZa-ct1ddE-7YRahJ-cZtAH9-5n3EuN-6KDsVR-8Szby-q5U5FY-2cHYb4-fjHU2S-cCxp4E-nac2Mw-5oh1AN-uWzN6-3aRRbX-85ZcvN-6jPyC1-oudwgE-8xYTVr-6B13jx-a6RZTN-5sn8t4-7smmD-992dQL-7ZZHTn-oupy5F-dP8sHj-bE8t-59G4t1-9fTpzb-9gKoEX-cvzKn5-qLC9TW-8s59Mp-2cHYtX-kKVeFB-rmWHuR-9xKv5y-8iAAWZ-guvBw9-9CuSNo

Non-Chef-Specific SecuritySupermarket content • Publicly shared cookbooks • Can be a point of entry • Read and review everything

https://www.flickr.com/photos/bcgovphotos/8113334531/in/photolist-637KYD-6rCqSw-ayHV31-h4FzN1-QtXAi-bz31qy-bz31ud-bz2ZPE-bMWH56-bMWH2e-bMWFmc-bz2ZDb-bz2ZvG-bz2ZyC-bMWFyH-bMWGTi-bz312W-bMWF5B-bMWGGM-bz2ZSY-bMWFH4-

aMGyv-dmWXG8-2TzBBJ-58mAsb-6bTs8x-hCCAGd-dmYcfU-akUamc-qw7r3R-b4uCGP-9fhgkr-knVUPZ-mrMJKx-mcYPuV-bMWH6P-bMWF9B-bMWGRg-bMWGMx-

bMWGWa-bz31gE-bz2YrY-bz2YA9-bMWE2z-bMWFak-bz2YuE-bMWDUx-h4FodQ-h4FoUD-5B38Au

Non-Chef-Specific SecuritySupermarket content • Publicly shared cookbooks • Can be a point of entry • Read and review everything • Test run in isolation-- if your security requires it

https://www.flickr.com/photos/riacale/4230091725/in/photolist-jxpBYJ-6nssFe-fMcDHU-piQkus-dwnAtk-84xhpn-apLYmH-4H6hbz-4T59iu-q5c9ra-5BK2bU-qHjXax-5Rb5eJ-piHCcJ-6HeMaE-6HaJxV-4zQbEB-buLxSf-iWs9-GiDTU-7rNjTX-4Q2u8A-9gFVjE-bpeN7A-dJGb51-

dGokV5-98ukrU-6334S-9gF9xM-7EpEWX-7vAbgF-dy4bk7-5NjDk8-4CvBbo-8TftWn-68aLxw-6v7j4D-6cuhSL-3bm4U-3bm4S-3bm4T-fz4T94-95m3Ti-6ugcDf-6NqRS7-dMPypX-p9tcgy-3bm4V-5RAXGm-5NoTru

Non-Chef-Specific SecurityGeneral security practice • Follow accepted guidelines

https://www.cisecurity.org

Non-Chef-Specific SecurityGeneral security practice • Follow accepted guidelines • CIS security benchmarks


Non-Chef-Specific SecurityGeneral security practice • Follow accepted guidelines • CIS security benchmarks • Benchmark documents • Metrics definitions


v0.2

Configuring Chef ClientsSSL Verification and chef-client Management

Lesson Objectives•After completing the lesson, you will be able to:

• Configure nodes to use SSL certificate validation

The Problem and Success Criteria•The problem: Make sure the Chef client service is is validating all SSL connections

•Success criteria: Use the chef-client cookbook to accomplish these tasks

SSL Certificate ValidationChef 11 defaults to not verifying certificates when it makes HTTPS connections - hence the SSL error when running chef-client • Enable SSL Verification by setting ssl_verify_mode :verify_peer in your config file

• If enabled, affects all SSL endpoints (eg., remote_files, etc.)

• Two useful knife commands: • knife ssl check - makes an SSL connection to your Chef server or any

other HTTPS server and tells you if the server presents a valid certificate • knife ssl fetch - allows you to automatically fetch a server’s certificates

to your trusted certs directory

Using the chef-client CookbookThe chef-client cookbook manages the Chef client: • client.rb configuration • service (runit, systemd, init, SMF, etc.) • Validation key deletion

Examining the chef-client CookbookWe’re already using two recipes on the node from the chef-client: •chef-client::service (via chef-client::default)

•chef-client::delete_validation

Managing client.rb settings

default_attributes( "chef_client" => { "config" => { "ssl_verify_mode" => ":verify_peer", "log_level" => ":info" } } )

chef_server_url "https://api.opscode.com/organizations/MYORG"

validation_client_name "MYORG-validator"

ssl_verify_mode :verify_peer

node_name "config-ubuntu-1204"

log_level :info

• For example, the following attributes (in a role)

• Will render the following configuration (/etc/chef/client.rb)

https://api.opscode.com/organizations/MYORG

OPEN IN EDITOR:

SAVE FILE!

Exercise: Edit base role; add chef-client::config

name "base"

description "Base Server Role"

run_list "recipe[chef-client::delete_validation]", "recipe[chef-client::config]", "recipe[chef-client]"

default_attributes(

"chef_client" => {

"config" => {

"ssl_verify_mode" => ":verify_peer",

"log_level" => ":info"

}

}

)

roles/base.rb

Updated Role base!

Exercise: Upload role$ knife role from file base.rb




- chef-client


...

Exercise: Run chef-clientchef@node1$ sudo chef-client

…

ssl_verify_mode :verify_peer

log_level :info

…

Exercise: Verify that Config is Updatedchef@node1$ sudo cat /etc/chef/client.rb

Special Handling for non-CA Signed Certs• Hosted Chef’s certificate is signed by a known CA • If you run your own CA, or your Chef server’s certificate is self-signed, you can use knife to import it

• If you do, it is also necessary to import to your Chef server, the certificate from the server used to generate your Chef certificate

Review Questions• How do you remove the validation .pem on the node? • What recipe configures chef-client? • How do you enable SSL verification on chef-client?

v0.2

Chef 12 SSL

Chef 12 SSLSSL certificate validation improvements: • New default settings for SSL certificate validation • Default in favor of validation • New knife ssl command

Chef 12 SSLInitial knife connections • You’ve just spun up a new Chef 12 server • You’ve just installed Chef 12 client on your local system

• You’ve created a user account and an organizationnode_name 'someNode' client_key 'clientKey.pem' validation_client_name 'org-validator' validation_key 'org-validator.pem' chef_server_url 'https://chef-server.some.domain/organization/org

ERROR: SSL Validation failure connecting to host: chef-server.example.com - SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed

ERROR: OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed

Chef 12 SSL$ knife client list

Chef 12 SSLWhy the error? • SSL verification enabled by default for all requests • Certificate is self-signed • No signing CA that can be verified • Hence, knife command fails

Chef 12 SSLknife ssl check: • Verify the connection and certificate

• Follow steps indicated to retrieve the server SSL certificate and store it locally as a ‘trusted’ certificate

• Or…

Connecting to host chef-server.example.com:443 ERROR: The SSL certificate of chef-server.example.com could not be verified Certificate issuer data: /C=US/ST=WA/L=Seattle/O=YouCorp/OU=Operations/CN=chef-server.example.com/[email protected]

Configuration Info: OpenSSL Configuration: * Version: OpenSSL 1.0.1j 15 Oct 2014 * Certificate file: /opt/chefdk/embedded/ssl/cert.pem

* Certificate directory: /opt/chefdk/embedded/ssl/certs

Chef SSL Configuration: * ssl_ca_path: nil

* ssl_ca_file: nil

* trusted_certs_dir: "/Users/jtimberman/Downloads/chef-repo/.chef/trusted_certs" TO FIX THIS ERROR: If the server you are connecting to uses a self-signed certificate, you must configure chef to trust that server's certificate. By default, the certificate is stored in the following location on the host where your chef-server runs: /var/opt/opscode/nginx/ca/SERVER_HOSTNAME.crt

Copy that file to your trusted_certs_dir (currently: /Users/<userName>/Downloads/chef-repo/.chef/trusted_certs) using SSH/SCP or some other secure method, then re-run this command to confirm that the server's certificate is now trusted.

WARNING: Certificates from chef-server.example.com will be fetched and placed in your trusted_cert directory (/Users/newUser/Downloads/chef-repo/.chef/trusted_certs).

Knife has no means to verify these are the correct certificates. You should verify the authenticity of these certificates after downloading.

Adding certificate for chef-server.example.com in /Users/newUser/Downloads/chef-repo/.chef/trusted_certs/chef-server.example.com.crt

Chef 12 SSL$ knife ssl fetch

Chef 12 SSL•The certificate should be verified to ensure that it is in fact the certificate on the Chef server.

•Compare checksums •Knife commands will now work!!

Chef 12 SSLDo I need to do this on every node?

http://pixabay.com/static/uploads/photo/2014/04/03/10/52/circle-311551_640.png

Chef 12 SSLknife ssh command •Invokes SSH commands (in parallel) on a subset of nodes within an organization

•Based on the results of a search query made to the Chef server

•Syntax: knife ssh SEARCH_QUERY SSH_COMMAND (options)

node-output.example.com WARNING: Certificates from chef-server-example.com will be fetched and placed in your trusted_cert node-output.example.com directory (/etc/chef/trusted_certs). node-output.example.com node-output.example.com Knife has no means to verify these are the correct certificates. You should node-output.example.com verify the authenticity of these certificates after downloading. node-output.example.com node-output.example.com Adding certificate for chef-server.example.com in /etc/chef/trusted_certs/chef-server.example.com.crt

Chef 12 SSL$ knife ssh 'name:*' 'sudo knife ssl fetch –c /etc/chef/client.rb

Chef 12 SSLSSL verification also works in Chef 11

SAVE FILE!

OPEN IN EDITOR:

Chef 12 SSL

Execute$ sudo chef-server-ctl reconfigure

nginx['ssl_certificate'] = "/etc/pki/tls/certs/your-host.crt" nginx['ssl_certificate_key'] = "/etc/pki/tls/private/your-host.key"

/etc/opscode/chef-server.rb

Chef 12 SSLCertificates expire and grow moldy

http://postimg.org/image/4cvafd4v1/full/ http://www.saveyourstuff.com/blog/wp-content/uploads/2010/04/dsc_0370-300x200.jpg

Chef 12 SSLRegenerate SSL certificates before expiration • Important aspect of protecting Chef server from vulnerabilities

•Helps prevent information stored in Chef server from becoming compromised

Chef 12 SSL•Stop the Chef server $ chef-server-ctl stop •Verify the FQDN for the Chef server $ hostname -f

•Delete certificate files $ rm /var/opt/opscode/nginx/ca/$<FQDN>.crt$ rm /var/opt/opscode/nginx/ca/$<FQDN>.key

Chef 12 SSL•If you are using custom SSL certificates, refer to the file names/location as referenced by the values:

nginx['ssl_certificate'] nginx['ssl_certificate_key'] within file:

/etc/opscode/chef-server.rb

•Delete those files; regenerate using same CA

Chef 12 SSL•Reconfigure server to create new SSL certs

$ chef-server-ctl reconfigure •Start the Chef server

$ chef-server-ctl start

Review Questions•What is the knife command used to acquire the SSL keys from the Chef server?

•What is the key difference between the Chef 11 server and Chef 12 server with regard to SSL certificates?

•How frequently do Chef server SSL certifications auto regenerate?

Link for Hipchat…http://goo.gl/MgTERT

http://goo.gl/MgTERT

v0.2

Chef-Vault

• Open source project published by Nordstrom

• Distributed as a RubyGem

• Bundled within chef-dk

• Gem needs to be installed on all workstations/servers that will either encrypt or decrypt data

105http://meta.wikimedia.org/wiki/

File:SLNSW_17216_Sliding_grille_at_the_entrance_to_the_Rural_Banks_vault_Martin_Place.jpg

http://flickr.com/photos/77947478@N00/2631203510

Not this

kind…

Chef Vault

*** LOCAL GEMS *** Chef-vault (2.4.0)

Check Chef Vault Version$ chef gem list chef-vault

$ Fetching: chef-vault-2.5.0.gem (100%) Successfully installed chef-vault-2.5.0 1 gem installed

Install Chef Vault (if necessary)…$ chef gem install chef-vault

• Command line structure

• Includes knife plugins

• Works just like normal data bags

• Secrets live in data bags on the Chef server knife vault create [VAULT] [ITEM] [VALUES] --mode MODE --search SEARCH --admins ADMINS --json FILE knife vault decrypt [VAULT] [ITEM] [VALUES] --mode MODE knife vault delete [VAULT] [ITEM] --mode MODE knife vault remove [VAULT] [ITEM] [VALUES] --mode MODE --search SEARCH --admins ADMINS knife vault update [VAULT] [ITEM] [VALUES] --mode MODE --search SEARCH --admins ADMINS --json FILE ...

108

Command Line Options…

We’ll use a new command here, knife vault create, to create the item that stores the secret info…

• We can directly pass the unencrypted secret without storing it

• Alternatively, we can store the unencrypted secret on disk in a JSON formatted file

109

Create the Item…

$ knife vault create secrets vaultuser \ '{"vaultuser":"<INSERT PASSWD>"}' \ --search 'role:base' \ --admins <YOUR USER> --mode client

110

Create Data Bag…

111

Alternate: Create JSON file…OPEN IN EDITOR: ./secrets_vaultuser.json

SAVE FILE!

{"vaultuser":"<SOME PASSWD>"}

112

Alternate: Create Data Bag…$ knife vault create secrets vaultuser \ --json secrets_vaultuser.json \ --search 'role:base' \ --admins <YOUR USER> --mode client

$ knife data bag show secrets

113

vaultuservaultuser_keys

Verify Content on Chef Server…

•The first item in the data bag, vaultuser, contains the actual secret.

•The second item indicates what nodes and user accounts have access to decrypt the password.

114

Examine the Data Bag Contents…

$ knife data bag show secrets vaultuser

115

id: vaultuservaultuser: cipher: aes-256-cbc encrypted_data: j+/fFM7ist6I7K360GNfzSgu6ix63HGyXN2ZAd99R6H4TAJ4pQKuFNpJXYnC SXA5n68xn9frxHAJNcLuDXCkEv+F/MnW9vMlTaiuwW/jO++vS5mIxWU170mR EgeB7gvPH7lfUdJFURNGQzdiTSSFua9E06kAu9dcrT83PpoQQzk= iv: cu2Ugw+RpTDVRu1QaaAfug== version: 1

Examine First Item in Data Bag…

$ knife vault show secrets vaultuser \ ’vaultuser’ --mode client

116

id: vaultuser vaultuser: <PASSWORD>

Decrypt First Item Using Knife…

Recall, the second item indicates what nodes and user accounts have access to decrypt the password.

117

Can Nodes Decrypt This?

$ knife data bag show secrets vaultuser_keys

118

admins: adminUserclients: os-945926465950316 os-2790002246935003id: vaultuser_keysadminUser: 0Q2bhw/kJl2aIVEwqY6wYhrrfdz9fdsf8tCiIrBih2ZORvV7EEIpzzKQggRX...XdmdfMKNYGaM6kzG8cwuKZXLAgGAgblVUB1HP8+8kQ==os-2790002246935003: kGQLsxsFmBe9uPuWxZpKiNBnqJq55hQZJLgaKdjG2Vvivv98RrFGz1y8Xbwe...rK3s5vD7v6jpkUW12Wj74Lz3Z6x3sKuIDzCtvEUnWw==os-945926465950316: XzTJrJ3TZZZ1u9L9p6DZledf3bo2ToH2yrLGZQKPV6/ANzElHXGcYrEdtP0q14Nz1NzsqEftzviAebUUnc6ke91ltD8s6hNQQrPJRqkUoDlM7lNEwiUiz/dD+sFI6CSzQptO3zPrUbAlUI1Zog5h7k/CCtiYtmFRD6wbAWnxmCqvLhO1jwqLVNJ1vfjlFsG77BDm2HFw7jgleuxRGYEgBfCCuBuW70FAdUTvNHIAwKQVkfU/Am75UYm7N4N0E+W76ZwojLoYtXXTV/iOGG1cw3C75SVAmCsBOuxUK/otub67zsNDsKToKa+laxzXGylrmkTricYXIqVpIQO8OL5nnw==

Examine Second Item in Data Bag…

NODE

NODE

Let’s use Chef Vault to manage a user resource with a password that is set to the value stored within our encrypted data bag. •Create a cookbook

•Add cookbook to base role

CD into your cookbooks directory

119

Manage a User Password

$ chef generate cookbook vault

120

Compiling Cookbooks…Recipe: code_generator::cookbook. . .

Create Cookbook

121

Create the Cookbook…OPEN IN EDITOR: /cookbooks/vault/recipes/default.rb

SAVE FILE!

. . .# All rights reserved - Do Not Redistribute#

chef_gem "chef-vault"require "chef-vault"

vault = ChefVault::Item.load("secrets", "vaultuser")hashed_password = vault['vaultuser'].crypt("$6$")

user "vaultuser" do password hashed_password home "/home/vaultuser" supports :manage_home => true shell "/bin/bash" comment "Chef Vault User"end

• This is where the decryption happens…

• Item.load takes two variables: the data bag (e.g. vault) and the item

• The value of the item is returned

122

Breaking It Down…vault = ChefVault::Item.load("secrets", "vaultuser")

• Setting a variable to equal the decrypted 'password' that has been re-encrypted with SHA-512 required by the operating system for user passwords.

• Using a case statement you could set the re-encryption to match that required by different operating systems.

123

Breaking It Down…hashed_password = vault['vaultuser'].crypt("$6$")

• Within the user resource the important attribute is password as we’re referencing the local variable vault and the vault user key

124

Breaking It Down…user "vaultuser" do password hashed_password home "/home/vaultuser" supports :manage_home => true shell "/bin/bash" comment "Chef Vault User"end

Uploading vault [0.1.0]

Uploaded 1 cookbook.

Upload Vault Cookbook$ knife cookbook upload vault

node1:

run_list:

role[base]

recipe[vault]

Exercise: Edit run list for managed node$ knife node run_list add node1 'recipe[vault]'




- chef-client

- vault


...


Password: <PASSWORD>

vaultuser@os-279000224593500d:~$

Use Newly Created Account…chef@node1$ su - vaultuser

Password: chef-vault

vaultuser@os-2790002246935003:~$ id

uid=1001(vaultuser) gid=1001(vaultuser)groups=1001(vaultuser)

vaultuser@os-2790002246935003:~$ pwd

/home/vaultuser

Use Newly Created Account…chef@node1$ su - vaultuser

id

homedir

What if one of our admins leaves and we want to change their associated password/secret?

130

http://sd.keepcalm-o-matic.co.uk/i/keep-calm-and-change-your-password-16.png

Chef Vault Edits

$ knife vault create secrets \ vaultuser '{"vaultuser": "<NEW PASSWORD>"}' \ --search 'role:base' \ --admins adminUser --mode client

131

ERROR: ChefVault::Exceptions::ItemAlreadyExists: secrets/vaultuser already exists, use 'knife encrypt remove' and 'knife encrypt update' to make changes.

Create Data Bag…

$ knife vault update secrets vaultuser \ '{"vaultuser":"<NEW PASSWORD>"}' \ --search 'role:base' \ --admins adminUser --mode client

132

Update Data Bag Instead…

$ knife vault update secrets vaultuser \ '{"vaultuser":"<NEW PASSWORD>"}' \ --admins adminUser --mode client

133

Shortened Update Data Bag Instead…

Secrets/vaultuser

Vaultuser: <NEW PASSWORD>

Decrypt First Item Using Knife…$ knife vault show secrets vaultuser vaultuser --mode client




- chef-client

- vault


...


Password: <NEW PASSWORD>

vaultuser@os-279000224593500d:~$

Reconverge and Validate…$ su - vaultuser

v0.2

Audit Controls and Rules

DependenciesChef Analytics Version 1.1.0 is current production release

Chef Server Version 12.0.3 or higher; Enterprise Chef 11.3 or higher

Chef Client Version 11 or 12 required; 12.1 required for audit-mode

Reporting Version 1.2.3 suggested for full feature/functionality

Analytics Components

Analytics FeaturesControls Automated tests, built into a cookbook and then used to test the state of the system for compliance

Audit Mode Execution mode of chef-client that evaluates control / control-groups that are defined within recipes •Can be run in combination with a standard run after resources converge

•As a separate chef-client run with no convergence

Analytics FeaturesChef Actions

A collection of data gathered and made available Keeps track of what happened during chef-client runs across your Chef managed systems

Reporting Reports can be generated for the entire organization and they can be generated for specific nodes.

Rules Track specific outcomes of audits and issue notifications to various endpoints and/or services

Writing ControlsMethods Two methods are built into the recipe DSL that can be used to configure tests for audit-mode: • control_group • control

Writing Controlscontrol_group •Defines a group of control methods that comprise a single audit

•Can contain several control methods

Writing Controlscontrol •Tests written within a recipe that are used to test the state of the system for compliance

•A single control block defines one or more validations

•Contains an it statement •Each it statement contains one or more expect statements

Writing Controlscontrol_group "audit name" do control "name" do it "should do something" do expect(something).to/.to_not be_something end endend

Writing Controlscontrol_group "audit name" do

control "name" do

it "should do something" do

expect(something).to/.to_not be_something

end

end

control "nameTwo" do

it "should do something new" do

expect(something).to/.to_not be_something

expect(somethingElse).to/.to_not be_something

end

end

end

Writing Controlsexpect statements can use matchers •Matchers are available for: o directory o file o package o port o service

Writing ControlsSome examples… directory: it "should be a directory" do expect(file("/var/directory")).to be_directory end

it "should be mounted with an ext4 partition" do expect(file("/")).to be_mounted.with( :type => 'ext4' ) end

Writing ControlsSome examples… file: it "should be executable by owner" do expect(file("/etc/file")).to be_executable.by("owner") end

it "should be mode 440" do expect(file("/etc/file")).to be_mode(440) end

Writing ControlsSome examples… package, port: it "should be installed" do expect(package("httpd")).to be_installed end

it "should not be listening" do expect(port(23)).to_not be_listening end

Writing ControlsSome examples… service: it "should be enabled" do expect(service("ntpd")).to be_enabled end

it "should be running" do expect(service("ntpd")).to be_running end

Writing ControlsPutting it together… control_group 'check sshd configuration' do

control 'sshd package' do it 'should be installed' do expect(package('openssh-‐server')).to be_installed end end

control 'sshd configuration' do let(:config_file) { file('/etc/ssh/sshd_config') } it 'should exist with the right permissions' do expect(config_file).to be_file expect(config_file).to be_mode(644) expect(config_file).to be_owned_by('root') expect(config_file).to be_grouped_into('root') end it 'should not permit RootLogin' do expect(config_file.content).to_not match(/^PermitRootLogin yes/) end it 'should explicitly not permit PasswordAuthentication' do expect(config_file.content).to match(/^PasswordAuthentication no/) end it 'should force privilege separation' do expect(config_file.content).to match(/^UsePrivilegeSeparation sandbox/) end end end

Let's Perform Audits…

http://2.bp.blogspot.com/---1IokhPScE/Ulb3PorYrJI/AAAAAAAAAik/MwNuvpkUmyU/s1600/AuditAllTheThings.png

. . .

tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN

. . .

Check Remote Node for Port 25 Listeninguser@centos$ netstat –plnt | grep :25


Recipe: code_generator::cookbook

* directory[/Users/larryeichenbaum/chef-repo-larryebaum/cookbooks/auditlab] action create

- create new directory /Users/larryeichenbaum/chef-repo-larryebaum/cookbooks/auditlab

* template[/Users/larryeichenbaum/chef-repo-larryebaum/cookbooks/auditlab/metadata.rb] action create_if_missing

- create new file /Users/larryeichenbaum/chef-repo-larryebaum/cookbooks/auditlab/metadata.rb

- update content in file /Users/larryeichenbaum/chef-repo-larryebaum/cookbooks/auditlab/metadata.rb from none to f8e5d5

. . .

* execute[initialize-git] action run

- execute git init .

* cookbook_file[/Users/larryeichenbaum/chef-repo-larryebaum/cookbooks/auditlab/.gitignore] action create

- create new file /Users/larryeichenbaum/chef-repo-larryebaum/cookbooks/auditlab/.gitignore

- update content in file /Users/larryeichenbaum/chef-repo-larryebaum/cookbooks/auditlab/.gitignore from none to dd37b2

Generate a Cookbook$ chef generate cookbook auditlab

OPEN IN EDITOR:

SAVE FILE!

Create Recipe

#

# Cookbook Name:: auditlab

# Recipe:: default

#

# Copyright (c) 2015 The Authors, All Rights Reserved.

control_group 'check smtp configuration' do

control 'smtp port' do it 'should not be listening' do expect(port(25)).to_not be_listening end end

end

/cookbooks/auditlab/recipes/default.rb

Uploading auditlab [0.1.0] Uploaded 1 cookbook.

Upload Cookbook$ knife cookbook upload auditlab

node1: run_list: role[base] recipe[vault] recipe[auditlab]

Set Node Runlist$ knife node run_list add <node> 'recipe[auditlab]'

New chef-client Run Option--audit-mode <MODE> •Audit mode evaluates custom rules, or audits, defined within recipes

•Can be run as part of normal chef-run, at the end •Can be run separately, where no converge occurs o --audit-mode disabled (default) o --audit-mode enabled o --audit-mode audit-only

[2015-03-06T22:29:50+00:00] WARN: Chef-client has been configured to skip converge and run only audits. Audit mode is an experimental feature currently under development. API changes may occur. Use at your own risk. * To enable audit mode after converge, use command line option `--audit-mode enabled` or set `:audit_mode = :enabled` in your config file. * To disable audit mode, use command line option `--audit-mode disabled` or set `:audit_mode = :disabled` in your config file. * To only run audit mode, use command line option `--audit-mode audit-only` or set `:audit_mode = :audit_only` in your config file. Audit mode is disabled by default. Starting Chef Client, version 12.1.0 resolving cookbooks for run list: ["auditlab"] Synchronizing Cookbooks: - auditlab Compiling Cookbooks... Starting audit phase

Execute client-run Audit Modeuser@centos$ sudo chef-client –-audit-mode audit-only

. . . check smtp configuration smtp port should not be listening (FAILED - 1)

Failures:

1) check smtp configuration smtp port should not be listening Failure/Error: expect(port(25)).to_not be_listening expected Port "25" not to be listening . . . Finished in 0.05565 seconds (files took 0.27206 seconds to load) 1 example, 1 failure

Failed examples:

rspec # check smtp configuration smtp port should not be listening

Execute client-run Audit Modeuser@centos$ sudo chef-client –audit-mode audit-only

OPEN IN EDITOR:

SAVE FILE!

Modify Audit

#

# Cookbook Name:: auditlab

# Recipe:: default

#

# Copyright (c) 2015 The Authors, All Rights Reserved.

control_group 'check smtp configuration' do

control 'smtp port' do it 'should be listening' do expect(port(25)).to be_listening end end

end

/cookbooks/auditlab/recipes/default.rb

Uploading auditlab [0.1.0] Uploaded 1 cookbook.

Re-upload Cookbook$ knife cookbook upload auditlab

Starting Chef Client, version 12.1.0 resolving cookbooks for run list: ["auditlab"] Synchronizing Cookbooks: - auditlab Compiling Cookbooks... Starting audit phase

check smtp configuration smtp port should be listening

Finished in 0.05616 seconds (files took 0.2733 seconds to load) 1 example, 0 failures Auditing complete

Running handlers: Running handlers complete Chef Client finished, 0/0 resources updated in 2.365794244 seconds 1/1 Audits succeeded


Who Tracks Audit Results?

http://s237.photobucket.com/user/dharma-putra/media/AuditReportandAuditorOpinion.png.html

Audit Rules…Audits are of limited value if the results are unknown or not communicated: •Rules track the events generated by audits •Rules track specific outcomes of the audit tests •Rules send notifications about these outcomes o http webhook o email o chat services, etc.

Audit Rule Details…Each rule must be associated with a message type •Messages sent to analytics as rules are triggered •Message types:

•action (msgs about actions that occur on Chef Server) •run_control (audit related) •run_control_group (audit related) •run_converge (msgs sent at end of chef-client run) •run_resource (msgs sent during chef-client run) •run_start (msgs sent at start of chef-client run)

Chef Analytics Login…



Let's Write Rules…

http://advisorspension.com/wp-content/uploads/2014/09/money-rules.jpg

rules 'pci check smtp rules'

rule on run_control

when

name = "should be listening" and

status = "success"

then

notify("Hipchat",<YOUR NAME>: "A server is running SMTP on port 25."

end

end

Create New Rule

Notifications and Reports…

https://openclipart.org/image/300px/svg_to_png/3743/zeimusu-Warning-notification.png

Data Flow

Event Processing and Rules Engine

User Interface

Notifications

IntegrationsChef Server

Event stream

Create Notification…



1345044

1345hEV944u5vEwPawKgUL8LOYaN2zqJGSwAUDdvHk00044


1345044

hEV944u5vEwPawKgUL8LOYaN2zqJGSwAUDdvHk00


hEV944u5vEwPawKgUL8LOYaN2zqJGSwAUDdvHk00

} ONE LINE, NO LINE BREAKS



[2015-03-06T22:29:50+00:00] WARN: Chef-client has been configured to skip converge and run only audits. Audit mode is an experimental feature currently under development. API changes may occur. Use at your own risk. * To enable audit mode after converge, use command line option `--audit-mode enabled` or set `:audit_mode = :enabled` in your config file. * To disable audit mode, use command line option `--audit-mode disabled` or set `:audit_mode = :disabled` in your config file. * To only run audit mode, use command line option `--audit-mode audit-only` or set `:audit_mode = :audit_only` in your config file. Audit mode is disabled by default. Starting Chef Client, version 12.1.0 resolving cookbooks for run list: ["auditlab"] Synchronizing Cookbooks: - auditlab Compiling Cookbooks... Starting audit phase


Verify Notification Received


Key Management / Rotation

http://upload.wikimedia.org/wikipedia/commons/c/c6/Llave_bronce.jpg

Key Management / Rotation•New in Chef server 12.0.3 (still in progress) •Multi-key authentication •Chef-server-ctl commands to build key rotation

Key Management / Rotation•Store and manage multiple keys for users and clients

•Server will check request against all valid keys for user or client

•Enables server to know if request is from a user or client

•Resolves user-client ambiguity •Optional expiration dates

Key Management / Rotation•Backwards compatible •API support underway •Still in progress, so don’t delete your ‘default’ keys just yet

Key Rotation CommandsSubcommands to chef-server-ctl •add-client-key •add-user-key •delete-client-key •delete-user-key • list-client-key • list-user-key

Key Rotation Syntax$ chef-‐server-‐ctl add-‐client-‐key ORG_NAME CLIENT_NAME \ PATH_TO_KEY [-‐-‐expiration-‐date DATE] [-‐-‐key-‐name NAME]

$ chef-‐server-‐ctl add-‐user-‐key USER_NAME PATH_TO_KEY [-‐-‐expiration-‐date DATE] [-‐-‐key-‐name NAME]

$ chef-‐server-‐ctl delete-‐client-‐key ORG_NAME CLIENT_NAME KEY_NAME

$ chef-‐server-‐ctl delete-‐client-‐key USER_NAME KEY_NAME

$ chef-‐server-‐ctl list-‐user-‐keys ORG_NAME CLIENT_NAME [-‐-‐hide-‐public-‐keys]

$ chef-‐server-‐ctl list-‐user-‐keys USER_NAME [-‐-‐hide-‐public-‐keys]

https://mastersreview.com/files/2014/06/summary.jpg

Summary•We’ve learned how native data bag encryption differs from the more complex data bag encryption we can accomplish via Chef Vault

• We’ve learned about how to lock down specific access points to Chef Server application and how to apply industry standards for security to further secure our Chef Server infrastructure

Summary•We’ve successfully learned how to use Chef Vault to generate, distribute and encrypt/decrypt secrets

•We’ve successfully performed a security audit of a Chef managed server and learned how we can extend this functionality in a broader capacity

Summary•We’ve learned about new Chef Server features for managing and rotating keys

ChefConf2015 Chef and Security (keyNote)...Exercise: Download your client .pem •You may reuse your Chef server setup from the morning Analytics workshop •Only do this if you don’t

Documents