Hardening Your Config Management - Security and Attack Vectors in Config Management

HARDENING YOUR CONFIG MANAGEMENT

SECURITY AND ATTACK VECTORS IN CONFIG MANAGEMENT

WHO AM I?

> Peter Souter > @petersouter

> @petems - IRC/GitHub> Professional Services Engineer at

Puppet Labs> Work with customers when they buy

services and teach Puppet classes

THIS IS MY 3RD FOSDEM!

WHAT IS THIS ALL ABOUT?

HTTPS://FLIC.KR/P/BHYT8B

SECURITY IS HARD

AND UNDER APPRECIATED!

HTTPS://TWITTER.COM/PETECHESLOCK/STATUS/595617204273618944

SPECIFIC REQUIREMENTSMULTIPLE SYSTEMS

EVERY OS HAS IT'S OWN QUIRKS AND NUANCES

CONFIG MANAGEMENT IS HERE TO SAVE THE DAY!

HOWEVER...

QUIS CUSTODIET IPSOS

CUSTODES?

A SYSTEM CAPABLE OF PERFORMING CHANGES FOR CONFIGURATION ACROSS

THOUSANDS OF SERVERS...

COULD CAUSE A LOT OF DAMAGE!

CONFIG MANAGEMENT: A PRETTY BIG ATTACK

VECTOR....

HOW DO WE HARDEN CONFIG MANAGEMENT

ITSELF?

DON'T WANT TO FOCUS TOO MUCH ON THE TOOLS

THEMSELVES

I HAVE BIASESBOTH CONSCIOUS AND SUBCONSCIOUS

THERE IS NO ONE-SIZE-FITS-ALL TO HARDEN CONFIG MANAGEMENT!

IT’S A LOT OF CHANGES TO PROCESSES

PEOPLE ARE HARDER TO CHANGE THAN COMPUTERS!

ACCEPT THAT YOU WILL FAIL, PLAN ACCORDINGLY

THE BADDIES HAVE MORE TIME/MONEY/ENERGY

THAN YOU DO!

YOU WILL FAIL AT SOME POINT. YOU NEED TO FAIL

SECURELY

A QUICK SURVEY

WHO HERE USES...ANSIBLE

CFENGINE

CHEF

PUPPET

SALTSTACK

WHERE TO START?

FIRST 3 RESULTS ARE FROM A COMPANY THAT RHYMES WITH RIPTIRE...

4TH RESULT: OWASP PRINCIPLES

5TH RESULT...

8TH RESULT: PRETTY GOOD BLOG POST

STILL, NOT SUPER IN-DEPTH...

GUESS I'LL HAVE TO ACTUALLY DO SOME

RESEARCH...

DATA

IT'S EASY TO LEAK DATA...

ESPECIALLY SOMETHING YOU CAN LOOK FOR

AUTOMATICALLY

BEST PRACTICESEPARATION OF CONCERNS

REMOVE DATA FROM CODEESPECIALLY COMPANY SPECIFIC DATA!

DATA ABSTRACTION:PUPPET - HIERA

CHEF - DATA BAGS/ATTRIBUTESANSIBLE - ROLES

SALT - GRAINS/PILLAR

BAD

GOOD

ADVANTAGE:NOT ONLY MORE SECURE, CLEANER CODE THAT'S

MORE REUSABLE!

THEORETICAL SCENARIO:

YOU SHOULD BE ABLE TO RELEASE MOST CODE YOU WRITE PUBLICALLY WITHOUT ANY SORT OF SECURITY

ISSUES

ANYTHING SENSITIVE SHOULD BE KEPT IN THE DATA ABSTRACTION LAYER

EXAMPLE: GDS

HTTPS://GITHUB.COM/ALPHAGOV/GOVUK-PUPPET

HTTPS://GDSTECHNOLOGY.BLOG.GOV.UK/2016/01/19/OPENING-GOV-UKS-

PUPPET-REPOSITORY/

YOUR DATA SHOULD IS NOW SEPARATED. HOORAY!

BUT IT'S PLAINTEXT. BOO!

ENCRYPTION

ENCRYPTING DATA WITH YOUR APPLICATION SPECIFIC TOOLS:

PUPPET - HIERA-EYAMLCHEF - CHEF-VAULT

ANSIBLE - ANSIBLE VAULTSALT - SALT.MODULES.GPGCFENGINE - CF-KEYCRYPT

TOOL-SPECIFIC VAULTS ARE GREAT, BUT ARE OFTEN LIMITED IN

FUNCTIONALITY OUTSIDE THAT TOOL.

YOU DON'T WANT TO STORE THE SAME PASSWORD IN 10 DIFFERENT SYSTEMS

IF YOU CAN HELP ITTHAT'S 10X MORE THAT NEEDS TO BE

SECURED

EXTERNAL SECRET

SERVERS?

OPEN SOURCE POTENTIALS CHOICES:OPENSTACK'S BARBICAN

CLOUDFLARE'S REDOCTOBERHASHICOP'S VAULT

GOING DEEPER:

SECURING DATA WITH SOURCE CONTROL

"I wanted to make a configuration management repository open for others to look at and contribute

to (à la Wikimedia's Puppet repository)...

However, the repository contained secret material, like SSL keys and

passwords...

git-crypt was developed so the secret material could be

protected without having to remove it from the repository (which is what Wikimedia had to

do).- ANDREW AYER

GIT-CRYPT?HTTPS://WWW.AGWA.NAME/

PROJECTS/GIT-CRYPT/

GIT-SUBMODULES OR SEPARATE REPOS

STAY IN (VERSION) CONTROL

GATE CONFIG MANAGEMENT CHANGES

BEHIND VERSION CONTROL

REMEMBER TO KEEP COMMITS CLEAN AS WELL!

commit 88a055c4c3dcec34d5r9054011963649be89d49cMerge: 783d425 1743488Author: Peter Souter <petems@users.noreply.github.com>Date: Mon April 1 23:47:43 2030 +0000

Turned off SSL, we don't need that right?

also password is now password123

RBAC FOR GIT REPOS CONTAINING THE DATA

RBAC

SPLIT ACCESS TO CONFIG MANAGEMENT TOOLS

BASED ON NEED

MOST APPLICATIONS HAVE SOME FORM OF RBAC HOOKS TO ANOTHER

AUTHORIZED SYSTEM (LDAP, AD, ETC.)

REVIEW PROCESSES(AUTOMATED AND MANUAL)

AUTOMATEDSPEC TESTING

AUTOMATED TESTING SUITESLINTING/SYNTAX CHECKING

MANUALCODE REVIEWS

GET SECURITY TEAM INVOLVED IN THE PROCESSES!

WORK WITH AUDITORS

PEOPLE LOVE TO HATE AUDITORS

ADVERSARIAL ENVIRONMENTS ARE NOT

FUN

IF YOU HAVE A GOOD WORKING RELATIONSHIP WITH THEM, THEY'RE LIKE

AN ADDITION TO YOUR TEAM.

LET'S FACE IT, YOU'LL HAVE TO DEAL WITH THEM ANYWAY, SO YOU MIGHT AS

WELL MAKE IT ENJOYABLE!

ASK AROUND

SOFT SKILL/CULTURAL SOLUTION

COMPARE YOUR SECURITY WITH OTHERS WHEN

POSSIBLE

A SECURITY MODEL MADE IN A VACUUM IS A SMELL

IF YOU'RE A CUSTOMER, ASK YOUR VENDOR

IF YOU'RE A FOSS USER, ASK ON MAILING LISTS

GAME DAYS AND DRILLS

IF SOMEONE HAD ACCESS TO THE VARIOUS PARTS OF

YOUR CONFIG MANAGEMENT INFRA...

HOW MUCH DAMAGE COULD THEY DO?HOW FAST COULD YOU REVOKE

ACCESS?HOW LONG WOULD IT TAKE YOU TO

NOTICE?

MONITOR, DON'T JUST LOG

GET A BASELINE OF WHAT YOUR CONFIG

MANAGEMENT DEPLOYS LOOK LIKE

ELK, STATSD, RIEMANN, COLLECTD, ETC.

GET DATA ON WHAT LOOKS SUSPICIOUS

ACTIVITY WHEN YOU DON'T EXPECT IT

4XX, 5XX ERRORS FROM YOUR CONFIG MANAGEMENT INFRA

UNEXPLAINED INCREASES IN THE TEMPERATURE OF YOUR MACHINES IN

THE DATA CENTREGENERAL ERRORS IN VARIOUS LOGS

COULD BE MALICIOUS, COULD BE ACCIDENTAL,

COULD BE A BUG...

ALL OF WHICH YOU SHOULD KNOW ABOUT!

REDUCE SURFACE LEVEL

OF ATTACK

NOT SECURITY THROUGH OBSCURITY!

A BASIC EXAMPLE AT THE APPLICATION LEVEL

> Chef: sensitive: true> Puppet: show_diff=false

> Ansible: no_log: True> Salt: --state-verbose=false

SECURITY BASELINE

USE THE SAME SECURITY BASELINE FOR ANY SORT OF SYSTEM:

NO DIRECT INTERNET ACCESS UNLESS ABSOLUTELY NECESSARY

USE BASTION HOSTS FOR DIRECT INTERNET ACCESS

MIRROR REPOS AND ARTIFACTS

KEEP PACKAGES UP TO DATE AND PATCHED

SENSIBLE FIREWALL RULES

HARDEN CONFIG MANAGEMENT

INFRASTRUCTURE WITH CONFIG MANAGEMENT!

CENTER FOR INTERNET SECURITY BENCHMARKS

HARDENING.IO

SOME 3 LETTER AGENCIES HAVE EVEN RELEASED

THEIR CONFIG MANAGEMENT CODE...

IN LIGHT OF RECENT EVENTS, THAT MIGHT BE NOT SUCH A GREAT THING

BUT HEY, IT'S CONFIG MANAGEMENT, SO YOU CAN INSPECT AND ADAPT WHERE

NECESSARY!

SSH

PRIMARILY FOR ANSIBLE

BUT SSH CAN BE USED FOR OTHER TOOLS AS

WELL...

PUPPET - SUPPLY DROP/CAPISTRANOCHEF - KNIFE SOLOSALT - SALT SSH

CUSTOM MADE SSH-LOOPS WRAPPING LOCAL MODES FOR TOOLS

SSH HARDENING STANDARDS

> Whitelisted access> Bastion hosts> Restrict users

> Increase key strength> Rotate keys

> Pre-populated knownhosts

HARDEN YOUR SSH WITH CONFIG MANAGEMENT! :)

IF YOU'RE USING ~/.ssh/id_rsa FOR EVERYTHING...

YOU'RE DOING IT WRONG :(

DEEPER SSH HARDENING...

SSH KEYS ON HARDWAREYUBIKEY

SMARTCARD

THOUGHT EXPERIMENT:DISABLE SSH COMPLETELY?

CONCLUSION

> Get your data out of your code> Encrypt it and control access

> Most normal security conventions apply> Follow best practices from communities and

organizations> Auditing and gating help

> Work together! :)

GOING TO CONFIG MANAGEMENT CAMP?

QUESTIONS? IDEAS?HOW ARE YOU HARDENING YOUR CONFIG

MANAGEMENT?