HARDENING YOUR CONFIG MANAGEMENT SECURITY AND ATTACK VECTORS IN CONFIG MANAGEMENT
Jan 15, 2017
HARDENING YOUR CONFIG MANAGEMENT
SECURITY AND ATTACK VECTORS IN CONFIG MANAGEMENT
WHO AM I?
> Peter Souter > @petersouter
> @petems - IRC/GitHub> Professional Services Engineer at
Puppet Labs> Work with customers when they buy
services and teach Puppet classes
THIS IS MY 3RD FOSDEM!
WHAT IS THIS ALL ABOUT?
HTTPS://FLIC.KR/P/BHYT8B
SECURITY IS HARD
AND UNDER APPRECIATED!
HTTPS://TWITTER.COM/PETECHESLOCK/STATUS/595617204273618944
SPECIFIC REQUIREMENTSMULTIPLE SYSTEMS
EVERY OS HAS IT'S OWN QUIRKS AND NUANCES
CONFIG MANAGEMENT IS HERE TO SAVE THE DAY!
HOWEVER...
QUIS CUSTODIET IPSOS
CUSTODES?
A SYSTEM CAPABLE OF PERFORMING CHANGES FOR CONFIGURATION ACROSS
THOUSANDS OF SERVERS...
COULD CAUSE A LOT OF DAMAGE!
CONFIG MANAGEMENT: A PRETTY BIG ATTACK
VECTOR....
HOW DO WE HARDEN CONFIG MANAGEMENT
ITSELF?
DON'T WANT TO FOCUS TOO MUCH ON THE TOOLS
THEMSELVES
I HAVE BIASESBOTH CONSCIOUS AND SUBCONSCIOUS
THERE IS NO ONE-SIZE-FITS-ALL TO HARDEN CONFIG MANAGEMENT!
IT’S A LOT OF CHANGES TO PROCESSES
PEOPLE ARE HARDER TO CHANGE THAN COMPUTERS!
ACCEPT THAT YOU WILL FAIL, PLAN ACCORDINGLY
THE BADDIES HAVE MORE TIME/MONEY/ENERGY
THAN YOU DO!
YOU WILL FAIL AT SOME POINT. YOU NEED TO FAIL
SECURELY
A QUICK SURVEY
WHO HERE USES...ANSIBLE
CFENGINE
CHEF
PUPPET
SALTSTACK
WHERE TO START?
FIRST 3 RESULTS ARE FROM A COMPANY THAT RHYMES WITH RIPTIRE...
4TH RESULT: OWASP PRINCIPLES
5TH RESULT...
8TH RESULT: PRETTY GOOD BLOG POST
STILL, NOT SUPER IN-DEPTH...
GUESS I'LL HAVE TO ACTUALLY DO SOME
RESEARCH...
DATA
IT'S EASY TO LEAK DATA...
ESPECIALLY SOMETHING YOU CAN LOOK FOR
AUTOMATICALLY
BEST PRACTICESEPARATION OF CONCERNS
REMOVE DATA FROM CODEESPECIALLY COMPANY SPECIFIC DATA!
DATA ABSTRACTION:PUPPET - HIERA
CHEF - DATA BAGS/ATTRIBUTESANSIBLE - ROLES
SALT - GRAINS/PILLAR
BAD
GOOD
ADVANTAGE:NOT ONLY MORE SECURE, CLEANER CODE THAT'S
MORE REUSABLE!
THEORETICAL SCENARIO:
YOU SHOULD BE ABLE TO RELEASE MOST CODE YOU WRITE PUBLICALLY WITHOUT ANY SORT OF SECURITY
ISSUES
ANYTHING SENSITIVE SHOULD BE KEPT IN THE DATA ABSTRACTION LAYER
EXAMPLE: GDS
HTTPS://GITHUB.COM/ALPHAGOV/GOVUK-PUPPET
HTTPS://GDSTECHNOLOGY.BLOG.GOV.UK/2016/01/19/OPENING-GOV-UKS-
PUPPET-REPOSITORY/
YOUR DATA SHOULD IS NOW SEPARATED. HOORAY!
BUT IT'S PLAINTEXT. BOO!
ENCRYPTION
ENCRYPTING DATA WITH YOUR APPLICATION SPECIFIC TOOLS:
PUPPET - HIERA-EYAMLCHEF - CHEF-VAULT
ANSIBLE - ANSIBLE VAULTSALT - SALT.MODULES.GPGCFENGINE - CF-KEYCRYPT
TOOL-SPECIFIC VAULTS ARE GREAT, BUT ARE OFTEN LIMITED IN
FUNCTIONALITY OUTSIDE THAT TOOL.
YOU DON'T WANT TO STORE THE SAME PASSWORD IN 10 DIFFERENT SYSTEMS
IF YOU CAN HELP ITTHAT'S 10X MORE THAT NEEDS TO BE
SECURED
EXTERNAL SECRET
SERVERS?
OPEN SOURCE POTENTIALS CHOICES:OPENSTACK'S BARBICAN
CLOUDFLARE'S REDOCTOBERHASHICOP'S VAULT
GOING DEEPER:
SECURING DATA WITH SOURCE CONTROL
"I wanted to make a configuration management repository open for others to look at and contribute
to (à la Wikimedia's Puppet repository)...
However, the repository contained secret material, like SSL keys and
passwords...
git-crypt was developed so the secret material could be
protected without having to remove it from the repository (which is what Wikimedia had to
do).- ANDREW AYER
GIT-CRYPT?HTTPS://WWW.AGWA.NAME/
PROJECTS/GIT-CRYPT/
GIT-SUBMODULES OR SEPARATE REPOS
STAY IN (VERSION) CONTROL
GATE CONFIG MANAGEMENT CHANGES
BEHIND VERSION CONTROL
REMEMBER TO KEEP COMMITS CLEAN AS WELL!
commit 88a055c4c3dcec34d5r9054011963649be89d49cMerge: 783d425 1743488Author: Peter Souter <[email protected]>Date: Mon April 1 23:47:43 2030 +0000
Turned off SSL, we don't need that right?
also password is now password123
RBAC FOR GIT REPOS CONTAINING THE DATA
RBAC
SPLIT ACCESS TO CONFIG MANAGEMENT TOOLS
BASED ON NEED
MOST APPLICATIONS HAVE SOME FORM OF RBAC HOOKS TO ANOTHER
AUTHORIZED SYSTEM (LDAP, AD, ETC.)
REVIEW PROCESSES(AUTOMATED AND MANUAL)
AUTOMATEDSPEC TESTING
AUTOMATED TESTING SUITESLINTING/SYNTAX CHECKING
MANUALCODE REVIEWS
GET SECURITY TEAM INVOLVED IN THE PROCESSES!
WORK WITH AUDITORS
PEOPLE LOVE TO HATE AUDITORS
ADVERSARIAL ENVIRONMENTS ARE NOT
FUN
IF YOU HAVE A GOOD WORKING RELATIONSHIP WITH THEM, THEY'RE LIKE
AN ADDITION TO YOUR TEAM.
LET'S FACE IT, YOU'LL HAVE TO DEAL WITH THEM ANYWAY, SO YOU MIGHT AS
WELL MAKE IT ENJOYABLE!
ASK AROUND
SOFT SKILL/CULTURAL SOLUTION
COMPARE YOUR SECURITY WITH OTHERS WHEN
POSSIBLE
A SECURITY MODEL MADE IN A VACUUM IS A SMELL
IF YOU'RE A CUSTOMER, ASK YOUR VENDOR
IF YOU'RE A FOSS USER, ASK ON MAILING LISTS
GAME DAYS AND DRILLS
IF SOMEONE HAD ACCESS TO THE VARIOUS PARTS OF
YOUR CONFIG MANAGEMENT INFRA...
HOW MUCH DAMAGE COULD THEY DO?HOW FAST COULD YOU REVOKE
ACCESS?HOW LONG WOULD IT TAKE YOU TO
NOTICE?
MONITOR, DON'T JUST LOG
GET A BASELINE OF WHAT YOUR CONFIG
MANAGEMENT DEPLOYS LOOK LIKE
ELK, STATSD, RIEMANN, COLLECTD, ETC.
GET DATA ON WHAT LOOKS SUSPICIOUS
ACTIVITY WHEN YOU DON'T EXPECT IT
4XX, 5XX ERRORS FROM YOUR CONFIG MANAGEMENT INFRA
UNEXPLAINED INCREASES IN THE TEMPERATURE OF YOUR MACHINES IN
THE DATA CENTREGENERAL ERRORS IN VARIOUS LOGS
COULD BE MALICIOUS, COULD BE ACCIDENTAL,
COULD BE A BUG...
ALL OF WHICH YOU SHOULD KNOW ABOUT!
REDUCE SURFACE LEVEL
OF ATTACK
NOT SECURITY THROUGH OBSCURITY!
A BASIC EXAMPLE AT THE APPLICATION LEVEL
> Chef: sensitive: true> Puppet: show_diff=false
> Ansible: no_log: True> Salt: --state-verbose=false
SECURITY BASELINE
USE THE SAME SECURITY BASELINE FOR ANY SORT OF SYSTEM:
NO DIRECT INTERNET ACCESS UNLESS ABSOLUTELY NECESSARY
USE BASTION HOSTS FOR DIRECT INTERNET ACCESS
MIRROR REPOS AND ARTIFACTS
KEEP PACKAGES UP TO DATE AND PATCHED
SENSIBLE FIREWALL RULES
HARDEN CONFIG MANAGEMENT
INFRASTRUCTURE WITH CONFIG MANAGEMENT!
CENTER FOR INTERNET SECURITY BENCHMARKS
HARDENING.IO
SOME 3 LETTER AGENCIES HAVE EVEN RELEASED
THEIR CONFIG MANAGEMENT CODE...
IN LIGHT OF RECENT EVENTS, THAT MIGHT BE NOT SUCH A GREAT THING
BUT HEY, IT'S CONFIG MANAGEMENT, SO YOU CAN INSPECT AND ADAPT WHERE
NECESSARY!
SSH
PRIMARILY FOR ANSIBLE
BUT SSH CAN BE USED FOR OTHER TOOLS AS
WELL...
PUPPET - SUPPLY DROP/CAPISTRANOCHEF - KNIFE SOLOSALT - SALT SSH
CUSTOM MADE SSH-LOOPS WRAPPING LOCAL MODES FOR TOOLS
SSH HARDENING STANDARDS
> Whitelisted access> Bastion hosts> Restrict users
> Increase key strength> Rotate keys
> Pre-populated knownhosts
HARDEN YOUR SSH WITH CONFIG MANAGEMENT! :)
IF YOU'RE USING ~/.ssh/id_rsa FOR EVERYTHING...
YOU'RE DOING IT WRONG :(
DEEPER SSH HARDENING...
SSH KEYS ON HARDWAREYUBIKEY
SMARTCARD
THOUGHT EXPERIMENT:DISABLE SSH COMPLETELY?
CONCLUSION
> Get your data out of your code> Encrypt it and control access
> Most normal security conventions apply> Follow best practices from communities and
organizations> Auditing and gating help
> Work together! :)
GOING TO CONFIG MANAGEMENT CAMP?
QUESTIONS? IDEAS?HOW ARE YOU HARDENING YOUR CONFIG
MANAGEMENT?