OpenStack Security CI/CD Way

OpenStack Security CI/CD Way

Jim FreemanMichael Xin

2

Jim FreemanDirector of Security Engineering

[email protected]

Michael Xin Manager of Security Engineering

[email protected]

It is all about SoftwareDevelopment

Testing Security

Deployment

Function Performance Security

Quality

Software Development Methodologies

Waterfall Methodology

Agile Development Methodology

Continuous Integration/Continuous Deployment (CI/CD)

6www.rackspace.com

Extensive Planning

Defined Scope

Better Design

7www.rackspace.com

BetterEngagement

Predictable Delivery

Improved Quality

8www.rackspace.com 8www.rackspace.com

Less Defects Fast Delivery Better Quality

9www.rackspace.com

Limited Resources

Priority Issue

Test Process

10www.rackspace.com

DevelopersVersion Control Server

Continuous Integration

Server

Configure

Static Analysis / Security

Unit/functional/ Security/

tests

Report

Report

Deploy

Smoke /Security/

Performance tests

commits triggers runs

runs

runs

logs

logs

logs

FAIL

Successlogs

FAIL

Success

•Reduce test time from weeks to hours

•Security defect fix time reduced from weeks to days

•Better security testing–Repeatable

–Consistent

–Auditable

•Build great working relationships

CI/CD Security Engineering Advantages

•Reduce test time from weeks to hours

•Security defect fix time reduced from weeks to days

•Better security testing–Repeatable

–Consistent

–Auditable

•Build great working relationships

CI/CD Security Engineering Advantages

Test Time: Weeks -> Days

Defect Fix time: Weeks-> Days

Better Security Tests

Test Time: Months -> Weeks

•Reduce test time from weeks to hours

•Security defect fix time reduced from weeks to days

•Better security testing–Repeatable

–Consistent

–Auditable

•Build great working relationships

CI/CD Security Engineering Advantages

Test Time: Weeks -> Days

Defect Fix time: Weeks-> Days

Better Security Tests

Defect Fix time: Weeks-> Days

Repeatable

Measurable

Auditable

Automation Efforts

Different CI/CD Pipelines

Mindset Change

How to integrate security into CI/CD pipeline?

What should we automate?

Security Code Review

API Security Tests

Infrastructure Test

NO PYTHON

Bandit a framework for performing security analysis of Python source code!

https://wiki.openstack.org/wiki/Security/Projects/Bandit

OpenStack Security Group

>> Issue: subprocess call without a subshell.

Severity: Low Confidence: High

Location: ./solum/worker/handlers/shell.py:494

493 try:

494 runtest = subprocess.Popen(command, env=user_env,

495 stdout=subprocess.PIPE)

496 returncode = runtest.wait()

>> Issue: Use of random is not suitable for security/cryptographic purposes.

Severity: Low Confidence: High

Location: ./solum/worker/handlers/shell.py:141

140 else:

141 str_assem = (''.join(random.choice(string.ascii_uppercase)

142 for i in range(20)))

143 user_env['ASSEMBLY_ID'] = str_assem

Customize the Configuration File: bandit.yaml

# optional: plugins discovery name pattern

plugin_name_pattern: '*.py’

exclude_dirs:

- '/tests/’

ShellInjection:

include:

- subprocess_popen_with_shell_equals_true

- start_process_with_no_shell

exclude:

SqlInjection:

include:

- hardcoded_sql_expressions

Extend Bandit using plugins

@takes_config('shell_injection')

@checks('Call')

def subprocess_popen_with_shell_equals_true(context, config):

if config and context.call_function_name_qual in config['subprocess']:

if context.check_call_arg_value('shell', 'True'):

return bandit.Issue(

severity=bandit.HIGH,

confidence=bandit.HIGH,

text="subprocess call with shell=True identified, security "

"issue. %s" % context.call_args_string

)

27www.rackspace.com

Commercial automatic Restful API scanner is limited

29www.rackspace.com

Quality Engineers

QE Framework

QE Test Codes

@tags("authorization", "security") def test_get_network_of_other_user(self): resp = self.one_network_client.get_network(self.two_network_id) assert resp.status_code != 200

@tags("authorization", "security") def test_update_network_of_other_user(self): resp = self.one_network_client.update_network(self.two_network_id, name="newname") assert resp.status_code != 200

32www.rackspace.com

POST /v2.0/subnets HTTP/1.1User-Agent: curl/7.30.0Host: xxx.xxx.xxx.xxxContent-Type: application/jsonAccept: application/jsonContent-Length: 189

{"subnet": {"network_id": "fc795965-cdad-40b5-8e7b-73ee174a9451", "name": "Sectest", "cidr": "11.168.200.0/24", "ip_version": 4, "dns_nameservers": ["11111111111111111111111111111111111"]}}

HTTP/1.1 503 Service Unavailable

Via: 1.1 Repose (Repose/2.12)

Content-Length: 0

Server: Jetty(8.0.y.z-SNAPSHOT)

CVE-2014-7821 (http://lists.openstack.org/pipermail/openstack-announce/2014-November/

000303.html )

CI/CD Evolve

Automate Contribute

Lessons Learned

CI/CD Opportunities

Automation Bandit

Collaboration

Questions?

[email protected]@rackspace.com

WE’RE HIRING!

bit.ly/RackerTalent

Expo Hall Booth P-11Python OpenStack EngineersC, C++ Linux Systems EngineersRuby DevOps EngineersJava Frontend & Backend Developers

C#, .NET Software Developer in TestJavaScript, CSS, HTML iOS/Android Development

Twisted, Backhone Data ScientistAngular.JS, Ember.js, Node.js Field Sales Specialist

Restful/JSON/XML Strategic Account ExecutiveClosure, Scala, Erlang

Hadoop, MongoDB, MySQLSolution Architect Data Visualization