Blending Automated and Manual Testing

Blending  Automated  and  Manual  Testing

Making  Application  Vulnerability  Management  Pay  Dividends

My  Background• Dan  Cornell,  founder  and  CTO  of  Denim  

Group

• Software  developer  by  background  (Java,  .NET,  etc)

• OWASP  San  Antonio@danielcornell

My  Background• Steve  Springett,  Application  Security  

Architect  for  Axway

• Software  developer  by  background

• Leader  of  OWASP  Dependency-­‐Track

• Contributor  to  OWASP  Dependency-­‐Check@stevespringett

Goal:  Continuous  Security

• Prerequisites– Standardization– Continuous  Integration– Continuous  Delivery

• Compliments– Continuous  Acceptance

Standardization

• All  projects  use  same  build  system• All  projects  built  the  same  way• Automated  onboarding  for  new  projects• Per-­‐project  build  expertise  not  required

MetricsArtifacts

Continuous  Integration

Continuous   Integration  Factory

Source  Code  (SCM)

Deliverables

Continuous  Delivery

Continuous   Delivery  Factory

Artifacts

Security  Metrics

Continuous  Security

Continuous   Security  Factory

Source  Code  (SCM) Deliverables

Automated  Security  Metrics

• Static  Analysis  Findings• Dynamic  Analysis  Findings• Component  Analysis  Findings• Attack  Surface  Analysis  Findings

Continuous  Security  Pipe

Jenkins  CI ThreadFix Defect  TrackerSCM

False  Positive

TargetApplication

12

ThreadFixAccelerate  Software  Remediation

ThreadFix   is  a  software  vulnerability   aggregation   and  management  system  that  helps  organizations   aggregate   vulnerability   data,  generate   virtual  patches,  and  interact  with  software  defect  tracking  systems.

ThreadFix

• Open  Source  (MPL)  application  vulnerability  management  platform

• Create  a  consolidated  view  of  your  applications  and  vulnerabilities

• Prioritize  application  risk  decisions  based  on  data

• Translate  vulnerabilities  to  developers  in  the  tools  they  are  already  using

ThreadFix Community  Edition• Main  ThreadFixwebsite:  www.threadfix.org

– General  information,  downloads

• ThreadFix GitHub site:  www.github.com/denimgroup/threadfix– Code,  issue  tracking

• ThreadFix GitHubwiki:  https://github.com/denimgroup/threadfix/wiki– Project  documentation

• ThreadFix Google  Group:  https://groups.google.com/forum/?fromgroups#!forum/threadfix– Community  support,  general  discussion

Vulnerability  AggregationAutomated

Automated Manual

Access  to  Vulnerability  Data

• Tradeoffs– The  more  places  the  vulnerability  data  lives,  the  more  likely  a  compromise

–Withholding  information  from  people  who  need  it  makes  remediation  more  challenging

Managing  All  Vulnerability  Data

• Manual  activities– Penetration  Testing– Code  Reviews

• 3rd Party  Data  Sources– Customer-­‐performed  Testing– External  auditor-­‐performed  Results

SSVL  and  Manual  Results

• SSVL  Data  Format:– https://github.com/owasp/ssvl

• SSVL  Conversion  Tool:– https://github.com/denimgroup/threadfix/wiki/SSVL-­‐Converter

RESTful API  to  Vulnerability  Data

CustomR&D  Monitoring

Dashboard

CustomDashboards

Key  Performance  Indicators

• Don’t  go  overboard  – Use  only  what  is  needed• Progress  and  velocity• Per  team  comparison• Min/max/avg time  to  close  per  severity• By  CWE

Lessons  Learned

• Always  automate  static  analysis• Always  automate  attack  surface  analysis• Always  automate  component  analysis• Always  automate  dynamic  analysis• Always  perform  manual  dynamic  analysis• Use  native  tools  &  workflow  for  static  analysis

Lessons  Learned

• Provide  as  much  visibility  as  possible– Varying  degrees  of  detail– Multiple  delivery  vehicles

• Set  clear  pass/fail  criteria  for  Security  Bars– Provide  custom  dashboard  to  provide  status  and  advanced  warning

Additional  Advice

• Automation  is  not  better  than  manual– It’s  faster  and  more  efficient– Both  are  necessary

• Don’t  forget  manual  assessments– Threat  Modeling– Secure  Design/Architecture  and  Code  Review– Penetration  Testing

Finally

• Vulnerabilities  in  CI  /  CD  /  CS  Infrastructure– Threat  Model– Secure  Architecture  Review– Patch  Management  – Configuration  Management– Key  Management– Always  use  TLS

Q  &  A