RSA Conference 2012 Security Metrics

Session ID: Session Classification:

John D. Johnson John Deere

Presenting Metrics to the Executive Team

SEM-003 Intermediate

Questions:

§  How do we define security metrics? §  How are security metrics useful? §  Where do get the information, and how do we

turn it into something meaningful? §  How do we present security metrics to our

management? §  Building a security metrics program §  Group Discussion: What works for you?

2  

Metrics In Real Life…

3  

Measurements & Metrics

§  Performance metrics measure how well an organization performs §  Drives process improvements and demonstrates

value-add §  Metrics can show how we compare to our peers

§  Metrics can help us break out of the cycle that comes from relying on products from vendors to rescue us from new threats: Detect à Report à Prioritize à Remediate

4  

Security Metrics

§  Make security metrics more meaningful to stakeholders §  We need to learn to ask the right questions, if our

results are going to be meaningful §  The best metrics are SMART: Specific, Measurable,

Attainable, Repeatable & Time-Dependent §  This is an inherently difficult problem

§  What is meaningful to stakeholders? §  Can we make metrics more quantitative? §  What can we measure? §  What are our peers doing?

5  

Motivations

§  Various Motivations for Developing Metrics §  Regulations - Compliance §  Audits (both internal and external) §  Money (security is rarely a profit center) §  Responding to new threats §  Enabling new technology and business processes §  Awareness: Making executives aware of trends

§  Example Compliance Metrics: §  Manager sign-off on access controls §  A&A control artifacts §  Audit reports/findings (number, severity, BU) §  Exception reporting/tracking §  PCI Compliance status, dates

6  

Example Security Metrics §  Application Security

§  # Applications, % Critical Applications, Risk Assessment Coverage, Security Testing Coverage

§  Configuration Change Management §  Mean-Time to Complete Changes, % Changes w/Security Review, % Changes w/Security

Exceptions

§  Financial §  Infosec Budget as % of IT Budget, Infosec Budget Allocation

§  Incident Management §  Mean-Time to Incident Discovery, Incident Rate, % Incidents Detected by Controls, Mean-

Time Between Security Incidents, Mean-Time to Recovery

§  Patch Management §  Patch Policy Compliance, Patch Management Coverage, Mean-Time to Patch

§  Vulnerability Management §  Vulnerability Scan Coverage, % Systems w/o Known Severe Vulnerabilities, Mean-Time to

Mitigate Vulnerabilities, # Known Vulnerability Instances

7  

* Source: Center for Internet Security

Gathering Data

§  Data can be qualitative or quantitative §  Data can be coarse-grained or fine-grained §  Data can involve ordinal or cardinal numbers §  Less mature programs often have historical data to use

§  Coarse-grained, qualitative, requires interpretation §  Examples: Audit findings, incident reports, viruses…

§  More mature programs use multiple data sources §  Data from different sources can provide context, it is

important to consider the type of meta data that can be gathered to add value later on

8  

Modeling Data

§  Some good standard assessment frameworks can be used to provide a standard taxonomy for describing risk

§  Common frameworks allow data to be shared and compared between companies

§  Good models allow better analysis of complex risk scenarios

§  Examples: CAPEC, FAIR and VERIS §  Example of Industry Data: Verizon DBIR

9  

Operational, Tactical & Strategic Metrics

§  Operational plans lead to accomplishing tactical plans, which in turn lead to accomplishing strategic plans (which in turn are aligned with business objectives).

§  Tactical & Operational: IDS, Forensics, Help Desk Tickets, Time to Patch, Viruses Blocked, Support, Change Management…

§  Strategic Metrics: Overall Compliance, Compared to Baseline, Identifies Gaps in Program, Shows Business Alignment & Value

10  

Learn Where Others Succeed & Fail

11  

§  Successful security leaders overcome confirmation bias and compare notes more often with peers

§  Standards and frameworks help a company establish a baseline

§  Results need to be translated into a context that is relevant for your business

§  Be aware that executives may downplay the significance of industry data and feel their company is the exception to the rule

Good or Bad?

12  

Good or Bad?

13  

© Pedro Monteiro of the What Type blog

Good or Bad?

14  

Good or Bad?

15  

Good or Bad?

16  

Good or Bad?

17  

Good or Bad?

18  

Applied Security Visualization, Raffael Marty

Good or Bad?

19  

Applied Security Visualization, Raffael Marty

Good or Bad?

20  

http://www.pentest-standard.org

Clear, Concise, Contextual

21  

© 2010 Institute of Operational Risk

Presenting to Executives

22  

© 2010 Institute of Operational Risk

Security Metrics for Management

§  Find a way to add business value §  Meeting regulatory requirements §  Consolidation of tools, reduction of resources §  Demonstrate reduced costs by reduction in help desk cases §  Business leaders take the loss of IP seriously §  Have security seen as a business enabler. New technologies

come with risks, but they may also lead to new innovations and competitive advantage.

§  Explain it in language business leaders understand §  Make presentations clear & concise §  Avoid IT jargon §  Provide the information executives need to make informed

decisions

23  

Building a Security Metrics Program

§  Decide on your goals and objectives at the onset §  Long-term and short-term goals

§  Identify key metrics (SMART) to generate §  Will these be qualitative or quantitative? §  Will these be manual or automated? §  Will these be based on a standard framework, or vetted against peers, or use

some other model? §  Will these be tactical, operational, strategic or business metrics?

§  Establish a baseline and targets

§  Determine how best to present metrics in a consistent way, for audience and frequency

§  Get stakeholder buy-in and feedback; deliver balanced scorecard

§  Develop a process for continuous improvement

24  

References §  CAPEC, http://capec.mitre.org

§  Verizon DBIR, http://www.verizonbusiness.com/go/2011dbir

§  Verizon VERIS Framework, https://www2.icsalabs.com/veris/

§  FAIR Framework, http://fairwiki.riskmanagementinsight.com/

§  Center for Internet Security, Security Metrics, http://benchmarks.cisecurity.org/en-us/?route=downloads.metrics

§  Trustwave SpiderLabs Global Security Report, https://www.trustwave.com/GSR

§  Ponemon Institute, http://www.ponemon.org

§  Security Metrics: Replacing Fear, Uncertainty, and Doubt, Andrew Jaquith (2007)

§  Metrics and Methods for Security Risk Management, Carl Young (2010)

§  Security Metrics, A Beginner’s Guide, Caroline Wong (2011)

§  Applied Security Visualization, Raffael Marty (2008)

§  The Visual Display of Quantitative Information, Edward Tufte (2001)

25  

References §  New School Security Blog, http://newschoolsecurity.com/

§  SecurityMetrics.org, http://securitymetrics.org/

§  A Few Good Metrics, http://www.csoonline.com/read/070105/metrics.html

§  Measuring Security, Dan Geer, http://geer.tinho.net/measuringsecurity.tutorial.pdf

§  CIS Consensus Security Metrics v1.0.0, https://community.cisecurity.org/download/?redir=/metrics/CIS_Security_Metrics_v1.0.0.pdf

§  Performance Measurement Guide for Information Security, http://csrc.nist.gov/publications/nistpubs/800-55-Rev1/SP800-55-rev1.pdf

§  Directions in Security Metrics Research, http://csrc.nist.gov/publications/drafts/nistir-7564/Draft-NISTIR-7564.pdf

§  A Guide to Security Metrics, http://www.sans.org/reading_room/whitepapers/auditing/a_guide_to_security_metrics_55

§  Patch Management and the Need for Metrics, http://www.sans.org/reading_room/whitepapers/bestprac/1461.php

26  

References §  The Security Metrics Collection,

http://www.csoonline.com/article/455463/The_Security_Metrics_Collection

§  Implementing a Network Security Metrics Program, http://www.giac.org/certified_professionals/practicals/gsec/1641.php

§  Choosing the Right Metric, http://www.juiceanalytics.com/writing/choosing-rightmetric/

§  Web Metrics Demystified, http://www.kaushik.net/avinash/2007/12/webmetrics-demystified.html

§  Blogs about: Security Metrics, http://en.wordpress.com/tag/security-metrics/

§  Standardizing metrics and their presentation, http://www.unifiedcompliance.com/it_compliance/metrics/reporting_standards/standardizing_metrics_and_thei.html

§  Getting to a Useful Set of Security Metrics, http://www.cert.org/podcast/show/20080902kreitner.html

§  Dashboards by Example, http://www.enterprise-dashboard.com/

§  Excel Charting Tips, http://peltiertech.com/Excel/Charts/index.html

27  

Group Discussion

28