Metrics evolution breakfast edition

YOUR SPEAKER – • 2014 HEAD OF INFORMATION SECURITY – WORLDLINE (ATOS GROUP)

• 2014 CISO LEVEL SECURITY, RISK & COMPLIANCE CONSULTANCY ACROSS EUROPE

• 2013 PCIDSS COMPLIANCE AT WALMART FOR ASDA & GEORGE (LEVEL ONE MERCHANT)

• 2011 - 2013 PCIDSS COMPLIANCE MANCHESTER AIRPORTS GROUP (LEVEL THREE MERCHANT)

• 2006-2011 PCIDSS COMPLIANCE HOMELOAN MANAGEMENT LIMITED (LEVEL ONE SERVICE PROVIDER)

• 2006 ECOMMERCE SECURITY– THOMAS COOK SCHEDULED BUSINESS

EXEC SUMMARY

• QUICK LOOK BACK OVER LAST TEN YEARS

• QUICK LOOK AT MY FAVOURITE BACKGROUND READING

• AT A FORK IN THE ROAD - KPI V SECVIZ

Looking Back…

35

INTRODUCTION (FROM 2003)

• IT SECURITY METRICS PROVIDE A PRACTICAL APPROACH TO MEASURING

INFORMATION SECURITY.

• EVALUATING SECURITY AT THE SYSTEM LEVEL, IT SECURITY METRICS ARE TOOLS THAT

FACILITATE DECISION MAKING AND ACCOUNTABILITY THROUGH COLLECTION,

ANALYSIS, AND REPORTING OF RELEVANT PERFORMANCE DATA.

34

OLD SCHOOL METRICS TUTORIAL

• DAN GEER

• 426 PAGES

• 2007

• GEER - HTTP://GEER.TINHO.NET/MEASURINGSECURITY.TUTORIAL.PDF

33

NIST SPECIAL PAPER

• NIST SP800 SERIES

• TECH STANDARDS

• US GOV

• 2008

32

CIS CONSENSUS ON IS METRICS

• HTTPS://BENCHMARKS.CISECURITY.ORG/DOWNLOADS/METRICS/

31

GOOD BOOKS • THERE ARE PLENTY OF BOOKS OUT THERE – NEW ONES BEING PUBLISHED ALL THE TIME

• CAROLYN WONG -2011

• LANCE HAYDEN -2010

30

WWW.SANS.ORG READING ROOM

• FAVOURITE DASHBOARD PAPER

• 2010

29

SANS PAPER • EXAMPLES

28

Know your enemy

27

HOW TO LIE WITH STATISTICS 1954

• DARRELL HUFF

26

HUFF HTLWS

• 2009

• AND NEW

• HTTP://CSEWEB.UCSD.EDU/~RICKO/CSE3/LIE_WITH_STATISTICS.PDF

25

24

BLACK HAT 2013 - TALK

• STEVE CHRISTIE

• BRIAN MARTIN

23

BLACK HAT TALK – UPDATE 2015

• HTTP://BLOG.OSVDB.ORG/CATEGORY/VULNERABILITY-STATISTICS/

22

Sample

21

Why Vulnerability Stats Suck • Stats are presented without understanding the limits of the data

• Even if explanations are provided, correlation is confused with causation:

20

Talking Points

• Defining Bias

• Researcher Bias

• Vendor Bias

• VDB Bias

• Bad Stats

• Good(ish) Stats

• Conclusion

19

Disease Research: Epidemiology vs. Vulnerability Research

Epidemiology Vulnerability Research

Goal Improve the public health SAVE ALL THE THINGZ ON THA INTERWEBZ! * (attention whoring)

Objects of Study People/Diseases Software/Vulnerabilities

Populations Groups of people Groups of vulnerabilities (as seen in multi-vuln disclosures)

Measurement Devices (Tools of the Trade)

Blood pressure monitors, thermometers, lab tests, observation

Automated code scanners w/high FP/FN rates, fuzzers, coffee-fueled malcontents staring at code at 3 AM

Publication Requirements

Refereed journals with peer review Ability to send email

Sampling Methods Using industry established methodologies and formal documentation.

Using wildly erratic methodologies, no standards for documentation or disclosure

* Goal not shared by all researchers. Please to be rolling with this, kthxbye

18

The Shocking Claim

• Bias and statistics in vulnerability research are far worse than it is in other disciplines

• At least people don’t die (yet?), but still use vulnerable equipment:

– SCADA

– Airplanes

– Automobiles

– Medical Devices

– Oh my…

17

KPI / KRI

16

SECURITY EFFORT / PERFORMANCE

• WITH KRI

15

SECURITY EFFORT / PERFORMANCE

• WITH KPI

14

lets get visual

13

THE BOOKS • APPLIED SECURITY VISUALIZATION DATA-DRIVEN SECURITY

12

SECVIZ AND AFTERGLOW

• SITE WWW.SECVIZ.ORG AND TOOL AFTERGLOW (PERL)

11

DATA DRIVEN SECURITY BLOG/PODCAST

10

INDEPENDENT REVIEW OF DDS • HTTP://HOLISTICINFOSEC.ORG/TOOLSMITH/PDF/SEPTEMBER2014.PDF

9

THE OTHER DATA DRIVEN SECURITY • HTTPS://WWW.TRUSTWORTHYINTERNET.ORG/DATA-DRIVEN-SECURITY/

8

PATERVA MALTEGO TRANSFORM TOOL • MALWARE INVESTIGATIONS EXAMPLES

2010 2013 2014

7

CROWDSTRIKE – DEEP PANDA

6

OPENDNS VIDEO AND GRAPHICS • HTTPS://WWW.OPENDNS.COM/2013

• HTTPS://WWW.OPENDNS.COM/2014

5

OPENDNS FREE TOOLS

4

REALTIME MAPS – PURE MARKETING • HTTP://CYBERMAP.KASPERSKY.COM/ HTTP://HTTP://MAP.IPVIKING.COM/

3

OTHER RT MAPS • HTTP://WWW.THREATMETRIX.COM/THREATMETRIX-LABS/WEB-FRAUD-MAP/

• HTTP://WWW.FIREEYE.COM/CYBER-MAP/THREAT-MAP.HTML

• HTTP://WWW.DIGITALATTACKMAP.COM/#ANIM=1&COLOR=0&COUNTRY=ALL&TIME=16352&VIEW=MAP

• HTTP://WWW.SICHERHEITSTACHO.EU/

• HTTP://MASTDB3.MCAFEE.COM/VIRUSMAP3.ASP?NAME=VIRUSMAP&B=IE&LEFT=-162.96&BOTTOM=13.2&RIGHT=-

42.96&TOP=73.2&LANG=EN&OVB=2&FT=JPEG&OCM=1&VIEWBY=2&TRACK=4&PERIOD=1&CHOOSEMAP=1&CMD=ZOO

MIN

• HTTP://CERT.EUROPA.EU/BIGSCREENMAP/

2

THREATBUTT

• CYBER POMPEII

1

Time is precious, thank you for yours

https://uk.linkedin.com/in/jmck4cybersecurity