Fix What Matters: !
Why CVSS Sucks And How To
Do Better
Once Jailbroke an Iphone 3G
Michael Roytman
Proud Owner of Remote Controlled AirplaneRecently a Naive Grad Student
Data Scientist, Risk I/ODoes Not Wake Up Before 11 CST
qualifications:
15x better than CVSS
Probability A Vuln Having Property X Has Observed Breaches
Random Vuln
CVSS 10
Exploit DB
Metasploit
MSP+EDB
0.0 0.1 0.2 0.2 0.3
PART 1: !
YOU SUCK AT YOUR JOB
!
(and don’t even know it yet)
Why Are We Here?
Empirical Failures of CVSSProper Remediation Frameworks
CVSS SUCKS
Analytical Failures of CVSS
(+Data Driven Alternatives)
Remove the Threat
RemediationAccept the Risk
Repair the Vulnerability
C(ommon) V(ulnerability) S(coring) S(ystem)
“CVSS is designed to rank information system vulnerabilities”
Exploitability/Temporal (Likelihood)
Impact/Environmental (Severity)
The Good: Open, Standardized Scores
“It is a capital mistake to theorize before one has data.
!
!
!
Insensibly, one begins to twist facts to suit theories, instead of
theories to suit facts.”
FAIL 1: A Priori Modeling“Following up my previous email, I have tweaked my equation to try to achieve better separation between adjacent scores and to have CCC have a perfect (storm) 10 score...There is probably a way to optimize the problem numerically, but doing trial and error gives one plausible set of parameters...except that the scores of 9.21 and 9.54 are still too close together. I can adjust x.3 and x.7 to get a better separation . . .”
F2: Data FundamentalismDon’t Ignore What a Vulnerability Is: Creation Bias http://blog.risk.io/2013/04/data-fundamentalism/ !
Jerico/Sushidude @ BlackHat https://www.blackhat.com/us-13/briefings.html#Martin
!
Luca Allodi - CVSS DDOS http://disi.unitn.it/~allodi/allodi-12-badgers.pdf
F2: Data FundamentalismSince 2006 Vulnerabilities have declined by 26 percent.” http://csrc.nist.gov/groups/SNS/rbac/documents/vulnerability-trends10.pdf !
!
The total number of vulnerabilities in 2013 is up 16 percent so far when compared to what we saw in the same time period in 2012. ” http://www.symantec.com/content/en/us/enterprise/other_resources/b-intelligence_report_06-2013.en-us.pdf
F3: Stochastic Ignorance
Attackers Change Tactics Daily
F3: Stochastic Ignorance
Empirical Failures of CVSSObjective: Remediate the riskiest vulnerabilities
Constraint: Can’t measure impact/priority
Need:
MOAR DATA!!!
Repair the Vulnerability
I Love It When You Call Me Big Data50,000,000 Live Vulnerabilities
1,500,000 Assets
2,000 Organizations
I Love It When You Call Me Big Data
3,000,000 Breaches
Baseline AllthethingsProbability (You Will Be Breached On A Particular Open Vulnerability)?
=(Open Vulnerabilities | Breaches Occurred On Their CVE) /(Total Open Vulnerabilities)
2%
Probability A Vuln Having Property X Has Observed Breaches
RANDOM VULN
CVSS 10
CVSS 9
CVSS 8
CVSS 6
CVSS 7
CVSS 5
CVSS 4
Has Patch
0.000 0.010 0.020 0.030 0.040
PART 2: !
FIX WHAT MATTERS
Empirical Failures of CVSSObjective: Remediate the riskiest vulnerabilities
Constraint: Can’t measure impact/priority
Need:
MOAR DATA!!!
Proper Framework
Know which vulnerabilities put you most at risk.
Uh, Sports?
Opposing Teams, Specific Players
Gameplay
Scouting Reports, Gametape
Roster, Player Skills
Learning from Losing
InfoSec?
Defend Like You’ve Done It Before
Groups, Motivations
Exploits
Vulnerability Definitions
Asset Topology, Actual Vulns on System
Learning from Breaches
Work With What You’ve Got:
Akamai, Safenet
ExploitDB, Metasploit
NVD, MITRE
Alternatives
Probability A Vuln Having Property X Has Observed Breaches
Random Vuln
CVSS 10
Exploit DB
Metasploit
MSP+EDB
0.0 0.1 0.2 0.2 0.3
Be Better Than The Gap
I Love It When You Call Me Big Data
!
Spray and Pray => 2% !
CVSS 10 => 4% !
Metasploit + ExploitDB => 30%
Holler!www.risk.io@mroytman
