SESSION ID: AIR-W02 PREDICTING … · -CVSS Temporal-Remote Code Execution-Availability-Integrity-Confidentiality-Authentication-Access Complexity-Access Vector-Publication Date-Patch

SESSION ID:

#RSAC

Michael Roytman

PREDICTING EXPLOITABILITY -FORECASTS FOR VULNERABILITY MANAGEMENT

AIR-W02

Chief Data ScientistKenna Security@mroytman

#RSAC

“Prediction is very difficult, especially about the future.”

-Niels Bohr

#RSAC

3 Types of Data-Driven

3

#RSAC

THE PROBLEM

4

Too many vulnerabilities.How do we derive risk from vulnerability in a data-driven manner?

#RSAC

EXPLOITABILITY

5

1. RETROSPECTIVE2. REAL-TIME3. PREDICTIVE

#RSAC

EXPLOITABILITY

6

1. RETROSPECTIVE2. REAL-TIME3. PREDICTIVE

#RSAC

Retrospective Model: CVSS

7

Analyst InputVulnerability Management Programs Augmenting Data

Temporal Score Estimation

Vulnerability Researchers

#RSAC

EXPLOITABILITY

8

1. RETROSPECTIVE2. REAL-TIME3. PREDICTIVE

#RSAC

Real-Time - The Data

9

Vulnerability Scans (Qualys, Rapid7, Nessus, etc):• 7,000,000 Assets (desktops, servers, urls, ips, macaddresses)• 1,400,000,000 Vulnerabilities (unique asset/CVE pairs)

Exploit Events - Successful Exploitations• ReversingLabs’ backend metadata

• Hashes for each CVE• Number of found pieces of malware corresponding to each hash

• Alienvault Backdoor• “attempted exploits” correlated with open vulnerabilities

#RSAC

Attackers Are Fast

10

#RSAC

Positive Predictive Value of Remediating:

11

#RSAC

DATA OF FUTURE PAST

12

Q: “Of my current vulnerabilities, which ones should I remediate?”

A: Old ones with stable, weaponized exploits

#RSAC

FUTURE OF DATA PAST

13

Q: “A new vulnerability was just released. Do we scramble?”

A:

#RSAC

EXPLOITABILITY

14

1. RETROSPECTIVE2. REAL-TIME3. PREDICTIVE

#RSAC

Learning Machine Learning

15

#RSAC

The Future

17

•Classification: output is qualitative

•prediction:

“Will this vulnerability have an exploit written for it?”(== cause more risk later)

Enter: AWS ML

#RSAC

The Data

19

N = 81303

All CVE. Described By:1. National Vulnerability Database2. Common Platform Enumeration3. Occurrences in Kenna Scan DataLabelled as Exploit Available/Not:1. Exploit DB2. Metasploit3. D2 Elliot/Canvas4. Blackhat Exploit Kits

#RSAC

70% Training, 30% Evaluation Split

N = 81303

All Models:

20

L2 regularizer

1 gb

100 passes over the data

Receiver operating characteristics for comparisons

#RSAC

Predictive - The Expectations

21

Distribution is not uniform. 77% of dataset is not exploited1. Accuracy of 77% would be bad

Precision matters more than Recall1. No one would use this model absent actual exploit available data.2. False Negatives matter less than false positives - wasted effort

We are not modeling when something will be exploited, just IFCould be tomorrow or in 6 months. Re-run the model every day

#RSAC

Model 1: Baseline

22

-CVSS Base-CVSS Temporal-Remote Code Execution-Availability-Integrity-Confidentiality-Authentication-Access Complexity-Access Vector-Publication Date

LMGTFY:

Moar Simple?

Measuring Performance

#RSAC

Model 2: Patches

26

-CVSS Base-CVSS Temporal-Remote Code Execution-Availability-Integrity-Confidentiality-Authentication-Access Complexity-Access Vector-Publication Date-Patch Exists

#RSAC

Model 3: Affected Software

27

-CVSS Base-CVSS Temporal-Remote Code Execution-Availability-Integrity-Confidentiality-Authentication-Access Complexity-Access Vector-Publication Date-Patch Exists-Vendors-Products

#RSAC

Model 4: Words!

28

-CVSS Base-CVSS Temporal-Remote Code Execution-Availability-Integrity-Confidentiality-Authentication-Access Complexity-Access Vector-Publication Date-Patch Exists-Vendors-Products-Description, Ngrams 1-5

#RSAC

Model 5: Vulnerability Prevalence

29

-CVSS Base-CVSS Temporal-Remote Code Execution-Availability-Integrity-Confidentiality-Authentication-Access Complexity-Access Vector-Publication Date-Patch Exists-Vendors

-Products-Description, Ngrams 1-5-Vulnerability Prevalence-Number of References

#RSAC

Model 6: ”Somewhat Likely”

30

#RSAC

Model 6: ”Highly Likely”

31

#RSAC

Model 6: ”Most Likely”

32

#RSAC

33

-Track Predictions vs. Real Exploits

-Integrate 20+ BlackHat Exploit Kits - FP reduction?

-Find better vulnerability descriptions - mine advisories for content? FN reduction?

Future Work

-Predict Breaches, not Exploits

-Attempt Models by Vendor

#RSAC

34

Too many vulnerabilities.How do we derive risk from vulnerability in a data-driven manner?

PROBLEM

#RSAC

35

1. Gather data about known successfulattack paths2. Issue forecasts where data is lacking in order to predict new exploits3. Gather MORE data about known successful attack paths

SOLUTION

#RSAC

36

2. When Risk is Rare, Precision is Difficult

Takeaways1. Simple, Power Questions make Machine Learning Useful in Security

3. When Precision is Difficult, Be Smart about Tradeoffs

#RSAC

38

The Takeaway

____ Machine Learn!

#RSAC

Putting It All Together

39

Thank You for waking up so early for this!

@mroytman

www.kennasecurity.com