8_Bay - CSIAC Cyber Metrics

10/19/15

1

Cyber&Metrics&in&the&DoD

or

How&Do&We&Know&What&We&Don’t&Know?

John S. Bay, Ph.D.Executive Director

Things& People&Have&Asked&MeThings& People&Have&Asked&Me

• How&much&money&should&I&spend&this&year&on&cyber&

defense&technologies?

• How&many&attacks&has&your&firewall&repelled&this&

month?

• If&I&only&had&a&dollar&to&spend&on&cyber,&where&

should&I&spend&it?

• Why&is&cyber&research&such&a&slog?

211/12/14

Answers

(which& did&not& go&over&well)

Answers

(which& did&not& go&over&well)

• How&much&money&have&you&got?

• We&repelled&all&of&them&…&except&that&one&you&read&

about&in&the&paper

• Spend&your&dollar&on&upgrades

• Cyber&research&is&a&slog&because&there&is&no&physics&

theory&underlying&it&all,& liker&Maxwells’&Equations&

or&Newton’s&Laws

311/12/14

10/19/15

2

But&really&…&it&DEPENDSBut&really&…&it&DEPENDS

• The&“threat”&factor&is&common&in&cybersecurity,&

but&mostly&not&elsewhere

• …&&and&it&IS& true&that&there&is&no&useful&PHYSICS&

for&the&problem

411/12/14

DoD&Taxonomy&of& ThreatsDoD&Taxonomy&of& Threats

5

From:.Defense.Science.Board,.Resilient(Military(Systems(and(the(Advanced(Cyber(Threat,.January.2013

Tier Description

I Practitioners . who. rely. on. others . to. develop. the. malicious . code,. delivery. mechanisms,. and. executionstrategy. (use.known. exploits ).

II Practitioners . with. a.greater. depth. of. experience,. with. the. ability. to.develop. their. own. tools . (from.publically. known. vulnerabilities ).

III Practitioners . who. focus . on.the. discovery. and.use. of. unknown. malicious . code,. are. adept. at. installing.user. and. kernel. mode. root. kits ,. frequently. use.data. mining. tools ,. target. corporate. executives . akey.users . (government. and. industry). for. the. purpose. of.s tealing. personal. and. corporate. data. with. the.expressed. purpose. of. selling. the. information. to. other. criminal. elements .

IV Criminal. or. s tate. actors . who. are. organized,. highly. technical,. proficient,. well.funded. profess ionals .working. in.teams. to. discover. new. vulnerabilities . and.develop. exploits .

V State. actors . who. create. vulnerabilities . through. an.active. program. to. “influence”. commercial. products .and. services .during. des ign,. development. or. manufacturing,. or. with. the. ability. to. impactproducts . while. in.the. supply. chain. to. enable. exploitation. of. networks . and. systems.of. interest

VI States . with. the. ability. to. success fully. execute. full. spectrum. (cyber. capabilities . in.combination. withall.of. their. military. and.intelligence. capabilities ). operations . to. achieve. a.specific.outcome. in. political,military,. economic,. etc.. domains . and. apply.at. scale.

11/12/14

And&The& Corresponding& CriticalityAnd&The& Corresponding& Criticality

611/12/14

10/19/15

3

What&Might&the&COSTS& Be?What&Might&the&COSTS& Be?

711/12/14

So&Then,&What&to&Measure?So&Then,&What&to&Measure?

• Qualitative

– Capabilities

– Missions&lost

• Quantitative

– Performance

– Cost

• To&achieve

• Not&achieving

811/12/14

Capabilities&and&MaturityCapabilities&and&Maturity

911/12/14

10/19/15

4

Dashboard&ApproachDashboard&Approach

1011/12/14

“Stoplight&Chart”&Assessments“Stoplight&Chart”&Assessments

1111/12/14

See:. SPIDERS.JCTD

Costs& to&UsCosts& to&Us

• All&vulnerabilities&are&bugs&

• All&code&has&bugs

• Bugs&are&expensive

• Exploits&are&cheap&! the&“asymmetry”&problem

1211/12/14

10/19/15

5

MissionUAssurance& ApproachMissionUAssurance& Approach

• Helps& focus& attention

• Requires& a&“map”& o& the&

mission

• Implies& a&prioritization&

on&missions& (something&

loses)

• Requires& reconfigurable&

systems& and& networks

• Is&not& cheap

13

From:. .DUSD(I&E). Office,. HANDBOOKFor( SELFAASSESSING ( SECURITY( VULNERABILITIES( &(RISKS( of(INDUSTRIAL( CONTROL(SYSTEMSOn(DOD( INSTALLATIONS,.December. 201211/12/14

Just& Good& Enough& (Incremental)

Approach

Just& Good& Enough& (Incremental)

Approach

• How& long&would& our& red&team& take&to&penetrate&the&

system?

– An&empirical&measure,&at&best.

– Implies&a&canonical&red&team

14

prob(first&

vulnerability& is&

discovered)

time

Bad&code

Better&code

Gamma(distribution?

11/12/14

The&Accountability&ApproachThe&Accountability&Approach

• NIST&800U53&guidelines

• The&“did&we&do&everything&we&know&how&to&do”&approach

15

From:.NIST.Special.Publication.800]53,.rev..4,.Security(and(Privacy(Controls(forFederal(Information(Systems(and(Organizations,.April.201311/12/14

10/19/15

6

Conclusions:&Which&is&Best?Conclusions:&Which&is&Best?

• None& of&them.& They& service&somewhat& orthogonal&

purposes.

– But&they&can&provide&applesUtoUapples&comparisons

• Can& they&answer& the&Generals’& questions?

– No

– …&except&maybe&the&one&about&the&firewall

– There& is&CERTAINLY&no&satisfactory&“physics”&to&guide&anybody

• Cyber&Metrics& is&still&an& extremely&important& and& highU

priority&problem& for&OSD!

1611/12/14