Test and Evaluation
of Cyber Systems
Aug 18, 2015
Arlington, VA
G.A. (Fred) Wright, PhD
404.407.7296
Cell: 404.840.7652
Contact Information
• Build intuition related to cyber security
technology, risks, and methodologies
• Investigate systems approaches, threat, risk
evaluation, and countermeasures
• Consider challenges and approaches of test and
evaluation (T&E) of cyber systems
Objectives and Themes
• Cyberspace and Cyber Systems
• Threats
• Definitions
• Business / Mission Assurance
• Information Technologies
• T&E Challenges
• Metrics and measures
• Planning
• Configuration / test execution
• Data Reduction and Analysis
• Technologies for Cyber Testing
Outline
Introduction to Cyber Systems
• Introduction to Cyberspace,
• Understanding the threat
• Cyber security definitions
• Business enterprise view
Cyber is such a perfect prefix. Because nobody
has any idea what it means, it can be grafted
onto any old word to make it seem new, cool --
and therefore strange, spooky.
New Yorker Magazine, Dec. 23, 1996
Reference: Wikipedia - Information Age - A Visualization of the various routes through a portion of the Internet.
All I knew about the word "cyberspace" when I
coined it, was that it seemed like an effective
buzzword. It seemed evocative and essentially
meaningless. It was suggestive of something,
but had no real semantic meaning, even for me,
as I saw it emerge on the page.
William Gibson
Current State, Unattributed Quotes
• “The state of cyber security today is a complete failure…If you haven’t been hacked you have nothing of interest to steal”
• “fundamental trust models in cyberspace are broken; there is no technology out there today that reflects trust; 100 years from now we will realize we were in a lawless state”
• “why do we lack systems understanding, holistic design principles, risk management, and training in our enterprise systems?”
• “we are our worst enemies…the problem is too huge…we cannot conceptualize it, cannot worry about it”
• “it’s going to take a ‘BP oil spill of data’ event to wake us up”
Agenda
• Introduction to Cyberspace
• Understanding the threat
• Security definitions
• Business Enterprise View
Current State is Rapidly Evolving & Expanding
• Hacker (1960’s)
• A person who enjoys exploring the details of programmable systems and stretching their capabilities
• “WarGames” (1983)
• A young hacker starts the countdown to World War 3.
• Computer Viruses (1980’s)
• Tool era - Self-replication & connectivity
• Hacktivism (1990’s)
• WANK Worm … to Anonymous & Lulz (2011)
• Cyber Criminals (2000’s)
• Financial theft, illicit trade
• Cyber Espionage (last decade)
• Characterized by persistence
• Cyber Kinetic Attacks (emerging)
• Primarily nation-state based, target physical systems
RQ-170 “Capture”
Current State is Rapidly Evolving
• Remarkable change in attack motivation from our IT Systems to our Enterprises
• Around 2005, saw attacks shift from individual IT systems to commercial enterprises
• Unprecedented transfer of wealth, not just IP but also enterprise strategies
• Organized crime and nation-state involvement
• Key threat shift: preparation and patience
• Not typical hacking – normal IT tradecraft used, but the technology is mainstream
• Espionage: reconnaissance, exfiltration, exploitation, profit
• New paradigms – “we have no idea what’s out there”
Hacking
• In computer security and everyday language, a hacker is someone who breaks into computers and computer networks
• Hackers may be multiple motivations, including profit, protest, or because of the challenge
• The subculture that has evolved around hackers is often referred to as the computer underground but it is now a somewhat open community
• Hacking is not necessarily bad
Reconn.
Scanning
Gaining Access
Maintaining Access
Covering Tracks
Generic Buffer Overflow “Exploit”
Buffer
Instruction Pointer
Malicious Code
• Target Credit Card
Info Incident
Recent Case
Target Point-of-Sale (PoS) breach:What do we know now?
Dell SecureWorks:“Inside a Targeted Point-of-SaleData Breach”
• Threat indicators reported twice by sensors, several weeks apart, before exfiltration began
• Began with Spear Phishing attack on Target’s HVAC provider
• Multi-step process to gain access to PoS network
• Multi-step process to aggregate and exfil data
• Limited information on how threat moved laterally within the network to PoS network
Agenda
• Introduction to Cyberspace
• Understanding the threat
• Cyber Security definitions
• Business enterprise view
What’s a Cyber System?
• Computer + Software + Internet = Cyber System?
• Local/Private Networks?
• Mobile/wireless- GSM, 3G, 4G?
• Combinations of these?
From US Air Force Brief on Cyberspace
Cyberspace/security Regimes
1: Physical
2: Data Link
3: Network
4: Transport
5: Session
6: Presentation
7: Application
Open Systems Interconnect
(OSI) Reference Model
Physical Network
Logical Network
Social/User Network
Multiple disciplines in
a complex system of
systems
8: User
Typical Cyber Attacks: Upper layers
1: Physical
2: Data Link
3: Network
4: Transport
5: Session
6: Presentation
7: Application
Cyber attacks occur at all layers; but attacks (on the internet)are prevalent at the application layer
Typical Cyber Attacks: Upper layers
1: Physical
2: Data Link
3: Network
4: Transport
5: Session
6: Presentation
7: Application
Many cyber attacks enter through the application layer with a goal of controlling computers, collectingdata, or inserting data
Typical Electronic Attack (EA): Lower layers
1: Physical
2: Data Link
3: Network
4: Transport
5: Session
6: Presentation
7: Application
Electronic attacks enter through the physical layer with a goal of disrupting or deceiving
Typical Electronic Attack (EA): Lower Layers
1: Physical
2: Data Link
3: Network
4: Transport
5: Session
6: Presentation
7: Application
Electronic Attack
Battlespace Components
C2 Center
Air Defenses
Cyber System
Any device or system participating in a local or
global network of interdependent information
technology infrastructures, telecommunications
networks, and computer processing systems
What is Cyber Security?
Computer security - protection of information and property from theft, corruption, or natural disaster, while allowing the information and property to remain accessible and productive to its intended users.
Reference: http://en.wikipedia.org/wiki/Computer_security, http://en.wikipedia.org/wiki/Information_security, http://en.wikipedia.org/wiki/Network_security, http://www.merriam-webster.com/dictionary/cybersecurity
Information security - protecting information and information systems from unauthorized access, use, disclosure, disruption, modification, perusal, inspection, recording or destruction.
Cyber security - measures taken to protect a computer or computer system (as on the Internet) against unauthorized access or attack.
Network security - consists of the provisions and policies adopted by the network administrator to prevent and monitor unauthorized access, misuse, modification, or denial of the computer network and network-accessible resources
Information Assurance (IA)
• Measures taken to protect and defend sensitive
information from an adversaries efforts to deny, destroy,
degrade or disrupt information or information systems.
• Measures taken to ensure that information is available,
reliable, defendable, and verifiable.
• Measures taken to ensure that information and
information systems implement requisite protection,
detection, and reaction capabilities.
IA Model is Risk and Threat-Based
Common Criteria for Information Technology Security Evaluationhttp://www.commoncriteriaportal.org/
Joint Pub 3-12
Cyberspace operations: The employment of cyberspace capabilities where the primary purpose is to achieve objectives in or through cyberspace
Cyberspace Operation
• Information Assurance – Making systems defendable
• Cyberspace Operations
• Enterprise/Network Operations (NetOps) – Running and managing the systems
• DoD Global Information Grid Operations (DGO)
• Defensive Cyberspace Operations (DCO) – Monitoring and responding to incidents (e.g., attacks, intrusions…)
• Cyberspace ISR – Discovering information (gathering intelligence)
• Offensive Cyber Operations (OCO) – Attacking systems
Cyberspace Operations Definitions
ISR in Cyberspace: Impacts of Attacks/Defense?
• Denial
• Disruption
• Degradation
• Destruction
• Deception
Issues with Information and Operations
• Cyberspace Attack: Cyberspace actions that create various direct denial effects in
cyberspace (i.e., degradation, disruption, or destruction) and manipulation that leads to denial
that is hidden or that manifests in the physical domains. These specific actions are:
• (a) Deny. To degrade, disrupt, or destroy access to, operation of; or availability of a target by
a specified level for a specified time. Denial prevents adversary use of resources.
• l. Degrade. To deny access (a function of amount) to, or operation of, a target to a level represented as
a percentage of capacity. Level of degradation must be specified. If a specific time is required, it can
be specified.
• 2. Disrupt. To completely but temporarily deny (a function of time) access to, or operation of, a target
for a period represented as a function of time. A desired start and stop time are normally specified.
Disruption can be considered a special case of degradation where the degradation level selected is
100 percent.
• 3. Destroy. To permanently, completely, and irreparably deny (time and amount are both maximized)
access to, or operation of, a target.
• (b) Manipulate. To control or change information, information systems, and for networks in a
manner that supports the commander’s objectives, including deception, decoying,
conditioning, spoofing, falsification, etc. Manipulation uses an adversary’s information
resources for friendly purposes.
From JP 3-12
• Computer Network Attack (CNA): Includes actions taken via computer networks to disrupt, deny, degrade, or destroy the information within computers and computer networks and/or the computers/networks themselves.
• Computer Network Defense (CND): Includes actions taken via computer networks to protect, monitor, analyze, detect, and respond to network attacks, intrusions, disruptions, or other unauthorized actions that would compromise or cripple defense information systems and networks.
• Computer Network Exploitation (CNE): Includes enabling actions and intelligence collection via computer networks that exploit data gathered from target or enemy information systems or networks.
Previous Terms (still widely used)
What is Cyber Warfare?
Network Centric Warfare is not Cyber Warfare
Network Centric Warfare is using cyber technology (computers/networks) to improve performance in Land/Sea/Air/Space domains
Tightening the (Observe, orient, decide, act (OODA) loop)
Electronic Warfare (EW) is not Cyber Warfare BUT… there is significant overlap
EW can have provide cyber warfare effects
And vice versa
In Cyber Warfare, the targets are in Cyberspace!
Whether defended targets or adversary targets
Cyber warfare might be considered a subset of Network centric warfare
When Information Becomes Digital Data
Concerned with:
• Data Access
• Data Structure
• Data NetworkSensitive
Data
ControlledData
PersonnelData
OperationalData
Confidentiality Integrity
Availability
Other Networks
PacketSwitch
Gateway
FileServer
Bridge
C-I-A Concerns: Access to the Data
• Confidentiality
• No disclosure
• Only those who need to see data should see it
• Integrity
• No alteration
• Only those allowed to alter data can modify it
• Availability
• No interruption
• Everyone who needs to access data can access it
Confidentiality Integrity
Availability
Cyberspace Visualization
Science concerned with the presentation of data
Understanding and extraction of information from data
Which techniques work best for Cyberspace?
Agenda
• Introduction to Cyberspace
• Understanding the threat
• Security definitions
• Business Enterprise View
• Objectives of IT enterprise is to support
“business” strategy and processes
• Business processes often utilize numerous IT
components
Business and Enterprise View
What processes and functions?
Image: http://jeffsutherland.org/oopsla97/hung.html
Co
nti
nu
um
of
Net
op
s Fu
nct
ion
s
Enterprise Management(Business Process or Mission Assurance)
EnterpriseAnd Network Management(Service Management)
Network/DeviceManagement
• Business-centric
• Service-centric
• Application-centric
• Infrastructure-centric
Incr
easi
ng
Mat
uri
ty o
f P
rod
uct
s &
Pro
cess
es
Trade Space: Progression and Complexity
• 26 processes for devising and managing IT services
• Focus on providing services levels (service level agreements (SLAs))
• Primarily addresses network infrastructure and telecom but SW management processes added
• Five parts, includes security processes in each:
• 1. Service Strategy
• 2. Service Design
• 3. Service Transition
• 4. Service Operation
• 5. Continual Service Improvement
Example Systems Engineering Process: IT Infrastructure Library (ITIL)
DoD Cyber Commands
USSTRATCOM
USCYBERCOM
24th AF 10th Fleet ARCYBER MARCYBER
AF Space
Command
Regional
COCOMs
Defense Information
Systems Agency
National Security
Agency
• IT services provide infrastructure for business processes
• Business performance metrics provide a basis for assessing security issues and incidents
Business management, IT enterprise management, and security must come together
Business Process and Metrics Provide Context
• Cyberspace and Cyber Systems
• Threats
• Definitions
• Business / Mission Assurance
• Information Technologies
• T&E Challenges
• Metrics and measures
• Planning
• Configuration / test execution
• Data Reduction and Analysis
• Technologies for Cyber Testing
Outline
Overarching Test Issues forNetwork-centric Systems
• Lack of operators
• Large number of nodes/Equipment
• Complex scenarios
• Distributed systems
• Variety of information exchanges
• Measuring effectiveness
Constructing the test environment!
Emerging/Evolving Complexity in Testing C2/C4I Systems
• Establishing and maintaining C2 system of systems and T&E capability• Numerous interfaces, message types, networks, comms,
and applications
• Distributed environment - geographic separation
• Multiple systems in joint environment – One system under test (SUT)
• Cost and bandwidth required to establish and maintain distributed M&S
• Software maintenance with changing C2 systems
System(s) Under Test (SUT)
Communications
(Transmissions) Systems
Applications (Information
Systems)
Workstations or Terminals
Sensors
Doctrine
System of Systems
Digital Networking
Equipment (Servers/
Gateways/Bridges/Routers…)
C2 Facilities (Buildings,
Vehicles, Enclosures…)
• Complex systems
• Users / operators – variable vulnerabilities
• Voluminous heterogeneous data
• Vulnerabilities are often difficult to predict/find
• Threat agents and vectors are not easy to characterize
• Metrics for some cyber operations
• mapping to mission effectiveness
• Offense / defense as symbiotic pair
Challenges with Cyber Systems
Refine
T&E as Part of Single-StepSystem Development Process
ConceptDefinition ?
Refine
Stop Development
SubsystemDevelopment ?
Refine
Stop Development
PrototypeDevelopment ?
Stop Development
T&E
T&E
T&E
ProductionSystem ?
Refine
Stop Development
T&E Release
Continue
Continue
Continue
Need
RISKACCEPT-
ABLE?
NO
DETERMINEOBJECTIVES
PRE-TESTANALYSIS
TEST EVALUATE
PRODUCT:KNOWN
RISKSOLUTION
IMPROVE
YES
T&E Phases
Pre-testTest
Event
Execution
Post-Test
• Risk Management Framework
• Artifacts should be helpful in T&E
• OSD-OT&E Memo, Aug 1, 2015: T&E Requirements
• Cooperative Vulnerability and Penetration Testing
• Adversarial Assessments
• Cybersecurity Test and Evaluation Guidebook, July 1, 2015
• Understand Cybersecurity Requirements
• Characterize the Cyber-Attack Surface
• Cooperative Vulnerability Identification
• Adversarial Cybersecurity DT&E
• Cooperative Vulnerability and Penetration Assessment
• Adversarial Assessment
DoD Guidance
• The PM will take full advantage of DoD ranges, labs, and other resources.
• DT&E activities will start when requirements are being developed to ensure that key technical
requirements are measurable, testable, and achievable.
• The DT&E program will support cybersecurity assessments and authorization.
• The PM will develop a strategy and budget resources for cybersecurity testing. The test program will
include, as much as possible, activities to test and evaluate a system in a mission environment with
a representative cyber threat capability (additional guidance is included in the DAG).
• For Major Defense Acquisition Programs, the DT&E T&E Master Plan (TEMP) approval authority will
provide the Milestone Decision Authority (MDA) with an assessment at each milestone review or
decision point.
• Beginning at Milestone (MS) A, the TEMP will document a strategy and define resources for
cybersecurity T&E.
• Beginning at MS B, appropriate measures will be included in the TEMP and used to evaluation
operational capability to protect, detect, react, and restore to sustain continuity of operation.
DoDI 5000.02, January 7, 2015
Cyber T&E Phases
• Cyberspace and Cyber Systems
• Threats
• Definitions
• Business / Mission Assurance
• Information Technologies
• T&E Challenges
• Metrics and measures
• Planning
• Configuration / test execution
• Data Reduction and Analysis
Outline
Measures and Metrics
ISSUES
MOE
1
MOE
2
MOE
n
MOP
1.1
MOP
1.2
MOP
1.n
.
.
.
.
.
.
TESTPARAMETER1.1.1
TESTPARAMETER1.1.2
TESTPARAMETER1.1.m
.
.
.
Pre-Test: From Issues
to ObservablesMOE = Measure of Effectiveness
MOP = Measure of Performance
• What questions do we need to answer?• How much testing?• How to test?• How to tailor test for life cycle?
. . .
. . .
. . .
. . .
ConductPlanning forOperation
Picture/Awareness
Target NominationProcess
Number/Types ofTracks
Interoperability ofLink feed interfaces
Timeliness/Accuracy ofTrack Updates
Commonality/Relaventnessof Awareness
Effectiveness ofSupportedCollab. Protocols
ProcessExecutionTime
Critical OperationalIssues
Measures ofPerformance
TESTING MEASURES
PERFORMANCE
SystemLevel
Mission
Does System provideFor effectiveMission execution
Measures of Effectiveness
Force levelTasks
Metrics Breakdown
…
MOP/Technical Performance Breakdown
Number/Type ofTracks
Interoperability ofLink feed interfaces
Timeliness/accuracy ofTrack Updates Common
Awareness
Effectiveness ofCollab. Tasks
ProcessExecutionTime
Measures of Performance
Link Message LatencyTask Latency
AlternativesAnalyzed
Number ofTargets in Plan
Technical PerformanceParameters
GeographicalDifferencesIn COP…
…
Effective Bandwidth … Quality of Service
Interactive/ExchangeAOC – AADC –JFACC
… …SystemPerformanceParameters
Basic Network Centric T&E Process
Define Objectives, Measures, and
Data Requirements
Define Test Network
Configuration
Generate/Select
Operational Scenario
Map Scenario
Players/Units to Test
Network Assets
Execute Test Event
Analyze Test Results
Compare Results to
Expectations Post Test
Analysis
Design Test/
Test
Preparation
Simulate/Preview
Test Event
67Reference: http://www.flickr.com/search/?w=all&q=castle+moat&m=text
68
Intrusion Detection
FirewallVPN Tunnel
• Compliance
• Standards / processes
• Service Level Agreements
• Defensive functions
• Offensive / exploitation functions
• Mission effectiveness
• Mapping cyber effects to mission effectiveness within cyberspace
• Mapping cyber effects to other warfighting domains
Categories of Cyber System Requirements
• International standards bodies:
• International Organization of Standardization (ISO)
• Payment Card Industry Security Standards Council (PCI)
• Information Security Forum
• The Open Group
• US-based standards bodies:
• National Institute of Standards and Technology (NIST)
• Federal Information Processing Standards (FIPS)
• DoD
• Department of Defense Information Assurance Certification and Accreditation Process (DIACAP) (and DITSCAP)
• Others (e.g., Joint Air Force Army Navy (JAFAN) , ICD 503)
Example Information Assurance Standards
• What is secure?
• Interpretation
• Death by standard
• Too many standards
• Conflicts
• Never underestimate the stupidity of people in large groups
• Typically seen as a low bar
Weaknesses
Polices, Standards, and Procedures
Governance Frameworks
• ISO
• SOx, PCI, HIPAA
• NIST, FIPS
• External
Policies
• High Level Guidance
Standards
• Implementation
Procedures
• Further Implementation
Implementation
• PCI-DSS 5: All systems storing credit card data must utilize anti-virus.Frameworks
• Policy-1: All systems will utilize with anti-virus,Policies
• Standard-1: All systems will utilize McAfee anti-virusStandards
• Procedure-1: All systems will utilize anti-virus with a given configuration.Procedures
Example
Risk Management Framework
(1) Categorize
(2) Select Security Controls
(3) Implement Security Controls
(4) Assess Security Controls
(5) Authorize System
(6) Monitor Security Controls
Data Owner
System Owner
System Security Office
•System Administrator
•Auditor, etc
Key People
System Security Plan
Security Control Matrix
System Security Office
Key Artifacts
Module 1.6 Information Assurance Standards 76
NIST 800-53 Control Families
ID Family Class
AC Access Control Technical
AT Awareness and Training Operational
AU Audit and Accountability Technical
CA Security Assessment and Authorization Management
CM Configuration Management Operational
CP Contingency Planning Operational
IA Identification and Authentication Technical
IR Incident Response Operational
MA Maintenance Operational
MP Media Protection Operational
PE Physical and Environmental Operational
PL Planning Management
PS Personnel Security Operational
RA Risk Assessment Management
SA System and Services Acquisition Management
SC System Communications Protection Technical
SI System and Information Integrity Operational
PM Program Management Management
Moderate-Low-Low Implementation Guidance
Identifier Family Class
Company Highly Confidential Baseline Controls
NIST 800-53 Full Control Family (455 Controls)
NIST 800-53 Control Tailoring
Not all baseline controls are appropriate for every system
Control may not be possible/feasible
Control may be overly burdensome
Control may not make sense
Tailoring process allows for system and risk specific implementation
Initial Security Control Baseline
Tailored Security Control Baseline
Data Owner Approved Set of Security Controls
Documented Agreed Upon Security Controls
(with rationale for any tailor in or tailor out)
Apply Tailoring Guidance
Data Owner
Acceptance
System Security Plan
Relationship of Risks to MissionT
hre
ats
/Vecto
rs
Information Assets
Thre
ats
/Vecto
rs
Countermeasures/Controls
Investment
Mission Threads/ Processes
Residual Risk
Assets
Summary of Network-Centric Measure Categories
• Interoperability
• Information Exchange Requirements (IERs)
• Message types/Formats
• Latencies
• C2/Planning timelines and effectiveness
• Information Security/Assurance
• Common Pictures (COP, CROP, CTP, SIAP…)
• Accuracy
• Number of tracks
• Timeliness
• …
• C&A - IA Compliance
• Controls/Countermeasure effectiveness
• Detection / Monitoring effectiveness
• Incident response effectiveness/timeliness and disaster recovery and business continuity
• Situation Awareness
• Ability to execute supported missions on networks/ information systems
Summary of Common Cyber Warfare Measure Categories
Test Results Mapped to Objectives
Obj # Summary of Objective
1.1Assess the ability of the SUT to detect attacks, probes and other CND events
1.2
Assess the SUT’s ability to manage, prioritize, filter and correlate CND related information and alerts from multiple disparate sources to distill voluminous detections into salient SA and detect threats otherwise undetectable
1.3Assess the ability of the SUT to store detailed data and allow for searches and queries to identify CND issues and past events
1.4Evaluate the SUT display to determine if necessary SA information is available to operators
1.5Assess the extensibility (ability to be modified and enhanced) of the SUT SA capability
Test Results Mapped to Objectives
Obj # Summary of Objective
2.1 Assess the ability of the SUT to detect emerging threats
2.2Assess the SUT’s ability to manage, prioritize, filter and correlate CND related information and alerts from multiple disparate sources to distill voluminous detections into SA on new and emerging threats
2.3Assess the ability of the SUT to store detailed information collected from multiple sources and allow for searches and queries to identify new and emerging threats
2.4Assess the ability of the SUT to display alerts and information so that operators and analysts can identify new and emerging CND threats
2.5Assess the extensibility of the SUT for collection, storage and analysis of CND information that can be used to identify and address new and emerging threats
Test Results Mapped to Objectives
Obj # Summary of Objective
3.1Assess the ability of the SUT to display information needed by the analysts and operators
3.2 Assess the usability of the SUT by the operators and analysts
3.3 Assess the sustainability and affordability of the SUT
3.4Assess the ability of the SUT to support the generation of administrative, summary and other reports
3.5Assess the interoperability of the SUT with other systems from the same vendor, other CND systems, and the enterprise network infrastructure as a whole
3.6Assess the ability of the SUT to facilitate responses to threats and attacks
Modification
Delivery
methods
Implementation
Reverse
Engineering
Discovery
Stealth &
Obfuscation
Approach /
Implementation
Shaping
Initial
Evaluation
Independent
Evaluation
Testing
As needed
Example Process:Vulnerability and Exploitation Process
Normal Dynamic Name Service (DNS) Operation
Web browser:
http://www.google.com
ISP’s DNS Server
Google’s Name Server
What is www.google.com?
What is www.google.com?
74.125.45.106
74.125.45.106
74.125.45.106
69.50.131.86
Poisoned Dynamic Name Service (DNS) Operation
Web browser:
http://www.google.com
ISP’s DNS Server
Google’s Name Server
What is www.google.com? 69.50.131.86
Attacker’s Name ServerAttacker
What is www.google.com?69.50.131.86
74.125.45.106
* On average, 2^16 attacks
Example Metric: (Beyond the basics)Duration of Attack Versus Conspicuousness
1
10
100
1000
0 100 200 300 400 500 600
Du
rati
on
(s)
[lo
gari
thm
ic s
cale
]
Packets Sent per Attempt
Duration vs. Packets Sent
The sweet spot-“Low and Slow”
• Cyberspace and Cyber Systems
• Threats
• Definitions
• Business / Mission Assurance
• Information Technologies
• T&E Challenges
• Metrics and measures
• Planning
• Configuration / test execution
• Data Reduction and Analysis
Outline
Basic Network Centric T&E Process
Define Objectives, Measures, and
Data Requirements
Define Test Network
Configuration
Generate/Select
Operational Scenario
Map Scenario
Players/Units to Test
Network Assets
Execute Test Event
Analyze Test Results
Compare Results to
Expectations Post Test
Analysis
Design Test/
Test
Preparation
Simulate/Preview
Test Event
Define Objectives, Measures & Data Requirements
Plan and Design Test
Execute Test Events
Analyze and Report Data
Elements of a Net-CentricTesting Methodology
Network Centric Test System Infrastructure Drivers
• Decouple operational processes and scenarios from network specifics
• Facilitate development of new SUT interfaces
• keep pace with C4I and network systems
• Accommodate various intra-system communications modes
• support for tactical environments
Network Centric T&E Concepts
• Stimulation and Virtual Representations
• C4I System Interfaces
• Mapping the Virtual to the Real
• Intra-Test System Communications
• Test Execution
• Data Reduction and Analysis
RISKACCEPT-
ABLE?
NO
DETERMINEOBJECTIVES
PRE-TESTANALYSIS
TEST EVALUATE
PRODUCT:KNOWN
RISKSOLUTION
IMPROVE
YES
Test and Evaluation
(T&E) Phases
Pre-testTest
Event
Execution
Post-Test
Objectives, Measures, and Data Requirements
Test objectives defined by the operational requirements
Measures and data requirements determined by user and tester
Generic Approach to Stimulation
InitiationMessage Type 32
ResponseMessage Type 35
Interactive Exchange
Type 33
Delay
between
messages
…
End Thread
Probabilities of branches
Completion of
Operational Task
• Background Traffic
• Interactions with Live
Players
Generate/Select Operational Scenario
Driven by operational objectives and measures
Operationally realistic
Live, virtual, or constructive
Simulation, scripted, or hybrid
Scenario generation tools
Simulation vs. Scripting Network/commtraffic (Sim vs. Stim.)
Simulation
Sophisticated battlefield simulations
Developed in training community
Large programs
Significant scenario development efforts
Often needed to assess or supplement “C2 effectiveness” measures with live forces
Scripting
Pre-built message database
Limited ability to adjust during an exercise
Simple and adaptable
More control of test events
Ideal for assessing IER measures
Ideal for generating “background” load
Host/Client-based
Virtual or bare metal
Generates end-to-end user traffic
Provides targets
Can represent threats
Packet/network-based
Packet streams sent through a network
Represents protocols
Does not typically represent full authentication, etc.
Does not typically represent a session properly
Can represent threat activity
Two Basic Types of Traffic Generation
Example Test Matrix
ID Description Threshold Data Requirements Form Event Sample Size
I-6 COI. Does the System allow the
embarked unit leader (Squad Leader
through Brigade Commander) to
command and control during
operations? [TEMP, para 2]
See App 6
to Annex
D
I-6 C-72 Critical Criterion (clarified).
PCLW. The vehicle's
communications system shall
provide for remote
transmission and monitoring
of any selected radio, and
for internal vehicle
communications (threshold).
App D-6,
paragrap
h 3
ALNO
Pending clarifications: 1) Must the SUT support an
"all-nets" broadcast over all radios in the
vehicle simultaneously from any one workstation?
2) Should each workstation operator be able to
monitor multiple radio nets simultaneously? If
so, what is the threshold number of nets? 3) Is
the second portion of the requirement, "and for
internal vehicle communications" redundant with
C-97 (ORD para 4.1.8.3.3.1)?
I-6 C-72 M-7 MOE. TD verification
that SUT can monitor
and transmit remotely
on any selected radio
from all
workstations.
Capability
Verified
TD Ver TD Ver AO/PT
AO/Comm
STE
NA
I-6 C-72 M-42 MOE. Percentage of
successful radio
access trials as
verified by VETT.
NFR Number of
successful radio
access trials
Form 27
VETT
AO/PT
AO/Comm
STE
36
Tri
als
I-6 C-72 M-43 MOE. Battalion Staff
ratings of the SUT's
capability for remote
transmission and
monitoring of radios.
> 50%
question
s with
>= 80%
favorabl
e
response
s
PTS Q#: 176, 177,
619
PTS AO/Comm
STE
AO/DegL
STE
AO/OMP 1
AO/OMP 2
AO/OMP 3
9
sol
ide
rs
Example Information Operations Attack System
System Under
Test Client
Communications
Node (CN)
PTT
PTT
VMF, C2PC, HTTP,FTP,
OTH Gold, VoIP
Communications
Node (CN)
Node Control Console
(Test Operator)
Radio Nets
IP Networks
Test and Engineering Network
System Under
Test Client
Systems Under Test
STARSHIP
Basic Control
Video Conferencing
Direct Injection
Client/Server
PTT
PTT
PTT
PTT
VMF, C2PC, HTTP,FTP,
OTH Gold, VoIP
IP or PSTN
Networks
PTTPTT
Voice/Video Emulation Test Tool (VETT)
Typical System Configuration
Two Nodes Plus Controller
Attacks
Denial of Service Attack
IRC Server (NI)
Wircd.exe
Bot Mater (I)
Subprogram Agent
launch_attack.exe
Image Get (I)
HTTP Agent
(Emulates Web
User)
Bots (I)
Subprogram Agent
gspot.exe
Web Server (I)
Subprogram Agent
LightTPD.exe
1.5 Mbps Link
LAN
Legend:
( I ) – Instrumented
( NI ) – Not Instrumented
LAN
IRC Channel(Botnet Control)
Results: Web Service Delay Cause by Attack
0.00
10.00
20.00
30.00
40.00
50.00
60.00
70.00
0.0
0
0.3
5
0.7
0
1.0
5
1.4
0
1.7
5
2.1
0
2.4
5
2.8
0
3.1
5
3.5
0
3.8
5
4.2
0
4.5
5
4.9
0
5.2
5
5.6
0
5.9
5
6.3
0
6.6
5
7.0
0
7.3
5
7.7
0
8.0
5
8.4
0
8.7
5
9.1
0
9.4
5
9.8
0
10
.15
10
.50
10
.85
11
.20
11
.55
11
.90
12
.25
12
.60
12
.95
13
.30
13
.65
14
.00
14
.35
14
.70
Min
seco
nd
s
Retrival Time
Processing Delay
Attack Bounds
• Joint Information Operations Range (JIOR or IO Range)
• Joint Mission Environment Test Capability 2.0
• National Cyber Range
• Lab ranges
• Some connected to JIOR
• Built from virtualized systems and networks
• Provide various traffic generation capabilities
• Threat representation / Red Teams
• Provide teams to attack systems
• Generally connected to the JIOR
• Examples:
• AF 346TS, Lackland AFB
• USN SSC PAC – Pearl City, HI
• USA Threat Systems Management Office (TSMO)
Cyber / IO Ranges
Typical Range Set Up: Attacks Through Gray Space
Servers
Endpoint Devices
Malware
Tools
Attacker
Endpoint
Servers
Defenses
Gray Space
Target Space orBlue Space
Malicious Actor Space orRed Space
User
Pivot in GraySpace
• Gray space is neutral networks/machines• Used to hide tracks• Often in countries other than target or attacker
• Cyberspace and Cyber Systems
• Threats
• Definitions
• Business / Mission Assurance
• Information Technologies
• T&E Challenges
• Metrics and measures
• Planning
• Configuration / test execution
• Data Reduction and Analysis
Outline
Plan and Design Test
Define Test System Configuration
Generate/Select Operational Scenario
Map Virtual Players/Units to Test Network Assets
Define Test System Configuration
Based on measures and data requirements
Stimulation and data capture
Distributed, Undistributed, or Hybrid Approach
Data collection management
Time synchronization
Generic Approach to Stimulation
InitiationMessage Type 32
ResponseMessage Type 35
Interactive Exchange
Type 33
Delay
between
messages
…
End Thread
Probabilities of branches
Completion of
Operational Task
• Background Traffic
• Interactions with Live
Players
Email(Thread Initiation)
…
Generate/Select Operational Scenario
Text Message(End of Thread)
VoIP Phone Call
Chat Session
…
Image Reference: http://www.grime.net/facets/air.htm
VMF via SADL(End of Thread)
C2 Message
Live Player(End of Thread)
…
Mapping the Virtual to the Real(Decoupling operations for IT specifics)
• Assign “operational” roles to real-world network assets
• Rapidly reconfigurable-Adapt to changes in networks
• Accommodate mixtures of Live/Virtual/Constructive
• Reuse of validated “scenarios”
G2 Free form
Intel messages
S3 Situation Reports
G3 Voice
Email Client
Email@address
GCCS COP Server
IP address
JVMF Message Process
Universal Reference Number
VoIP Call Manager
Phone number
…
Execute Test Events
Control scenario events and threads
Data aggregation
Time synchronization controls
Define Test System Configuration
Two methods: Direct injection and C2 application/system stimulation
C4I and Cyber System Interfaces
Direct Injection on Network
Must validate messages/formats
Protocol/login/encryption issues
Streamlined approach –typically yields more volume
C2 Application/system stimulation
End-to-end system test!
Validation “extends” from C2 app/system
Protocol/login/ encryption handled naturally
Volume can be limited by C2 app/system
Verification & Validation Issues
• Correct network load/data
• Broadcasts, unintended services, etc.
• Message format verification
• Protocol verification
• Security certification
• Scenario Validation
Intra-T&E System Communications
• Dedicated T&E “out-of-band” network
• T&E monitoring/control transparent to SUT
• Connectivity using SUT networks
• Limited use of bandwidth
• Use often limited to pre and post-test event
• Disconnected mode (“Sneaker” net)
• Hybrids - Use of all three modes simultaneously
Practical Issue: Time Synchronization
• Synchronizing test system
• Synchronizing SUT
• GMT or local
• Use test network or SUT network
• Time sync section in test plan/procedures
• Synchronization checks on daily check list
• Cyberspace and Cyber Systems
• Threats
• Definitions
• Business / Mission Assurance
• Information Technologies
• T&E Challenges
• Metrics and measures
• Planning
• Configuration / test execution
• Data Reduction and Analysis
Outline
Analyze and Report Data
Analyze Test Results
Compare Results to Expectations
0
20
40
60
80
100
120
140
160
180
200
Task
Ex
ec
uti
on
Tim
e (
se
c)
Multi-ship
Geolocation
CAS Mission Time Sensitive
Targeting
End-to-end
track update
Planning
Pre-contact
Contact
Post-contact
Analyze and Report Data
• System Logs
• Network Devices
• Defensive Devices
• Sniffers/Protocol analyzers
• Management system reports
• Voice / Video
• Modeling and simulation/stimulation
• Command and Control Systems
• Cyber / IO ranges control
• Red team logs / penetration test reports
• Electro-magnetic environment
Types of Data Sources
• Leverage SUT for instrumentation
• Leverage existing instrumentation within environment
• Add instrumentation to validate data
Instrumentation Tricks
Data Reduction/Analysis Concepts
• Systematic tie-in with Simulation/Stimulation and Instrumentation
• Identifiers (or serial numbers) to associate inputs (sent messages) with outputs (received)
• Database driven approach allows for recalculation of measures for different requirements
Data Reduction/Analysis Concept (Continued)
• Automated and rapid data reduction
• Need for speed!
• Quick look analysis during test execution
• Allows tester to adapt between test events
Cyber T&E AnalysisProblem Definition:
Finding the “needles” in the
digital haystack
• Agile C2 concepts, particularly at the Joint Operational Level of warfare, loosely define operational task threads
• TTPs can change as an operation is conducted
• Numerous applications and protocols (VoIP, chat, http, email, etc.) can be used to accomplish tasks
• Threat representations can be difficult to instrument and process
• Heterogeneous network traffic is voluminous
• Operationally significant transactions are “needles” in the digital hay stack
• Tester must track network operations in near-real-time
• Ensure the right data/events are being collected
• Adjust test plans as operation dictates
• Quick look analysis
Cyber T&E AnalysisProblem Definition:
• Distinguishing threat activity from benign activity
• Tracking and “profiling” threat sequences
• Correlating network data with red team logs
• Reconciling the “good guys” view from the red team view
• What really happened
• Cyberspace and Cyber Systems
• Threats
• Definitions
• Business / Mission Assurance
• Information Technologies
• T&E Challenges
• Metrics and measures
• Planning
• Configuration / test execution
• Data Reduction and Analysis
• Technologies for Cyber Testing
Outline
Technologies for
Cyber Testing
Technologies for Testing in Cyberspace
• Building the environment
• Threat Representations
• Instrumentation and
situation awareness (SA)
• Data reduction and analysis
Parallels between operational requirements and testing requirements!
Building Representative Environments: Key Technologies
• Virtual machines and networks (VMWare®, Citrix XenServer™)
• Replicate user machines and LANs on server farms
• Lower cost/ footprint
• Numerous open source tools for threat and defense representation (nmap, metasploit, snort…)
• Nessus and other commercial vulnerability scanners
• Ranges (IO Range, National Cyber Range)
• Varying levels of classification
• Numerous facilities and capabilities for testing on the range
• Connectivity to “open air” (non-virtual)
• Traffic generation (benign and threat)
• Generation server to generation server
• Represent traffic on the wire
• High volumes
• End-user machines
• Represent attacks on targets (bare metal or VMs)
• Lower volumes
• Threat “teams” and Penetration testers with various specialties
• Each service has team(s) and some Joint teams
• Specialties may include IP-based, SCADA, C2 systems, social engineering, etc.
Gap: Up-to-date, faithful threat representations
Traffic generation and threat representation
Ingest Processing Threat Reps/Automated Intelligence
Malware
Web
Presence
Social
Engineering
Open
Source
Network
monitor
Closed
Sources
Plug-able
Architecture
Traffic
Generators
Threat Actor Characteristics
Wide Array of
Open and Closed
Intelligence Inputs
Correlation &
Profiling
Threat
Representations
Automated Cyber Threat Representations (ACToR): Project Overview
135
Capability Summaries
Engagement Guidance
Cyberspace Data Reduction and Analysis Requirements
Measures for “Agile Cyber Warfare”: Timely and effective
decisions
Methods to nail down effectiveness/operational impacts
Technology needed to “mine” measures from
heterogeneous and voluminous network traffic
Correlation of events on disparate and distributed media
Tracking and making sense of cyber warriors’ agility
Timeline (SV-10c) Visualization
Log File Event
Node-node transactions
Color-coded type(e.g., https, http, ssh)
Mouse hover over transaction or log pops
up more detailNode (e.g., server, client,
application, service)
High light threat activity
Timeline Visualization (Continued)
Select TCP Payload to get transaction
message
Transaction message Popup
Copy Payload and paste into file
StatusMap
Mouse over headingto get full description
Mouse over demo caseto get associated task
Mouse over heading, then click on + to expand view into heading specific items
Inspect data for selected run time
Select run-time of interest