Exercises in Defending Cyberspace: The Capstone of Education, Training, and Awareness Craig E. Kaucher LTC, U.S. Army Professor of Information Operations.

Exercises in Exercises in Defending Defending

Cyberspace: The Cyberspace: The Capstone of Capstone of

Education, Training, Education, Training, and Awarenessand Awareness

Craig E. KaucherCraig E. KaucherLTC, U.S. ArmyLTC, U.S. Army

Professor of Information Operations and AssuranceProfessor of Information Operations and AssuranceInformation Resources Management CollegeInformation Resources Management College

National Defense UniversityNational Defense [email protected]@ndu.edu

““My opinions: not necessarily the USG, My opinions: not necessarily the USG, DOD, or NDUDOD, or NDU!”!”

AgendaAgendaAgendaAgenda

• Why Exercises ?Why Exercises ?• Developing ExercisesDeveloping Exercises

– SponsorSponsor– ObjectivesObjectives– Scope and FormatScope and Format– ParticipantsParticipants– ScenarioScenario– ControllersControllers– ModelsModels– Testing and ValidationTesting and Validation

• Case StudiesCase Studies

• Why Exercises ?Why Exercises ?• Developing ExercisesDeveloping Exercises

– SponsorSponsor– ObjectivesObjectives– Scope and FormatScope and Format– ParticipantsParticipants– ScenarioScenario– ControllersControllers– ModelsModels– Testing and ValidationTesting and Validation

• Case StudiesCase Studies

Why Exercises ?Why Exercises ?

• ““To test civilian agencies’ security preparedness To test civilian agencies’ security preparedness and contingency planning, and contingency planning, DHS will use exercises DHS will use exercises to evaluate the impact of cyberattacks on to evaluate the impact of cyberattacks on governmentwide processesgovernmentwide processes. Weaknesses . Weaknesses discovered will be included in agency corrective discovered will be included in agency corrective action plans and submitted to the OMB. DHS also action plans and submitted to the OMB. DHS also will will explore such exercises as a way to test the explore such exercises as a way to test the coordination of public and private incident coordination of public and private incident management, response and recovery capabilitiesmanagement, response and recovery capabilities.” .” (A/R 1-3)(A/R 1-3)

• ““Corporations are encouraged to Corporations are encouraged to regularly review regularly review and exercise IT continuity plansand exercise IT continuity plans and to consider and to consider diversity in IT service providers as a way of diversity in IT service providers as a way of mitigating risk.” (A/R 1-4)mitigating risk.” (A/R 1-4)

Appendix, Actions and Recommendations (A/R) Summary, The National Strategy to Secure Cyberspace, February 2003.

Why else…Why else…(Obligatory Dead Guy Quote)(Obligatory Dead Guy Quote)

• ““To rely on rustics and To rely on rustics and not prepare is the not prepare is the greatest of crimes; to greatest of crimes; to be prepared beforehand be prepared beforehand for any contingency is for any contingency is the greatest of virtues.”the greatest of virtues.”

Sun Tzu, on the need to wargameSun Tzu, on the need to wargame

strategies, from “Sun Tzu and the Art strategies, from “Sun Tzu and the Art ofof

Business: Six Strategic Principles”, Business: Six Strategic Principles”, MarkMark

McNeilly, Oxford University Press, McNeilly, Oxford University Press, 1996.1996.

Developing ExercisesDeveloping ExercisesThe Role of the SponsorThe Role of the Sponsor

Developing ExercisesDeveloping ExercisesThe Role of the SponsorThe Role of the Sponsor

• What does the sponsor want What does the sponsor want to learn or demonstrate?to learn or demonstrate?

• What does the sponsor want What does the sponsor want the participants to learn or the participants to learn or demonstrate?demonstrate?

• How can the exercise best How can the exercise best assure that the sponsor’s assure that the sponsor’s goals are met?goals are met?

• What information must be What information must be provided by the sponsor?provided by the sponsor?• What information will be What information will be

gathered for the sponsor?gathered for the sponsor?

• What does the sponsor want What does the sponsor want to learn or demonstrate?to learn or demonstrate?

• What does the sponsor want What does the sponsor want the participants to learn or the participants to learn or demonstrate?demonstrate?

• How can the exercise best How can the exercise best assure that the sponsor’s assure that the sponsor’s goals are met?goals are met?

• What information must be What information must be provided by the sponsor?provided by the sponsor?• What information will be What information will be

gathered for the sponsor?gathered for the sponsor?

Developing ExercisesDeveloping ExercisesSpecifying ObjectivesSpecifying ObjectivesDeveloping ExercisesDeveloping ExercisesSpecifying ObjectivesSpecifying Objectives

• Educational and Training ObjectivesEducational and Training Objectives– Teach or train new tasks and proceduresTeach or train new tasks and procedures– Reinforce previous training and educationReinforce previous training and education– Evaluate training and educationEvaluate training and education

• Research and Procedural ObjectivesResearch and Procedural Objectives– Develop new strategies, plans, proceduresDevelop new strategies, plans, procedures– Test execution of strategies, plans, proceduresTest execution of strategies, plans, procedures– Identify issues and gaps in current strategies, Identify issues and gaps in current strategies,

plans, proceduresplans, procedures– Build consensus for strategies, plans, Build consensus for strategies, plans,

proceduresprocedures

• Educational and Training ObjectivesEducational and Training Objectives– Teach or train new tasks and proceduresTeach or train new tasks and procedures– Reinforce previous training and educationReinforce previous training and education– Evaluate training and educationEvaluate training and education

• Research and Procedural ObjectivesResearch and Procedural Objectives– Develop new strategies, plans, proceduresDevelop new strategies, plans, procedures– Test execution of strategies, plans, proceduresTest execution of strategies, plans, procedures– Identify issues and gaps in current strategies, Identify issues and gaps in current strategies,

plans, proceduresplans, procedures– Build consensus for strategies, plans, Build consensus for strategies, plans,

proceduresprocedures

Full Scale or “Live”Exercise

Command Post Exercise

Tabletop

Full Scale or “Live”Exercise

Command Post Exercise

Tabletop

Developing Exercises Developing Exercises Scope of ActivitiesScope of Activities

Developing Exercises Developing Exercises Scope of ActivitiesScope of Activities

Exercises

Education

Training

Awareness

Exercises

Education

Training

Awareness

Developing ExercisesDeveloping Exercises Tabletop ExercisesTabletop Exercises

Developing ExercisesDeveloping Exercises Tabletop ExercisesTabletop Exercises

“One step that any organization can take is to reach out to other publicand private entities in its region toconduct joint tabletop exercises.”

( Andrews, 2003)

“One step that any organization can take is to reach out to other publicand private entities in its region toconduct joint tabletop exercises.”

( Andrews, 2003)

• Normally very low cost• Anyplace, anytime• Small number of participants• Could be for any type of objective• Could be the first phase of a larger exercise

• Normally very low cost• Anyplace, anytime• Small number of participants• Could be for any type of objective• Could be the first phase of a larger exercise

• Many organizations, Many organizations, not many peoplenot many people

• Frequently examinesFrequently examines existing or new existing or new

proceduresprocedures• Also could be part ofAlso could be part of an exercise an exercise

“buildup”“buildup”• More costs, more More costs, more

disruption to regular disruption to regular activitiesactivities

Developing ExercisesDeveloping Exercises Command Post ExercisesCommand Post Exercises

Developing ExercisesDeveloping Exercises Command Post ExercisesCommand Post Exercises

• Highest costHighest cost• Most people Most people

involvedinvolved• Inter-agency, inter-Inter-agency, inter-

governmental, governmental, inter-sectorinter-sector

• Occasional (but Occasional (but required)required)

• Impressions and Impressions and perceptions countperceptions count

Developing ExercisesDeveloping Exercises Full Scale ExercisesFull Scale Exercises

Developing ExercisesDeveloping Exercises Full Scale ExercisesFull Scale Exercises

“That’s why the most comprehensivecyberpreparedness exercises bringtogether people from different, interdependent sectors and governmentagencies and include practicing howinformation will be shared.”Dr. Craig Koerner, Naval War College

Developing ExercisesDeveloping ExercisesIdentifying ParticipantsIdentifying Participants

Developing ExercisesDeveloping ExercisesIdentifying ParticipantsIdentifying Participants

•Organization(Organization(s)s)

•IndividualsIndividuals•Who is Who is

essential?essential?•ControllersControllers

•Organization(Organization(s)s)

•IndividualsIndividuals•Who is Who is

essential?essential?•ControllersControllers

Developing ExercisesDeveloping ExercisesDeveloping the ScenarioDeveloping the Scenario

Developing ExercisesDeveloping ExercisesDeveloping the ScenarioDeveloping the Scenario

• The ScenarioThe Scenario– A situation into which A situation into which

participants are placed that participants are placed that requires them to make requires them to make decisionsdecisions

• Scenario-related informationScenario-related information– Who and what will decisions Who and what will decisions

affect?affect?– What operational informationWhat operational information

is required?is required?– How will the scenario be How will the scenario be

changed or updated?changed or updated?

• The ScenarioThe Scenario– A situation into which A situation into which

participants are placed that participants are placed that requires them to make requires them to make decisionsdecisions

• Scenario-related informationScenario-related information– Who and what will decisions Who and what will decisions

affect?affect?– What operational informationWhat operational information

is required?is required?– How will the scenario be How will the scenario be

changed or updated?changed or updated?

“The scenario can have asignificant, if not overwhelmingeffect on the decisions playersare able to make.” (Perla, 1990)

Developing ExercisesDeveloping ExercisesThe Role of ControllersThe Role of ControllersDeveloping ExercisesDeveloping Exercises

The Role of ControllersThe Role of Controllers

• Monitor Monitor participant participant actionsactions

• Assess Assess interactionsinteractions

• Inform Inform participants participants about outcomesabout outcomes

• Monitor Monitor participant participant actionsactions

• Assess Assess interactionsinteractions

• Inform Inform participants participants about outcomesabout outcomes

Developing ExercisesDeveloping ExercisesUsing ModelsUsing Models

Developing ExercisesDeveloping ExercisesUsing ModelsUsing Models

• Models can have several purposesModels can have several purposes– Provide inputs to the exerciseProvide inputs to the exercise– Keep the exercise movingKeep the exercise moving– Replicate realistic organizations, events or Replicate realistic organizations, events or

functionsfunctions

• ExamplesExamples– Physical or logical environmentPhysical or logical environment– Functional activities (logistics, intelligence)Functional activities (logistics, intelligence)– SensorsSensors– Command and controlCommand and control– WeaponsWeapons

• Models can have several purposesModels can have several purposes– Provide inputs to the exerciseProvide inputs to the exercise– Keep the exercise movingKeep the exercise moving– Replicate realistic organizations, events or Replicate realistic organizations, events or

functionsfunctions

• ExamplesExamples– Physical or logical environmentPhysical or logical environment– Functional activities (logistics, intelligence)Functional activities (logistics, intelligence)– SensorsSensors– Command and controlCommand and control– WeaponsWeapons

Developing ExercisesDeveloping ExercisesTesting and ValidationTesting and ValidationDeveloping ExercisesDeveloping Exercises

Testing and ValidationTesting and Validation

•Model, data, Model, data, and scenario and scenario validationvalidation

•Play testingPlay testing•Pre-playPre-play•Final RulesFinal Rules

•Model, data, Model, data, and scenario and scenario validationvalidation

•Play testingPlay testing•Pre-playPre-play•Final RulesFinal Rules

Exercise Case Study: Eligible Exercise Case Study: Eligible ReceiverReceiver

Exercise Case Study: Eligible Exercise Case Study: Eligible ReceiverReceiver

• ““The eye-opener exercise”The eye-opener exercise”• Live cyberattacks involvedLive cyberattacks involved• DOD focused and directedDOD focused and directed• No notice to “participants”No notice to “participants”• Key lesson learned: DOD Key lesson learned: DOD

networks are highly vulnerablenetworks are highly vulnerable• Led to the formation of Joint Task Led to the formation of Joint Task

Force Computer Network DefenseForce Computer Network Defense

• ““The eye-opener exercise”The eye-opener exercise”• Live cyberattacks involvedLive cyberattacks involved• DOD focused and directedDOD focused and directed• No notice to “participants”No notice to “participants”• Key lesson learned: DOD Key lesson learned: DOD

networks are highly vulnerablenetworks are highly vulnerable• Led to the formation of Joint Task Led to the formation of Joint Task

Force Computer Network DefenseForce Computer Network Defense

• Focused on regional CIP in preparation Focused on regional CIP in preparation for 2002 Winter Olympics, co-sponsored for 2002 Winter Olympics, co-sponsored by Utah Dept. of Public Safety, US DOE by Utah Dept. of Public Safety, US DOE Office of CIP, Utah Olympic Public Safety Office of CIP, Utah Olympic Public Safety CommandCommand

• Tabletop exerciseTabletop exercise• Used to surface issues, develop and Used to surface issues, develop and

implement an action plan for “disaster implement an action plan for “disaster resistant Olympics”resistant Olympics”

• Key lessons learned in understanding Key lessons learned in understanding interdependencies, communication, interdependencies, communication, coordination, and resource allocationcoordination, and resource allocation

• Focused on regional CIP in preparation Focused on regional CIP in preparation for 2002 Winter Olympics, co-sponsored for 2002 Winter Olympics, co-sponsored by Utah Dept. of Public Safety, US DOE by Utah Dept. of Public Safety, US DOE Office of CIP, Utah Olympic Public Safety Office of CIP, Utah Olympic Public Safety CommandCommand

• Tabletop exerciseTabletop exercise• Used to surface issues, develop and Used to surface issues, develop and

implement an action plan for “disaster implement an action plan for “disaster resistant Olympics”resistant Olympics”

• Key lessons learned in understanding Key lessons learned in understanding interdependencies, communication, interdependencies, communication, coordination, and resource allocationcoordination, and resource allocation

Exercise Case Study: Exercise Case Study: Black IceBlack Ice

Exercise Case Study: Exercise Case Study: Black IceBlack Ice

• US Air Force exercise focused on US Air Force exercise focused on internal networks and operatorsinternal networks and operators

• Used to evaluate detection, Used to evaluate detection, response, recovery proceduresresponse, recovery procedures

• Live and simulated (range) playLive and simulated (range) play• Validated operational Validated operational

procedures, and gathered best procedures, and gathered best practicespractices

• US Air Force exercise focused on US Air Force exercise focused on internal networks and operatorsinternal networks and operators

• Used to evaluate detection, Used to evaluate detection, response, recovery proceduresresponse, recovery procedures

• Live and simulated (range) playLive and simulated (range) play• Validated operational Validated operational

procedures, and gathered best procedures, and gathered best practicespractices

Exercise Case Study: Black Exercise Case Study: Black DemonDemon

Exercise Case Study: Black Exercise Case Study: Black DemonDemon

• Pacific Northwest critical infrastructure Pacific Northwest critical infrastructure owners with federal, state, local owners with federal, state, local governments (US and Canada)governments (US and Canada)

• Tabletop exerciseTabletop exercise• Physical attacks (in the scenario) led to IT Physical attacks (in the scenario) led to IT

failuresfailures• Key lessons learned:Key lessons learned:

– Number/degree of interdependencies unknownNumber/degree of interdependencies unknown– Regional and US/Canada coordination lackingRegional and US/Canada coordination lacking– Unanticipated loss of communicationsUnanticipated loss of communications– No mechanism for cross-border analysis and No mechanism for cross-border analysis and

reportingreporting– Roles, missions, role of law enforcement not Roles, missions, role of law enforcement not

understoodunderstood

• Pacific Northwest critical infrastructure Pacific Northwest critical infrastructure owners with federal, state, local owners with federal, state, local governments (US and Canada)governments (US and Canada)

• Tabletop exerciseTabletop exercise• Physical attacks (in the scenario) led to IT Physical attacks (in the scenario) led to IT

failuresfailures• Key lessons learned:Key lessons learned:

– Number/degree of interdependencies unknownNumber/degree of interdependencies unknown– Regional and US/Canada coordination lackingRegional and US/Canada coordination lacking– Unanticipated loss of communicationsUnanticipated loss of communications– No mechanism for cross-border analysis and No mechanism for cross-border analysis and

reportingreporting– Roles, missions, role of law enforcement not Roles, missions, role of law enforcement not

understoodunderstood

Exercise Case Study: Blue Exercise Case Study: Blue CascadesCascades

Exercise Case Study: Blue Exercise Case Study: Blue CascadesCascades

• Local/regional exercise involving Local/regional exercise involving federal, state, local govt., industry, federal, state, local govt., industry, academia, militaryacademia, military

• ““Congressionally” directedCongressionally” directed• Three PhasesThree Phases

– TabletopTabletop– Lessons learned implementationLessons learned implementation– Live exerciseLive exercise

• Key lessons learned: Start small and Key lessons learned: Start small and build, broadest participation is best, build, broadest participation is best, many information gaps existmany information gaps exist

• Local/regional exercise involving Local/regional exercise involving federal, state, local govt., industry, federal, state, local govt., industry, academia, militaryacademia, military

• ““Congressionally” directedCongressionally” directed• Three PhasesThree Phases

– TabletopTabletop– Lessons learned implementationLessons learned implementation– Live exerciseLive exercise

• Key lessons learned: Start small and Key lessons learned: Start small and build, broadest participation is best, build, broadest participation is best, many information gaps existmany information gaps exist

Exercise Case Study: Exercise Case Study: Dark ScreenDark Screen

Exercise Case Study: Exercise Case Study: Dark ScreenDark Screen

• Department of Homeland Security and Department of Homeland Security and Dartmouth University sponsored/run exerciseDartmouth University sponsored/run exercise

• Simulated attacks (physical and cyber)Simulated attacks (physical and cyber)• Focus on banking and financial sector, with Focus on banking and financial sector, with

other sector involvementother sector involvement• Government performance “certainly a B+, Government performance “certainly a B+,

better than my personal expectations” – Amit better than my personal expectations” – Amit YoranYoran

• Key Lessons Learned: inter-sector Key Lessons Learned: inter-sector coordination and information sharing need coordination and information sharing need improvement improvement

• Department of Homeland Security and Department of Homeland Security and Dartmouth University sponsored/run exerciseDartmouth University sponsored/run exercise

• Simulated attacks (physical and cyber)Simulated attacks (physical and cyber)• Focus on banking and financial sector, with Focus on banking and financial sector, with

other sector involvementother sector involvement• Government performance “certainly a B+, Government performance “certainly a B+,

better than my personal expectations” – Amit better than my personal expectations” – Amit YoranYoran

• Key Lessons Learned: inter-sector Key Lessons Learned: inter-sector coordination and information sharing need coordination and information sharing need improvement improvement

Exercise Case Study: Exercise Case Study: LivewireLivewire

Exercise Case Study: Exercise Case Study: LivewireLivewire

Other Views of ExercisesOther Views of ExercisesOther Views of ExercisesOther Views of Exercises

• How do exercises affect industry ?How do exercises affect industry ?– ParticipationParticipation

• Scope (number of participants)Scope (number of participants)• Business ImpactBusiness Impact• RepetitionRepetition

– CostCost• Who pays ?Who pays ?• Overhead & overtimeOverhead & overtime

– Interrelated sectorsInterrelated sectors

• How do exercises affect industry ?How do exercises affect industry ?– ParticipationParticipation

• Scope (number of participants)Scope (number of participants)• Business ImpactBusiness Impact• RepetitionRepetition

– CostCost• Who pays ?Who pays ?• Overhead & overtimeOverhead & overtime

– Interrelated sectorsInterrelated sectors

Closing ThoughtsClosing ThoughtsClosing ThoughtsClosing Thoughts• Education, training, Education, training,

and awareness are and awareness are valuable valuable countermeasures, but countermeasures, but exercises are where exercises are where “the rubber meets the “the rubber meets the road”road”

• ““If you’ve never been If you’ve never been under mass fire and under mass fire and suddenly you are, the suddenly you are, the odds are that your brain odds are that your brain will shut down and will shut down and you’ll do everything you’ll do everything wrong.” - Stephen wrong.” - Stephen Northcutt, SANS Northcutt, SANS Institute.Institute.

• Education, training, Education, training, and awareness are and awareness are valuable valuable countermeasures, but countermeasures, but exercises are where exercises are where “the rubber meets the “the rubber meets the road”road”

• ““If you’ve never been If you’ve never been under mass fire and under mass fire and suddenly you are, the suddenly you are, the odds are that your brain odds are that your brain will shut down and will shut down and you’ll do everything you’ll do everything wrong.” - Stephen wrong.” - Stephen Northcutt, SANS Northcutt, SANS Institute.Institute.

Graphic courtesy of US Naval Postgraduate School,Winners of the 2002 DOD Cyber Defense ExerciseDownloaded from:www.nps.navy.mil/PAO/Internal/ Cyber_Defense.htm

ReferencesReferencesReferencesReferences• ““The Art of Wargaming”, Peter P. Perla, Naval Institute Press, Annapolis, MD, The Art of Wargaming”, Peter P. Perla, Naval Institute Press, Annapolis, MD,

1990.1990.• ““How can information exchange be enhanced”, Richard Andrews, Security How can information exchange be enhanced”, Richard Andrews, Security

Management, vol. 47/6, pg. 162. Arlington, VA, 2003.Management, vol. 47/6, pg. 162. Arlington, VA, 2003.• ““More than a game”, Deborah Padcliff, Computerworld, vol. 36/37, September More than a game”, Deborah Padcliff, Computerworld, vol. 36/37, September

2002.2002.• ““Blue Cascades” Final Report, Pacific Northwest Economic Region, 18 July 2002.Blue Cascades” Final Report, Pacific Northwest Economic Region, 18 July 2002.• ““Infrastructure Interdependencies Tabletop Exercise: Summary of Key Issues Infrastructure Interdependencies Tabletop Exercise: Summary of Key Issues

and Actions to Date”, Paula Scalingi, DOE Office of CIP, May 2001and Actions to Date”, Paula Scalingi, DOE Office of CIP, May 2001• ““Black demon tests tactics, improves network defense”, Dom Cardonita, HQ Black demon tests tactics, improves network defense”, Dom Cardonita, HQ

AIA/PA, Lackland AFB, Texas, Summer 2002.AIA/PA, Lackland AFB, Texas, Summer 2002.• ““Dark Screen: A Cyber Security Exercise for San Antonio/Bexar County. Final Dark Screen: A Cyber Security Exercise for San Antonio/Bexar County. Final

Report”, Gregory B. White, University of Texas – San Antonio, 26 September Report”, Gregory B. White, University of Texas – San Antonio, 26 September 2003.2003.

• ““Simulated terrorist Attack Exposes Problems”, Ted Bridis, Associated Press, Simulated terrorist Attack Exposes Problems”, Ted Bridis, Associated Press, downloaded from downloaded from http://www.informationweek.comhttp://www.informationweek.com, 25 November 2003., 25 November 2003.

• ““Cyberexercises”, Seth Cowand, University of Texas-San Antonio, unpublished Cyberexercises”, Seth Cowand, University of Texas-San Antonio, unpublished manuscript, December 2003.manuscript, December 2003.

• ““Current Issues in US Homeland Security and Critical Infrastructure Protection”, Current Issues in US Homeland Security and Critical Infrastructure Protection”, Cristin L. Flynn, MCI, Inc., briefing at National Defense University, November 6, Cristin L. Flynn, MCI, Inc., briefing at National Defense University, November 6, 20032003