Defense Against The Dark Arts: Using Computer Security To Teach Core Computer Science Concepts Mark W. Bailey Associate Professor, Hamilton College Visiting.

Defense Against Defense Against The Dark Arts:The Dark Arts:Using Computer Security To Using Computer Security To Teach Core Computer Science Teach Core Computer Science ConceptsConcepts

Mark W. Bailey

Associate Professor, Hamilton CollegeVisiting Professor, University of Virginia

OutlineOutline

Goals, Context

Course Overview

Tools

Course Material

Assignment Examples

Core CS Concepts

Summary

ObservationsObservations

Declining CS enrollments nationwide

Need to demonstrate CS relevance

Difficult to include new courses in limited curricula

Security issues are of concern to everyone

Future decision makers must be literate to make informed policy decisions

Computer security provides an ideal framework for attracting and engaging students

GoalsGoals

Develop an innovative set of computer security courses

Include courses for students without computer programming expertise

Courses should be suitable across the spectrum of colleges and universities

Package and document courses to facilitate widespread adoption

Participating SchoolsParticipating Schools

Hamilton College (Liberal Arts)

Mark Bailey

University of Virginia(Research University)

Jack Davidson

Virginia State University(HBCU)

Jeff Zadeh

Proposed CoursesProposed Courses

C4: Advanced Topicsin Computer

Systems Security

C1: Introduction toComputer Systems

Security

C3: CounteringNetwork Threats:

Principles and Techniques

C2: Anti-virusPrinciples and

Techniques

C4: Advanced Topicsin Computer

Systems Security

C1: Introduction toComputer Systems

Security

C3: CounteringNetwork Threats:

Principles and Techniques

C2: Anti-virusPrinciples and

Techniques

Proposed CoursesProposed Courses

C1 and C4 require no programming experience

Suitable for liberal arts and engineering students wishing to obtain technical literacy

Use case study approach

CS majors could take all four to obtain a strong background in security

Courses C2 and C3 have programming assignments

C4: Advanced Topicsin Computer

Systems Security

C1: Introduction toComputer Systems

Security

C3: CounteringNetwork Threats:

Principles and Techniques

C2: Anti-virusPrinciples and

Techniques

C4: Advanced Topicsin Computer

Systems Security

C1: Introduction toComputer Systems

Security

C3: CounteringNetwork Threats:

Principles and Techniques

C2: Anti-virusPrinciples and

Techniques

OutlineOutline

Goals, Context

Course Overview

Tools

Course Material

Assignment Examples

Core CS Concepts

Summary

C2: Defense Against The Dark C2: Defense Against The Dark ArtsArtsFocuses on anti-virus principles and techniques

Prerequisites: Programming and assembly

Reinforces assembly language

Introduces programming vulnerabilities

Emphasizes virus prevention, detection, and disinfection

Designed for third and fourth year students

Anti-Virus Course Anti-Virus Course MotivationMotivation

Threats to computers systems, such as viruses and worms, are a serious problem

CS students need to understand malware schemes and defenses against them

Teaches how to detect and defeat malicious software

Analyzing programs demands application of core theoretical concepts of CS

Teaches these concepts in an application area of great topical interest

OutlineOutline

Goals, Context

Course Overview

Tools

Course Material

Assignment Examples

Core CS Concepts

Summary

ToolsTools

Anti-virus programs need toDisassemble binary codeAnalyze and reason about codeModify, or fix codeReassemble binary code

Many of these operations are performed by compilers

Phoenix Compiler SuitePhoenix Compiler Suite

A cutting-edge suite of compilers and tools from Microsoft Research

Scalable, configurable, extensible, compilation infrastructure

Configurable for new tools, and purposesEasy insertion of plug-ins at any point in analysis sequenceWell defined, API’s encouraging analysis and transformation reuse

Supports binary manipulation

Phoenix IR Phoenix IR Raising/LoweringRaising/Lowering

Notice that the flow arrows go in both directionsA binary (in EIR form) can be:

Raised all the way to HIR, transformed

Lowered to MIR, transformed

Lowered to LIR, transformed

Then written back out as a new binary

ASTs (Abstract Syntax Trees)

Source Code (C++)

C++ Front End

C# Front End VB Front End

Source Code (C#)

Source Code (Visual Basic)

HIR (High-level IR)

MIR (Medium-level IR)

LIR (Low-level IR)

EIR (Encoded IR)

OutlineOutline

Goals, Context

Course Overview

Tools

Course Material

Assignment Examples

Core CS Concepts

Summary

TopicsTopics

Introduction, ethics, threat modelsTerminology, x86 architecture

Tools: Disassembly tools, Phoenix intro

Phoenix binary analysis tools

Viruses: Boot, interrupt, memory resident, executable file

Detecting viruses, regular expressions, lex, Chomsky hierarchy

Ken Thompson’s Turing Award Lecture

Topics (Continued)Topics (Continued)

Obfuscation, SSA form and PhoenixAnti-anti-virus schemes, analyzing systems

Retroviruses, tunneling, armor, encryption, oligomorphic, polymorphic, metamorphic

Software Dynamic Translation (SDT)Strata SDT framework; SDT security applicationsCode vulnerabilities and exploits; secure coding, static security analyzersRoot kits

Antivirus AssignmentsAntivirus Assignments

“Tricky jump” illustrated (C/assembly)

Reinforces assembly, introduces DUMPBIN

Dumping Phoenix IRDemonstrates construction of Phoenix analysis and instrumentation tools

Virus code detection using lexFinding junk instructions using SSA formRemoving obfuscating jumps using Phoenix control flow informationUsing Phoenix to prevent stack smashingStudent research presentations

OutlineOutline

Goals, Context

Course Overview

Tools

Course Material

Assignment Examples

Core CS Concepts

Summary

Tricky Jump IllustratedTricky Jump Illustrated

Application code:xor eax, eaxret

Can be replaced withpush offset malicious_funcret

Causing a jump instead of a returnStudents build tricky jump program

assembly language and debugging skill building

Virus Code DetectionVirus Code Detection

A common virus excerpt changes the IVT:

mov eax, 4CH

mov dword ptr [eax], edx

Which register is used is irrelevant

Disassemble executable using dumpbin

Recognize pattern using lex

Introduces regular expressions and their limitations

Finding Junk InstructionsFinding Junk Instructions

Viruses obfuscate using junk instructions

code SSA form

x = 2; x1 = 2; (useless)

y = 3; y1 = 3;

x = 4; x2 = 4;

y = y*x; y2 = y1*x2;

Students use Phoenix’s SSA to find junk

Removing Obfuscating Removing Obfuscating JumpsJumps

Simple sequences like:

x += 4;y +- (z – x);z -= 3;printf(“%d\n”, x);

Can be obfuscated using jumps:

x += 4;goto lab2;

lab3:z -= 3;goto lab4;

lab2:y += (z – x);goto lab3;

lab4:printf(“%d\n”,

x);

OutlineOutline

Goals, Context

Course Overview

Tools

Course Material

Assignment Examples

Core CS Concepts

Summary

Core CS ConceptsCore CS Concepts

Viruses often detected by pattern matchingRegular expressions in context of suspicious code patternsCode obfuscations make pattern matching inadequate in practiceChomsky language hierarchy is used to understand this limitationEquivalence of obfuscated code applies concepts from computability and theory of computation Students learn anti-virus software must often approximately solve an infeasible problem

Core CS Concepts Core CS Concepts ContinuedContinued

Pattern matching limitations suggest semantic analyses found in compilers

Simple dataflow analysis and SA form for de-obfuscation

Simple register allocation/assignment used to defeat register renumbering—another obfuscation technique

The Ongoing BattleThe Ongoing Battle

Endless advances in both malicious software and the tools that combat itAs each generation is defeated by security software, new techniques are developed that defeat the security toolsExamples: Armoring of viruses, obfuscation techniques, evolutionary viruses such as polymorphic, and metamorphic virusesThis “warfare” between good and bad forces has been found to intensify student interest

OutcomesOutcomes

Taught twice at Virginia, once at Hamilton and VSU (Spring 2007)UVa course overenrolled (had to turn away students)Used Phoenix infrastructure as a vehicle for teaching anti-virus techniques (compilers in disguise)Student feedback very positive

Students wanted more Phoenix assignments. “Would have liked to do more with Phoenix and less with Lex, but I guess time was too much of a limitation.”

SummarySummary

Course focuses on topic of concern to everyoneUses core CS concepts in an interesting application areaStudents use state-of-the-art tools to analyze real code (but not real viruses…)Course theme, title, and subject helps attract and fill coursesCourse materials suitable at a wide range of institutionsCourse materials will be made available in the Microsoft Academic Alliance Curriculum Repository

For More InformationFor More Information

Mark Bailey ([email protected])

Jack Davidson ([email protected])

Jeff Zadeh ([email protected])

Microsoft Research Faculty Summit 2007