Page 1
DynInst Security© 2001 Barton P. Miller December 2001
Playing Inside the Blackbox:Using Dynamic Instrumentation to Create
Security Holes
Barton P. [email protected]
Computer Sciences DepartmentUniversity of Wisconsin
Madison, Wisconsin 53705USA
Page 2
– 2 – DynInst Security© 2001 Barton P. Miller
Overview
1. How to easily do dangerous and malicious things to a running program.
2. How to detect when someone does something evil to your program.
Page 3
– 3 – DynInst Security© 2001 Barton P. Miller
A New View
Running programs are objects to be easily manipulated. Kinds of manipulations might include:
Instrumentation
Optimization
Control
Page 4
– 4 – DynInst Security© 2001 Barton P. Miller
The Vehicle:The DynInst API
A machine-independent library for machine level code patching.
Eases the task of building new tools.
Provides the basic abstractions to patch code on-the-fly
Page 5
– 5 – DynInst Security© 2001 Barton P. Miller
Dynamic InstrumentationDoes not require recompiling or relinking
• Saves time: compile and link times are significant in real systems.
• Can instrument without the source code (e.g., proprietary libraries).
• Can instrument without linking (relinking is not always possible.
Instrument optimized code.
Page 6
– 6 – DynInst Security© 2001 Barton P. Miller
Dynamic Instrumentation (con’d)Only instrument what you need, when you
need• No hidden cost of latent instrumentation.• Enables “one pass” tools.
Can instrument running programs (such as Web or database servers)• Production systems.• Embedded systems.• Systems with complex start-up procedures.
Page 7
– 7 – DynInst Security© 2001 Barton P. Miller
The Basic MechanismApplicationProgramApplicationProgram
Function fooFunction foo
TrampolineTrampoline
Pre-InstrumentationPre-Instrumentation
RelocatedRelocatedInstructionInstruction
Post-InstrumentationPost-Instrumentation
Page 8
– 8 – DynInst Security© 2001 Barton P. Miller
The DynInst Interface
Machine independent representationObject-based interface to build Abstract
Syntax Trees (AST’s)Write-once, instrument-many (portable)Hides most of the complexity in the API
• Process Hijacker: only 700 lines of user code!
• MPI tracer: 250 lines
Page 9
– 9 – DynInst Security© 2001 Barton P. Miller
Basic DynInst OperationsProcess control:
• Attach/create process• Monitor process status changes• Callbacks for fork/exec/exit
Image (executable program) routines:• Find procedures/modules/variables• Call graph (parent/child) queries• Intra-procedural control-flow graph
Page 10
– 10 – DynInst Security© 2001 Barton P. Miller
Basic DynInst OperationsInferior (application processor) operations:
• Malloc/free– Allocate heap space in application process
• Inferior RPC– Asynchronously execute a function in the
application.
• Load module– Cause a new .so/.dll to be loaded into the
application.
Page 11
– 11 – DynInst Security© 2001 Barton P. Miller
Basic DynInst OperationsInferior operations (continued):
• Remove Function Call– Disable an existing function call in the
application
• Replace Function Call– Redirect a function call to a new function
• Replace Function– Redirect all calls (current and future) to a
function to a new function.
Page 12
– 12 – DynInst Security© 2001 Barton P. Miller
Basic DynInst OperationsBuilding AST code sequences:
• Control structures: if and goto• Arithmetic and Boolean expressions• Get PID/TID operations• Read/write registers and global variables• Read/write parameters and return value• Function call
Page 13
– 13 – DynInst Security© 2001 Barton P. Miller
Security Applications of DynInst
Lots of tool applications of Dyninst by lots of
groups. Here are two security-oriented ones:
License server bypassing
Condor security attacks
Page 14
– 14 – DynInst Security© 2001 Barton P. Miller
License Server Attack: The Bypass
Program License Data
Network
License
Server
Normal: licensed program runs after communicates with license server.
Program License Data
Network
License
Server
Undesired: licensed program refuses to run if license server does not respond.
Page 15
– 15 – DynInst Security© 2001 Barton P. Miller
Example: Adobe FrameMaker
Two-step license verification:• retrieve license data from server [once]• check license data for correctness [often]
In practice:• allow FM to time-out waiting for server• allow FM to attempt to go into “demo” mode• switch FM back to normal mode• insure that future license checks always
succeed
Page 16
– 16 – DynInst Security© 2001 Barton P. Miller
StrategiesComplete reverse engineering:
• not an option– legal problems– complexity (FrameMaker is a 7 MB binary!)
Focus on certain characteristics:• I/O (network sockets) traffic• execution trace
Page 17
– 17 – DynInst Security© 2001 Barton P. Miller
ToolsHigh-level language translators:
• Dyner: interactive, interpreted C subset• Jdyninst: Java to DynInst compiler
Bypasser: an interactive call graph browser• Search and walk application call graph• Resolves function pointers at runtime• Call follow caller or callee paths• Can generate call trace
Page 18
– 18 – DynInst Security© 2001 Barton P. Miller
UseDetermining where to apply changes:
• get trace for a successful run• get trace for a (forced-)failure run• compare to find differences• repeat as needed
Page 19
– 19 – DynInst Security© 2001 Barton P. Miller
DetailsFM calls NlOpenlicenses on start up
• Contacts license server and caches credential if successful
At end of main, FM calls NluiCheckLicense• If credential is not present, call ChangeProductToDemo (cannot save files)
Frequently, during operation, FM will check for cached credentials.
Page 20
– 20 – DynInst Security© 2001 Barton P. Miller
DetailsFM calls NlOpenlicenses on start up
• Contacts license server and caches credential if successful
• Allow this to fail.At end of main, FM calls NluiCheckLicense
• If credential is not present, call ChangeProductToDemo (cannot save files)
Frequently, during operation, FM will check for cached credentials.
Page 21
– 21 – DynInst Security© 2001 Barton P. Miller
DetailsFM calls NlOpenlicenses on start up
• Contacts license server and caches credential if successful
• Allow this to fail.At end of main, FM calls NluiCheckLicense
• If credential is not present, call ChangeProductToDemo (cannot save files)
• Delete the call to ChangeProductToDemo.Frequently, during operation, FM will check
for cached credentials.
Page 22
– 22 – DynInst Security© 2001 Barton P. Miller
DetailsFM calls NlOpenlicenses on start up
• Contacts license server and caches credential if successful
• Allow this to fail.At end of main, FM calls NluiCheckLicense
• If credential is not present, call ChangeProductToDemo (cannot save files)
• Delete the call to ChangeProductToDemo.Frequently, during operation, FM will check
for cached credentials.• Change this call to always return “true”.
Page 23
– 23 – DynInst Security© 2001 Barton P. Miller
Condor Attack: Lurking Jobs
Condor schedules jobs on idle workstationsIn a normal mode, jobs run as a common,
low-privilege user ID: “nobody”.This common user ID provides an
opportunity for an evil lurking process to ambush subsequent jobs (from other users):
Page 24
– 24 – DynInst Security© 2001 Barton P. Miller
Condor Job Structure
Submitting HostSubmitting Host
Shadow Process
Execution HostExecution Host
User Jobsystem calls
Page 25
– 25 – DynInst Security© 2001 Barton P. Miller
Condor Job Structure
Submitting HostSubmitting Host
Shadow Process
Execution HostExecution Host
EvilUser Job
system calls
LurkerProcess
forkfork
Page 26
– 26 – DynInst Security© 2001 Barton P. Miller
Condor Job Structure
Submitting HostSubmitting Host Execution HostExecution Host
LurkerProcess
Page 27
– 27 – DynInst Security© 2001 Barton P. Miller
Condor Job Structure
Submitting HostSubmitting Host
Shadow Process
Execution HostExecution Host
InnocentUser Job
system calls
LurkerProcess
Page 28
– 28 – DynInst Security© 2001 Barton P. Miller
Condor Job Structure
Submitting HostSubmitting Host
Shadow Process
Execution HostExecution Host
InnocentUser Job
system calls
LurkerProcess
attach
Page 29
– 29 – DynInst Security© 2001 Barton P. Miller
Condor Job Structure
Submitting HostSubmitting Host
Shadow Process
Execution HostExecution Host
InnocentUser Job
system calls
Control remotesystem calls
LurkerProcess
rm -rf *
Page 30
– 31 – DynInst Security© 2001 Barton P. Miller
Page 31
– 32 – DynInst Security© 2001 Barton P. Miller
Can We Trust a Remote Job?The threats:
1. Cause the job to make improper remote system calls.
2. Cause the job to calculate an incorrect answer.
3. Steal data from the remote job.
Threat protection strategies:• File sand-boxing (#1)• System call sand-boxing (#1)• Obscure and encode binary (#1)• Replicate remote job (#2)
Page 32
– 33 – DynInst Security© 2001 Barton P. Miller
Sand-Boxing
Shadow process selectively rejects system calls:• Restrict access to specific files or directories• Disallow certain system calls• Disallow certain system call parameter values
Submitting HostSubmitting Host
Shadow Process
Execution HostExecution Host
User Jobsystem calls
Page 33
– 34 – DynInst Security© 2001 Barton P. Miller
Obscuring the Executable
User Job
ModifiedUser Job
Checking Shadow
Modifier/Obscurer
Page 34
– 35 – DynInst Security© 2001 Barton P. Miller
Obscuring the Executable
Goal:Even if an intruder can see, examine, and fully
control the remote job, no harm can come to the local machine.
Page 35
– 36 – DynInst Security© 2001 Barton P. Miller
Obscuring the Executable
Modify the executable:• Replace each system call site with call to a
unique random function nameThis makes understanding the executable harder and
makes checking the system call stream easier.
• Delete any constant parametersValues like file names or descriptor numbers are hidden.
• Pad parameter other lists so all calls have equal number
• Permute the order of parameters• Add dummy system calls throughout code
Page 36
– 37 – DynInst Security© 2001 Barton P. Miller
Obscuring the Executable
Generate a Checking Shadow:• Build a FSM that reflects the inter-procedural
control-flow of the job.Note that this FSM has increased accuracy, since each
call site has a unique name
• Map random function names back to their actual system calls.
• Insert parameter values and present in correct order.
• Ignore dummy calls.
Page 37
– 38 – DynInst Security© 2001 Barton P. Miller
Obscuring the Executable
Some notes of interest:• All hiding and replacement is done before the
job is submitted to Condor.No knowledge of the encryption or keys is contained in
the remote job.
• All replacing is done in the Checking Shadow, which only resides on your local machine.
• FSM accuracy and computational complexity is simplified by the function renaming.
Our early results look good. A tech report will be available by the end of November.
Page 38
– 39 – DynInst Security© 2001 Barton P. Miller
How to Get a Copy of DynInst:Release 2.3 (release 3.0 imminent)
• Free for research use.• Runs on Solaris (SPARC & x86), Windows
NT, AIX/SP2, Linux (x86), Irix (MIPS),Tru64 Unix (Alpha).
http://www.paradyn.org
http://www.dyninst.org
[email protected]