Research in Next-Generation Digital Forensics

Research in Next-Generation Digital Forensics

Golden G. Richard III, Ph.D.Associate Professor

Dept. of Computer Science [email protected]

http://www.cs.uno.edu/~golden

Digital Forensics Research Group

• Fall 2006:– Thursdays @ 1pm in NSSAL (Math 322)

• Primary Collaborators:– Vassil Roussev [UNO CS]– Vico Marziale [UNO Ph.D. student]– Frank Adelstein [ATC-NY]

Digital Forensics

Definition: “Tools and techniques to recover, preserve, and examine digital evidence on or transmitted by digital devices.”

Devices include computers, PDAs, cellular phones, videogame consoles, copy machines, printers, …

Examples of Digital Evidence• Threatening emails• Documents (e.g., in places they shouldn’t be)• Suicide notes• Bomb-making diagrams• Malicious Software

– Viruses– Worms– …

• Child pornography (contraband)• Evidence that network connections were made

between machines• Cell phone SMS messages

Facts (or: Why Digital Forensics?)

• Deleted files aren’t securely deleted– Recover deleted file + when it was deleted!

• Renaming files to avoid detection is pointless

• Formatting disks doesn’t delete much data• Web-based email can be (partially)

recovered directly from a computer• Files transferred over a network can be

reassembled and used as evidence

Facts (2)• Uninstalling applications is much more difficult than it

might appear…• “Volatile” data hangs around for a long time (even across

reboots)• Remnants from previously executed applications• Using encryption properly is difficult, because data isn’t

useful unless decrypted• Anti-forensics (privacy-enhancing) software is mostly

broken• “Big” magnets (generally) don’t work• Media mutilation (except in the extreme) doesn’t work• Basic enabler: Data is very hard to kill

Privacy Through Media Mutilation

degausser

or

orforensically-securefile deletion software(but make sure it works!)

or

Digital Forensics Process• Identification of potential digital evidence

– Where might the evidence be?– Which devices did the suspect use?

• Preservation and copying of evidence– On the crime scene…– First, stabilize evidence…prevent loss and contamination– If possible, make identical copies of evidence for

examination• Careful examination of evidence• Presentation

– “The FAT was fubared, but using a hex editor I changed the first byte of directory entry 13 from 0xEF to 0x08 to restore ‘HITLIST.DOC’…”

– “The suspect attempted to hide the Microsoft Word document ‘HITLIST.DOC’ but I was able to recover it without tampering with the file contents.”

• Legal: Balance of need to investigate vs. privacy

“Traditional” Digital Forensics

• Pull the plug• “Image” (make bit-perfect copies) of hard drives,

floppies, USB keys, etc.• Use forensics software to analyze copies of

drives• Investigator typically uses a single computer to

perform investigation in the lab• Present results to client, to officer-in-charge,

court

Traditional: Where’s the evidence?

• Undeleted files, expect some names to be incorrect• Deleted files• Windows registry• Print spool files• Hibernation files• Temp files (all those .TMP files!)• Slack space• Swap files• Browser caches• Alternate or “hidden” partitions• On a variety of removable media (floppies, ZIP,

Jazz, tapes, …)

But Evidence is Also…

• In RAM• “In” the network• On machine-critical machines

– Can’t turn off without severe disruption– Can’t turn them ALL off just to see!

• On huge storage devices– 1TB server: image entire machine and drag it

back to the lab to see if it’s interesting?– 10TB?

Next Generation: Needs• Broad:

– Better design, better software• Yes, some of it is engineering (and hacking)• Someone has to do it

– Better vision, application of ‘real’ CS to problems• More specific:

– Need for speed– Machine correlation– Machine profiling– Better auditing of investigative process– On-the-spot forensics: Triage– Live forensics– Network forensics– Specific tools for detection and remediation of malware– Phishing investigation– …

Next Generation: UNO

• Better file carving• Forensic-aware OS components• In-place file carving • Forensic accountability• On-the-spot forensics• Distributed digital forensics

File Carving: Basic Ideaone cluster

one sector

header, e.g., 0x474946e8e761(GIF)

unrelated disk blocks interesting file

footer, e.g., 0x003B(GIF)

“milestones”or “anti-milestones”

File Carving: Fragmentation

header, e.g., 0x474946e8e761(GIF)

footer, e.g., 0x003B(GIF)

“milestones”or “anti-milestones”

File Carving: Fragmentation

header, e.g., 0x474946e8e761(GIF)

footer, e.g., 0x003B(GIF)

File Carving: Damaged Files

header, e.g., 0x474946e8e761(GIF)

“milestones”or “anti-milestones”

No footer

File Carving: Doing a Better Job• Better design • Faster• Distributed implementation• More flexible description of file types• Automatic generation of type descriptions

– Patterns– Rule sets

• Multiple-pass carving– Carve, “remove” validated files from block list, re-

carve, hope that some fragmented files coalesce• Block-sniffing

File Carving: Block Sniffing

header, e.g., 0x474946e8e761(GIF)

Do these blocks “smell” right?

• N-gram analysis• entropy tests• parsing

Better Software: File Carving: Scalpel

• Two-pass design• Minimizes:

– Reads– Seeks– Writes– Data copying– Memory usage

• Doesn’t yet incorporate all of the carving wizardry we have in mind

G. G. Richard III, V. Roussev, "Scalpel: A Frugal, High Performance File Carver," Proceedings of the 2005 Digital Forensics Research Workshop (DFRWS 2005), New Orleans, LA.

Some Scalpel Results (1)

Big targets, large carve sizes, huge improvement (over 5 hours faster)

Tread + 238,270,750,000 bytes

Some Scalpel Results (2)

Big targets, large carve sizes, huge improvement (over 7 hours faster)

Tread + 117,622,357,936 bytes

OS Support for Digital Forensics

• Export raw disk devices across network for processing– Others: network block device (NBD)– Us: optimization

• “In-place” file carving– Us: Export results from file carving as a

filesystem, w/ minimal extra storage• Better auditing of investigative process

– Us: “digital evidence bag”-aware filesystems

FUSE (Filesystem in User Space)

user space

kernel space

LinuxVirtual File System

Interface(VFS)

C library

dd if=/evidence/DEC/img.dd of=copy.dd

read()

FUSE

ext3

reiserFS

C library

FUSE library

Filesystem Implementation

In-Place File Carving

preview databaseFUSE

scalpel_fs

client applications

nbd server

nbd client

networklocal drive

remote drive

G. G. Richard III, V. Roussev, V. Marziale, “In-Place File Carving,” submitted to the Third Annual IFIP WG 11.9 International Conference on Digital Forensics, 2007.

Scalpel

Better Auditing

Want: Digital Evidence Bags

See: P. Turner, “Unification of Digital Evidence from Disparate Sources (Digital Evidence Bags),” DFRWS 2005See: Common Digital Evidence Storage Format (CDESF) working group, http://www.dfrws.org/CDESF/.

Better Auditing (2)

DEC

(DEB, AFF,Gfzip …)

FDAM

dd scalpel FTK…

VFS Interface

TSK

EvidenceData

Audit Log

Import/Export

Applications(User space)

(Kernel)OperatingSystem

Block-levelData Access

FilesystemData Access

FDAM Block Device

G. G. Richard III, V. Roussev, "Toward Secure, Audited Processing of Digital Evidence: Filesystem Support for Digital Evidence Bags," Research Advances in Digital Forensics, Springer, 2006.

DigitalEvidenceContainer

Bluepipe: On the Spot Digital Forensics

Cu Bootable Bluepipe CD Removable media

Target

Bluetooth or 802.11dongle 3G/VPN

Remote investigator(s)

Handheld Bluepipe client

Y. Gao, G. G. Richard III, V. Roussev, “Bluepipe: An Architecture for On-the-Spot Digital Forensics,” International Journal of Digital Evidence (IJDE), 3(1), 2004.

Bluepipe Patterns

<BLUEPIPE NAME=”findcacti”> <!-- find illegal cacti pics using MD5 hash dictionary --> <DIR TARGET=”/pics/” /> <FINDFILE USEHASHES=TRUE LOCALDIR=”cactus” RECURSIVE=TRUE RETRIEVE=TRUE MSG="Found cactus %s with hash %h "> <FILE ID=3d1e79d11443498df78a1981652be454/> <FILE ID=6f5cd6182125fc4b9445aad18f412128/> <FILE ID=7de79a1ed753ac2980ee2f8e7afa5005/> <FILE ID=ab348734f7347a8a054aa2c774f7aae6/> <FILE ID=b57af575deef030baa709f5bf32ac1ed/> <FILE ID=7074c76fada0b4b419287ee28d705787/> <FILE ID=9de757840cc33d807307e1278f901d3a/> <FILE ID=b12fcf4144dc88cdb2927e91617842b0/> <FILE ID=e7183e5eec7d186f7b5d0ce38e7eaaad/> <FILE ID=808bac4a404911bf2facaa911651e051/> <FILE ID=fffbf594bbae2b3dd6af84e1af4be79c/> <FILE ID=b9776d04e384a10aef6d1c8258fdf054/> </FINDFILE> </BLUEPIPE>

Distributed Digital Forensics

V. Roussev, G. G. Richard III, "Breaking the Performance Wall: The Case for Distributed Digital Forensics,“ Proceedings of the 2004 Digital Forensics Research Workshop (DFRWS 2004), Baltimore, MD

750GB750GB

300GB 300GB

Distributed Digital Forensics• Scalable

– Want to support at least IMAGE SIZE / RAM_PER_NODE nodes• Platform independent

– Want to be able to incorporate any (reasonable) machine that’s available

• Lightweight– Horsepower is for forensics, not the framework—less fat

• Highly interactive• Extensible

– Allow incorporation of existing sequential tools– e.g., stegdetect, image thumbnailing, file classification, hashing,

…• Robust

– Must handle failed nodes smoothly

Distributed Digital Forensics (2)

SHUTDOWN STARTUP

CACHE

FETCH

JOIN / LEAVE

CANCEL / EXITHASH / GREP / COMMAND/ …

DONE / REPORT / ERROR

CoordinatorWorker Worker

Common Store

Local Store

LOAD

STORE

SHUTDOWN STARTUP

CACHE

FETCH

JOIN / LEAVE

CANCEL / EXITHASH / GREP / COMMAND/ …

DONE / REPORT / ERROR

CoordinatorWorker Worker

Common Store

Local Store

LOAD

STORE

Distributed Digital Forensics (3)

SCSIRAID: 504GBFile Server

CPU: 2x1.4GHz XeonRAM: 2GB

Switch96-port, 10/100/1000 Mb24 Gb Backplane

1Gb

NodeCPU: 2.4 GHz Pentium 4RAM: 1 GB

SCSIRAID: 504GBFile Server

CPU: 2x1.4GHz XeonRAM: 2GB

Switch96-port, 10/100/1000 Mb24 Gb Backplane

1Gb

NodeCPU: 2.4 GHz 4RAM: 1 GB

Beowulf [RIP], Slayer of Computer Criminals…

DDF: Results (1)

• Live string search:“Vassil Roussev”

• Regular expression search:

v[a-z]*i[a-z]*a[a-z]*g[a-z]*r[a-z]*a

Search time:String Expression

(mm:ss)

Search time:Regular Expression

(mm:ss)

FTK 08:27 41:508-node System 00:27 00:28

DDF: Results (2)• Stego detection using Stegdetect 0.5 under RH9 Linux

on the cluster• Traditional:

– 6GB image mounted using loopback device– find /mnt/loop –exec ./stegdetect ‘{}’ \;– 790 seconds == 13:10 minutes

• Using the distributed framework– Stegdetect 0.5 code incorporated into framework– Detection against cached files– “STEGO” command (after IMAGE/CACHE)– 82 seconds == 1:22 minutes

• 9.6X faster with 8 machines• CPU bound operation

DDF: To Do List

• User interface! (unless you love Putty)

DDF: To Do (2)• Case persistence• Secure support for overlapping cases• Better fault tolerance• Intelligent caching schemes to support larger

images• Collaboration with colleagues (you?) working in:

– Image analysis/classification– Speech recognition– More stego– Other CPU horsepower-intensive, forensics-

applicable stuff– We provide cycles…you provide…

Current: Live Forensics• Physical memory dumps

– Hard to do when adversarial OS is present– Via USB hacking?– Firewire proof of concept developed by Maximillian

Dornseif • Defeating process hiding techniques, e.g., FU

“rootkit”– Check OS components from many angles

• Remnants of applications (executed) past…– e.g., instant messenger fragments– e.g., recent invocations of process hiding– e.g., fingerprints of recently executed (or executing)

malware

Conclusion: Lots of Work To Do

• Benevolent hacking (engineering) meets science

• Desperately need methods for pipelining investigative process

• Live forensics critically important– volatile computing– whole disk encryption– hardware-based whole disk encryption!– nasty malware

Conclusion (2)• Arguably, almost any field in CS can collaborate

– All media handling needs work– Algorithms for dealing with huge, partially-organized

datasets– Attribution– Correlation– Profiling– Document similarity measures– Databases– High-performance computing– OS Internals

Random Bedside Reading…• http://www.dfrws.org (Digital Forensics Research Workshop)• http://www.ijde.org/ (International Journal of Digital Evidence)• F. Adelstein, “Live Forensics: Diagnosing Your System Without Killing it First,” Communications

of the ACM, February 2006.• M. A. Caloyannides, Privacy Protection and Computer Forensics, Second Edition, 2004.• B. Carrier, File System Forensic Analsis, Addison-Wesley, 2005.• B. Carrier, “Risks of Live Digital Forensics Analysis,” Communications of the ACM, February

2006. • E. Casey, Digital Evidence and Computer Crime, Academic Press, 2004.• J. Chow, B. Pfaff, T. Garfinkel, M. Rosenblum, “Shredding Your Garbage: Reducing Data

Lifetime Through Secure Deallocation,” 14th USENIX Security Symposium, 2005.• M. Geiger, “Evaluating Commercial Counter-Forensic Tools,” 5th Annual Digital Forensic

Research Workshop (DFRWS 2005), New Orleans, 2005. • G. G. Richard III, V. Roussev, "Next Generation Digital Forensics," Communications of the ACM,

February 2006. • G. G. Richard III, V. Roussev, “Digital Forensics Tools: The Next Generation,” invited chapter in

Digital Crime and Forensic Science in Cyberspace, IDEA Group Publishing, 2005.• A. Schuster, “Searching for Processes and Threads in Microsoft Windows Memory Dumps,” 6th

Annual Digital Forensic Research Workshop (DFRWS 2006), West Lafayette, IN, 2006. • S. Sparks, J. Butler, “Raising the Bar for Windows Rootkit Detection,” Phrack Issue # 63. • G. Hoglund, J. Butler, “Rootkits: Subverting the Windows Kernel,” Addison-Wesley, 2005.

Presentation available:http://www.cs.uno.edu/~golden/teach.html

[email protected]

Security Lab (NSSAL): Math 322

?