Audits, Triage,And the Future of Hashing

8/12/2019 Audits, Triage,And the Future of Hashing

http://slidepdf.com/reader/full/audits-triageand-the-future-of-hashing 1/39

Audits, Triage,

and the Future of Hashing

Jesse Kornblum



2

Outline

• Identical Files

• Hash Collisions

• Why you should stop using MD5

• SHA-3

• NSRL

• Hash Set Auditing

• Fast Hash Matching

• Questions



Identical Files

• Bit for bit identical

– Identical content

– Timestamps, filenames, may be different

• Cryptographic hashing does NOT prove files are identical

3



Hash Collisions

• When two different inputs hash to the same value

• When A ≠ B, H(A) = H(B)

• Extension of the Pigeon Hole Principle

4



Hash Collisions

• Pigeon-Hole Principle

• 2128 possible MD5 values

• For a file of length n, 2n

possible inputs

• For a 128KB file, there are2131,072 possible inputs

• 131,072 >> 128

• Therefore there will be

hash collisions

5

Picture courtesy Flickr user addedentry and used under a Creative Commons license, http://www.flickr.com/photos/addedentry/3273096118/



Types of Attacks

• Collision Attack

– Find m1 and m2 such that H(m1) = H(m2)

– Can be used to create apps with different functionality but the

same hash

• But you can't choose the hash output

• Chosen Prefix Collision Attack

– Given p1 and p2, find m1 and m2 such that H(p1||m1) = H(p2||m2)

– Can be used to forge code signatures

• But you can't choose the hash output

6



Types of Attacks

• Preimage Attack

– Given hash output h, find m such that H(m) = h

– Find a new input which matches a chosen hash

• That new input may not be meaningful

• Second Preimage Attack – Given m1, find m2 such that H(m1) = H(m2)

– From existing exe, generate new file which has the same hash

• That new input may not be meaningful

7



MD5 Attacks

• Published in 1992

• Cryptographically broken in 1996

– Different from a practical break

• Collision technique developed in 2004 by Wang et al.

• Chosen prefix attack published in 2007

8



MD5 Attack Demo

9



Types of Attacks



– Takes seconds on a netbook


– Given p1 and p2, find m1 and m2 such that H(p1||m1) = H(p2 ||m2)

• Preimage Attack


• Second Preimage Attack

– Given m1, find m2 such that H(m1) = H(m2)

10



Types of Attacks



– Takes seconds on a netbook


– Given p1 and p2, find m1 and m2 such that H(p1||m1) = H(p2 ||m2) – Takes a cluster of Playstation 3s

• Preimage Attack


• Second Preimage Attack

– Given m1, find m2 such that H(m1) = H(m2)

11



The Future of MD5

12

Picture courtesy Flickr user katerha and used under a Creative Commons license, http://www.flickr.com/photos/katerha/4526272937/



Hash Algorithms Families

• MD5 is based on Merkle –Damgård construction

– Method to turn one-way compression functions into collision-

resistant hash functions

– So are SHA-1 and SHA-2

• Weaknesses found in MD5 may be applicable to SHA-1 and SHA-2 – They may not survive long!

13



That Algorithm is Our Last Hope

“No, there is another”

14



Hash Algorithm Families

• There are many other families

• Davies –Meyer

• Matyas –Meyer –Oseas

• Hirose

• Miyaguchi –Preneel – Whirlpool algorithm

– Part of md5deep suite

15



SHA-3

• National Institute of Standards and Technology (NIST)

• Competition for SHA-3 standard

– Much like Rijndael became “AES”

• Three year process

• Finalists as of 23 Jan 2012:

– BLAKE, Grøstl, JH, Keccak, Skein

• Final conference in Spring 2012 – http://csrc.nist.gov/groups/ST/hash/sha-3/index.html

16



NSRL

• National Software Reference Library (NSRL)

– Created by NIST in 2001

• Known files

– Not guaranteed to be known good

– NIST will hash anything – Vendor contact for each file

17



NSRL

• Data set is HUGE

– 78 million hashes in the set

– 21 million unique hashes

– About 1.6GB of data

• Best resource that nobody uses• Lots of programs can parse the file format

– But have trouble with the full data set

18





nsrlquery

Displays unknown files as default

$ md5deep –br * | nsrllookup

305e40dee29d261d0a3dc466f2184e35 unknown.exe

607e033a16006ed1e9987cfc62562f72 EVILEVIL.exe

Can also display known files

$ md5deep –br * | nsrllookup -k

e97295de2a9fde547feab4fe41df16ca mspaint.exe

eee470f2a771fc0b543bdeef74fceca0 msiexec.exe

20



Kyrus NSRL Server

• Server requires about 1GB of RAM

– Takes a while to start

• Kyrus is testing a public nsrlquery server with MD5 hashes

– nsrl.kyr.us

– Add -s flag for remote server

C:\> md5deep * | nsrllookup -s nsrl.kyr.us

305e40dee29d261d0a3dc466f2184e35 unknown.exe

607e033a16006ed1e9987cfc62562f72 EVILEVIL.exe

21



nsrlquery Demo

22



Hash Set Auditing

• Hash sets are used to detect changes

– Verifying the contents of downloaded file

– Determining if your forensics tool has made any changes

• Hash set tools are great for detecting identical files

– Break down when asked to detect changes

• Current Approaches

– Report known files found

– Or report unknown files found

– Or report known files not found

23



Hash Set Auditing

24



Example - Bogocopy

C:\> dir /b src

foo.txt bar.txt

C:\> md5deep src\* > known.txt

C:\> bogocopy src dest

C:\> md5deep -lm known.txt dest\*

dest\foo.txt

dest\bar.txt

C:\> dir /b dest

foo.txt bar.txt CONFESSION.DOCX

25



Hash Set Auditing

• Current Approaches

– Report known files found

– Or report known files not found

– Or report unknown files found

• We want all three of these!• Along with

– Report known files found in new location

• Determine what is there

• Determine what's supposed to be there

• Highlight any mismatches

26



Hash Set Auditing

• Hashdeep

– Part of the md5deep suite

– http://md5deep.sf.net/

• Can do positive and negative matching

• Multihashing• Hash set audits

– Reports any mismatches

– Finds new files, moved files, files not found

27



Example – Bogocopy with Hashdeep

C:\> dir /b src

foo.txt bar.txt

C:\> hashdeep -b src\* > known.txt

C:\> bogocopy src dest

C:\> hashdeep -bak known dest\*

hashdeep: Audit failed

C:\> dir /b dest

foo.txt bar.txt CONFESSION.DOCX

28



Example – Bogocopy with Hashdeep

C:\> hashdeep–

vvbak known.txt dest\*CONFESSION.DOCX: No match

hashdeep: Audit failed

Files matched: 2

Files partially matched: 0

Files moved: 0

New files found: 1

Known files not found: 0

29



WoW64 Gotcha

• Windows on Windows64

– x86 emulator for x64 based Windows systems

– 32-bit view for 32-bit programs running on a 64-bit OS

– http://msdn.microsoft.com/en-us/library/aa384249(v=vs.85).aspx

• So what?

• 32-bit programs have a different view

– For example C:\Windows\System32

– md5deep vs. md5deep64

30



WoW64 Gotcha

• On a 64-bit OS, 32-bit programs see a different file

C:> md5deep Windows\System32\ieapfltr.dll

ee9d715af1b928982f417238b9914484

C:\Windows\System32\ieapfltr.dll

(This is actually C:\Windows\SYSWOW64\ieapfltr.dll)

C:\> md5deep64 Windows\System32\ieapfltr.dll

8eada158d964e3fd1999ad96c9c507ff

C:\Windows\System32\ieapfltr.dll

31





Fast Hash Matching

• I did not invent this

• Several other names

– Fibonnacci hashing

– AccessData Triage Hashing

– Piecewise Hashing

– Partial hashing – And many more

33

Picture courtesy Flickr user nvarvel and used under a Creative Commons li cense, http://www.flickr.com/photos/nvarvel/6269179660/



Fast Hash Matching

• Traditional Approach:

• For each known file

– Read and compute hash, H(known)

• For each unknown file: – Read and compute hash, H(unknown)

– For each known hash:

• If H(unknown) == H(known)

– Match!

34



Fast Hash Matching

• Traditional Approach:

• For each known file

– Read and compute hash, H(known)

• For each unknown file: – Read and compute hash, H(unknown)

– For each known hash:

• If H(unknown) == H(known)

– Match!

35



Assumptions

• Searching for identical files

– Based on content

• If any part of the content is not the same, the files are not identical

• Example:

– Identical files are the same size – If two files are not the same size, they are not identical

• Therefore we should compare file sizes first

– Fast!

• Then part of the file

• Then the whole file

36



Fast Hash Matching Demo

37



38

Outline

• Identical Files

• Hash Collisions

• Why you should stop using MD5

• SHA-3

• NSRL

• Hash Set Auditing• Fast Hash Matching



Questions?

Jesse Kornblum

[email protected]

39

Audits, Triage,And the Future of Hashing

Documents