Yet Another Heapspray Detector Danny Kovach Raytheon SI
Dec 31, 2015
Introduction
• Currently we monitor an application in a VM for such behavior as:– Loading drivers– Creating executable files– Network activity
• Heap sprays are very hard to detect.
What is a heapspray?
• Technique used to put executable code onto the heap.
• Consists of– NOP sled– Shellcode
• Goal: direct execution flow to the NOP sled; shellcode.
Assumptions
• Bytes on a normal heap should be randomly distributed (white noise)
• Fourier transform of white noise has constant magnitude.
Visualizing the Heap(normal program operation)
256 512 768 1024 1280 1536 1792 2048
50 000
100 000
150 000
Fourier Transforms of the Heap(normal program operation)
200 400 600
500 000
1 .0 106
1 .5 106
2 .0 106
2 .5 106
3 .0 106
Fourier Transforms of the Heap(normal program operation)
200 400 600 800 1000
500 000
1 .0 106
1 .5 106
2 .0 106
2 .5 106
3 .0 106
3 .5 106
Visualizing the Heap(heap spray)
256 512 768 1024
20 000
40 000
60 000
80 000
100 000
120 000
140 000
Visualizing the Heap(heap spray)
256 512 768 1024 1280 1536 1792 2048
20 000
40 000
60 000
80 000
100 000
Results
• 100% accurate for all our test cases.
• Rushed into production (without further testing).
• FAIL!
Advantages of a Statistical Approach
• Easy to code
• Friendly to system resources
• More general than hard coded approach
• Theoretically sound
Results
• Out of over 500 files tested, we had 100% success.
• 0 false positives
• 0 false negatives
How to defeat
• Write shellcode so as to minimally alter normal distribution.
• Most likely will leave some signature.
• Invites cat and mouse game.
References
1. http://research.microsoft.com/en-us/projects/nozzle/
2. http://en.wikipedia.org/wiki/Heap_spraying
3. https://lirias.kuleuven.be/bitstream/123456789/265421/1/fulltext.pdf
4. http://www.mathnstuff.com/math/spoken/here/2class/90/normal.htm