Top Banner
Browser Bug Hunting Memoirs of a last man standing Atte Kettunen (@attekett) OUSPG https://code.google.com/p/ouspg/
51

44CON 2013 - Browser bug hunting - Memoirs of a last man standing - Atte Kettunen

May 10, 2015

Download

Technology

44CON

Just like drinking is not a game in Finland; neither is browser bug hunting - it’s serious business! Browser bugs have been supporting Atte Kettunen (@attekett) traditional Finnish way of living since late 2011 and he’s going to tell you all about how he has been living the dream browser bug hunting - focusing on one of the most secure browser around, Google Chrome!

He’ll tell you a tale of his experiences with bounty programs and how those have evolved since he started way back (vendors can show the love too!) and how he’s managed to survive in the harsh environment of browser bug hunting. He’ll impart some important bug hunting social skills by showing you how and how NOT to step on the others guys toes - very competitive cottage industry is browser bug hunting. ;)

Atte is also going to share with you how and why he selected his current target feature *(still full of bugs!), how he built his fuzzer-module(s) and the results produced. We’ll all walk a mile in a bug hunters shoes together and take a peek at the tool sets, as well as the infrastructures that are used to find browser bugs by individuals and vendors!
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: 44CON 2013 - Browser bug hunting - Memoirs of a last man standing - Atte Kettunen

Browser Bug HuntingMemoirs of a last man standing

Atte Kettunen (@attekett)OUSPG

https://code.google.com/p/ouspg/

Page 2: 44CON 2013 - Browser bug hunting - Memoirs of a last man standing - Atte Kettunen

Picture by @dominic_sim

Page 3: 44CON 2013 - Browser bug hunting - Memoirs of a last man standing - Atte Kettunen

Started at OUSPG in summer 2011

First security bug from Chrome 2011-12

Since then ~100 Vulns~60 Rewards 39 CVEs

Atte Kettunen

Page 4: 44CON 2013 - Browser bug hunting - Memoirs of a last man standing - Atte Kettunen

Mozilla since 2004 - Sec-High/Critical $3,000

Google since 2010 - Typical security bugs $1,000-$3,133.7

- Possibility for bonus rewards● PoC, exploit, awesomeness

(Microsoft 2013 June 25 - July 25)

Browser Bug Bounty Programs

Page 5: 44CON 2013 - Browser bug hunting - Memoirs of a last man standing - Atte Kettunen

Easy to get started - Lots of bugs \o/

Helpful vendor security teams and supportive responses to first bug submissions

Supportive (secretive/competitive) community of other bounty hunters

Browser Bug Bounty Programs

Page 6: 44CON 2013 - Browser bug hunting - Memoirs of a last man standing - Atte Kettunen

● Use-after-free○ DOM○ CSS○ Rendering

● Buffer-overflow○ Media formats○ Parsers○ Decoders○ Coordinates

Where the bugs are

Page 7: 44CON 2013 - Browser bug hunting - Memoirs of a last man standing - Atte Kettunen

AddressSanitizer - global-buffer-overflow - READ of size 2 #0 nsCharTraits<unsigned short>::length() #1 nsAString_internal::Assign()

.Repro-file:

<link rel="stylesheet" href="data:text/css;charset=utf-16,p#two%1%7Bbackground-color%65535A%3B%7D%0D%0A"/>

Some bug - CVE-2012-4185 - Firefox

Page 8: 44CON 2013 - Browser bug hunting - Memoirs of a last man standing - Atte Kettunen

AddressSanitizer - heap-use-after-free - READ of size 2 #0 WebCore::nextBreakablePosition() #1 ...::RenderBlock::LineBreaker::nextLineBreak() .Repro-file:<html><body><ruby>

<q style="column-gap:2;">a</ruby><cite style="word-break: break-all;">a<q style="text-transform:uppercase;">a<sup style="text-overflow:ellipsis;"></body></html>

Some bug - Regression - Chrome

Page 9: 44CON 2013 - Browser bug hunting - Memoirs of a last man standing - Atte Kettunen

==3213== ERROR: AddressSanitizer heap-buffer-overflow on address 0x7f50cd6ffcf8 at pc 0x7f50dd159dde bp 0x7fff3e0accd0 sp 0x7fff3e0accc8READ of size 2 at 0x7f50cd6ffcf8 thread T0 #0 0x7f.de in WebCore::CSSParser::lex(void*) ???:0 #1 0x7f.78 in cssyyparse(void*) ???:0 #2 0x7f.40 in WebCore::CSSParser::parseDeclaration().

Repro-file:<a style=top:-1px>

Some bug - Regression - Chrome

Page 10: 44CON 2013 - Browser bug hunting - Memoirs of a last man standing - Atte Kettunen

Three golden rules:

Hunting for living

Page 11: 44CON 2013 - Browser bug hunting - Memoirs of a last man standing - Atte Kettunen

Three golden rules:

1. Stay green

Hunting for living

Page 12: 44CON 2013 - Browser bug hunting - Memoirs of a last man standing - Atte Kettunen

Three golden rules:

1. Stay green - Features

2. Stay green

Hunting for living

Page 13: 44CON 2013 - Browser bug hunting - Memoirs of a last man standing - Atte Kettunen

Three golden rules:

1. Stay green - Features

2. Stay green - Competition

3. Stay green

Hunting for living

Page 14: 44CON 2013 - Browser bug hunting - Memoirs of a last man standing - Atte Kettunen

Three golden rules:

1. Stay green - Features

2. Stay green - Competition

3. Stay green - Tools

Hunting for living

Page 15: 44CON 2013 - Browser bug hunting - Memoirs of a last man standing - Atte Kettunen

1. Stay green - Features● New features are published all the time

○ New code \o/

● Some changes are not highlighted○ Minor updates to JavaScript API support etc.

● Old bugs fixed○ New code \o/

● Old features can change○ Prefixes disappear(-webkit,-moz), ○ Features can get disabled

Hunting for living

Page 16: 44CON 2013 - Browser bug hunting - Memoirs of a last man standing - Atte Kettunen

1. Stay Green - Features

● Firefox Aurora - Release note: "Partial support for Web Audio, targeted at web developers for testing" (May 17, 2013)

Hunting for living

Page 17: 44CON 2013 - Browser bug hunting - Memoirs of a last man standing - Atte Kettunen

2. Stay green - Competition

● Tools○ Different approach -> Different bugs?

● Targets○ Find new minefields

● Platforms○ Different code on different platforms

Hunting for living

Page 18: 44CON 2013 - Browser bug hunting - Memoirs of a last man standing - Atte Kettunen

2. Stay green - Competition

@cevans: "@j00ru has melted polar ice with his PDF fuzzing on 9k cores."

Hunting for living

Page 19: 44CON 2013 - Browser bug hunting - Memoirs of a last man standing - Atte Kettunen

3. Stay green - Tools

● Instrumentations○ New instrumentation -> detect new issues

● Build environments○ Broken builds @#!¤#...

● Fuzzers○ New techniques

Hunting for living

Page 20: 44CON 2013 - Browser bug hunting - Memoirs of a last man standing - Atte Kettunen

3. Stay green - Tools<Q>: WTF??? On Chromium startup:==25254== ERROR: AddressSanitizer: global-buffer-overflow on address 0x000011d3dde5 at pc 0x5ab21a bp 0x7fff00659450 sp 0x7fff00659428READ of size 10 at 0x000011d3dde5 thread T0 #0 0x5ab219 in __interceptor_memcmp _asan_rtl_ #1 0xa1edc08 in fillInUnixFile .../sqlite/amalgamation/sqlite3.c:28654 #2 0xa1efe7c in unixOpen .../sqlite/amalgamation/sqlite3.c:29294<A>: Diff of /trunk/tools/build/scripts/slave/runtest.py:+ # Avoid aggressive memcmp checks until http://crbug.com/178677 is fixed.+ os.environ['ASAN_OPTIONS'] = 'strict_memcmp=0'

Hunting for living

Page 21: 44CON 2013 - Browser bug hunting - Memoirs of a last man standing - Atte Kettunen

● Instrumentation

● Fuzzers

● Hardware/Infrastructure

Tools

Page 22: 44CON 2013 - Browser bug hunting - Memoirs of a last man standing - Atte Kettunen

● Clang compiler plugin

● Adds instrumentation to check memory access at runtime

● Similar to Valgrind

● Only 2x slowdown

● Created at Google

● Used by Google & Mozilla

● Linux & OS X● http://www.chromium.org/developers/testing/addresssanitizer

AddressSanitizer

Page 23: 44CON 2013 - Browser bug hunting - Memoirs of a last man standing - Atte Kettunen

● Awesome with use-after-frees

● Very good for buffer-overflows and out of bounds access

● Good but confused with type confusions

AddressSanitizer

Page 24: 44CON 2013 - Browser bug hunting - Memoirs of a last man standing - Atte Kettunen

==6==ERROR: AddressSanitizer: heap-use-after-free on address 0x6070000268d0 at pc 0x7f845771029f bp 0x7fff...2a0 sp 0x7fffc7eea298READ of size 8 at 0x6070000268d0 thread T0 (chrome) #0 0x7f845771029e (... /asan-linux-release-209136/chrome+0x96f229e) #1 0x7f84576aacea (... /asan-linux-release-209136/chrome+0x968ccea) #2 0x7f8451ce00f3 (... /asan-linux-release-209136/chrome+0x3cc20f3)

.0x6070000268d0 is located 64 bytes inside of 72-byte region [0x607000026890,0x6070000268d8)freed by thread T19 (AudioOutputDevi) here: #0 0x7f844f58e101 (... /asan-linux-release-209136/chrome+0x1570101) #1 0x7f845887b5ec (... /asan-linux-release-209136/chrome+0xa85d5ec)

.

AddressSanitizer

Page 25: 44CON 2013 - Browser bug hunting - Memoirs of a last man standing - Atte Kettunen

==6==ERROR: AddressSanitizer: heap-use-after-free on address 0x6070000268d0 at pc 0x7f845771029f bp 0x7fff...2a0 sp 0x7fffc7eea298READ of size 8 at 0x6070000268d0 thread T0 (chrome) #0 0x7f845771029e in WebCore::WaveShaperDSPKernel::lazyInitializeOversampling(...) .../WebKit/Source/wtf/OwnPtr.h:138 #1 0x7.a in WebCore::WaveShaperProcessor::setOversample(...) .../WebKit/Source/modules/webaudio/WaveShaperProcessor.cpp:70 .0x6070000268d0 is located 64 bytes inside of 72-byte region [0x607000026890,0x6070000268d8)freed by thread T19 (AudioOutputDevi) here: #0 0x7.1 in operator delete(void*) _asan_rtl_ #1 0x7.c in WebCore::AudioDSPKernelProcessor::uninitialize() src/third_party/WebKit/Source/wtf/OwnPtrCommon.h:47 .

AddressSanitizer

Page 26: 44CON 2013 - Browser bug hunting - Memoirs of a last man standing - Atte Kettunen

● Used to instrument binaries

● Redirects heap-related calls to own run-

time library

● Currently only heap-instrumentation

● Chrome/Chromium only atm.

● About 3x Slowdown

● Windows only● https://code.google.com/p/sawbuck/wiki/SyzyASanDesignDocument

SyzyASan

Page 27: 44CON 2013 - Browser bug hunting - Memoirs of a last man standing - Atte Kettunen

SyzyASAN error: heap-buffer-overflow on address 0x0379D1A7 (stack_id=0x44CB69D7)READ of size 8 at 0x0379D000 #0 0x000068ef23be in (unknown) #1 0x000068f387f4 in (unknown) #2 0x000068eeb486 in (unknown) #3 0x000068e8add7 in (unknown)...

SyzyASan

Page 28: 44CON 2013 - Browser bug hunting - Memoirs of a last man standing - Atte Kettunen

Bad access information: +0x000 alloc_stack : [62] 0x0f999970 Void +0x0f8 alloc_stack_size : 0x3c '<' +0x0fc alloc_tid : 0x14a8 +0x100 free_stack : [62] (null) +0x1f8 free_stack_size : 0 '' +0x1fc free_tid : 0 +0x200 error_type : 3 ( HEAP_BUFFER_OVERFLOW ) +0x204 access_mode : 0 ( ASAN_READ_ACCESS ) +0x208 access_size : 8 +0x20c shadow_info : [128] "06499E3F is 23 bytes beyond 384-byte block [06499CA8,06499E28)." +0x290 microseconds_since_free : 0

SyzyASan

Page 29: 44CON 2013 - Browser bug hunting - Memoirs of a last man standing - Atte Kettunen

Crash stack:chrome_dll!SkOpSegment::addTCoincident+0x18echrome_dll!SkOpContour::calcCoincidentWinding+0x9fchrome_dll!CoincidenceCheck+0x3c chrome_dll!Op+0x26a.Allocation stack:asan_rtl!asan_HeapAlloc+0x48chrome_dll!malloc+0x17chrome_dll!realloc+0x15chrome_dll!SkOpSegment::addT+0x9bchrome_dll!AddIntersectTs+0xcebchrome_dll!Op+0x244

SyzyASan

Page 30: 44CON 2013 - Browser bug hunting - Memoirs of a last man standing - Atte Kettunen

● Heap allocation monitoring for Windows

● No feedback - Only crash :(

● “Works” on Chrome/Chromium

● env: CHROME_ALLOCATOR="winheap"

● Enable Chrome error reporting ->

minidumps

● Firewall Chrome( No free 0-days for Google ;) )

● Debugging tools x86● http://www.chromium.org/developers/testing/page-heap-for-chrome

Page-Heap

Page 31: 44CON 2013 - Browser bug hunting - Memoirs of a last man standing - Atte Kettunen

ExceptionAddress: 564a0cd7 (chrome_..!WebCore::WaveShaperDSPKernel::lazyInitializeOversampling+0x0...06) ExceptionCode: c0000005 (Access violation) ExceptionFlags: 00000000NumberParameters: 2 Parameter[0]: 00000000 Parameter[1]: 27261fe4Attempt to read from address 27261fe4.STACK_TEXT: chrome_...!WebCore::WaveShaperDSPKernel::lazyInitializeOversampling+0x6 [...\webkit\source\modules\webaudio\waveshaperdspkernel.cpp @ 53]chrome_...!WebCore::WaveShaperProcessor::setOversample+0x29 .APPLICATION_FAULT_INVALID_POINTER_READ_chrome!WebCore::WaveShaperDSPKernel::lazyInitializeOversampling+6

Dump-analysis

Page 32: 44CON 2013 - Browser bug hunting - Memoirs of a last man standing - Atte Kettunen

● Dumb fuzzing ○ Yes, still works○ Yes, you can still find bugs with bit-flipping of

image-files

● Smart fuzzing ○ Finds bugs fast but runs out of bugs faster. :(

Fuzzers

Page 33: 44CON 2013 - Browser bug hunting - Memoirs of a last man standing - Atte Kettunen

Dumb fuzzing● Radamsa || Surku \o/

○ https://code.google.com/p/ouspg/

● Mutate old repros ( find ./src/ -type d -name *crashtest* | xargs ls; )

● Collect winnings

Fuzzers

Page 34: 44CON 2013 - Browser bug hunting - Memoirs of a last man standing - Atte Kettunen

Smart fuzzing● W3C/MDN(/MSDN)● Again stay green ● Most of the JavaScript APIs in

browsers are really similar● Some of the public tools have the logic

in them already● W3C spec + grep + sed = $$$

Fuzzers

Page 35: 44CON 2013 - Browser bug hunting - Memoirs of a last man standing - Atte Kettunen

Smart fuzzing

WebAudio API - PannerNode - Specification

interface PannerNode : AudioNode { void setPosition(double x, double y, double z); void setOrientation(double x, double y, double z); void setVelocity(double x, double y, double z);

attribute double refDistance; attribute double maxDistance; attribute double rolloffFactor;};

Fuzzers

Page 36: 44CON 2013 - Browser bug hunting - Memoirs of a last man standing - Atte Kettunen

Smart fuzzing

2D Canvas API - Specification // rectsvoid clearRect(unrestricted double x, unrestricted double y, unrestricted double w, unrestricted double h);void fillRect(unrestricted double x, unrestricted double y, unrestricted double w, unrestricted double h);

// shadowsattribute unrestricted double shadowOffsetX; // (default 0)attribute unrestricted double shadowOffsetY; // (default 0)attribute unrestricted double shadowBlur; // (default 0)

Fuzzers

Page 37: 44CON 2013 - Browser bug hunting - Memoirs of a last man standing - Atte Kettunen

Individuals:● Physical machines -> sysadmining <3

● SSD <3

● RAM++

● Vicious cycle of

Bug->Bounty->New HW->Bug->Bounty...

Hardware/Infrastructure

Page 38: 44CON 2013 - Browser bug hunting - Memoirs of a last man standing - Atte Kettunen

8x Dual Core CPU, 2GB ram, USB-stick, aka. Badgers

OUSPG - 2011

Page 39: 44CON 2013 - Browser bug hunting - Memoirs of a last man standing - Atte Kettunen

University HW:Badgers6x Quad core AMD A10, 16GB ram, SSD6x Dual Dual Core AMD antique, 8GB, 10k rpm

BYOD:4x Quad core i7-3770K, 16GB ram, SSDAnd additional 30+ cores misc hardware with 133.7+ GB of ram and bunch of SSDs

OUSPG - 2013

Page 40: 44CON 2013 - Browser bug hunting - Memoirs of a last man standing - Atte Kettunen

OUSPG - 2013

Page 41: 44CON 2013 - Browser bug hunting - Memoirs of a last man standing - Atte Kettunen
Page 42: 44CON 2013 - Browser bug hunting - Memoirs of a last man standing - Atte Kettunen

ClusterFuzz aka. CF

● Google fuzzing cluster● 2012 -

○ 6000 Chrome instances ○ 50m+ test cases per day○ Plans for quadrupling at that time

● ASAN, multiple fuzzers, minimization, regression ranges, verify fixes, dupes & dupes & dupes...

Hardware/Infrastructure

Page 43: 44CON 2013 - Browser bug hunting - Memoirs of a last man standing - Atte Kettunen

“cluster-fuzz is a soulless bug hunting machine. It has no want or need for your gratitude. It lives only to feed on bugs.”

ClusterFuzz

Page 44: 44CON 2013 - Browser bug hunting - Memoirs of a last man standing - Atte Kettunen

● 12 machines running 24/7

● ~50 cores, ~133.7GB of RAM

● approx. 20m test cases per day

● 19 file-formats

● git, scp, auto-update, auto-minimize

● Radamsa and ...

My stuff

Page 45: 44CON 2013 - Browser bug hunting - Memoirs of a last man standing - Atte Kettunen

● Browser fuzzer harness● Written in JavaScript ( Node.js ) ● Linux, Windows, OS X● Test case generators and instrumentations

loaded as modules● Uses WebSockets for test case injection to

browser● Stable - https://code.google.com/p/ouspg/downloads/list● Trunkish - https://github.com/attekett/NodeFuzz

NodeFuzz

Page 46: 44CON 2013 - Browser bug hunting - Memoirs of a last man standing - Atte Kettunen

Requirements: Google Chrome installed

$ sudo apt-get install nodejs

$ git clone https://github.com/attekett/NodeFuzz.git

$ cd NodeFuzz

$ npm install

$ vim config.js #Optional

$ node nodefuzz.js

NodeFuzz - Setup - Ubuntu

Page 47: 44CON 2013 - Browser bug hunting - Memoirs of a last man standing - Atte Kettunen

● Fairly new JS API (Chrome 2011, FF 2013)

● "The API has been designed to allow modular routing.(UAF) Basic audio operations are performed by audio nodes that are linked together to form an audio routing graphs.(UAF/BOF) Inside a same context, several sources are supported, with different kind of channel layout.(UAF/BOF) This modular design allows for great flexibility and for the creation of complex audio functions and of dynamic effects.(BOF)" - MDN

NodeFuzz - module - WebAudio

Page 48: 44CON 2013 - Browser bug hunting - Memoirs of a last man standing - Atte Kettunen

Bugs found:● Chrome - 4 UAF, 3 BOF● Firefox - 1 UAF, 8 BOF

NodeFuzz - module - WebAudio

Page 49: 44CON 2013 - Browser bug hunting - Memoirs of a last man standing - Atte Kettunen

CVE-2013-0879 - Chrome - BOF<script>try{var context= new webkitAudioContext()}catch(e){}try{var oscillator= context.createOscillator()}catch(e){}

try{oscillator.start(0.701,0.7,0.7)}catch(e){}

setInterval(function(){try{oscillator.connect(context.destination);}catch(e){}},4)

try{oscillator.stop(0.70)}catch(e){}</script>

NodeFuzz - module - WebAudio

Page 50: 44CON 2013 - Browser bug hunting - Memoirs of a last man standing - Atte Kettunen

CVE-2013-2845 - Chrome - UAF<script>var Context0= new webkitAudioContext()var Analyser0=Context0.createAnalyser();var WaveShaper0=Context0.createWaveShaper();var Convolver3=Context0.createConvolver();Analyser0.connect(WaveShaper0);WaveShaper0.connect(Context0.destination);Convolver3.connect(Analyser0);

setInterval(function(){Analyser0.disconnect();},4)</script>

NodeFuzz - module - WebAudio

Page 51: 44CON 2013 - Browser bug hunting - Memoirs of a last man standing - Atte Kettunen

DEMO!!!&&

Q&A