Android Crash analysis and The Dalvik Garbage collector – Tools and Tips

Android Crash Analysis and The DalvikGarbage Collector - Tips

Oren BarAd

Mobile Security Team Leader

AVG

2

100,000,000

3

Intro

• App on the market

• Crashes will happen

• Need to Fix the Bugs!

• Identify the root issues

4

Who?

• Developing for Android since 2008

• The first employee of DroidSecurity

– First company to release An antivirus to the Android market (2009)

– Later acquired by AVG (2011)

• Mobile Security Research Team Leader @AVG

• B.Sc Information System Engineering from BGU

5

Agenda

• Android app crash Collection & Analysis

• The Android Garbage Collector

• Memory analysis Tools

• Tips and tricks

6

How to collect crash information?

7

What to fix first?

• Easy to fix

• Affects most users

• The ones users complain most about

• Affects the most popular device

8

Root cause analysis

• Stare at data:

–Crash type

– Exception Type

–Device

–Manufacturer

–OS version

– Locale

• And stare some more…

9

Ultimate debugging method!

• What is the answer to this bug?

– 42

• We don’t know the answer

– Brainstorming

– Developing tools

• Learning something new…

10

Root cause analysis – Real world example

• Concurrent.Timeout exception– ~10% of total crashes for AVG antivirus.

– Unhelpful stack traces

– Second place in top 10 crashes (and number 5, and number 9)

java.util.concurrent.TimeoutException: android.os.BinderProxy.finalize() timed out after 10 seconds

at java.lang.Object.wait(Native Method)

at java.lang.Object.wait(Object.java:401)

at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:102)

at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:73)

at java.lang.Daemons$FinalizerDaemon.run(Daemons.java:170)

at java.lang.Thread.run(Thread.java:841)

java.util.concurrent.TimeoutException: com.android.internal.os.BinderInternal$GcWatcher.finalize() timed out after 10 seconds

java.util.concurrent.TimeoutException: android.os.Parcel.finalize() timed out after 10 seconds

java.util.concurrent.TimeoutException: java.io.FileInputStream.finalize() timed out after 10 seconds

11

Device and OS analysis

33%

66%

Device

Galaxy S3

49%

38%

13%

19%

10%

71%

OSCrash Dashboard

4.2.2

4.3Other OSVersions

12

What next?

• After going deep into the Code…

• Get S3 and use

13

Android GC – little bit of history

Gingerbread

14

Complex Object

Tricolor collector at work

roots rootsroots

object object object

object object objectobject

object

object object

15

object

Tricolor collector at work

roots rootsroots

object object object

object object objectobject

object

object object

16

object

Tricolor collector at work

roots rootsroots

object object object

object object objectobject

object

object object

17

object

Tricolor collector at work

roots rootsroots

object object object

object object objectobject

object

object object

18

Complex Object

Tricolor collector at work

roots rootsroots

object object object

object object objectobject

object

object object

19

Complex Object

Tricolor collector at work

roots rootsroots

object object object

object object objectobject

object

object object

20

Garbage collector observed behavior

Cause Effect

GC_FOR_MALLOC Stop The world GC

Use complex Objects GC takes longer

A lot of small objects GC take Longer

Use more memory GC takes longer

USE_LARGE_HEAP All of the above

21

Trying to replicate the problem

• The GC_Test project– Test the GC performance under load

– Android app to test the behavior of the GC on android

– https://github.com/oba2cat3/GCTest

• Useful logcat python script – https://github.com/oba2cat3/logcat2memorygraph

• Available from GitHub

22

Logcat output

02-04 12:40:33.185: D/dalvikvm(2496): GC_EXPLICIT freed <1K, 19% free 14383K/17720K, paused 1ms+1ms, total 17ms

02-04 12:40:33.265: D/dalvikvm(3279): GC_CONCURRENT freed 661K, 27% free 14934K/20336K, paused 5ms+3ms, total 21ms

02-04 12:40:33.460: D/dalvikvm(5637): GC_CONCURRENT freed 834K, 22% free 15516K/19644K, paused 6ms+3ms, total 33ms

02-04 12:40:33.480: D/dalvikvm(3279): GC_CONCURRENT freed 313K, 27% free 15022K/20336K, paused 3ms+9ms, total 30ms

02-04 12:40:33.665: D/dalvikvm(5637): GC_CONCURRENT freed 369K, 21% free 15658K/19644K, paused 2ms+2ms, total 18ms

02-04 12:40:33.725: D/dalvikvm(5730): GC_CONCURRENT freed 1166K, 23% free 15188K/19644K, paused 4ms+2ms, total 25ms

02-04 12:40:33.765: D/dalvikvm(5637): GC_CONCURRENT freed 430K, 20% free 15740K/19644K, paused 7ms+2ms, total 22ms

02-04 12:40:33.865: D/dalvikvm(5730): GC_CONCURRENT freed 325K, 23% free 15256K/19644K, paused 2ms+3ms, total 22ms

02-04 12:40:34.065: D/dalvikvm(5637): GC_CONCURRENT freed 1811K, 25% free 15934K/21040K, paused 3ms+2ms, total 29ms

02-04 12:40:34.160: D/dalvikvm(5637): GC_CONCURRENT freed 99K, 23% free 16354K/21040K, paused 2ms+4ms, total 31ms

02-04 12:40:34.250: D/dalvikvm(3403): GC_EXPLICIT freed 172K, 24% free 14956K/19628K, paused 3ms+4ms, total 28ms

02-04 12:40:34.315: D/dalvikvm(5766): GC_CONCURRENT freed 1185K, 23% free 15148K/19628K, paused 3ms+3ms, total 28ms

02-04 12:40:34.350: D/dalvikvm(5637): GC_CONCURRENT freed 1140K, 26% free 15766K/21040K, paused 3ms+3ms, total 30ms

02-04 12:40:34.495: D/dalvikvm(4155): GC_CONCURRENT freed 1274K, 24% free 15093K/19660K, paused 7ms+5ms, total 56ms

02-04 12:40:34.635: D/dalvikvm(3333): GC_CONCURRENT freed 574K, 24% free 14982K/19620K, paused 3ms+2ms, total 19ms

02-04 12:40:34.780: D/dalvikvm(5694): GC_CONCURRENT freed 1268K, 24% free 15100K/19656K, paused 3ms+1ms, total 25ms

02-04 12:40:34.840: D/dalvikvm(2960): GC_EXPLICIT freed 1236K, 10% free 53524K/58860K, paused 4ms+7ms, total 130ms

02-04 12:40:34.915: D/dalvikvm(5637): GC_FOR_ALLOC freed 143K, 25% free 15978K/21040K, paused 26ms, total 26ms

02-04 12:40:33.480: D/dalvikvm(3279): GC_CONCURRENT freed 313K, 27% free 15022K/20336K, paused 3ms+9ms, total 30ms

02-04 12:40:33.665: D/dalvikvm(5637): GC_CONCURRENT freed 369K, 21% free 15658K/19644K, paused 2ms+2ms, total 18ms

02-04 12:40:33.725: D/dalvikvm(5730): GC_CONCURRENT freed 1166K, 23% free 15188K/19644K, paused 4ms+2ms, total 25ms

02-04 12:40:33.765: D/dalvikvm(5637): GC_CONCURRENT freed 430K, 20% free 15740K/19644K, paused 7ms+2ms, total 22ms

02-04 12:40:33.865: D/dalvikvm(5730): GC_CONCURRENT freed 325K, 23% free 15256K/19644K, paused 2ms+3ms, total 22ms

02-04 12:40:34.065: D/dalvikvm(5637): GC_CONCURRENT freed 1811K, 25% free 15934K/21040K, paused 3ms+2ms, total 29ms

23

Script result - Heap size and GC delays

0

0.2

0.4

0.6

0.8

1

1.2

0

20000

40000

60000

80000

100000

120000

140000

160000

Time in MSheap size in KB

dalvikvm event timestamps

current heap

max heap

24

Script result - Heap size and GC delays

0

50

100

150

200

250

300

350

400

0

20000

40000

60000

80000

100000

120000

140000

160000

Time in MSheap size in KB

dalvikvm event timestamps

current heap

max heap

ui pause time

total pause

25

Script result - Heap size and GC delays

0

50

100

150

200

250

300

350

400

0

20000

40000

60000

80000

100000

120000

140000

160000

Time in MSheap size in KB

dalvikvm event timestamps

current heap

max heap

ui pause time

total pause

webview

start

26

Memory Analyzer Tool tricks

27

Memory Analyzer Tool tricks

28

Memory Analyzer Tool tricks

29

Memory Analyzer Tool tricks

30

Memory Analyzer Tool tricks

31

Memory Analyzer Tool tricks

32

Memory Analyzer Tool tricks

33

Memory Analyzer Tool tricks

34

Tips for managing GC

Do How

Avoid GC_FOR_ALLOC Load stuff in Splash

Use Object Pools For large/heavy objects

Avoid Complex Objects Use simpler Objects

Perform Heap analysis Use MAT and DDMS

Reduce memory use …

35

Summary

• Application will crash

–Collect and Analyze

–Understand and Fix

• Fixing the bug is half the fun

36

Back to the problem

• JNI objects are a special case

• Dalvik will finalize and Destroy

• Timeouts will happen... And they Do!

37

Q&A

• Questions?

• Contact: Oren.barad@avg.com

• We are hiring!

• Come talk to us in the AVG booth

38

Links and resources

• http://android-developers.blogspot.co.il/2011/03/memory-analysis-for-android.html

• http://www.brpreiss.com/books/opus5/html/page424.html

• http://java-is-the-new-c.blogspot.co.il/2013/07/tuning-and-benchmarking-java-7s-garbage.html

• http://www.slideshare.net/muhammedshakirmisarwala/java-performance-monitoring-tuning

• http://www.slideshare.net/VikasBalikai/gc-in-android

• http://www.javacodegeeks.com/2012/01/practical-garbage-collection-part-1.html

39

Links and resources

• https://android.googlesource.com/platform/dalvik/+/android-4.3_r2/vm/alloc/MarkSweep.cpp

• https://android.googlesource.com/platform/dalvik/+log/android-4.4.2_r2/vm/alloc/MarkSweep.cpp

• https://android.googlesource.com/platform/dalvik/+/gingerbread/vm/alloc/MarkSweep.c

• https://android.googlesource.com/platform/dalvik/+/froyo-release/vm/alloc/MarkSweep.c