GeeCon2016- High Performance Instrumentation (handout)

HIGH PERFORMANCE INSTRUMENTATION

Jaroslav Bachorík@yardus, @btraceio

Prague, 20-21 October 2016

ABOUT ME

Jaroslav Bachorík Prague, 20-21 October 2016

Jaroslav Bachorík, [email protected], [email protected]@yardus

mailto:[email protected]


PERFORMANCE


PERFORMANCE● Quantifiable

○ startup time○ request latency○ CPU usage○ Memory usage

● Reproducible○ controlled environment○ consistent results

● Measurable○ strictly defined target goals

● Benchmarking


INSTRUMENTATION


INSTRUMENTATIONint method() {

MyObject o = new MyObject();

int x = o.getCount();

logger.debug(“Instance “ + o “ has count “ + x;

//

return x;

}


INSTRUMENTATION● APIs and code providing means to monitor and control application

○ loggers○ stat counters○ profilers

● Decoupled from the application○ application works properly without instrumentation○ same instrumentation may work for multiple applications


SOURCE LEVEL INSTRUMENTATION● Instrumentation part of the source base

○ OS■ dtrace■ systemtap

○ Runtime■ JFR■ jstat counters

○ Application■ logging

● Difficult to modify and extend○ requires access to sources○ rebuild & redistribution


BYTECODE LEVEL INSTRUMENTATION● No source code modifications● Modifying bytecode

○ result of Java source compilation○ binary executable consumed by JVM

● Bytecode Injection (BCI)○ during compilation

■ eg. maven AOP plugins■ same drawbacks as static

instrumentation○ during class loading

■ JVM agent and class transformers


JVM JVM Agent

Classes

ClassloaderTransformer

Transformer

Transformer

CLASS TRANSFORMERS


java.lang.instrument.ClassTransformer

byte[] transform(ClassLoader l, String name, Class<?> cls,

ProtectionDomain pd, byte[] classfileBuffer)

● Inspect and modify the class data○ complex task

■ constant pool■ stack frame map

○ better delegate to specialized tools■ ASM■ ByteBuddy■ CGLIB

DYNAMIC INSTRUMENTATION /w BCI● Required steps

○ Create and register JVM agent○ Create and register class transformers○ Prepare injected bytecode

■ create bytecode■ validate bytecode

○ Inject bytecode■ merge class bytecode /w injected bytecode■ validate merged bytecode■ redefine/retransform class using merged bytecode


BTRACE● Bytecode level instrumentation simplified

○ JVM agent○ Class Transformers○ Optimized bytecode injection○ Safety guarantees

● Injected code as POJO○ annotations specify where injection should go○ code specifies what should be injected

● Started as a research project at Sun JDK Serviceability


BTRACE SCRIPT● Easy access to

○ class and method name and parameters○ enclosing instance○ return value○ method duration○ fields via reflection

■ immutable, guarded, access only● Interfacing via

○ stdout○ file○ JMX (MXBean)○ jstat counters


BTRACE SCRIPT@BTrace public class AllMethods {

@OnMethod(clazz="/javax\\.swing\\..*/", method="/.*/")

public static void m(@Self Object o, @ProbeClassName String probeClass,

@ProbeMethodName String probeMethod) {

println("this = " + o);

print("entered " + probeClass);

println("." + probeMethod);

}

}


> this = DerivedColor(color=192,192,193)

> entered javax.swing.plaf.nimbus.DerivedColor.getRGB

INSTRUMENTATION PERFORMANCE


PERFORMANCE IMPACT● Class (re)transformation

○ application startup time● Injected bytecode instructions

○ CPU usage○ JIT optimizer decisions○ heap usage○ GC activity

● Instrumentation framework○ additional drain of resources (CPU, RAM)


SPARK SPECIFICS● Distributed environment● Worker JVMs come and go

○ startup time is important● The inner parts are frequently executed

○ RDD (Resilient Distributed Dataset) iterators○ latency/overhead of injected code is important

● Startup time equally important as latency/overhead


CLASS (RE)TRANSFORMATION● Affects application startup time● Major impact on short lived applications ● Usually a small number of classes will be instrumented

○ optimize class filter for non-match● Minimize overhead of parsing class files

○ register as few transformers as possible○ consider smart caching of the transformed class data

● Example: Spark driver○ lifespan easily just a few minutes○ 104+ classes loaded at startup○ optimizing class transformation decreased overall overhead by >1.5%


INJECTED BYTECODE● Affects the application runtime performance● Keep injected code as simple as possible

○ no non-deterministic loops○ minimize external method calls

■ escape analysis○ prefer working with stack instead of fields

■ method arguments■ local variables

● Smart activation of injected code○ sampling○ injection guards


ESCAPING OBJECTS● A local instance escapes via injected instrumentation

○ affects GC and JIT optimizer decisions


int method() {



return x;

}

int method() {



// inspect the instance providing the count

// a local instance 'o' escapes the method scope

Instrumentation.inspect(o);

//

return x;

}

GC INTERFERENCE● Minimize instrumentation interference with GC

○ use off-heap data structures where possible○ specialized primitive collections○ specialized queues in runtime (eg. JCTools)

● Reduce instantiations to minimum○ boxing○ string concatenation○ varargs

● Collect only raw data○ aggregations on different JVM or host


STACK UNWINDING● Reuse the values stored on stack


Java StackGETSTATIC TestClass.name : Ljava/lang/String;LLOAD 2

INVOKESPECIAL C.m (Ljava/lang/String;J)J String: “name”Long : 2 (H)Long : 2 (L)

DUP_X2

String: “name”Long : 2 (H)Long : 2 (L)

DUP2_X1INVOKESTATIC Probe.p(Ljava/lang/String;J)V

TIMESTAMP FOLDING● Timestamps are expensive

○ TSC correlated across cores○ monotonic counter values adjusted for core frequencies

● Minimize number of requested timestamps○ fold in subsequent calls to System.nanoTime()

● BTrace will optimize timestamps for @Duration parameters


INVOCATION SAMPLING● Instrumented methods are frequently executed

○ injected code causing high overhead● Short methods experiencing unproportional overhead● Rely on statistically relevant sample instead

○ execute only on each Nth pass○ adjust N for acceptable overhead and detail

● Use @Sampled annotation in BTrace ○ fixed N○ dynamically adjusted N for guaranteed overhead


SAMPLING IN BTRACE@BTrace

public class ArgsDurationSampled {

@OnMethod(clazz="/.*\\.OnMethodTest/", method="args", location=@Location(value=Kind.RETURN))

@Sampled(kind = Sampled.Sampler.Const, mean = 20)

public static void args(@Self Object self, @Return long retVal, @Duration long dur) {

println("args");

}

}


// Adaptive sampler keeps ‘mean’ nanoseconds between samples in average

@Sampled(kind = Sampled.Sampler.Adaptive, mean = 300)

INJECTION GUARDS● Fastest code is the one never executed● Think of Logger levels● Class retransformation is costly● Introducing injection guards

○ injected code executed only when a condition is met○ minimal overhead when not executing injected code

■ fast field check● Use @Level annotation in BTrace


@OnMethod(clazz="org.apache.spark.rdd.RDD",

method="iterator",

enableAt=@Level(">=" + SAMPLING_LEVEL),

location=@Location(Kind.RETURN))

LESSONS LEARNED


HIGH PERFORMANCE INSTRUMENTATION● Fast filters for identifying injection points● Minimal and optimized code for injection

○ use timestamps sparsely○ beware of callbacks from injected code○ prefer stack manipulation above field retrievals

● Be gentle to GC● Use sampling when possible

○ getting overhead down○ still obtaining valid insights

● Enable turning off injection when not needed○ class retransformation is slow○ injection guards


Resources● BTrace (https://github.com/btraceio/btrace)

○ Contributors welcomed!● ASM (http://asm.ow2.org/index.html)● CGLIB (https://github.com/cglib/cglib)● ByteBuddy (http://bytebuddy.net/#/)● JCTools (https://github.com/JCTools/JCTools)


https://github.com/btraceio/btrace

http://asm.ow2.org/index.html

https://github.com/cglib/cglib

http://bytebuddy.net/#/

https://github.com/JCTools/JCTools

Q&A


THANK YOU!


[email protected], @yardus



GeeCon2016- High Performance Instrumentation (handout)

Documents