Adaptive Optimization with On-Stack Replacement
Stephen J. Fink IBM T.J. Watson Research Center
Feng Qian (presenter)Sable Research Group, McGill University
http://www.sable.mcgill.ca
Motivation
Modern VM uses adaptive recompilation strategies
VM replaces entry in dispatching table with newly compiled code
Switching to new code can only happen at the next invocation
On-stack replacement (OSR) allows transformation happen in the middle of method execution
What is On-stack Replacement?
Transfer execution from compiled code m1 to compiled code m2 even while m1 runs on some thread’s stack
stack
PC
frame
m1
m1
stack
PC
frame
m2
m2
Why On-Stack Replacement (OSR)?
Debugging optimized code via dynamic de-optimization [SELF-93]
Deferred compilation of cold paths in a method [SELF-91, HotSpot, Whaley 2001]
Promotion of long-run activations [SELF-93]
Safe invalidation for speculative optimization [HotSpot, SELF-91]
Related Work
Holzle, Chambers, and Ungar (SELF-91, SELF-93) deferred compilation, de-optimization for debugging, promotion of long-run loops, safe invalidation [OOPSLA’91, PLDI’92, OOPSLA’94]
HotSpot server compiler [JVM’01]
Partial method compilation [OOPSLA’01]
OSR Challenges
Engineering Complexity How to minimize disruption to VM code base? How to constrain optimizations?
Policies for applying OSR How to make rational decisions for applying OSR?
Effectiveness How does OSR improve/constrain dataflow
optimizations? How effective are online OSR-based optimizations?
Outline
Motivation OSR Mechanism Applications Experimental Results Conclusion
OSR Mechanism Overview
Extract compiler-independent state from a suspended activation for m1
Generate specialized code m2 for the suspended activation
Compile and transfer execution to the new code m2
m2
stack
PC
frame
m1
m1
compiler-
independent state
stack
PC
frame
m2
m2
1 2 3
JVM Scope Descriptor
Compiler-independent state of a running activation
Based on Java Virtual Machine Architecture Five components:
1) Thread running the activation2) Reference to the activation's stack frame3) Program Counter (as a bytecode index)4) Value of each local variable5) Value of each stack location
class C { static int sum(int c) { int y = 0; for (int i=0; i<c; i++) { y += i; } return y; }}
Running thread: MainThreadFrame Pointer: 0xSomeAddressProgram Counter: 16Local variables: L0(c) = 100; L1(y) = 1225; L2(i) = 50;Stack Expressions: S0 = 50; S1 = 100;
JVM Scope Descriptor 0 iconst_0 1 istore_1 2 iconst_0 3 istore_2 4 goto 14 7 iload_1 8 iload_2 9 iadd 10 istore_1 11 iinc 2 1 14 iload_2 15 iload_0 16 if_icmplt 7 19 iload_1 20 ireturn
Bytecode
JVM Scope Descriptor Example
Suspend after50 loop iterations(i = 50)
Extracting JVM Scope Descriptor
Trivial from interpreter Optimizing Compiler
Insert OSR Point (safe-point) instructions in initial IR OSR Point uses stack, local state needed to recover
scope descriptor OSR Point is treated as a call, transfers control to exit
block Aggregate OSR points to an OSR map when generating
machine instructionsstack
PC
frame
m1
m1
compiler-
independent state
1
Specialized Code Generation
Prepend a specialized prologue to original bytecode
Prologue will• Save JVM Scope Descriptor values into local variables• Push JVM Scope Descriptor values onto the stack• Jump to the desired program counter
m2
compiler-
independent state
2
Running thread: MainThreadFrame Pointer: 0xSomeAddressProgram Counter: 16Local variables: L0(c) = 100; L1(y) = 1225; L2(i) = 50;Stack Expressions: S0 = 50; S1 = 100;
JVM Scope Descriptor
ldc 100 istore_0 ldc 1225 istore_1 ldc 50 istore_2 ldc 50 ldc 100 goto 160 iconst_0 ...16 if_icmplt 7 ...20 ireturn
Specialized Bytecode 0 iconst_0
1 istore_1 2 iconst_0 3 istore_2 4 goto 14 7 iload_1 8 iload_2 9 iadd 10 istore_1 11 iinc 2 1 14 iload_2 15 iload_0 16 if_icmplt 7 19 iload_1 20 ireturn
Original Bytecode
Transition Example
m2
stack
PC
frame
m2
m2
3
Transfer Execution to the New Code
Compile m2 as a normal method System unfolds the stack frame of m1 Reschedule the thread to execute m2 By construction, executing specialized m2 sets up
target stack frame and continues execution
m2
stack
PC
frame
m2
m2
3
Suppose optimizer inlines A -> B -> C:
A'
stack
PC
frameA
A
1 2 3
JVM ScopeDescriptor A
JVM ScopeDescriptor C
JVM ScopeDescriptor B
C'
B'
stack
PC
frame
m2
C'
A'
B'
AA
frame
C'frame
A'
frame
B'
frame
Recovering from Inlining
Inlining Example
foo_prime() { <specialized foo prologue> call bar_prime() goto A; ... bar(); A: ...}bar_prime() { <specialized bar prologue> goto B: ... B: ...}
void foo() { bar(); A: ... } void bar() { ... B: ... }
Wipe stackto caller C and call foo_prime
frame
A
stack
PC
frame
m2
foo'
bar'
C
frame
bar'
frame
foo'
Suspendat B: inA -> B
Implementation Details
Target Compiler unmodified, except for .... New pseudo-bytecodes
Load literals (to avoid inserting new constants in constant pool)
Load an address/bytecode index: JSR return address on stack
Fix bytecode indices for GC maps, exception tables, line number tables
Pros and Cons
Advantages mostly compiler-independent avoid multi-entry points of compiled code target compiler can exploit run-time constants
Disadvantage must compile target method twice (once for transition,
once for next invocation)
Outline
MotivationOSR Mechanism Applications Experimental Results Conclusion
Two OSR Applications
Promotion (see the paper for details) recompile a long-running activation
Deferred Compilation don't compile uncommon paths saves compile-time
x = 1; x = foo();
return x;
if (foo is currently final)
trap/OSR;
Deferred Compilation
What's "infrequent"? static heuristics profile data
Adaptive recompilation decision is modified to consider OSR factors
Feng Qian:
Class initialization is called by a class loader, when do we need OSR
for it?
Feng Qian:
Class initialization is called by a class loader, when do we need OSR
for it?
Outline
MotivationOSR MechanismApplications Experimental Results Conclusion
Online Experiments
Eager : (by default) no deferred compilation OSR/static: deferred compilation for CHA-based inlining
only OSR/edge counts: deferred compilation w/online profile
data & CHA-based inlining
Adaptive System Performance
First Run
0.8
0.9
1
1.1
1.2
com
pres
s
jess db
java
c
mpe
gaud
io
mtr
t
jack
g. m
ean
Per
form
ance
Rel
ativ
e to
Eag
er
OSR/ edge counts OSR/ static
bett
er
Adaptive System Performance
Best Run of 10
0.8
0.9
1
1.1
1.2co
mpre
ss
jess db
java
c
mpegau
dio
mtr
t
jack
g.m
ean
Perf
orm
an
ce R
ela
tive t
o E
ag
er OSR/ edge counts OSR/ static
bett
er
Promotions Invalidations
compress 3 6
jess 0 0
db 0 1
javac 0 10
mpegaudio 0 1
mtrt 0 5
jack 0 1
total 3 24
OSR ActivitiesSPECjvm98 size 100 First Run
Outline
MotivationOSR MechanismApplicationsExperimental Results Conclusion
Summary
A new On-stack replacement mechanism Online profile-directed deferred compilation Evaluation of OSR applications in JikesRVM
Conclusion
Should a VM implement OSR?+Can be done with minimal intrusion to code
baseModest gains from deferred compilationNo benefit for class-hierarchy-based inlining+Debugging with dynamic de-optimization
valuable TODO: More advanced speculative
optimizations
Implementation is available to public in JikesRVM under CPL:
Linux/x86, Linux/PPC, and AIX/PPC
http://www-124.ibm.com/developerworks/oss/jikesrvm/
Backup Slides
Compile RateOffline Profile
Compile RateOffline Profile
Machine Code SizeOffline Profile
Machine Code SizeOffline Profile
Code QualityOffline Profile
Code QualityOffline Profile
better
Jikes RVM Analytic Recompilation Model
Definecur, current optimization level for method mTj, expected future execution time at level jCj, compilation cost at opt level j
Choose j > cur that minimizes Tj + CjIf Tj + Cj < Tcur recompile at level jAssumptions
Method will execute for twice its current duration Compilation cost and speedup based on offline average Sample data determines how long a method has executed
Jikes RVM OSR Promotion Model
Given: Outdated activation A of method mDefine
L, last optimization level for any compiled version of mcur, current optimization level for activation A
Tcur , expected future execution time of A at level cur
CL , compilation cost for method m at opt level L
TL , expected future execution time of A at level L
If TL + CL < Tcur specialize A at level LAssumption
Outdated activation will execute for twice its current duration
Jikes RVM Recompilation Model, with Profile-Driven Deferred Compilation
Definecur, current optimization level for method mTj, expected future execution time at level jCj, compilation cost at opt level j
P, percentage of code in m that profile data indicates was reached
Choose j > cur that minimizes Tj + P*CjIf Tj + P*Cj < Tcur recompile at level jAssumptions
Method will execute for twice its current duration Compilation cost and speedup based on offline average Sample data determines how long a method has executed
Offline Profile experiments
Collect "perfect" profile data offline Mark any block never reached as "uncommon" Defer compilation of "uncommon" blocks Four configurations
Ideal: deferred compilation trap keeps no state liveIdeal-OSR: deferred compilation trap is valid OSR pointStatic-OSR: no profile data; defer compilation for CHA-based
inlining; trap is valid OSR pointEager: (default) no deferred compilation
Compile RateOffline Profile
Machine Code SizeOffline Profile
Code QualityOffline Profile
OSR Challenges
Engineering ComplexityHow to minimize disruption to VM code base?How to constrain optimizations?
Policies for applying OSRHow to make rational decisions for applying OSR?
EffectivenessHow does OSR improve/constrain dataflow optimizations?
How effective are online OSR-based optimizations?
Recompilation ActivitiesFirst Run
O0 O1 O2 total O0 O1 O2 total
compress 17 7 2 26 13 9 6 28
jess 49 20 1 70 39 17 4 60
db 8 4 2 14 8 4 5 17
javac 171 19 2 192 168 16 3 187
mpegaudio
68 32 7 107 66 29 6 101
mtrt 57 14 3 74 61 11 3 75
jack 59 25 8 92 54 26 5 85
total 429 121 25 575 409 112 32 553
With OSR Without OSR
Summary of Study (1)
Engineering ComplexityHow to minimize disruption to VM code base?
°Compiler-independent specialized source code to manage transition transparently
How to constrain optimizations?°Model OSR Points like CALLS in standard transformations
Policies for applying OSRHow to make rational decisions for applying OSR?
°Simple modifications to cost-benefit analytic model
Summary of Study (2)
Effectiveness (for an implementation of online profile-directed deferred compilation)
How does OSR improve/constrain dataflow optimizations?
°small ideal benefit from dataflow merges (0.5 - 2.2%)°negligible benefit when constraining optimization for potential invalidation°negligible benefit for just CHA-based inlining
patch points + splitting + pre-existence good enough
How effective are online OSR-based optimizations? °average performance improvement of 2.6% on first run SPECjvm98 s=100°individual benchmarks range from +8% to -4%°negligible impact on steady state performance (best of 10 iterations)°adaptive recompilation model relatively insensitive, compiles 4% more methods
Experimental Details
SPECjvm98, size 100Jikes RVM 2.1.1
FastAdaptiveSemispace configurationone virtual processor500MB heap
separate VM instance for each benchmarkIBM RS/6000 Model F80
six 500 MHz PowerPC 630'sAIX 4.3.34 GB memory
Specialized Code Generation
Generate specialized m2 that sets up new stack frame and continues execution, preserving semantics.
Express the transition to new stack frame in source code (bytecode)
m2
compiler-
independent state
2
Deferred Compilation
Don't compile "infrequent" blocks
x = 1; trap/OSR;
return x;
if (foo is currently final)
x = 1; x = foo();
return x;
if (foo is currently final)
Experimental Results
Online profile-directed deferred compilation Evaluation
How much do OSR points improve optimization by eliminating merges?How much do OSR points constrain optimization?How effective is online profile-directed deferred compilation?
Adaptive System Performance
Adaptive System Performance
Online Experiments
Before optimizing, collect intraprocedural edge countersDefer compilation at blocks that profile data says not reachedIf deferred block reached
Trigger OSR and deoptimizeInvalidate compiled code
Modify analytic recompilation modelPromotion from baseline to optimizedCompile-time cost estimate modified according to profile data