Detecting ROP with Statistical Learning of Program ... · system to detect ROP payloads based on unsupervised statistical learning of program characteristics. We study, for the first

Department of Computer ScienceGeorge Mason University Technical Reports

4400 University Drive MS#4A5Fairfax, VA 22030-4444 USAhttp://cs.gmu.edu/ 703-993-1530

Detecting ROP with Statistical Learning of ProgramCharacteristics

Mohamed [email protected]

Daniel [email protected]

Dan [email protected]

Angelos [email protected]

Technical Report GMU-CS-TR-2016-5

Abstract

Return-Oriented Programming (ROP) has emerged asone of the most widely used techniques to exploit soft-ware vulnerabilities. Unfortunately, existing ROP pro-tections suffer from a number of shortcomings: theyrequire access to source code and compiler support, fo-cus on specific types of gadgets, depend on accuratedisassembly and construction of Control Flow Graphs,or use hardware-dependent (microarchitectural) charac-teristics. In this paper, we propose EigenROP, a novelsystem to detect ROP payloads based on unsupervisedstatistical learning of program characteristics. We study,for the first time, the feasibility and effectiveness of us-ing microarchitecture-independent program characteristics— namely, memory locality, register traffic, and memoryreuse distance — for detecting ROP. We propose a noveldirectional statistics based algorithm to identify devia-tions from the expected program characteristics duringexecution. EigenROP works transparently to the pro-tected program, without requiring debug information,source code or disassembly. We implemented a dynamicinstrumentation prototype of EigenROP using Intel Pinand measured it against in-the-wild ROP exploits and onpayloads generated by the ROP compiler ROPC. Overall,EigenROP achieved significantly higher accuracy thanprior anomaly-based solutions. It detected the executionof the ROP gadget chains with 81% accuracy, 80% truepositive rate, only 0.8% false positive rate, and incurredcomparable overhead to similar Pin-based solutions.

1 Introduction

Since its introduction by Shacham in 2007 [38], Return-Oriented Programming (ROP) has become an increas-ingly popular technique for bypassing Data ExecutionPrevention (DEP) defenses on modern operating sys-tems. DEP ensures that all writable memory pages of aprogram are non-executable, which prevents the execu-

tion of any input data, effectively mitigating all classiccode injection attacks. In a ROP attack, on the other hand,the attacker does not inject new code. Instead, existingsequences of instructions in the process executable mem-ory, called gadgets, are chained together to perform theintended computation. While the traditional AddressSpace Layout Randomization (ASLR) randomizes thelocation of most libraries and executables, ROP attackscan still bypass ASLR by finding a few code segmentsin statically known locations, or through brute-forcingand de-randomization by exploiting memory disclosurevulnerabilities.

Over the past few years, research in ROP defenseshas become an arms race, where emerging defenses arecountered by new subtle variations of ROP attacks. De-fenses can be categorized into two broad categories. Thefirst category attempts to prevent ROP attacks at compiletime, by eliminating gadgets from binaries [32] or enforc-ing Control-Flow Integrity (CFI) [9]. The second categoryaims at detecting ROP attacks at runtime, by monitoringthe execution of programs [33, 13, 15, 30, 16, 41].

Defenses in the second category can further be clas-sified based on the detection approach into signature-based and anomaly-based. Signature-based solutionsdetect ROP attacks by identifying static signatures (pat-terns) in the execution trace of programs. The mostcommon method is to detect gadgets execution by en-forcing predefined constraints over the program counterand the call stack, either through dynamic instrumen-tation [15, 23, 18] or by leveraging existing hardwarebranch tracing features [13]. These solutions incur verylow overhead, but the employed signatures are oftenincomplete due to strong constraints on the ROP struc-ture, allowing the defenses to be bypassed by attack-ers [14, 12, 19].

Anomaly-based detection, on the other hand, learns abaseline of normal (clean) behavior and detects attacksby measuring statistical deviations from the normal be-havior. This approach has the significant advantage of

1

being able to protect against a broad spectrum of attacks,including zero-day. Until recently, anomaly-based ap-proaches have only leveraged software characteristics,e.g., network traffic and system call sequences [31, 24].Meanwhile, attacks have increased in complexity, becom-ing stealthier and harder to detect. Therefore, researchershave explored the potential of using hardware character-istics, such as instruction mixes and branch predictionrate, to detect ROP attacks [30, 16, 33, 41, 34].

Using hardware characteristics has a major advantageover software characteristics: it is harder for attackersto gain sufficient control over the hardware in order toevade detection. For example, it is easy to craft ROPpayloads that mimic the behavior of clean software exe-cution by chaining gadgets that invoke benign sequencesof system calls, while still executing the attack payload.On the other hand, it is hard to craft payloads that, whilestill attacking the system, maintain precise control of thebranch prediction rate of the hardware. This is becauseattacks, by definition, have to go against the normal flowof the program, inevitably resulting in misprediction ofbranches and returns by the hardware branch predictor.

Prior work that utilized hardware characteristics usedtwo classes of characteristics: 1) architectural characteris-tics, which are dependent on the instruction set architec-ture (ISA), such as the number of load and store instruc-tions retired. And, 2) microarchitectural characteristics,meaning characteristics that depend on the underlyingmicroarchitecture configurations, such as branches mis-prediction rate and cache misses. These characteristicswere typically measured by reading the hardware per-formance counters (HPC) of the underlying processor.However, a common pitfall is that characteristics mea-sured using HPC may actually hide the underlying pro-gram behavior, making the HPC-based metrics appearsimilar for inherently different behaviors [20, 45].

In this paper, we introduce EigenROP, a novel systemfor detecting ROP attacks. We study, for the first time,the feasibility and value of using microarchitecture-in-dependent program characteristics for the detection ofROP attacks. We propose a new type of anomaly-basedROP detectors that leverages microarchitecture-indepen-dent program characteristics, including memory reusedistance [47], register traffic load [17], memory local-ity [27], among others, in addition to traditional hard-ware characteristics (see Section 4).

EigenROP employs a novel anomaly detection algo-rithm that builds on concepts from directional statistics.The fundamental idea is that strong relationships amongthe different program characteristics will appear as prin-cipal axes in some high-dimensional space. Since ROPexecutes against the control flow of the program, it isreasonable to assume that it causes some unexpectedchanges in the relationships between the program char-acteristics learned from benign runs. Such changes canbe detected as statistically significant deviations in thedirections of the axes in the high-dimensional space. We

investigate if and to what extent ROP causes changes inprogram characteristics, and verify our hypothesis withextensive experiments using multiple in-the-wild ROPpayloads and payloads generated by the ROPC ROPcompiler.

EigenROP operates in two phases: a learning phaseand a detection phase. During the learning (offline)phase, programs are executed over benign inputs un-der EigenROP, where it collects different characteristics.The characteristics are measured periodically, every Ninstructions retired. A model is then constructed usingKernel Principal Component Analysis (KPCA) [36] anddirectional statistics (see Section 5). EigenROP uses atemporal model, where both the current snapshot ofcharacteristics and the history are taken into account.This concludes the learning phase. In the detectionphase, EigenROP monitors the execution of the targetprogram, collects the characteristics, and tests for devia-tion from the trained model.

We implemented a prototype of EigenROP on Linux,using the dynamic instrumentation framework Pin [29].We conducted several experiments to quantify the accu-racy of EigenROP, the effect of involved parameters andthe incurred performance overhead (see Section 7). Inour experiments, microarchitecture-independent charac-teristics resulted in 11% increase on average in detectionaccuracy, relative to using only microarchitectural char-acteristics. EigenROP achieved an overall accuracy of81%, 80% true positive rate, and only 0.8% false positiverate. The incurred performance overhead decayed expo-nentially as the sampling interval increases, and fasterthan the deterioration in accuracy. Overall, the overheadincurred matches with prior Pin-based solutions (seeSection 9).

To summarize, we make the following contributions:

• We study the effectiveness of combining microarchi-tecture-independent program characteristics withtypical hardware characteristics for the detection ofROP attacks.

• We propose a novel anomaly detection algorithm us-ing directional statistics of program characteristics,embedded in high-dimensional space.

• We present EigenROP, a working prototype of ourapproach.

• We quantify the security effectiveness of Eigen-ROP using in-the-wild ROP attacks against commonLinux programs.

• We quantify the runtime accuracy-performancetradeoff of EigenROP.

2

2 Background

2.1 Return-Oriented Programming

Return Oriented Programming (ROP) [38] enables at-tackers to execute arbitrary code without injecting newcode into the victim process, by returning to arbitraryinstruction sequences in the executable memory of theprogram.

The basic idea is to use indirect jumps (e.g., ret in-structions) to return to arbitrary points in the executableprocess memory that execute sequences of instructionsending in another indirect jump instruction. The lastindirect jump instruction allows executing one such se-quence after another. Multiple sequences can be com-bined into “gadgets” that perform an atomic task, suchas load, store and system call. The attacker then “chains”the gadgets together, to perform the intended maliciousfunctionality. Typically, gadgets end with a ret instruc-tion, which returns to the stack. The attacker chains thegadgets by hijacking the stack and writing appropriateaddresses to the beginning of the desired instructionsequence.

A typical ROP attack operates as follows: first, theattacker overwrites the stack contents with addresses ofthe desired ROP gadgets. Once the ret instruction of thecurrent routine is executed, the first return address ofthe current stack frame is used as a return target. Instruc-tion sequences at that address will execute, till the nextret instruction. Upon execution of the ret instruction,control is transferred to the next gadget. This processrepeats, jumping from one gadget to the next, till thegadget chain terminates.

In Fig. 1, we give an example of a gadget that stores aconstant 0x1 at the target memory address 0xa0de1b6e.The gadget starts by loading the constant 0x1 from thestack to the register eax. It then loads the target memoryaddress to register ebx. Finally, it moves the contents ofeax back to the memory pointed at by ebx.

Conventional ROP attacks use ret instructions tochain the gadgets [38]. In [11], a ROP variant was pre-sented that uses indirect jump (e.g., jmp eax) instruc-tions to chain the gadgets. While we mainly evaluateEigenROP using conventional ROP attacks, our solutionis applicable regardless of how the gadgets are chained.We discuss in Section 8 how different variants of ROPpayloads can be detected by EigenROP.

It has been shown that ROP can perform Turing-Complete computations if the attacker can find sufficientgadgets to perform memory, arithmetic, logical opera-tions and system calls [42]. An infamous example onthat is the recent ROP-only Adobe Reader exploit [1].We refer the reader to [38, 35] for more details on ROP.

Finally, it is worth mentioning that overwriting thereturn address on the stack is not the only way to hijackthe execution of the target process. Other vulnerabilitiessuch as format string and integer overflow vulnerabil-

Figure 1: Example of a ROP gadget that stores a constantvalue 0x1 at memory location 0xa0de1b6e.

ities can allow the attackers to write arbitrary valuesto function pointers that are used as jump targets bythe program, thereby redirecting the execution to theattacker’s instructions of choice. For example, a verycommon approach is overwriting the Global Offset Ta-ble on Linux systems, which holds absolute addresses tofunctions in dynamically linked libraries.

2.2 Microarchitecture-independent Charac-teristics

It has been shown that microarchitecture-indepen-dent characteristics have higher discrimination powerbetween different inherent program behaviors, com-pared to architectural and microarchitectural character-istics [20, 45]. Microarchitecture-independent character-istics are program characteristics that are unique to agiven instruction set architecture (ISA) and a given com-piler but are independent of a given microarchitecture.In other words, the characteristics are invariant of theunderlying hardware cache size, pipeline size, branchpredictors size and algorithm, number of cores and theirconfigurations, and so on. In the context of ROP detec-tion, several microarchitecture-independent characteris-tics can prove useful in discriminating between benignexecution behavior and gadget execution, such as mem-ory locality and reuse distance, and register traffic (seeSection 4 for details). Note that while characteristicsdependent on the ISA, i.e., architectural characteristics,can be regarded as a subset of microarchitecture-inde-pendent characteristics, we keep them distinct in thiswork as is the trend in prior program characterizationwork [20, 45, 30, 41].

The main downside of using microarchitecture-inde-pendent characteristics is that it requires runtime instru-

3

mentation to measure the characteristics. However, theoverhead decays over time as more efficient algorithmsand tools are developed [8].

In the following section, we outline the big picture ofhow EigenROP works.

3 Overview of EigenROP

The key idea of EigenROP is to identify anomalies in pro-gram characteristics, due to the execution of ROP gad-gets. In this context, it is difficult to precisely define whatanomalies are since that depends on the characteristicsof both the monitored program and the ROP. However,it is reasonable to assume that some unexpected changeoccurs in the relationships among the different programcharacteristics due to the execution of the ROP. By ex-tracting and learning arbitrary relationships among theprogram characteristics, EigenROP detects ROP by look-ing for unexpected changes in the learned relationships.

Given our definition of anomaly, strong relationshipsamong the measured program characteristics should ap-pear as principal directions in some high-dimensionalspace [36]. Such directions can be extracted using Ker-nel Principal Component Analysis (KPCA) [36]. Morespecifically, the principal component vectors of the mea-surements mapped into the high-dimensional space canbe interpreted as the relationships among the programcharacteristics.

The general workflow of EigenROP is illustrated inFig. 2. First, the target program is loaded and executed.During execution, EigenROP takes a snapshot of thedifferent program characteristics, every N instructionsretired. Each snapshot is a d−dimensional vector ofcharacteristics. The snapshots are pushed to a bufferthat EigenROP iterates over using a sliding window.

In the learning phase, the target program is executedover benign inputs. For each window of measured char-acteristics, EigenROP maps the measurements into ahigh-dimensional space and extracts the principal com-ponents of the measurements in that space. EigenROPthen estimates a representative direction from all the prin-cipal components, and estimates the density of the dis-tances of all principal components around the represen-tative direction. Recall, the idea here is that any strongrelationships among the measured characteristics willappear as principal components in the high-dimensionalspace. In the detection phase, EigenROP computes thedistances of the principal components of incoming mea-surements, in the high-dimensional space, to the repre-sentative direction. If the distance exceeds some thresh-old, then an alarm is raised.

In the following, we define the characteristics usedby EigenROP and explain in detail how learning anddetection work.

4 Which Characteristics to Mea-sure?

To choose the most relevant characteristics for ROP detec-tion, we conducted several experiments to collect cleanand infected measurements from a variety of programsand exploits (see Section 7.3). We considered most of thecharacteristics used in previous program characteriza-tion work [20, 45, 30, 41]. Then, we used the Fisher Scoreto quantify the discriminative power of each characteris-tic. The following is the shortlisted categories of charac-teristics we measured. The letters between brackets de-note the type of the characteristics: Architectural [A], Mi-croarchitecture-Independent [I], and Microarchitectural[M]. We emphasize that all the characteristics used inthis work are computed in software.

• Branch predictability [M]. Since ROP attacks dis-turb the normal control flow of execution, they mayincrease the number of mispredicted branches bythe processor branch predictor.

• Instruction mix [A]. This is a traditional architec-tural characteristic that measures the frequency ofdifferent classes of instructions (branch, call, stack,load and store, arithmetic, among others). SinceROP attacks depend on chaining blocks of instruc-tions that load data from the hijacked program stackto registers, and for returning to the stack, they mayexhibit different usage of ret and call instructionsas well as stack pop and push instructions.

• Memory locality [I]. Given a set of instructions,memory locality is the difference in the data ad-dresses between subsequent memory accesses [27].Here, it is typical that a distinction is made betweenmemory reads (loads) and writes (stores). SinceROP attacks depend on chaining gadgets from ar-bitrary memory locations, the attacks may exhibitlow memory locality when compared to clean exe-cution. The memory distance between subsequentreads and writes may indicate the execution of aROP attack.

• Register traffic [I]. Two useful register traffic char-acteristics can be measured [17]: 1) the average num-ber of register input operands to an instruction; and2) the register reuse distance, i.e., the number ofinstructions between writing a register and readingit. ROP attacks load data from the hijacked stack toregisters typically using pop instructions that takea single operand. Therefore, the number of instruc-tion operands could be an indicator of the presenceof a gadget chain. Additionally, the usage degreeof the registers themselves could be different fromthat of clean execution.

• Memory reuse [I]. This is an important metric thatcharacterizes the cache behavior of programs. It

4

Figure 2: Workflow of EigenROP. EigenROP periodically interrupts the monitored process to measure the character-istics. It embeds each window of measurements into a high-dimensional space and extracts the principal directionsin that space. Then, in the learning phase, it computes a representative (mean) direction and estimates the density ofdistances of all principal directions to the mean direction. In the detection phase, the principal directions of incomingmeasurements are compared to the mean direction for significant deviation.

measures the number of unique cache blocks refer-enced between subsequent memory reads [47]. Foreach memory read, the corresponding cache blockis retrieved (assuming LRU cache). For each cacheblock, the number of unique cache blocks accessedsince the last time it was referenced is determined.Since ROP attacks operate by using the stack forchaining the gadgets, and the gadgets are typicallyspread out across the memory of the program, theyshall exhibit abnormal reuse of the same memoryblocks when compared to clean execution.

Table 1 shows the top 15 characteristics, ranked bytheir Fisher scores. For each characteristic i, its FisherScore is computed by:

scorei =m(+)

(x(+)

i − xi

)2+ m(−)

(x(−)i − xi

)2

m(+)s2(+)i + m(+)s2(−)

i

, (1)

where (+) and (−) are the infected and clean classesof measurements, respectively; x(y)i and s2(y)

i are themean and variance of characteristic i in class y ∈ {+,−},and xi is the overall mean of feature i over both the in-fected and clean measurements. The Fisher Score is awidely established feature filtering method that assignshigher scores to features that result in greater separationbetween the means of clean and infected samples. Notethat we used infected and clean measurements here toquantify the discriminative power of the selected charac-teristics. The infected measurements are not used duringthe learning phase of EigenROP.

Table 1: Top 15 characteristics sorted by discriminationpower (highest to lowest). Chosen characteristics aremarked with ?. Types A, I and M stand for “architec-tural,” “microarchitecture-independent” and “microar-chitectural,” respectively. All counts are for instructions(insns) retired.Rank Type Name Description

? 1 A INST RET # leave and ret insns.? 2 A INST CALL # near call insns.? 3 I MEM REUSE Memory reuse distance.? 4 A INST STACK # pop and push insns.? 5 I MEM RDIST Memory read distance.

6 A INST LOAD # memory read insns.? 7 I REG OPS Avg. # register operands.? 8 M MISP CBR Mispredicted branches.

9 A INST ARITH # arithmetic insns.? 10 M MISP RET Mispredicted ret insns.

11 A INST STORE # memory write insns.? 12 I MEM WDIST Memory write distance.? 13 A INST NOP # nop insns.

14 I REG REUSE Register reuse distance.15 I ILP Instruction level

parallelism.

5

Since the Fisher Score ignores mutual information,some of the scored characteristics might be redundant.Therefore, we picked 10 features out of the top 15 asfollows. First, we excluded Instruction Level Paral-lelism (a measure of how many instructions of a pro-gram can be executed in parallel) since it added signifi-cant performance overhead and is highly dependent onthe type of application. For example, cryptography ap-plications may exhibit low instruction level parallelism,while a scientific computation program may exhibit highparallelism. Similarly, we excluded INST LOAD andINST ARITH. Via experimentation, we found that REG -REUSE does not increase the accuracy of the model, sowe excluded it as well.

5 Learning and Detection

Given a sequence T of d-dimensional measurements, wedivide T into n subsequences using a sliding window ofwidth m. Let us denote the resulting subsequences by:

S(j) =

xT(j)

1xT(j)

2...

xT(j)m

, (2)

for j = 1 . . . n. Note that each x(j)i is a vector of d

measured characteristics.Next, each S(j) is embedded (implicitly mapped) into

a higher dimension spaceH with Φ : Rd → H, and theprincipal component vectors of S(j) in H are extracted.This is done using Kernel PCA [36], which solves thefollowing eigenvalue problem:

λ(j)i v(j)

i = Kv(j)i , (3)

where λ(j)i are the eigenvalues of K, v(j)

i are the nor-malized eigenvectors of K, and K is the m × m kernelmatrix

[k(

x(j)i , x(j)

l

)]for i = 1 . . . m; l = 1 . . . m. Here, k

is the kernel function, which we set to the Radial BasisFunction (RBF) given by:

k(x1, x2) = Φ(x1)Φ(x2)T (4)

= exp(−γ ‖x1 − x2‖2

), (5)

where γ = 1d . We assume K is centered [36], i.e., K =

K− 1mK− K1m + 1mK1m, where 1m is an m×m matrixfor which each element takes the value 1

m .Using the eigenvalues and eigenvectors in H, the re-

sultant direction v(j) of the data S(j), embedded inH, isthen computed by:

v(j) = cm

∑i=1

λ(j)i v(j)

i , (6)

where c is a normalizing factor such that v(j)Tv(j) =1. This direction can be perceived as a representativedirection of all the principal axes of S(j) in the kernelspaceH.

We then compute the mean direction µµµ of T by:

µµµ =∑n

j=1 v(j)∥∥∥∑nj=1 v(j)

∥∥∥ . (7)

The direction µµµ is the representative direction for theentire trace of characteristics, where the extracted direc-tions v(j) distribute around µµµ.

Hence, the following similarity vector Z is con-structed:

Z =

v(1)Tµµµ

v(2)Tµµµ...

v(n)Tµµµ

, (8)

where each row corresponds to the angular distancebetween each direction v(j) and µµµ.

Next, a kernel density is estimated over Z using thestandard normal kernel density estimator, given by:

fh(z) =1

nh

n

∑i=1

N

(z− zi

h

), (9)

where h is the smoothing parameter (the bandwidth),zi ∈ Z, and N is the standard normal function. In our im-plementation, we chose the value of h using grid search.

We expect the resulting density to be close to expo-nential since the directions extracted from clean mea-surements are expected to be concentrated (tightly dis-tributed around µµµ), resulting in a skewed density witha peak around high similarity values. Therefore, wereduce the skewness of fh by applying the followinglogarithmic transform:

fh(z) = fh(z) log ( fh(z)) , (10)

where the area under the curve of fh(z) gives the en-tropy η of fh. This transforms the bulk of the densitytowards the peak, resulting in a shorter (easier to thresh-old) tail.

This concludes the learning phase. The following sub-section explains the anomaly metric and the detectionphase of EigenROP.

5.1 Anomaly Metric

Given an incoming subsequence of measurements S′(j),an anomaly is detected if the direction of S′(j), in theHspace, is significantly different from the learned direc-tions around µµµ. The decision r is computed by:

6

v′(j) from Eq. (6) (11)

z′(j)= v′(j)T

µµµ (12)

ζ =∫ z′(j)

−1fh(z) dz (13)

r = sgn(ζ − θη) , (14)

where θ ∈ (0, 1) is the detection threshold, which setsthe fraction of the entropy that the model leaves out fordetecting attacks. This concludes the detection phase.

To summarize, EigenROP operates as follows:

Learning Phase

1. Periodically, collect program characteristics{S(j)}n

j=1 of the target program.

2. Extract the principal directions {v(j)}nj=1 in a

higher-dimension kernel space.

3. Compute a representative direction µµµ from{v(j)}n

j=1.

4. Estimate η of the distance between the principaldirections and µµµ.

Detection Phase

5. Repeat steps 1 and 2.

6. Compute the anomaly metric r, if r equals−1 thenan attack is present.

5.2 Detection Time and Space Complexity

Computing the anomaly metric requires performing theKPCA computation (Eq. (3)) in O(m3) [36]. Computingthe resultant vector (Eq. (6)) takes O(m2). The distancein Eq. (12) is computed in O(m). Thus, it takes a to-tal time of O(m3) to compute the anomaly metric. Ourmodel requires space m · d for the incoming measure-ments window S(j), m for the representative directionµµµ, and c for the transformed density (Eq. (10)), wherec is the number of points of the density. Thus, it takesa total space of O(md + c). Note that all terms in ourprototype implementation of EigenROP are bounded:d = 10, m ≤ 10 and c ≤ 1000.

5.3 Handling Multiple Runs

The algorithm discussed so far focused on a single runof the monitored program. To handle multiple runs, weproceed as follows. Given a set {T(i)}k

i=1 of sequences,where each T(i) corresponds to a different run of themonitored program, we compute the family of sets ofdirections {{v(j)}n(i)

j=1}ki=1, then compute µµµ over the en-

tire family. Here, storing the entire set of directions is

Figure 3: Architecture of EigenROP within Pin.

not necessary, since µµµ and the distance density can becomputed iteratively using online (streaming) mean anddensity algorithms.

6 Implementation

We implemented a proof-of-concept prototype of Eigen-ROP on top of MICA [21], a Pintool for collecting pro-gram characteristics. The EigenROP module is imple-mented in ∼700 lines of Python, with the aid of theSciKit-Learn [6] machine learning toolkit. Pin [29] is ageneric dynamic instrumentation framework with a richAPI that Pintools use to specify own instrumentationcode. Pintools are written in C/C++. We chose Pin sinceit achieves the best performance among various dynamicinstrumentation platforms [29].

Fig. 3 shows the architecture of EigenROP within Pin.MICA uses the instrumentation API of Pin to specify itsown instrumentation code, which computes the differentcharacteristics. As the program executes, the JIT com-piler in Pin intercepts the program traces and compilesthe instrumentation code into the program, where thecharacteristics are computed over the program traces.A program trace is a chain of multiple basic blocks thatend with an unconditional jump. The measurementsreported by MICA are stored in a d-dimensional circularbuffer, one row at a time. The EigenROP module con-sumes and processes the buffer using a sliding windowas explained in Section 5. Finally, the learned directionsand densities are stored on disk for usage in the detec-tion phase, where the same procedure is followed inaddition to computing the anomaly metric. If a ROP isdetected, EigenROP logs an alarm and terminates thetarget process.

7 Evaluation

We evaluate the security effectiveness, the added valueof using microarchitecture-independent characteristics,

7

and the tradeoff between runtime overhead and the de-tection accuracy of EigenROP. For security evaluation,we conducted several experiments using in-the-wildROP attacks and attacks generated by the ROPC [5]compiler. For performance evaluation, we used theUnixBench [7] systems benchmark. We ran our experi-ments on an Intel Core i7-4870HQ 2.5 GHZ machine with4 GB of RAM, running 32-bit Linux Ubuntu 12.04, IntelPin version 2.14, MICA version 0.40 and GCC version4.6.3.

7.1 Evaluation Metrics

To evaluate our approach, we use Receiver OperatingCharacteristics (ROC) curves and the area under thecurve (AUC) scores. The x-axis of the ROC curve givesthe false positive rate (FPR), and the y-axis gives thetrue positive rate (TPR). The FPR (eqv. with 1− speci-ficity) represents the probability of false alarm, i.e., thelikelihood of mislabeling a clean execution as an attack,given by FP/(FP+ TN). The TPR (eqv. with sensitivity)represents the probability of correct detection of ROPexecution, given by TP/(TP + FN). Each point on theROC curve corresponds to the FPR and TPR, for a spe-cific value of θ ∈ (0, 1). The area under the curve (AUC)of the ROC is also computed, which provides a quantita-tive single value measure of the accuracy of the systemfor a variable θ. The higher the AUC, the higher thedetection accuracy. The AUC reaches its best value at 1and its worst at 0.

Table 2: Data set used in our experiments.

Program Avg. Payload Length # of Samples

cmp 800 80cpio 650 210diff 910 140file 700 315grep 631 150hteditor 60 100openssl 1021 195php 400 265sed 570 350sort 712 110stat 673 110wget 813 90

Total Samples: 2115

7.2 Dataset and Evaluation Procedure

We used two publicly available ROP exploits: OSVDB-ID:87289 [2] and OSVDB-ID:72644 [4], for the Linux HexEditor (hteditor) version 2.0.20 and PHP version 5.3.6,respectively. We also used a number of exploits gener-ated by the ROP gadgets finder and compiler ROPC [5],

for common Linux programs (4 different exploits perprogram). Table 2 shows the programs used in our eval-uation, the average payload length (the number of in-structions) of each exploit, and the number of samplesper program.

We collected clean samples for each target programby running the functionality tests that shipped with theprogram. In the case of hteditor, as it did not ship withfunctionality tests, we ran it on 100 random PDF filesdownloaded from the web. We collected infected sam-ples following a similar approach to [13, 34]: assumethat the attacker had successfully compromised the tar-get process, and inject code into the target process toload a given exploit payload into memory and execute it.The payload (gadgets) is executed by directly jumpingto the beginning of the payload at random points duringthe execution of the process. Each payload executionwas considered an infected (attack) sample.

For each program, we used 5-fold cross-validation:4 clean folds for training, and 1 clean fold for testingalong with infected samples. We used the same numberof clean and infected samples in the testing fold. Themean of the resulting five TPRs and FPRs is then used incomputing the ROC and its AUC. We stress that labeledmeasurements were collected strictly for testing; Eigen-ROP uses only the clean measurements for training.

7.3 Detection Accuracy

7.3.1 Hteditor OSVDB-ID:87289 and PHP OSVDB-ID:72644

EigenROP successfully detected the hteditor ROP ex-ploit with sampling intervals up to 16k instructions re-tired and detected the PHP ROP with sampling intervalsup to 32k. In both cases, EigenROP resulted in zero falsepositives. We emphasize that the focus here is on the de-tection of the ROP stage of the exploits, i.e., the executionof a gadget chain, rather than the execution of a shellcode or a different process (both were shown to be easilydetectable [41, 30]). Despite the very small ROP length(only ∼60 instructions in the case of hteditor) whencompared to the sampling window size, EigenROP stilldetected the deviation in the programs characteristics.

7.3.2 Overall Detection Accuracy

Fig. 4 shows the overall ROC of all experiments, fora sampling interval of 16k instructions. EigenROPachieved an overall accuracy (AUC) of 81%. The bestpoint of performance had 80% TPR and 0.8% FPR. Notethat EigenROP focuses on the detection of ROP. This isdifferent from relevant prior work [41, 30], where it wasassumed that the attacks undergo multiple stages suchthat only the first stage is a ROP chain, while the rest arenormally injected code or a different process. Since theROP chain length is usually in the order of a few hun-dred instructions, it is significantly more challenging for

8

��

����

����

����

����

��

�� ���� ���� ���� ���� ��

������������������

�������������������

�������������

����������

Figure 4: Overall ROC of EigenROP. The sampling inter-val was set to 16k instructions. The AUC is 0.81.

it to be detected. While the authors in [41, 30] detectedthe non-ROP stages of the attack with high accuracy, andas they noted, their proposed models performed poorlyin the detection of the ROP chains alone (AUC rangedfrom 49% to 68%). In contrast, EigenROP focuses on thedetection of the execution of the ROP gadget chain itself.

7.3.3 Sampling Granularity

The breakdown of the detection accuracy for differentsampling intervals is shown in Fig. 5. As expected, theaccuracy drops for very large sampling intervals, giventhe small number of instructions of the attacks. Out ofall the programs, wget had the worst detection accuracydue to excessive use of signals, which exhibits poor local-ity and reuse (see Section 8 for discussion). The densityestimate of wget was very heavy-tailed, which resultedin low discrimination between clean runs and attacks.On the other hand, openssl had the highest detectionaccuracy, as its characteristics had higher concentrationaround the mean direction. The bulk of the distributionof the AUC curves neared the best accuracy curve (theAUC was skewed towards the worst accuracy curve),indicating that the behavior of wget was possibly anoutlier.

7.3.4 Microarchitecture-independent vs. Other Char-acteristics

Fig. 6 shows the difference in accuracy with and with-out the microarchitecture-independent characteristics.By including microarchitecture-independent characteris-tics, an increase of 9% to 15% in accuracy was achieved.This indicates that microarchitecture-independent char-acteristics contribute significantly to the detection per-formance of EigenROP.

����

����

����

����

����

��

�� ��� ��� ��� ���� ����

���

���������������������������������������������

�����������

�����

Figure 5: AUC for different sampling intervals. Thehigher the AUC curve, the better the detection accuracy.

����

�����

����

�����

����

�����

�� ��� ��� ��� ���� ����

���

���������������������������������������������

���������������

Figure 6: AUC for different sampling intervals, with andwithout the microarchitecture-independent characteris-tics.

7.3.5 Sliding Window Size

Fig. 7 shows the effect of changing the sliding windowsize m on the detection accuracy. Note that the windowsize controls the amount of temporal information avail-able to the model. We observe that the effect of the win-dow size on accuracy goes through three stages. First,too small window sizes hurt the detection accuracy, sincesmall windows give higher variances in principal direc-tions, resulting in higher FPR. Second, as the windowsize increases, the detection accuracy improves since thedirections become more stable around µµµ. Finally, theaccuracy deteriorates for too large window sizes sincethe influence of clean measurements on the principaldirections dominates that of the ROP payload, resultingin lower TPR.

7.4 Overhead-Accuracy Tradeoff

We quantified the overhead of EigenROP for differentsampling intervals by measuring the overall percent-

9

����

�����

����

�����

����

�����

� � � �� �� ��

���

�������������������

Figure 7: AUC for different sliding window sizes. Bothtoo small and too large windows result in lower detec-tion accuracy.

age slowdown in execution of UnixBench [7]. Fig. 8shows the overhead and accuracy tradeoff. The over-head incurred by EigenROP exponentially decreases asthe sampling interval increases. We also observe thatthe reduction in overhead outpaces the decay in accuracy.The overhead incurred by MICA is approximately con-stant as MICA analyzes the individual instructions oftarget programs, and the total number of instructionsof each execution is invariant of the sampling interval.Overall, the incurred runtime overhead is comparableto similar dynamic instrumentation and HPC-based de-fenses [15, 34, 41]. Note that we did not perform anyoptimization attempts to reduce the overhead of Eigen-ROP or MICA. Our work is orthogonal to how the pro-gram characteristics are collected. While we used MICAand Pin in our prototype implementation of EigenROP,they may not be the best tools for full build-out and fullproduction. Finally, we emphasize that the memory andspace overhead incurred by EigenROP are bounded andnegligible (see Sections 5.2 and 6).

8 Discussion and Improvements

8.1 False Positives and Negatives

The detection approach of EigenROP (and relevant HPC-based solutions [30, 16, 41]) is based on the hypothesisthat programs exhibit characteristics that are relativelyconcentrated around some statistic – in our case, themean direction. However, if a program exhibits behaviorthat has a large spread, it becomes harder to separateanomalies from benign executions, resulting in a higherfalse positive rate (or a lower true positive rate).

From our experience with EigenROP, we observedthat programs that use far jumps (e.g., setjmp/longjmp,signal) or extensively multiplex between data sources(e.g., using select for socket multiplexing) are more

��

���

���

���

���

���

���

���

���

���

�� ��� ��� ��� ���� ���������

����

�����

����

�����

����

�����

����

�����

����

������������������

���

���������������������������������������������

������������

���

Figure 8: Overhead-accuracy tradeoff. The runtime over-head of MICA is measured relative to the overhead ofPin.

likely to suffer from false positives. The reason is thatsuch programming constructs access far code and data,which inherently exhibits poor branch predictability,memory locality, and reuse. A possible workaround is toidentify the entry and exit points of such code sites andbuild a separate model for the characteristics exhibitedby those code sites. ROP chains missed by EigenROPwere very short chains (<40 instructions) with small gad-gets (2-4 instructions per gadget). This is mainly due tothe relatively large sampling interval compared to thechain size. To handle such very short chains, EigenROPcan be complemented by low-overhead solutions thattarget short gadgets and chains (e.g., kBouncer [33] andROPecker [13]).

8.2 ROP Variants

In our evaluation of EigenROP, we used conventionalROP payloads that use return instructions to chain thegadgets. However, several variants of ROP were dis-covered by researchers. For example, in [11], Jump-Oriented Programming (JOP) was introduced where in-direct jumps are employed to chain the gadgets ratherthan using return instructions. In [37], COOP was intro-duced where a loop in the program code that invokesattacker-controlled virtual function calls in C++ binariesis used to dispatch and chain the gadgets.

The goal is to simulate ret using a sequence of instruc-tions that pops an address from the stack then jumps tothat address using an indirect jump instruction, i.e., apop-jump gadget. To use the pop-jump gadget, othergadgets have to end in an indirect jmp that transfers con-trol to the pop-jump gadget, e.g., [add; mov; ...; jmp

eax; pop ebx; jmp ebx;] where [jmp eax;] jumps tothe pop-jump gadget, and [pop ebx; jmp ebx;] exe-cutes the pop-jump gadget and transfers control to thenext gadget. In EigenROP, we picked the characteristicsthat cover the behavior of all ROP variants (branches,

10

calls and returns, memory locality and reuse, stack usage,and nop sleds) regardless of how the gadgets are chained.Also, it is easy and straightforward to include other rel-evant characteristics if need be, such as the number ofindirect jump instructions retired. Overall, EigenROPhas the advantage that the detection is robust against at-tack variations, since it captures the execution behaviorof benign runs, and does not put strong assumptions onhow the gadgets are chained at the ISA level.

8.3 Evasion and Mimicry Attacks

Three recent attack gadgets were presented [12] thatbypass ROP defenses through evasion and mimicry: call-preceded gadgets, evasion gadgets, and history-flushinggadgets.

Call-preceded gadgets are constructed from sequencesof instructions that are preceded by a call instruction inthe program memory. Such gadgets violate the assump-tion made by the majority of defenses [33, 13, 15, 23]that a sequence ending in ret must be legitimate, if itwas preceded by any call. Since EigenROP does notdepend on branch tracing, it is not vulnerable to attacksbased on call-preceded gadgets. Moreover, the returnaddress will be mispredicted, regardless of the gadgettype, unless the call-ret are strictly paired. Since Eigen-ROP takes the misprediction rate of returns into account(see Section 4), call-preceded gadgets will result in abnor-mal mispredictions, potentially increasing the detectionaccuracy.

Evasion gadgets were introduced for evading ROPdetectors that use heuristics based on the length of thegadget chain (e.g., [33, 13]). Such detectors detect ROPby identifying gadget chains within some window of theexecution trace. The heuristics are based on the lengthof the gadgets within the chain, with the presumptionthat short gadgets are likely part of an executing ROP.Evasion gadgets violate that assumption by using longenough gadgets to violate such constraints. Since Eigen-ROP does not depend on the gadget chain length, ratheron the characteristics of the gadgets, it is not vulnerableto attacks based on evasion gadgets.

History-flushing gadgets target defenses that onlykeep a limited history about execution (typically de-pendent on the available hardware buffer size wherethe history is recorded). History is flushed by utiliz-ing innocuous gadgets to fill up the history. For exam-ple, kBouncer [33] uses the Last Branch Record (LBR), ahardware feature that records the most recent 16 takenbranches by the processor. While kBouncer is very ef-ficient against short ROP chains, it can be evaded by aROP chain that executes any 16 valid indirect jumps tofill the LBR with legitimate branches completely[12].

In our context, flushing the history means manipulat-ing all affected characteristics by the ROP, such that theyappear normal. The attacker would need to chain gad-gets that exhibit similar characteristics to benign code,

in addition to achieving the attack goal. While this istheoretically possible, we argue that it is hard to realizesuch attacks in practice. First, chaining more gadgetswould require larger attacker-controlled memory space.Second, if the attacker includes benign code in the ROPto mimic normal behavior, the benign code would berequired to either have no effect on the actual ROP ex-ecution or be undone by chaining, even more, gadgets.Third, and As noted in [12, 34], history flushing comes atthe expense of significant slowdown (reported 20-timesslowdown) in the execution of the ROP payload.

Randomization has been proposed as a defenseagainst evasion and mimicry attacks in anomaly-basedintrusion detection systems [44, 43], and more re-cently [39] where it was shown that mimicry attackscould be efficiently detected by judging the quality ofdetection using an ensemble of classifiers. In the con-text of EigenROP, a potential defense strategy is to ran-domize the set of characteristics measured by EigenROPand build multiple detectors using different subsets ofcharacteristics. The detectors can be constructed usingdifferent models, where a subset of the models is chosenat runtime at random. Additionally, we can randomlychoose between the models at different points in theprogram. For example, using 15 characteristics and 5models where each model randomly uses 5 characteris-tics, there will be 5 · (15

5 ) = 15015 possible configurations.Since the attacker does not have direct control over theprogram characteristics, she would need to craft ROPpayloads that bypass all possible configurations of detec-tors and characteristics, significantly increasing the costto attack.

8.4 Overhead Reduction

The current downside of using michroarchitecture-inde-pendent characteristics is the need for dynamic instru-mentation to compute the characteristics. As shown inSection 7.4, this may incur a non-negligible overheadpenalty. However, this is an active research area, andmore efficient program characterization algorithms andtools are being developed [8]. The need for dynamicinstrumentation can be eliminated if the hardware orthe kernel provide support by computing the requiredcharacteristics. Rather than instrumenting the processin user space, the characteristics can be computed (bythe kernel or the hardware) and written to a memory-mapped ring buffer that is readable in user space. Incase the buffer is not consumed quickly enough, an inter-rupt can be triggered to pause the monitored process. Asimilar approach is adopted by the Linux performancecounter subsystem [3], which already provides supportfor a wide range of architectural and microarchitecturalcharacteristics.

11

8.5 Input Coverage

In the learning phase of EigenROP, the target program isexecuted over benign inputs. Sufficient input coveragecould arguably be a challenging task for the deploymentof EigenROP. In our evaluation, we used the positivefunctionality tests that shipped with the programs totrain EigenROP, which are integral to the software de-velopment lifecycle. In addition to functionality tests,EigenROP models can be constructed from successfuldry runs during internal acceptance and pre-release test-ing. Additionally, EigenROP can even be trained by endusers. To avoid learning bad behavior, the learned mod-els can be aggregated from clusters of users and aver-aged (by computing the mean directions and densities),then filtered (cleaned) from outliers. Further, EigenROPcan continue learning even after deployment by iter-atively updating the learned directions and densities.This can be a privilege that is tied to the user group, forinstance, update the models only from processes ownedby admin users. The effectiveness of training by endusers is currently in our future work.

9 Related Work

While the literature on ROP is vast, due to space con-straints we briefly only mention solutions that used hard-ware or software characteristics, as well as anomaly-based solutions.

9.1 Binary Rewriting and Instrumentation

Some solutions were presented that used binary rewrit-ing and dynamic instrumentation to detect ROP attacks.ROPDefender [15] enforces call-ret pairing by maintain-ing a shadow stack of call and ret targets. When aret instruction is executed, ROPDefender compares theshadow stack to the actual system stack. If the two stacksdo not match, then a ROP is detected. ROPStop [23] usesstatic binary rewriting to insert instrumentations thatcheck two main constraints on the program counter andthe call stack: 1) the program counter must point to avalid intended instruction, and 2) the call stack heightis valid. The 2nd constraint is checked by analyzingthe CFG and computing the set of all possible call stackheights from function entry points to branching points.If any of the constraints is not satisfied, a ROP is de-tected.

Similarly, ROPGuard [18] checks a set of constraintsover the call stack, call and ret instructions at entrypoints to system calls, e.g., ret instructions must be pre-ceded by call instructions, the call instruction mustlead back to current entry point, etc. While such solu-tions are easy to deploy and require no system modifi-cations, they are limited by some factors: 1) using CFGsis constrained by the speed and accuracy of binary dis-assembly and CFG construction. 2) Binary rewriting

breaks self-checksumming and signed code. 3) Framepointers that are required to traverse the stack are usuallyomitted by compilers during optimization. And, 4) call-ret pairing restricts valid, commonly used, call-without-return assembly constructs, e.g., using [setjmp; ...;

longjmp;] for exception handling, and [call; pop;]

for retrieving the program counter.

9.2 Hardware Branch Tracing

Recently, ROP defenses that leverage existing hardwarebranch tracing features were introduced. kBouncer [33]uses the Last Branch Record (LBR) on modern Intel pro-cessors to check for sequences of consecutive call-ret

instructions. The LBR stores the most recent 4-16 indirectbranches executed by the processor. kBouncer checks,at the entry of every system call, if 1) call instructionspreceded ret targets, and 2) there is no call-ret se-quence of length greater than 8. ROPecker [13] extendskBouncer by also checking at arbitrary points during theprogram execution, and counting the number of possi-ble gadget-like sequences ahead of the program counter.Similarly, Eunomia [46] utilizes the Branch Trace Store(BTS) to check for unpaired call-ret sequences. Unfor-tunately, while these approaches incur very low over-head, attackers have bypassed them by violating thelength based heuristics [14, 12, 19]. Nevertheless, theyprovide a solid defense against very short ROP chains(due to the limited hardware buffer sizes) and are com-plementary to this work.

9.3 Anomaly-based Solutions

In [26], Krugel et al. introduced an application specificapproach that uses network traffic to detect maliciousactivities. Mazeroff et al. [31] described methods forinferring and using probabilistic models for detectinganomalous sequences of API calls. In [24], Jyostna et al.proposed a system for detecting anomalous program be-havior by clustering critical system calls. While networktraffic and system call defenses are simple and easy todeploy, they are susceptible to mimicry attacks [25].

One of the first works on using hardware architecturalcharacteristics of programs was the work of Malone etal. [30]. They showed that hardware performance coun-ters (HPC) could be utilized to detect unauthorized soft-ware changes. The authors recorded HPC measurementsof the original programs and used linear regression todetect if the program was modified at runtime. Demmeet al. [16] ported the idea to Android, and proposed hard-ware modifications to detect malware using HPC mea-surements from good and malicious samples. Stewin etal. [40] proposed detecting DMA attacks by monitoringthe number of transactions on the memory bus.

In [41], Tang et al. combined microarchitectural charac-teristics with architectural characteristics to detect drive-by attacks. They assumed that attacks consist of three

12

stages: ROP stage disables DEP, stage 1 downloads amalicious program, and stage 2 executes the maliciousprogram. By training a one-class Support Vector Ma-chine (oc-SVM) over the architectural and microarchitec-tural characteristics of benign samples, they showed thatstage 1 of the attacks could be detected with high accu-racy, while their model performed poorly on stage 2 ofthe attacks. This is because the oc-SVM is very sensitiveto tuning parameters, and the chosen features did nothave sufficient discrimination power to detect the exe-cution of ROP chains. This is different from EigenROPsince we solely focus on stage 2 of the attack. Similarly,in [10, 34], two solutions were presented that trained atwo-class SVM using the architectural characteristics ofboth clean executions and attacks.

In contrast, EigenROP does not require any analysisof the binaries and operates using measurements onlyfrom clean executions. It does not need source code ordebug information, and does not depend on branch trac-ing. EigenROP introduces a new class of anomaly-baseddetectors that utilize both hardware characteristics andmicroarchitecture-independent characteristics of moni-tored programs.

10 Conclusion

We presented EigenROP, a novel anomaly-based ROPdetector that utilizes program characteristics and direc-tional statistics. To the best of our knowledge, we arethe first to study the effectiveness of using microarchitec-ture-independent program characteristics versus typicalarchitectural and microarchitectural characteristics, inthe detection of ROP. We demonstrated the ability ofEigenROP to detect both in-the-wild and pure ROP ex-ploits, despite the short payload length. EigenROP isunsupervised, fully transparent, and does not requireany side information about the protected programs. Onelimitation of using microarchitecture-independent char-acteristics is the need for dynamic instrumentation tocollect the measurements. One potential avenue to sig-nificantly reduce the overhead is by implementing therun-time monitors in hardware. Also, hardware sup-port would probably increase the detection accuracy byenabling low-cost fine granularity monitoring. Whileour work demonstrates that ROP payloads can be de-tected using simple program characteristics, there arestill needed improvements concerning detection accu-racy and overhead reduction. Despite that, EigenROPraises the bar for ROP attacks, and can be easily coupledwith hardware-based defenses to detect ROP transpar-ently without program changes [28, 22].

References

[1] Analyzing the first ROP-only, sandbox-escapingPDF exploit. https://blogs.mcafee.com/mcafee-

labs/analyzing-the-first-rop-only-sandbox-

escaping-pdf-exploit.

[2] HT Editor 2.0.20 - Buffer Overflow (ROP). https://www.exploit-db.com/exploits/22683/.

[3] Linux performance counter subsystem.https://github.com/torvalds/linux/blob/

master/tools/perf/design.txt.

[4] PHP 5.3.6 - Buffer Overflow PoC (ROP). https:

//www.exploit-db.com/exploits/17486/.

[5] ropc: A turing complete ROP compiler. https://github.com/pakt/ropc.

[6] Scikit. http://scikit-learn.org/stable/.

[7] Unixbench. https://github.com/kdlucas/byte-

unixbench.

[8] Applications, tools and techniques on the roadto exascale computing. In K. de Bosschere,E. H. D’Hollander, G. R. Joubert, D. Padua, andF. Peters, editors, Advances in Parallel Computing,volume 22. IOS Press, 2012.

[9] M. Abadi, M. Budiu, U. Erlingsson, and J. Ligatti.Control-flow integrity. In Proceedings of the 12thACM conference on Computer and communications se-curity, pages 340–353. ACM, 2005.

[10] M. B. Bahador, M. Abadi, and A. Tajoddin. Hpc-malhunter: Behavioral malware detection usinghardware performance counters and singular valuedecomposition. In Computer and Knowledge Engi-neering (ICCKE), 2014 4th International eConferenceon, pages 703–708. IEEE, 2014.

[11] T. Bletsch, X. Jiang, V. W. Freeh, and Z. Liang. Jump-oriented programming: a new class of code-reuseattack. In Proceedings of the 6th ACM Symposium onInformation, Computer and Communications Security,pages 30–40. ACM, 2011.

[12] N. Carlini and D. Wagner. Rop is still dangerous:Breaking modern defenses. In USENIX SecuritySymposium, 2014.

[13] Y. Cheng, Z. Zhou, Y. Miao, X. Ding, H. DENG,et al. Ropecker: A generic and practical approachfor defending against rop attack. In Network andDistributed System Security (NDSS) Symposium, 2014.

[14] L. Davi, D. Lehmann, A.-R. Sadeghi, and F. Mon-rose. Stitching the gadgets: On the ineffectivenessof coarse-grained control-flow integrity protection.In USENIX Security Symposium, 2014.

13

[15] L. Davi, A.-R. Sadeghi, and M. Winandy. Ropde-fender: A detection tool to defend against return-oriented programming attacks. In Proceedings of the6th ACM Symposium on Information, Computer andCommunications Security, pages 40–51. ACM, 2011.

[16] J. Demme, M. Maycock, J. Schmitz, A. Tang,A. Waksman, S. Sethumadhavan, and S. Stolfo. Onthe feasibility of online malware detection with per-formance counters. ACM SIGARCH Computer Ar-chitecture News, 41(3):559–570, 2013.

[17] M. Franklin and G. S. Sohi. Register traffic analy-sis for streamlining inter-operation communicationin fine-grain parallel processors. In ACM SIGMI-CRO Newsletter, volume 23, pages 236–245. IEEEComputer Society Press, 1992.

[18] I. Fratric. Ropguard: Runtime prevention of return-oriented programming attacks, 2012.

[19] E. Goktas, E. Athanasopoulos, M. Polychronakis,H. Bos, and G. Portokalidis. Size does matter: Whyusing gadget-chain length to prevent code-reuseattacks is hard. In 23rd USENIX Security Symposium,San Diego, CA, pages 417–432, 2014.

[20] K. Hoste and L. Eeckhout. Comparing benchmarksusing key microarchitecture-independent charac-teristics. In Workload Characterization, 2006 IEEEInternational Symposium on, pages 83–92. IEEE, 2006.

[21] K. Hoste and L. Eeckhout. Microarchitecture-independent workload characterization. IEEE Mi-cro, 3:63–72, 2007.

[22] J. F. Hughes, A. Van Dam, M. Morgan, D. F. Sklar,J. D. Foley, and S. K. Feiner. Computer Graphics:Principles and Practice. Pearson Education, 2013.

[23] E. R. Jacobson, A. R. Bernat, W. R. Williams, and B. P.Miller. Detecting code reuse attacks with a modelof conformant program execution. In EngineeringSecure Software and Systems, pages 1–18. Springer,2014.

[24] G. Jyostna, P. Himanshu, and P. Eswari. Detectinganomalous application behaviors using a systemcall clustering method over critical resources. InAdvances in Network Security and Applications, pages53–64. Springer, 2011.

[25] H. G. Kayacik et al. Mimicry attacks demystified:What can attackers do to evade detection? In Pri-vacy, Security and Trust, 2008. PST’08. Sixth AnnualConference on, pages 213–223. IEEE, 2008.

[26] C. Krugel, T. Toth, and E. Kirda. Service specificanomaly detection for network intrusion detection.In Proceedings of the 2002 ACM symposium on Appliedcomputing, pages 201–208. ACM, 2002.

[27] J. Lau, S. Schoemackers, and B. Calder. Structuresfor phase classification. In Performance Analysis ofSystems and Software, 2004 IEEE International Sympo-sium on-ISPASS, pages 57–67. IEEE, 2004.

[28] G. Lavou’e and R. Mantiuk. Quality assessment incomputer graphics. In Visual Signal Quality Assess-ment, pages 243–286. Springer, 2015.

[29] C.-K. Luk, R. Cohn, R. Muth, H. Patil, A. Klauser,G. Lowney, S. Wallace, V. J. Reddi, and K. Hazel-wood. Pin: Building customized program analysistools with dynamic instrumentation. In Proceedingsof the 2005 ACM SIGPLAN Conference on Program-ming Language Design and Implementation, pages 190–200. ACM, 2005.

[30] C. Malone, M. Zahran, and R. Karri. Are hardwareperformance counters a cost effective way for in-tegrity checking of programs. In Proceedings of thesixth ACM workshop on Scalable trusted computing,pages 71–76. ACM, 2011.

[31] G. Mazeroff, J. Gregor, M. Thomason, and R. Ford.Probabilistic suffix models for API sequence anal-ysis of Windows XP applications. Pattern Recogn.,41(1):90–101, Jan 2008.

[32] K. Onarlioglu, L. Bilge, A. Lanzi, D. Balzarotti, andE. Kirda. G-free: defeating return-oriented pro-gramming through gadget-less binaries. In Proceed-ings of the 26th Annual Computer Security ApplicationsConference, pages 49–58. ACM, 2010.

[33] V. Pappas, M. Polychronakis, and A. D. Keromytis.Transparent rop exploit mitigation using indirectbranch tracing. In USENIX Security, pages 447–462,2013.

[34] D. Pfaff, S. Hack, and C. Hammer. Learning howto prevent return-oriented programming efficiently.In Engineering Secure Software and Systems, pages68–85. Springer, 2015.

[35] M. Prandini and M. Ramilli. Return-oriented pro-gramming. Security & Privacy, IEEE, 10(6):84–87,2012.

[36] B. Scholkopf, A. Smola, and K.-R. Muller. Kernelprincipal component analysis. In Artificial NeuralNetworks - ICANN, pages 583–588. Springer, 1997.

[37] F. Schuster, T. Tendyck, C. Liebchen, L. Davi, A.-R.Sadeghi, and T. Holz. Counterfeit object-orientedprogramming: On the difficulty of preventing codereuse attacks in c++ applications. In Security andPrivacy (SP), 2015 IEEE Symposium on, pages 745–762. IEEE, 2015.

14

[38] H. Shacham. The geometry of innocent flesh on thebone: Return-into-libc without function calls (onthe x86). In Proceedings of the 14th ACM conferenceon Computer and communications security, pages 552–561. ACM, 2007.

[39] C. Smutz and A. Stavrou. When a tree falls: Usingdiversity in ensemble classifiers to identify evasionin malware detectors. In Network and DistributedSystem Security (NDSS) Symposium, 2016.

[40] P. Stewin. A primitive for revealing stealthyperipheral-based attacks on the computing plat-forms main memory. In Research in Attacks, Intru-sions, and Defenses, pages 1–20. Springer, 2013.

[41] A. Tang, S. Sethumadhavan, and S. J. Stolfo. Unsu-pervised anomaly-based malware detection usinghardware features. In Research in Attacks, Intrusionsand Defenses, pages 109–129. Springer, 2014.

[42] M. Tran, M. Etheridge, T. Bletsch, X. Jiang, V. Freeh,and P. Ning. On the expressiveness of return-into-libc attacks. In Recent Advances in Intrusion Detection,pages 121–141. Springer, 2011.

[43] K. Wang, J. J. Parekh, and S. J. Stolfo. Anagram:A content anomaly detector resistant to mimicry

attack. In Recent Advances in Intrusion Detection,pages 226–248. Springer, 2006.

[44] H. Xu, W. Du, and S. J. Chapin. Context sensitiveanomaly monitoring of process control flow to de-tect mimicry attacks and impossible paths. In RAID,pages 21–38. Springer, 2004.

[45] J. J. Yi, H. Vandierendonck, L. Eeckhout, and D. J.Lilja. The exigency of benchmark and compilerdrift: designing tomorrow’s processors with yester-day’s tools. In Proceedings of the 20th annual inter-national conference on Supercomputing, pages 75–86.ACM, 2006.

[46] L. Yuan, W. Xing, H. Chen, and B. Zang. Securitybreaches as pmu deviation: detecting and identi-fying security attacks using performance counters.In Proceedings of the Second Asia-Pacific Workshop onSystems, page 6. ACM, 2011.

[47] Y. Zhong, X. Shen, and C. Ding. Program localityanalysis using reuse distance. ACM Transactionson Programming Languages and Systems (TOPLAS),31(6):20, 2009.

15