Dynamic Backdoor Attacks Against Machine Learning Models

Dynamic Backdoor Attacks AgainstMachine Learning Models

Ahmed Salemlowast Rui Wenlowast Michael BackeslowastShiqing Madagger Yang Zhanglowast

lowastCISPA Helmholtz Center for Information SecuritydaggerRutgers University

AbstractmdashMachine learning (ML) has made tremendousprogress during the past decade and is being adopted in variouscritical real-world applications However recent research hasshown that ML models are vulnerable to multiple security andprivacy attacks In particular backdoor attacks against MLmodels that have recently raised a lot of awareness A successfulbackdoor attack can cause severe consequences such as allowingan adversary to bypass critical authentication systems

Current backdooring techniques rely on adding static triggers(with fixed patterns and locations) on ML model inputs Inthis paper we propose the first class of dynamic backdooringtechniques Random Backdoor Backdoor Generating Network(BaN) and conditional Backdoor Generating Network (c-BaN)Triggers generated by our techniques can have random patternsand locations which reduce the efficacy of the current backdoordetection mechanisms In particular BaN and c-BaN are thefirst two schemes that algorithmically generate triggers whichrely on a novel generative network Moreover c-BaN is the firstconditional backdooring technique that given a target label itcan generate a target-specific trigger Both BaN and c-BaN areessentially a general framework which renders the adversary theflexibility for further customizing backdoor attacks

We extensively evaluate our techniques on three bench-mark datasets MNIST CelebA and CIFAR-10 Our techniquesachieve almost perfect attack performance on backdoored datawith a negligible utility loss We further show that our techniquescan bypass current state-of-the-art defense mechanisms againstbackdoor attacks including Neural Cleanse ABS and STRIP

I INTRODUCTION

Machine learning (ML) represented by Deep Neural Net-work (DNN) has made tremendous progress during the pastdecade and ML models have been adopted in a wide rangeof real-world applications including those that play criticalroles For instance Applersquos FaceID [1] is using ML-basedfacial recognition systems for unlocking the mobile deviceand authenticating purchases in Apple Pay However recentresearch has shown that machine learning models are vulner-able to various security and privacy attacks such as evasionattacks [33] [32] [48] membership inference attacks [39][37] model stealing attacks [44] [29] [46] data poisoningattacks [5] [17] [42] Trojan attacks [22] and backdoorattacks [49] [12]

In this work we focus on backdoor attacks against DNNmodels on image classification tasks which are among themost successful ML applications deployed in the real world Inthe backdoor attack setting an adversary trains an ML modelwhich can intentionally misclassify any input with an added

trigger (a secret pattern constructed from a set of neighboringpixels eg a white square) to a specific target label To mounta backdoor attack the adversary first constructs backdooreddata by adding the trigger to a subset of the clean data andchanging their corresponding labels to the target label Nextthe adversary uses both clean and backdoored data to trainthe model The clean and backdoored data are needed so themodel can learn its original task and the backdoor behaviorsimultaneously Backdoor attacks can cause severe securityand privacy consequences For instance an adversary canimplant a backdoor in an authentication system to grant herselfunauthorized access

Current state-of-the-art backdoor attacks [12] [22] [49]generate static triggers in terms of fixed trigger pattern andlocation (on the input) For instance Figure 1a shows anexample of triggers constructed by Badnets [12] one ofthe most popular backdoor attack methods As we can seeBadnets in this case uses a white square as a trigger and alwaysplaces it in the top-left corner of all inputs Recent proposeddefense mechanisms [47] [21] leverage the static property oftriggers to detect whether an ML model is backdoored or not

A Our Contributions

In this work we propose the first class of backdooringtechniques against ML models that generate dynamic triggersin terms of trigger pattern and location We refer to ourtechniques as dynamic backdoor attacks Dynamic backdoorattacks offer the adversary more flexibility as they allowtriggers to have different patterns and locations Moreover ourtechniques largely reduce the efficacy of the current defensemechanisms demonstrated by our empirical evaluation Fig-ure 1b shows an example of our dynamic backdoor attacksimplemented in a model trained on the CelebA dataset [23]In addition we extend our techniques to work for all labels ofthe backdoored ML model while the current backdoor attacksonly focus on a single or a few target labels This furtherincreases the difficulty of our backdoors being mitigated

In total we propose 3 different dynamic backdoor tech-niques namely Random Backdoor Backdoor Generating Net-work (BaN) and conditional Backdoor Generating Network(c-BaN) In particular the latter two attacks algorithmicallygenerate triggers to mount backdoor attacks which are first oftheir kind In the following we abstractly introduce each ofour techniques

arX

iv2

003

0367

5v1

[cs

CR

] 7

Mar

202

0

(a) Static backdoor

(b) Dynamic backdoor

Fig 1 A comparison between static and dynamic backdoors Figure 1a shows an example for static backdoors with a fixedtrigger (white square at top left corner of the image) Figure 1b show examples for the dynamic backdoor with different triggersfor the same target label As the figures show the dynamic backdoor trigger have different location and patterns compared tothe static backdoor where there is only a single trigger with a fixed location and pattern

Random Backdoor In this approach we construct triggersby sampling them from a uniform distribution Then we placeeach randomly generated trigger at a random location foreach input which is then mixed with clean data to train thebackdoor model

Backdoor Generating Network (BaN) In our second tech-nique we propose a generative ML model ie BaN togenerate triggers To the best of our knowledge this is thefirst backdoor attack which uses a generative network toautomatically construct triggers which increases the flexibilityof the adversary to perform backdoor attacks BaN is trainedjointly with the backdoor model it takes a latent code sampledfrom a uniform distribution to generate a trigger then placeit at a random location on the input thus making the triggerdynamic in terms of pattern and location Moreover BaN isessentially a general framework under which the adversarycan change and adapt its loss function to her requirementsFor instance if there is a specific backdoor defense in placethe adversary can evade the defense by adding a tailoreddiscriminative loss in BaN

conditional Backdoor Generating Network (c-BaN) Bothof our Random Backdoor and the BaN techniques can imple-ment a dynamic backdoor for either a single target label ormultiple target labels However for the case of the multipletarget labels both techniques require each target label to haveits unique trigger locations In other words a single locationcannot have triggers for different target labels

Our last and most advanced technique overcomes the previ-ous two techniquesrsquo limitation of having disjoint location setsfor the multiple target labels In this technique we transformthe BaN into a conditional BaN (c-BaN) to force it to generatelabel specific triggers More specifically we modify the BaNrsquosarchitecture to include the target label as the input to generatea trigger for this specific label This target specific triggersproperty allows the triggers for different target labels to be

positioned at any location In other words each target labeldoes not need to have its unique trigger locations

To demonstrate the effectiveness of our proposed tech-niques we perform empirical analysis with three ML modelarchitectures over three benchmark datasets All of our tech-niques achieve almost a perfect backdoor accuracy ie the ac-curacy of the backdoored model on the backdoored data is ap-proximately 100 with a negligible utility loss For instanceour BaN trained models on CelebA [23] and MNIST [2]datasets achieve 70 and 99 accuracy respectively whichis the same accuracy as the clean model Also c-BaN BaNand Random Backdoor trained models achieve 92 921and 92 accuracy on the CIFAR-10 [3] dataset respectivelywhich is almost the same as the performance of a clean model(924) Moreover we evaluate our techniques against threeof the current state-of-the-art backdoor defense techniquesnamely Neural Cleanse [47] ABS [21] and STRIP [10] Ourresults show that our techniques can bypass these defenses

In general our contributions can be summarized as thefollowing

bull We broaden the class of backdoor attacks by introducingthe dynamic backdoor attacks

bull We propose both Backdoor Generating Network (BaN)and conditional Backdoor Generating Network (c-BaN)which are the first algorithmic backdoor paradigm

bull Our dynamic backdoor attacks achieve strong perfor-mance while bypassing the current state-of-the-art back-door defense techniques

B Organization

We first present the necessary background knowledgein Section II then we introduce our different dynamic back-door techniques in Section III Section IV evaluates theperformance of our different techniques and the effect oftheir hyperparameters Finally we present the related worksin Section V and conclude the paper in Section VI

2

II PRELIMINARIES

In this section we first introduce the machine learningclassification setting Then we formalize backdoor attacksagainst ML models and finally we discuss the threat modelwe consider throughout the paper

A Machine Learning Classification

A machine learning classification model M is essentially afunction that maps a feature vector x from the feature spaceX to an output vector y from the output space Y ie

M(x) = y

Each entry yi in the vector y corresponds to the posteriorprobability of the input vector x being affiliated with the label`i isin L where L is the set of all possible labels In this workinstead of y we only consider the output of M as the labelwith the highest probability ie

M(x) = argmax`iy

To train M we need a dataset D which consists of pairs oflabels and features vectors ie D = (xi `i)iisinN with Nbeing the size of the dataset and adopt some optimizationalgorithm such as Adam to learn the parameters of Mfollowing a defined loss function

B Backdoor in Machine Learning Models

Backdooring is the general technique of hiding a -usually-malicious functionality in the system that can be only trig-gered with a certain secretbackdoor For instance an adver-sary can implement a backdoor into an authentication systemto access any desired account An example trigger in this usecase can be a secret password that works with all possibleaccounts An important requirement of backdoors is that thesystem should behave normally on all inputs except the oneswith triggers

Intuitively a backdoor in the ML settings resembles ahidden behavior of the model which only happens when it isqueried with an input containing a secret trigger This hiddenbehavior is usually the misclassification of an input featurevector to the desired target label

A backdoored modelMbd is expected to learn the mappingfrom the feature vectors with triggers to their correspondingtarget label ie any input with the trigger ti should have thelabel `i as its output To train such a model an adversaryneeds both clean data Dc (to preserve the modelrsquos utility) andbackdoored data Dbd (to implement the backdoor behaviour)where Dbd is constructed by adding triggers on a subset ofDc

Current backdoor attacks construct backdoors with statictriggers in terms of fixed trigger pattern and location (on theinput) In this work we introduce dynamic backdoors wherethe trigger pattern and location are dynamic In other words adynamic backdoor should have triggers with different values(pattern) and can be placed at different positions on the input(location)

More formally a backdoor in an ML model is associatedwith a set of triggers T set of target labels Lprime and abackdoor adding function A We first define the backdooradding function A as follows

A(x ti κ) = xbd

where x is the input vector ti isin T is the trigger κ is thedesired location to add the backdoor -more practically thelocation of the top left corner pixel of the trigger- and xbd isthe input vector x with the backdoor inserted at the locationκ

Compared to the static backdoor attacks dynamic backdoorattacks introduce new features for the triggers which givethe adversary more flexibility and increase the difficulty ofdetecting such backdoors Namely dynamic backdoors intro-duce different locations and patterns for the backdoor triggersThese multiple patterns and locations for the triggers hardenthe detection of such backdoors since the current design ofdefenses assumes a static behavior of backdoors Moreoverthese triggers can be algorithmically generated ones as willbe shown later in Section III-B and Section III-C which allowsthe adversary to customize the generated triggers

C Threat Model

As previously mentioned backdooring is a training timeattack ie the adversary is the one who trains the ML modelTo achieve this we assume the adversary can access the dataused for training the model and control the training processThen the adversary publishes the backdoored model to thevictim To launch the attack the adversary first adds a triggerto the input and then uses it to query the backdoored modelThis added trigger makes the model misclassify the input tothe target label In practice this can allow an adversary tobypass authentication systems to achieve her goal This threatmodel follows the same one used by previous works suchas [12]

III DYNAMIC BACKDOORS

In this section we propose three different techniques forperforming the dynamic backdoor attack namely RandomBackdoor Backdoor Generating Network (BaN) and condi-tional Backdoor Generating Network (c-BaN)

A Random Backdoor

We start with our simplest approach ie the RandomBackdoor technique Abstractly the Random Backdoor tech-nique constructs triggers by sampling them from a uniformdistribution and adding them to the inputs at random locationsWe first introduce how to use our Random Backdoor techniqueto implement a dynamic backdoor for a single target label thenwe generalize it to consider multiple target labels

Single Target Label We start with the simple case of consid-ering dynamic backdoors for a single target label Intuitivelywe construct the set of triggers (T ) and the set of possiblelocations (K) such that for any trigger sampled from T and

3

Fig 2 An illustration of our location setting technique for 6target labels (for the Random Backdoor and BaN techniquesin the multiple target labels case) The red dotted line demon-strates the boundary of the vertical movement for each targetlabel

added to any input at a random location sampled from K themodel will output the specified target label More formallyfor any location κi isin K any trigger ti isin T and any inputxi isin X

Mbd(A(xi ti κi)) = `

where ` is the target label T is the set of triggers and K isthe set of locations

To implement such a backdoor in a model an adversaryneeds first to select her desired trigger locations and createthe set of possible locations K Then she uses both cleanand backdoored data to update the model for each epochMore concretely the adversary trains the model as mentionedin Section II-B with the following two differences

bull First instead of using a fixed trigger for all inputs eachtime the adversary wants to add a trigger to an inputshe samples a new trigger from a uniform distributionie t sim U(0 1) Here the set of possible triggers Tcontains the full range of all possible values for thetriggers since the trigger is randomly sampled from auniform distribution

bull Second instead of placing the trigger in a fixed locationshe places it at a random location κ sampled from thepredefined set of locations ie κ isin K

Finally this technique is not only limited to uniform dis-tribution but the adversary can use different distributions likethe Gaussian distribution to construct the triggers

Multiple Target Labels Next we consider the more complexcase of having multiple target labels Without loss of gener-ality we consider implementing a backdoor for each label inthe dataset since this is the most challenging setting Howeverour techniques can be applied for any smaller subset of labelsThis means that for any label `i isin L there exists a trigger t

which when added to the input x at a location κ will makethe model Mbd output `i More formally

forall`i isin L exist t κ Mbd(A(x t κ)) = `i

To achieve the dynamic backdoor behaviour in this settingeach target label should have a set of possible triggers and aset of possible locations More formally

forall`i isin L exist TiKi

where Ti is the set of possible triggers and Ki is the set ofpossible locations for the target label `i

We generalize the Random Backdoor technique by dividingthe set of possible locations K into disjoint subsets for eachtarget label while keeping the trigger construction method thesame as in the single target label case ie the triggers arestill sampled from a uniform distribution For instance for thetarget label `i we sample a set of possible locations Ki whereKi is subset of K (Ki sub K)

The adversary can construct the disjoint sets of possiblelocations as follows

1) First the adversary selects all possible triggers locationsand constructs the set K

2) Second for each target label `i she constructs the setof possible locations for this label Ki by sampling theset K Then she removes the sampled locations fromthe set K

We propose the following simple algorithm to assign thelocations for the different target labels However an adver-sary can construct the location sets arbitrarily with the onlyrestriction that no location can be used for more than onetarget label

We uniformly split the image into non-intersecting regionsand assign a region for each target label in which the triggersrsquolocations can move vertically Figure 2 shows an example ofour location setting technique for a use case with 6 targetlabels As the figure shows each target label has its ownregion for example label 1 occupies the top left region of theimage We stress that this is one way of dividing the locationset K to the different target labels However an adversary canchoose a different way of splitting the locations inside K tothe different target labels The only requirement the adversaryhas to fulfill is to avoid assigning a location for different targetlabels Later we will show how to overcome this limitationwith our more advanced c-BaN technique

B Backdoor Generating Network (BaN)

Next we introduce our second technique to implement dy-namic backdoors namely the Backdoor Generating Network(BaN) BaN is the first approach to algorithmically generatebackdoor triggers instead of using fixed triggers or samplingtriggers from a uniform distribution (as in Section III-A)

BaN is inspired by the state-of-the-art generative model ndashGenerative Adversarial Networks (GANs) [11] However itis different from the original GANs in the following aspectsFirst instead of generating images our BaN generator gen-erates backdoor triggers Second we jointly train the BaN

4

BaN

Uniform Distribution

120373i

120013

120013bd 9

(a) BaN

c-BaN

Uniform Distribution

120373i

120013

120013bd 9

[000000001](9)

(b) c-BaN

Fig 3 An overview of the BaN and c-BaN techniques The main difference between both techniques is the additional input(the label) in the c-BaN For the BaN on the input of a random vector z it outputs the trigger ti This trigger is then addedto the input image using the backdoor adding function A Finally the backdoored image is inputted to the backdoored modelMbd which outputs the target label 9 For the c-BaN first the target label (9) together with a random vector z are input tothe c-BaN which outputs the trigger ti The following steps are exactly the same as for the BaN

generator with the target model instead of the discriminatorto learn (the generator) and implement (the target model) thebest patterns for the backdoor triggers

After training the BaN can generate a trigger (t) for eachnoise vector (z sim U(0 1)) This trigger is then added toan input using the backdoor adding function A to createthe backdoored input as shown in Figure 3a Similar to theprevious approach (Random Backdoor) the generated triggersare placed at random locations

In this section we first introduce the BaN technique for asingle target label then we generalize it for multiple targetlabels

Single Target Label We start with presenting how to imple-ment a dynamic backdoor for a single target label using ourBaN technique First the adversary creates the set K of thepossible locations She then jointly trains the BaN with thebackdoored Mbd model as follows

1) The adversary starts each training epoch by queryingthe clean data to the backdoored modelMbd Then shecalculates the clean loss ϕc between the ground truthand the output labels We use the cross-entropy loss forour clean loss which is defined as followssum

i

yi log(yi)

where yi is the true probability of label `i and yi is ourpredicted probability of label `i

2) She then generates n noise vectors where n is the batchsize

3) On the input of the n noise vectors the BaN generatesn triggers

4) The adversary then creates the backdoored data byadding the generated triggers to the clean data usingthe backdoor adding function A

5) She then queries the backdoored data to the backdooredmodelMbd and calculates the backdoor loss ϕbd on themodelrsquos output and the target label Similar to the cleanloss we use the cross-entropy loss as our loss functionfor ϕbd

6) Finally the adversary updates the backdoor modelMbd

using both the clean and backdoor losses (ϕc+ϕbd) andupdates the BaN with the backdoor loss (ϕbd)

One of the main advantages of the BaN technique is itsflexibility Meaning that it allows the adversary to customizeher triggers by plugging any customized loss to it In otherwords BaN is a framework for a more generalized class ofbackdoors that allows the adversary to customize the desiredtrigger by adapting the loss function

Multiple Target Labels We now consider the more complexcase of building a dynamic backdoor for multiple target labelsusing our BaN technique To recap our BaN generates generaltriggers and not label specific triggers In other words thesame trigger pattern can be used to trigger multiple targetlabels Thus similar to the Random Backdoor we depend onthe location of the triggers to determine the output label

We follow the same approach of the Random Backdoortechnique to assign different locations for different targetlabels (Section III-A) to generalize the BaN technique Moreconcretely the adversary implements the dynamic backdoorfor multiple target labels using the BaN technique as follows

1) The adversary starts by creating disjoint sets of locationsfor all target labels

2) Next she follows the same steps as in training thebackdoor for a single target label while repeating fromstep 2 to 5 for each target label and adding all theirbackdoor losses together More formally for the multipletarget label case the backdoor loss is defined as

|Lprime|sumi

ϕbdi

where Lprime is the set of target labels and ϕbdi is thebackdoor loss for target label `i

C conditional Backdoor Generating Network (c-BaN)

So far we have proposed two techniques to implement dy-namic backdoors for both single and multiple target labels ie

5

Fig 4 An illustration of the structure of the c-BaN The targetlabel `i and noise vector z are first input to separate layersThen the outputs of these two layers are concatenated andapplied to multiple fully connected layers to generate the targetspecific trigger ti

Random Backdoor (Section III-A) and BaN (Section III-B)To recap both techniques have the limitation of not havinglabel specific triggers and only depending on the triggerlocation to determine the target label We now introduce ourthird and most advanced technique the conditional BackdoorGenerating Network (c-BaN) which overcomes this limitationMore concretely with the c-BaN technique any location κinside the location set K can be used to trigger any target labelTo achieve this location independency the triggers need to belabel specific Therefore we convert the Backdoor GeneratingNetwork (BaN) into a conditional Backdoor Generating Net-work (c-BaN) More specifically we add the target label asan additional input to the BaN for conditioning it to generatetarget specific triggers

We construct the c-BaN by adding an additional input layerto the BaN to include the target label as an input Figure 4represents an illustration for the structure of c-BaN As thefigure shows the two input layers take the noise vector andthe target label and encode them to latent vectors with thesame size (to give equal weights for both inputs) These twolatent vectors are then concatenated and used as an input tothe next layer It is important to mention that we use one-hotencoding to encode the target label before applying it to thec-BaN

The c-BaN is trained similarly to the BaN with the follow-ing two exceptions

1) First the adversary does not have to create disjoint setsof locations for all target labels (step 1) she can use thecomplete location set K for all target labels

2) Second instead of using only the noise vectors as an

input to the BaN the adversary one-hot encodes thetarget label then use it together with the noise vectorsas the input to the c-BaN

To use the c-BaN the adversary first samples a noise vectorand one-hot encodes the label Then she inputs both of themto the c-BaN which generates a trigger The adversary usesthe backdoor adding function A to add the trigger to thetarget input Finally she queries the backdoored input to thebackdoored model which will output the target label Wevisualize the complete pipeline of using the c-BaN techniquein Figure 3b

In this section we have introduced three techniques forimplementing dynamic backdoors namely the Random Back-door the Backdoor Generating Network (BaN) and the con-ditional Backdoor Generating Network (c-BaN) These threedynamic backdoor techniques present a framework to generatedynamic backdoors for different settings For instance ourframework can generate target specific triggersrsquo pattern usingthe c-BaN or target specific triggersrsquo location like the RandomBackdoor and BaN More interestingly our framework allowsthe adversary to customize her backdoor by adapting thebackdoor loss functions For instance the adversary can adaptto different defenses against the backdoor attack that can bemodeled as a machine learning model This can be achieved byadding any defense as a discriminator into the training of theBaN or c-BaN Adding this discriminator will penalizeguidethe backdoored model to bypass the modeled defense

IV EVALUATION

In this section we first introduce our datasets and experi-mental settings Next we evaluate all of our three techniquesie Random Backdoor Backdoor Generating Network (BaN)and conditional Backdoor Generating Network (c-BaN) Wethen evaluate our three dynamic backdoor techniques againstthe current state-of-the-art techniques Finally we study theeffect of different hyperparameters on our techniques

A Datasets Description

We utilize three image datasets to evaluate our tech-niques including MNIST CelebA and CIFAR-10 These threedatasets are widely used as benchmark datasets for varioussecurityprivacy and computer vision tasks We briefly describeeach of them below

MNIST The MNIST dataset [2] is a 10-class dataset consist-ing of 70 000 grey-scale 28times28 images Each of these imagescontains a handwritten digit in its center The MNIST datasetis a balanced dataset ie each class is represented with 7 000images

CIFAR-10 The CIFAR-10 dataset [3] is composed of 60 00032 times 32 colored images which are equally distributed on thefollowing 10 classes Airplane automobile bird cat deerdog frog horse ship and truck

6

CelebA The CelebA dataset [23] is a large-scale face at-tributes dataset with more than 200K colored celebrity im-ages each annotated with 40 binary attributes We select thetop three most balanced attributes including Heavy MakeupMouth Slightly Open and Smiling Then we concatenate theminto 8 classes to create a multiple label classification taskFor our experiments we scale the images to 64 times 64 andrandomly sample 10 000 images for training and another10 000 for testing Finally it is important to mention thatunlike the MNIST and CIFAR-10 datasets this dataset ishighly imbalanced

B Experimental Setup

First we introduce the different modelsrsquo architecture forour target models BaN and c-BaN Then we introduce ourevaluation metrics

Models Architecture For the target modelsrsquo architecture weuse the VGG-19 [40] for the CIFAR-10 dataset and build ourown convolution neural networks (CNN) for the CelebA andMNIST datasets More concretely we use 3 convolution layersand 5 fully connected layers for the CelebA CNN And 2convolution layers and 2 fully connected layers for the MNISTCNN Moreover we use dropout for both the CelebA andMNIST models to avoid overfitting

For BaN we use the following architectureBackdoor Generating Network (BaN)rsquos architecture

z rarr FullyConnected(64)

FullyConnected(128)

FullyConnected(128)

FullyConnected(|t|)

Sigmoidrarr t

Here FullyConnected(x) denotes a fully connected layerwith x hidden units |t| denotes the size of the required triggerand Sigmoid is the Sigmoid function We adopt ReLU as theactivation function for all layers and apply dropout after alllayers except the first and last ones

For c-BaN we use the following architectureconditional Backdoor Generating Network (c-BaN)rsquos archi-tecture

z `rarr 2times FullyConnected(64)

FullyConnected(128)

FullyConnected(128)

FullyConnected(128)

FullyConnected(|t|)

Sigmoidrarr t

The first layer consists of two separate fully connected layerswhere each one of them takes an independent input ie thefirst takes the noise vector z and the second takes the targetlabel ` The outputs of these two layers are then concatenatedand used as an input to the next layer (see Section III-C)

CIFAR-10 CelebA MNIST60

65

70

75

80

85

90

95

100

Acc

urac

y

Clean ModelBaNRandom Backdoor

Fig 5 [Higher is better] The result of our dynamic backdoortechniques for a single target label We only show the accuracyof the models on the clean testing datasets as the backdoorsuccess rate is approximately always 100

Similar to BaN we adopt ReLU as the activation function forall layers and apply dropout after all layers except the firstand last one

All of our experiments are implemented using Pytorch [4]and our code will be published for reproducibility purposes

Evaluation Metrics We define the following two metricsto evaluate the performance of our backdoored models Thefirst one is the backdoor success rate which is measured bycalculating the backdoored modelrsquos accuracy on backdooreddata The second one is model utility which is used tomeasure the original functionality of the backdoored modelWe quantify the model utility by comparing the accuracy ofthe backdoored model with the accuracy of a clean model onclean data Closer accuracies implies a better model utility

C Random Backdoor

We now evaluate the performance of our first dynamicbackdooring technique namely the Random Backdoor Weuse all three datasets for the evaluation First we evaluate thesingle target label case where we only implement a backdoorfor a single target label in the backdoored model Mbd Thenwe evaluate the more generalized case ie the multiple targetlabels case where we implement a backdoor for all possiblelabels in the dataset

For both the single and multiple target label cases we spliteach dataset into training and testing datasets The trainingdataset is used to train the MNIST and CelebA models fromscratch For CIFAR-10 we use a pre-trained VGG-19 modelWe refer to the testing dataset as the clean testing dataset andwe first use it to construct a backdoored testing dataset byadding triggers to all of its images To recap for the RandomBackdoor technique we construct the triggers by samplingthem from uniform distribution and add them to the imagesusing the backdoor adding function A We use the backdooredtesting dataset to calculate the backdoor success rate and thetraining dataset to train a clean model -for each dataset- toevaluate the backdoored modelrsquos (Mbd) utility

7

(a) Random Backdoor

(b) BaN

(c) BaN with higher randomness

Fig 6 The result of our Random Backdoor (Figure 6a) BaN(Figure 6b) and BaN with higher randomness (Figure 6c)techniques for a single target label (0)

We follow Section III-A to train our backdoored modelMbd

for both the single and multiple target labels cases Abstractlyfor each epoch we update the backdoored model Mbd usingboth the clean and backdoor losses ϕc + ϕbd For the set ofpossible locations K we use four possible locations

The backdoor success rate is always 100 for both thesingle and multiple target labels cases on all three datasetshence we only focus on the backdoored modelrsquos (Mbd) utility

Single Target Label We first present our results for the singletarget label case Figure 5 compares the accuracies of thebackdoored modelMbd and the clean modelM -on the cleantesting dataset- As the figure shows our backdoored modelsachieve the same performance as the clean models for boththe MNIST and CelebA datasets ie 99 for MNIST and70 for CelebA For the CIFAR-10 dataset there is a slightdrop in performance which is less than 2 This shows thatour Random Backdoor technique can implement a perfectlyfunctioning backdoor ie the backdoor success rate of Mbd

is 100 on the backdoored testing dataset with a negligibleutility loss

To visualize the output of our Random Backdoor techniquewe first randomly sample 8 images from the MNIST datasetand then use the Random Backdoor technique to constructtriggers for them Finally we add these triggers to the imagesusing the backdoor adding function A and show the resultin Figure 6a As the figure shows the triggers all lookdistinctly different and are located at different locations asexpected

Multiple Target Labels Second we present our resultsfor the multiple target label case To recap we consider allpossible labels for this case For instance for the MNISTdataset we consider all digits from 0 to 9 as our target labelsWe train our Random Backdoor models for the multiple targetlabels as mentioned in Section III-A

We use a similar evaluation setting to the single targetlabel case with the following exception To evaluate the

CIFAR-10 CelebA MNIST60

65

70

75

80

85

90

95

100

Acc

urac

y

Clean Modelc-BaNBaNRandom Backdoor

Fig 7 [Higher is better] The result of our dynamic backdoortechniques for multiple target label Similar to the singletarget label case we only show the accuracy of the modelson the clean testing dataset as the backdoor success rate isapproximately always 100

performance of the backdoored model Mbd with multipletarget labels we construct a backdoored testing dataset foreach target label by generating and adding triggers to the cleantesting dataset In other words we use all images in the testingdataset to evaluate all possible labels

Similar to the single target label case we focus on theaccuracy on the clean testing dataset since the backdoorsuccess rate for all models on the backdoored testing datasetsare approximately 100 for all target labels

We use the clean testing datasets to evaluate the backdooredmodelrsquos Mbd utility ie we compare the performance of thebackdoored modelMbd with the clean modelM in Figure 7As the figure shows using our Random Backdoor techniquewe are able to train backdoored models that achieve similarperformance as the clean models for all datasets For instancefor the CIFAR-10 dataset our Random Backdoor techniqueachieves 92 accuracy which is very similar to the accuracyof the clean model (924) For the CelebA dataset theRandom Backdoor technique achieves a slightly (about 2)better performance than the clean model We believe this is dueto the regularization effect of the Random Backdoor techniqueFinally for the MNIST dataset both models achieve a similarperformance with just 1 difference between the clean model(99) and the backdoored one (98)

To visualize the output of our Random Backdoor techniqueon multiple target labels we construct triggers for all possiblelabels in the CIFAR-10 dataset and use A to add them toa randomly sampled image from the CIFAR-10 clean testingdataset Figure 8a shows the image with different triggers Thedifferent patterns and locations used for the different targetlabels can be clearly demonstrated in Figure 8a For instancecomparing the location of the trigger for the first and sixthimages the triggers are in the same horizontal position but adifferent vertical position as previously illustrated in Figure 2

Moreover we further visualize in Figure 9a the dynamicbehavior of the triggers generated by our Random Backdoortechnique Without loss of generality we generate triggers for

8

the target label 5 (plane) and add them to randomly sampledCIFAR-10 images To make it clear we train the backdoormodel Mbd for all possible labels set as target labels but wevisualize the triggers for a single label to show the dynamicbehaviour of our Random Backdoor technique with respectto the triggersrsquo pattern and locations As Figure 9a showsthe generated triggers have different patterns and locations forthe same target label which achieves our desired dynamicbehavior

D Backdoor Generating Network (BaN)

Next we evaluate our BaN technique We follow the sameevaluation settings for the Random Backdoor technique exceptwith respect to how the triggers are generated We train ourBaN model and generate the triggers as mentioned in Sec-tion III-B

Single Target Label Similar to the Random Backdoor theBaN technique achieves perfect backdoor success rate with anegligible utility loss Figure 5 compares the performance ofthe backdoored models trained using the BaN technique withthe clean models on the clean testing dataset As Figure 5shows our BaN trained backdoored models achieve 99924 and 70 accuracy on the MNIST CIFAR-10 andCelebA datasets respectively which is the same performanceof the clean models

We visualize the BaN generated triggers using the MNISTdataset in Figure 6b To construct the figure we use the BaNto generate multiple triggers -for the target label 0- then weadd them on a set of randomly sampled MNIST images usingthe backdoor adding function A

The generated triggers look very similar as shown in Fig-ure 6b This behaviour is expected as the MNIST dataset issimple and the BaN technique does not have any explicitloss to enforce the network to generate different triggersHowever to show the flexibility of our approach we increasethe randomness of the BaN network by simply adding onemore dropout layer after the last layer to avoid the overfittingof the BaN model to a unique pattern We show the resultsof the BaN model with higher randomness in Figure 6c Theresulting model still achieves the same performance ie 99accuracy on the clean data and 100 backdoor success ratebut as the figure shows the triggers look significantly differentThis again shows that our framework can easily adapt to therequirements of an adversary

These results together with the results of the RandomBackdoor (Section IV-C) clearly show the effectiveness of bothof our proposed techniques for the single target label caseThey are both able to achieve almost the same accuracy ofa clean model with a 100 working backdoor for a singletarget label

Multiple Target Labels Similar to the single target labelcase we focus on the backdoored modelsrsquo performance on thetesting clean dataset as our BaN backdoored models achievea perfect accuracy on the backdoored testing dataset ie the

backdoor success rate for all datasets is approximately 100for all target labels

We compare the performance of the BaN backdoored mod-els with the performance of the clean models on the cleantesting dataset in Figure 7 Our BaN backdoored models areable to achieve almost the same accuracy as the clean modelfor all datasets as can be shown in Figure 7 For instancefor the CIFAR-10 dataset our BaN achieves 921 accuracywhich is only 03 less than the performance of the cleanmodel (924) Similar to the Random Backdoor backdooredmodels our BaN backdoored models achieve a marginallybetter performance for the CelebA dataset More concretelyour BaN backdoored models trained for the CelebA datasetachieve about 2 better performance than the clean model onthe clean testing dataset We also believe this improvement isdue to the regularization effect of the BaN technique Finallyfor the MNIST dataset our BaN backdoored models achievestrong performance on the clean testing dataset (98) whichis just 1 lower than the performance of the clean models(99)

Similar to the Random Backdoor we visualize the resultsof the BaN backdoored models with two figures The first(Figure 8b) shows the different triggers for the differenttarget labels on the same CIFAR-10 image and the second(Figure 9b) shows the different triggers for the same targetlabel (plane) on randomly sampled CIFAR-10 images As bothfigures show the BaN generated triggers achieves the dynamicbehaviour in both the location and patterns For instance forthe same target label (Figure 9b) the patterns of the triggerslook significantly different and the locations vary verticallySimilarly for different target labels (Figure 8b) both thepattern and location of triggers are significantly different

E conditional Backdoor Generating Network (c-BaN)

Next we evaluate our conditional Backdoor GeneratingNetwork (c-BaN) technique For the c-BaN technique we onlyconsider the multiple target labels case since there is only asingle label so the conditional addition to the BaN techniqueis not needed In other words for the single target label casethe c-BaN technique will be the same as the BaN technique

We follow a similar setup as introduced for the BaNtechnique in Section IV-D with the exception on how totrain the backdoored model Mbd and generate the triggersWe follow Section III-C to train the backdoored model andgenerate the triggers For the set of possible locations K weuse four possible locations

We compare the performance of the c-BaN with the othertwo techniques in addition to the clean model All of our threedynamic backdoor techniques achieve an almost perfect back-door success rate on the backdoored testing datasets hencesimilar to the previous sections we focus on the performanceon the clean testing datasets

Figure 7 compares the accuracy of the backdoored andclean models using the clean testing dataset for all of ourthree dynamic backdoor techniques As the figure shows allof our dynamic backdoored models have similar performance

9

(a) Random Backdoor

(b) BaN

(c) c-BaN

Fig 8 The visualization result of our Random Backdoor (Figure 8a) BaN (Figure 8b) and c-BaN (Figure 8c) techniques forall labels of the CIFAR-10 dataset

as the clean models For instance for the CIFAR-10 datasetour c-BaN BaN and Random Backdoor achieves 92 921and 92 accuracy respectively which is very similar to theaccuracy of the clean model (924) Also for the MNISTdataset all models achieve very similar performance with nodifference between the clean and c-BaN models (99) and 1difference between the BaN and Random Backdoor (98) andthe clean model

Similar to the previous two techniques we visualize thedynamic behaviour of the c-BaN backdoored models firstby generating triggers for all possible labels and addingthem on a CIFAR-10 image in Figure 8c More generallyFigure 8 shows the visualization of all three dynamic backdoortechniques in the same settings ie backdooring a singleimage to all possible labels As the figure shows the RandomBackdoor Figure 8a has the most random patterns which isexpected as they are sampled from a uniform distribution Thefigure also shows the different triggersrsquo patterns and locationsused for the different techniques For instance each target labelin the Random Backdoor (Figure 8a) and BaN (Figure 8b)techniques have a unique (horizontal) location unlike the c-BaN (Figure 8c) generated triggers which different targetlabels can share the same locations as can be shown forexample in the first second and ninth images To recap boththe Random Backdoor and BaN techniques split the locationset K on all target labels such that no two labels share alocation unlike the c-BaN technique which does not have thislimitation

Second we visualize the dynamic behaviour of our tech-niques by generating triggers for the same target label 5(plane) and adding them to a set of randomly sampled CIFAR-10 images Figure 9 compares the visualization of our threedifferent dynamic backdoor techniques in this setting To makeit clear we train the backdoor model Mbd for all possible

labels set as target labels but we plot for a single labelto visualize how different the triggers look like for eachtarget label As the figure shows the Random Backdoor (Fig-ure 9a) and BaN (Figure 9b) generated triggers can movevertically however they have a fixed position horizontallyas mentioned in Section III-A and illustrated in Figure 2The c-BaN (Figure 9c) triggers also show different locationsHowever the locations of these triggers are more distant andcan be shared for different target labels unlike the other twotechniques Finally the figure also shows that all triggers havedifferent patterns for our techniques for the same target labelwhich achieves our targeted dynamic behavior concerning thepatterns and locations of the triggers

F Evaluating Against Current State-Of-The-Art Defenses

We now evaluate our attacks against the current state-of-the-art backdoor defenses Backdoor defenses can be classifiedinto the following two categories data-based defenses andmodel-based defenses On one hand data-based defenses focuson identifying if a given input is clean or contains a triggerOn the other hand model-based defenses focus on identifyingif a given model is clean or backdoored

We first evaluate our attacks against model-based defensesthen we evaluate them against data-based ones

Model-based Defense We evaluate all of our dynamic back-door techniques in the multiple target label case against twoof the current state-of-the-art model-based defenses namelyNeural Cleanse [47] and ABS [21]

We start by evaluating the ABS defense We use the CIFAR-10 dataset to evaluate this defense since it is the only sup-ported dataset by the published defense model As expectedrunning the ABS model against our dynamic backdoored onesdoes not result in detecting any backdoor for all of our models

10

(a) Random Backdoor

(b) BaN

(c) c-BaN

Fig 9 The result of our Random Backdoor (Figure 9a) BaN(Figure 9b) and c-BaN (Figure 9c) techniques for the targettarget label 5 (plane)

For Neural Cleanse we use all three datasets to evaluateour techniques against it Similar to ABS all of our modelsare predicted to be clean models Moreover in multiple casesour models had a lower anomaly index (the lower the better)than the clean model

We believe that both of these defenses fail to detect ourbackdoors for two reasons First we break one of their mainassumption ie that the triggers are static in terms of locationand pattern Second we implement a backdoor for all possiblelabels which makes the detection a more challenging task

Data-based Defense Next we evaluate the current state-of-the-art data-based defense namely STRIP [10] STRIP triesto identify if a given input is clean or contains a trigger Itworks by creating multiple images from the input image byfusing it with multiple clean images one at a time Then STRIPapplies all fused images to the target model and calculates theentropy of predicted labels Backdoored inputs tend to havelower entropy compared to the clean ones

We use all of our three datasets to evaluate the c-BaNmodels against this defense First we scale the patterns byhalf while training the backdoored models to make themmore susceptible to changes Second for the MNIST datasetwe move the possible locations to the middle of the imageto overlap with the image content since the value of theMNIST images at the corners are always 0 All trained scaledbackdoored models achieve similar performance to the non-scaled backdoored models

Our backdoored models successfully flatten the distributionof entropy for the backdoored data for a subset of targetlabels In other words the distribution of entropy for ourbackdoored data overlaps with the distributions of entropy ofthe clean data This subset of target labels makes picking a

02505

007

510

012

515

017

520

022

500

05

10

15

20

25CleanBD

(a) CIFAR-10

00 05 10 15 2000

05

10

15

20 CleanBD

(b) MNIST02

505

007

510

012

515

017

520

000

05

10

15

20 CleanBD

(c) CelebA

Fig 10 The histogram of the entropy of the backdoored vsclean input for our best performing labels against the STRIPdefense for the CIFAR-10 (Figure 10a) MNIST (Figure 10b)and CelebA (Figure 10c) datasets

threshold to identify backdoored from clean data impossiblewithout increasing the false positive rate ie various cleanimages will be detected as backdoored ones We visualizethe entropy of our best performing labels against the STRIPdefense in Figure 10

Moreover since our dynamic backdoors can generate dy-namic triggers for the same input and target label The adver-sary can keep querying the target model while backdooring theinput with a fresh generated trigger until the model accepts it

These results against the data and model-based defensesshow the effectiveness of our dynamic backdoor attacks andopens the door for designing backdoor detection systems thatwork against both static and dynamic backdoors which weplan for future work

G Evaluating Different HyperparametersWe now evaluate the effect of different hyperparameters for

our dynamic backdooring techniques We start by evaluatingthe percentage of the backdoored data needed to implementa dynamic backdoor into the model Then we evaluate theeffect of increasing the size of the location set K Finally weevaluate the size of the trigger and the possibility of making itmore transparent ie instead of replacing the original valuesin the input with the backdoor we fuse them

Proportion of the Backdoored Data We start by evaluatingthe percentage of backdoored data needed to implement adynamic backdoor in the model We use the MNIST datasetand the c-BaN technique to perform the evaluation First weconstruct different training datasets with different percentagesof backdoored data More concretely we try all proportionsfrom 10 to 50 with a step of 10 10 means that 10of the data is backdoored and 90 is clean Our results showthat using 30 is already enough to get a perfectly workingdynamic backdoor ie the model has a similar performancelike a clean model on the clean dataset (99 accuracy) and100 backdoor success rate on the backdoored dataset Forany percentage below 30 the accuracy of the model onclean data is still the same however the performance on thebackdoored dataset starts degrading

Number of Locations Second we explore the effect ofincreasing the size of the set of possible locations (K) for

11

Fig 11 An illustration of the effect of using different trans-parency scales (from 0 to 1 with step of 025) when adding thetrigger Scale 0 (the most left image) shows the original inputand scale 1 (the most right image) the original backdooredinput without any transparency

the c-BaN technique We use the CIFAR-10 dataset to traina backdoored model using the c-BaN technique but withmore than double the size of K ie 8 locations The trainedmodel achieves similar performance on the clean (92) andbackdoored (100) datasets We then doubled the size again tohave 16 possible locations in K and the model again achievesthe same results on both clean and backdoored datasets Werepeat the experiment with the CelebA datasets and achievesimilar results ie the performance of the model with a largerset of possible locations is similar to the previously reportedone However when we try to completely remove the locationset K and consider all possible locations with a sliding win-dow the performance on both clean and backdoored datasetssignificantly dropped

Trigger Size Next we evaluate the effect of the trigger sizeon our c-BaN technique using the MNIST dataset We traindifferent models with the c-BaN technique while setting thetrigger size from 1 to 6 We define the trigger size to be thewidth and height of the trigger For instance a trigger size of3 means that the trigger is 3times 3 pixels

We calculate the accuracy on the clean and backdooredtesting datasets for each trigger size and show our resultsin Figure 12 Our results show that the smaller the trigger theharder it is for the model to implement the backdoor behaviourMoreover small triggers confuse the model which results inreducing the modelrsquos utility As Figure 12 shows a triggerwith the size 5 achieves a perfect accuracy (100) on thebackdoored testing dataset while preserving the accuracy onthe clean testing dataset (99)

Transparency of the Triggers Finally we evaluate the effectof making the trigger more transparent More specifically wechange the backdoor adding function A to apply a weightedsum instead of replacing the original inputrsquos values Ab-stractly we define the weighted sum of the trigger and theimage as

xbd = s middot t+ (1minus s) middot x

where s is the scale controlling the transparency rate x isthe input and t is the trigger We implement this weightedsum only at the location of the trigger while maintaining theremaining of the input unchanged

We use the MNIST dataset and c-BaN technique to evaluatethe scale from 0 to 1 with a step of 025 Figure 11 visualizes

1 2 3 4 5 6Trigger Size

20

40

60

80

100

Acc

urac

y

Clean DataBackdoored Data

Fig 12 [Higher is better] The result of trying different triggersizes for the c-BaN technique on the MNIST dataset Thefigure shows for each trigger size the accuracy on the cleanand backdoored testing datasets

the effect of varying the scale when adding a trigger to aninput

Our results show that our technique can achieve the sameperformance on both the clean (99) and backdoored (100)testing datasets when setting the scale to 05 or higherHowever when the scale is set below 05 the performancestarts degrading on the backdoored dataset but stays the sameon the clean dataset We repeat the same experiments for theCelebA dataset and find similar results

V RELATED WORKS

In this section we discuss some of the related work We startwith current state-of-the-art backdoor attacks Then we discussthe defenses against backdoor attacks and finally mentionother attacks against machine learning models

Backdoor Attacks Gu et al [12] introduce BadNets the firstbackdoor attack on machine learning models BadNets uses theMNIST dataset and a square-like trigger with a fixed locationto show the applicability of the backdoor attacks in themachine learning settings Liu et al [22] later propose a moreadvanced backdooring technique namely the Trojan attackThey simplify the threat model of BadNets by eliminating theneed for Trojan attack to access the training data The Trojanattack reverse-engineers the target model to synthesize trainingdata Next it generates the trigger in a way that maximizesthe activation functions of the target modelrsquos internal neuronsrelated to the target label In other words the Trojan attackreverse-engineers a trigger and training data to retrainupdatethe model and implement the backdoor

The main difference between these two attacks (BadNetsand Trojan attacks) and our work is that both attacks onlyconsider static backdoors in terms of triggersrsquo pattern andlocation Our work extends the backdoor attacks to considerdynamic patterns and locations of the triggers

Defenses Against Backdoor Attacks Defenses against back-door attacks can be classified into model-based defenses anddata-based defenses

12

First model-based defenses try to find if a given model con-tains a backdoor or not For instance Wang et al [47] proposeNeural Cleanse (NC) a backdoor defense method based onreverse engineering For each output label NC tries to generatethe smallest trigger which converts the output of all inputsapplied with this trigger to that label NC then uses anomalydetection to find if any of the generated triggers are actually abackdoor or not Later Liu et al [21] propose another model-based defense namely ABS ABS detects if a target modelcontains a backdoor or not by analyzing the behaviour of thetarget modelrsquos inner neurons when introducing different levelsof stimulation

Second data-based defenses try to find if a given input isclean or backdoored For instance Gao et al [10] proposeSTRIP a backdoor defense method based on manipulating theinput to find out if it is backdoored or not More concretelySTRIP fuses the input with multiple clean data one at a timeThen it queries the target model with the generated inputs andcalculate the entropy of the output labels Backdoored inputstend to have lower entropy than the clean ones

Attacks Against Machine Learning Poisoning attack [17][42] [5] is another training time attack in which the adversarymanipulates the training data to compromise the target modelFor instance the adversary can change the ground truth for asubset of the training data to manipulate the decision boundaryor more generally influence the modelrsquos behavior Shafahi etal [38] further introduce the clean label poisoning attackInstead of changing labels the clean label poisoning attackallows the adversary to modify the training data itself tomanipulate the behaviour of the target model

Another class of ML attacks is the adversarial examplesAdversarial examples share some similarities with the back-door attacks In this setting the adversary aims to trick atarget classifier into miss classifying a data point by addingcontrolled noise to it Multiple works have explored the privacyand security risks of adversarial examples [32] [45] [6] [20][43] [33] [48] Other works explore the adversarial exam-plersquos potentials in preserving the userrsquos privacy in multipledomains [30] [18] [51] [19] The main difference betweenadversarial examples and backdoor attacks is that backdoorattacks are done in training time while adversarial examplesare done after the model is trained and without changing anyof the modelrsquos parameters

Beside the above there are multiple other types of at-tacks against machine learning models such as membershipinference [39] [16] [13] [34] [35] [24] [14] [25] [50][27] [41] [37] [28] model stealing [44] [31] [46] modelinversion [8] [7] [15] propoerty inference [9] [26] anddataset reconstruction [36]

VI CONCLUSION

The tremendous progress of machine learning has lead toits adoption in multiple critical real-world applications suchas authentication and autonomous driving systems Howeverit has been shown that ML models are vulnerable to various

types of security and privacy attacks In this paper we focus onbackdoor attack where an adversary manipulates the trainingof the model to intentionally misclassify any input with anadded trigger

Current backdoor attacks only consider static triggers interms of patterns and locations In this work we propose thefirst set of dynamic backdoor attack where the trigger canhave multiple patterns and locations To this end we proposethree different techniques

Our first technique Random Backdoor samples triggers froma uniform distribution and place them at a random location ofan input For the second technique ie Backdoor GeneratingNetwork (BaN) we propose a novel generative network toconstruct triggers Finally we introduce conditional BackdoorGenerating Network (c-BaN) to generate label specific trig-gers

We evaluate our techniques using three benchmark datasetsEvaluation shows that all our techniques can achieve almosta perfect backdoor success rate while preserving the modelrsquosutility Moreover we show that our techniques successfullybypass state-of-the-art defense mechanisms against backdoorattacks

REFERENCES

[1] httpswwwapplecomiphoneface-id 1[2] httpyannlecuncomexdbmnist 2 6[3] httpswwwcstorontoedusimkrizcifarhtml 2 6[4] httpspytorchorg 7[5] B Biggio B Nelson and P Laskov ldquoPoisoning Attacks against Support

Vector Machinesrdquo in International Conference on Machine Learning(ICML) JMLR 2012 1 13

[6] N Carlini and D Wagner ldquoTowards Evaluating the Robustness of NeuralNetworksrdquo in IEEE Symposium on Security and Privacy (SampP) IEEE2017 pp 39ndash57 13

[7] M Fredrikson S Jha and T Ristenpart ldquoModel Inversion Attacks thatExploit Confidence Information and Basic Countermeasuresrdquo in ACMSIGSAC Conference on Computer and Communications Security (CCS)ACM 2015 pp 1322ndash1333 13

[8] M Fredrikson E Lantz S Jha S Lin D Page and T Ristenpart ldquoPri-vacy in Pharmacogenetics An End-to-End Case Study of PersonalizedWarfarin Dosingrdquo in USENIX Security Symposium (USENIX Security)USENIX 2014 pp 17ndash32 13

[9] K Ganju Q Wang W Yang C A Gunter and N Borisov ldquoPropertyInference Attacks on Fully Connected Neural Networks using Per-mutation Invariant Representationsrdquo in ACM SIGSAC Conference onComputer and Communications Security (CCS) ACM 2018 pp 619ndash633 13

[10] Y Gao C Xu D Wang S Chen D C Ranasinghe and S NepalldquoSTRIP A Defence Against Trojan Attacks on Deep Neural Networksrdquoin Annual Computer Security Applications Conference (ACSAC) ACM2019 pp 113ndash125 2 11 13

[11] I Goodfellow J Pouget-Abadie M Mirza B Xu D Warde-FarleyS Ozair A Courville and Y Bengio ldquoGenerative Adversarial Netsrdquo inAnnual Conference on Neural Information Processing Systems (NIPS)NIPS 2014 4

[12] T Gu B Dolan-Gavitt and S Grag ldquoBadnets Identifying Vul-nerabilities in the Machine Learning Model Supply Chainrdquo CoRRabs170806733 2017 1 3 12

[13] I Hagestedt Y Zhang M Humbert P Berrang H Tang X Wang andM Backes ldquoMBeacon Privacy-Preserving Beacons for DNA Methy-lation Datardquo in Network and Distributed System Security Symposium(NDSS) Internet Society 2019 13

[14] J Hayes L Melis G Danezis and E D Cristofaro ldquoLOGANEvaluating Privacy Leakage of Generative Models Using GenerativeAdversarial Networksrdquo Symposium on Privacy Enhancing TechnologiesSymposium 2019 13

13

[15] B Hitaj G Ateniese and F Perez-Cruz ldquoDeep Models Under theGAN Information Leakage from Collaborative Deep Learningrdquo in ACMSIGSAC Conference on Computer and Communications Security (CCS)ACM 2017 pp 603ndash618 13

[16] N Homer S Szelinger M Redman D Duggan W TembeJ Muehling J V Pearson D A Stephan S F Nelson and D W CraigldquoResolving Individuals Contributing Trace Amounts of DNA to HighlyComplex Mixtures Using High-Density SNP Genotyping MicroarraysrdquoPLOS Genetics 2008 13

[17] M Jagielski A Oprea B Biggio C Liu C Nita-Rotaru and B LildquoManipulating Machine Learning Poisoning Attacks and Countermea-sures for Regression Learningrdquo in IEEE Symposium on Security andPrivacy (SampP) IEEE 2018 1 13

[18] J Jia and N Z Gong ldquoAttriGuard A Practical Defense Against At-tribute Inference Attacks via Adversarial Machine Learningrdquo in USENIXSecurity Symposium (USENIX Security) USENIX 2018 13

[19] J Jia A Salem M Backes Y Zhang and N Z Gong ldquoMemGuardDefending against Black-Box Membership Inference Attacks via Ad-versarial Examplesrdquo in ACM SIGSAC Conference on Computer andCommunications Security (CCS) ACM 2019 pp 259ndash274 13

[20] B Li and Y Vorobeychik ldquoScalable Optimization of Randomized Oper-ational Decisions in Adversarial Classification Settingsrdquo in InternationalConference on Artificial Intelligence and Statistics (AISTATS) PMLR2015 pp 599ndash607 13

[21] Y Liu W-C Lee G Tao S Ma Y Aafer and X Zhang ldquoABSScanning Neural Networks for Back-Doors by Artificial Brain Stimula-tionrdquo in ACM SIGSAC Conference on Computer and CommunicationsSecurity (CCS) ACM 2019 pp 1265ndash1282 1 2 10 13

[22] Y Liu S Ma Y Aafer W-C Lee J Zhai W Wang and X ZhangldquoTrojaning Attack on Neural Networksrdquo in Network and DistributedSystem Security Symposium (NDSS) Internet Society 2019 1 12

[23] Z Liu P Luo X Wang and X Tang ldquoDeep Learning Face Attributesin the Wildrdquo in IEEE International Conference on Computer Vision(ICCV) IEEE 2015 1 2 7

[24] Y Long V Bindschaedler and C A Gunter ldquoTowards MeasuringMembership Privacyrdquo CoRR abs171209136 2017 13

[25] Y Long V Bindschaedler L Wang D Bu X Wang H Tang C AGunter and K Chen ldquoUnderstanding Membership Inferences on Well-Generalized Learning Modelsrdquo CoRR abs180204889 2018 13

[26] L Melis C Song E D Cristofaro and V Shmatikov ldquoExploiting Unin-tended Feature Leakage in Collaborative Learningrdquo in IEEE Symposiumon Security and Privacy (SampP) IEEE 2019 13

[27] M Nasr R Shokri and A Houmansadr ldquoMachine Learning withMembership Privacy using Adversarial Regularizationrdquo in ACM SIGSACConference on Computer and Communications Security (CCS) ACM2018 13

[28] mdashmdash ldquoComprehensive Privacy Analysis of Deep Learning Passive andActive White-box Inference Attacks against Centralized and FederatedLearningrdquo in IEEE Symposium on Security and Privacy (SampP) IEEE2019 13

[29] S J Oh M Augustin B Schiele and M Fritz ldquoTowards Reverse-Engineering Black-Box Neural Networksrdquo in International Conferenceon Learning Representations (ICLR) 2018 1

[30] S J Oh M Fritz and B Schiele ldquoAdversarial Image Perturbation forPrivacy Protection ndash A Game Theory Perspectiverdquo in IEEE InternationalConference on Computer Vision (ICCV) IEEE 2017 pp 1482ndash149113

[31] T Orekondy B Schiele and M Fritz ldquoKnockoff Nets Stealing Func-tionality of Black-Box Modelsrdquo in IEEE Conference on Computer Visionand Pattern Recognition (CVPR) IEEE 2019 13

[32] N Papernot P D McDaniel I Goodfellow S Jha Z B Celik andA Swami ldquoPractical Black-Box Attacks Against Machine Learningrdquoin ACM Asia Conference on Computer and Communications Security(ASIACCS) ACM 2017 pp 506ndash519 1 13

[33] N Papernot P D McDaniel S Jha M Fredrikson Z B Celik andA Swami ldquoThe Limitations of Deep Learning in Adversarial Settingsrdquoin IEEE European Symposium on Security and Privacy (Euro SampP)IEEE 2016 pp 372ndash387 1 13

[34] A Pyrgelis C Troncoso and E D Cristofaro ldquoKnock Knock WhorsquosThere Membership Inference on Aggregate Location Datardquo in Networkand Distributed System Security Symposium (NDSS) Internet Society2018 13

[35] mdashmdash ldquoUnder the Hood of Membership Inference Attacks on AggregateLocation Time-Seriesrdquo CoRR abs190207456 2019 13

[36] A Salem A Bhattacharya M Backes M Fritz and Y ZhangldquoUpdates-Leak Data Set Inference and Reconstruction Attacks in On-line Learningrdquo in USENIX Security Symposium (USENIX Security)USENIX 2020 13

[37] A Salem Y Zhang M Humbert P Berrang M Fritz and M BackesldquoML-Leaks Model and Data Independent Membership Inference At-tacks and Defenses on Machine Learning Modelsrdquo in Network andDistributed System Security Symposium (NDSS) Internet Society 20191 13

[38] A Shafahi W R Huang M Najibi O Suciu C Studer T Dumitrasand T Goldstein ldquoPoison Frogs Targeted Clean-Label Poisoning At-tacks on Neural Networksrdquo in Annual Conference on Neural InformationProcessing Systems (NIPS) NIPS 2018 pp 6103ndash6113 13

[39] R Shokri M Stronati C Song and V Shmatikov ldquoMembership Infer-ence Attacks Against Machine Learning Modelsrdquo in IEEE Symposiumon Security and Privacy (SampP) IEEE 2017 pp 3ndash18 1 13

[40] K Simonyan and A Zisserman ldquoVery Deep Convolutional Networksfor Large-Scale Image Recognitionrdquo in International Conference onLearning Representations (ICLR) 2015 7

[41] C Song and V Shmatikov ldquoThe Natural Auditor How To Tell If Some-one Used Your Words To Train Their Modelrdquo CoRR abs1811005132018 13

[42] O Suciu R Marginean Y Kaya H D III and T Dumitras ldquoWhenDoes Machine Learning FAIL Generalized Transferability for Evasionand Poisoning Attacksrdquo CoRR abs180306975 2018 1 13

[43] F Tramer A Kurakin N Papernot I Goodfellow D Boneh andP McDaniel ldquoEnsemble Adversarial Training Attacks and Defensesrdquoin International Conference on Learning Representations (ICLR) 201713

[44] F Tramer F Zhang A Juels M K Reiter and T Ristenpart ldquoStealingMachine Learning Models via Prediction APIsrdquo in USENIX SecuritySymposium (USENIX Security) USENIX 2016 pp 601ndash618 1 13

[45] Y Vorobeychik and B Li ldquoOptimal Randomized Classification in Ad-versarial Settingsrdquo in International Conference on Autonomous Agentsand Multi-agent Systems (AAMAS) 2014 pp 485ndash492 13

[46] B Wang and N Z Gong ldquoStealing Hyperparameters in MachineLearningrdquo in IEEE Symposium on Security and Privacy (SampP) IEEE2018 1 13

[47] B Wang Y Yao S Shan H Li B Viswanath H Zheng and B YZhao ldquoNeural Cleanse Identifying and Mitigating Backdoor Attacks inNeural Networksrdquo in IEEE Symposium on Security and Privacy (SampP)IEEE 2019 pp 707ndash723 1 2 10 13

[48] W Xu D Evans and Y Qi ldquoFeature Squeezing Detecting AdversarialExamples in Deep Neural Networksrdquo in Network and Distributed SystemSecurity Symposium (NDSS) Internet Society 2018 1 13

[49] Y Yao H Li H Zheng and B Y Zhao ldquoLatent Backdoor Attacks onDeep Neural Networksrdquo in ACM SIGSAC Conference on Computer andCommunications Security (CCS) ACM 2019 pp 2041ndash2055 1

[50] S Yeom I Giacomelli M Fredrikson and S Jha ldquoPrivacy Risk inMachine Learning Analyzing the Connection to Overfittingrdquo in IEEEComputer Security Foundations Symposium (CSF) IEEE 2018 13

[51] Y Zhang M Humbert T Rahman C-T Li J Pang and M BackesldquoTagvisor A Privacy Advisor for Sharing Hashtagsrdquo in The WebConference (WWW) ACM 2018 pp 287ndash296 13

14