Testing Robustness Against Unforeseen Adversaries Daniel Kang * Stanford University [email protected] Yi Sun * Columbia University [email protected] Dan Hendrycks UC Berkeley [email protected] Tom Brown OpenAI [email protected] Jacob Steinhardt OpenAI, UC Berkeley [email protected] Abstract Most existing adversarial defenses only measure robustness to L p adversarial attacks. Not only are adversaries unlikely to exclusively create small L p perturbations, adversaries are unlikely to remain fixed. Adversaries adapt and evolve their attacks; hence adversarial defenses must be robust to a broad range of unforeseen attacks. We address this discrepancy between research and reality by proposing a new evaluation framework called ImageNet-UA. Our framework enables the research community to test ImageNet model robustness against attacks not encountered during training. To create ImageNet-UA’s diverse attack suite, we introduce a total of four novel adversarial attacks. We also demonstrate that, in comparison to ImageNet-UA, prevailing L ∞ robustness assessments give a narrow account of adversarial robustness. By evaluating current defenses with ImageNet-UA, we find they provide little robustness to unforeseen attacks. We hope the greater variety and realism of ImageNet-UA enables development of more robust defenses which can generalize beyond attacks seen during training. 1 Introduction Neural networks perform well on many datasets [24] yet can be consistently fooled by minor adversarial distortions [22]. The research community has responded by quantifying and developing adversarial defenses against such attacks [33], yet these defenses and metrics have two key limitations. First, the vast majority of existing defenses exclusively defend against and quantify robustness to L p -constrained attacks [33, 11, 43, 58]. Though real-world adversaries are not L p constrained [19] and can attack with diverse distortions [5, 49], the literature largely ignores this and evaluates against the L p adversaries already seen during training [33, 58], resulting in optimistic robustness assessments. The attacks outside the L p threat model that have been proposed [51, 42, 14, 61, 15, 48] are not intended for general defense evaluation and suffer from narrow dataset applicability, difficulty of optimization, or fragility of auxiliary generative models. Second, existing defenses assume that attacks are known in advance [21] and use knowledge of their explicit form during training [33]. In practice, adversaries can deploy unforeseen attacks not known to the defense creator. For example, online advertisers use attacks such as perturbed pixels in ads to defeat ad blockers trained only on the previous generation of ads in an ever-escalating arms race [54]. However, current evaluation setups implicitly assume that attacks encountered at test-time are the same as those seen at train-time, which is unrealistic. The reality that future attacks are unlike those encountered during training is akin to a train-test distribution mismatch—a problem studied outside of adversarial robustness [45, 25]—but we now bring this idea to the adversarial setting. * Equal contribution Preprint. Under review. arXiv:1908.08016v2 [cs.LG] 9 Jun 2020
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Testing Robustness Against Unforeseen Adversaries
Daniel Kang∗Stanford University
Yi Sun∗Columbia University
Dan HendrycksUC Berkeley
Tom BrownOpenAI
Jacob SteinhardtOpenAI, UC Berkeley
Abstract
Most existing adversarial defenses only measure robustness to Lp adversarialattacks. Not only are adversaries unlikely to exclusively create small Lp
perturbations, adversaries are unlikely to remain fixed. Adversaries adapt andevolve their attacks; hence adversarial defenses must be robust to a broad rangeof unforeseen attacks. We address this discrepancy between research and realityby proposing a new evaluation framework called ImageNet-UA. Our frameworkenables the research community to test ImageNet model robustness against attacksnot encountered during training. To create ImageNet-UA’s diverse attack suite,we introduce a total of four novel adversarial attacks. We also demonstrate that,in comparison to ImageNet-UA, prevailing L∞ robustness assessments give anarrow account of adversarial robustness. By evaluating current defenses withImageNet-UA, we find they provide little robustness to unforeseen attacks. Wehope the greater variety and realism of ImageNet-UA enables development ofmore robust defenses which can generalize beyond attacks seen during training.
1 Introduction
Neural networks perform well on many datasets [24] yet can be consistently fooled by minoradversarial distortions [22]. The research community has responded by quantifying and developingadversarial defenses against such attacks [33], yet these defenses and metrics have two key limitations.
First, the vast majority of existing defenses exclusively defend against and quantify robustnessto Lp-constrained attacks [33, 11, 43, 58]. Though real-world adversaries are not Lp constrained[19] and can attack with diverse distortions [5, 49], the literature largely ignores this and evaluatesagainst the Lp adversaries already seen during training [33, 58], resulting in optimistic robustnessassessments. The attacks outside the Lp threat model that have been proposed [51, 42, 14, 61, 15, 48]are not intended for general defense evaluation and suffer from narrow dataset applicability, difficultyof optimization, or fragility of auxiliary generative models.
Second, existing defenses assume that attacks are known in advance [21] and use knowledge of theirexplicit form during training [33]. In practice, adversaries can deploy unforeseen attacks not knownto the defense creator. For example, online advertisers use attacks such as perturbed pixels in ads todefeat ad blockers trained only on the previous generation of ads in an ever-escalating arms race [54].However, current evaluation setups implicitly assume that attacks encountered at test-time are thesame as those seen at train-time, which is unrealistic. The reality that future attacks are unlike thoseencountered during training is akin to a train-test distribution mismatch—a problem studied outsideof adversarial robustness [45, 25]—but we now bring this idea to the adversarial setting.
∗Equal contribution
Preprint. Under review.
arX
iv:1
908.
0801
6v2
[cs
.LG
] 9
Jun
202
0
Prev
ious
Atta
cks
L∞ L2 L1 Elastic
Our
New
Atta
cks
JPEG Fog Snow GaborFigure 1: Adversarially distorted chow chow dog images created with old attacks and our new attacks.The JPEG, Fog, Snow, and Gabor adversarial attacks are visually distinct from previous attacks andserve as unforeseen attacks in the ImageNet-UA attack suite.The present work addresses these limitations by proposing an evaluation framework ImageNet-UAto measure robustness against unforeseen attacks. ImageNet-UA assesses a defense which may havebeen created with knowledge of the commonly used L∞ or L2 attacks with six diverse attacks (fourof which are novel) distinct from L∞ or L2. We intend these attacks to be used at test-time onlyand not during training. Performing well on ImageNet-UA thus demonstrates generalization to adiverse set of distortions not seen during defense creation. While ImageNet-UA does not providean exhaustive guarantee over all conceivable attacks, it evaluates over a diverse unforeseen testdistribution similar to those used successfully in other studies of distributional shift [44, 25, 45].ImageNet-UA works for ImageNet models and can be easily used with our code available at https://github.com/ddkang/advex-uar.
Designing ImageNet-UA requires new attacks that are strong and varied, since real-world attacks arediverse in structure. To meet this challenge, we contribute four novel and diverse adversarial attacks,in contrast to prior papers offering only one [4, 1, 14, 57]. Our new attacks produce distortions withocclusions, spatial similarity, and simulated weather, all of which are absent in previous attacks.Performing well on ImageNet-UA thus demonstrates that a defense generalizes to a diverse set ofdistortions distinct from the commonly used L∞ or L2.
With ImageNet-UA, we show marked weaknesses in existing evaluation practices and defensesthrough a study of 8 attacks against 48 models adversarially trained on ImageNet-100, a 100-classsubset of ImageNet [46]. While most adversarial robustness evaluations use only L∞ attacks,ImageNet-UA reveals that models with high L∞ attack robustness can remain susceptible to otherattacks. This implies that L∞ evaluations are a narrow measure of robustness, even though much ofthe literature treats this evaluation as comprehensive [33, 40, 47, 60]. We address this deficiency byusing the novel attacks in ImageNet-UA to evaluate robustness to a more diverse set of unforeseenattacks. Moreover, our results demonstrate that L∞ adversarial training, the current state-of-the-artdefense, has limited generalization to unforeseen adversaries, and is not easily improved by trainingagainst more attacks. This adds to the evidence that achieving robustness against a few train-timeattacks is insufficient to impart robustness to unforeseen test-time attacks [29, 30, 53].
In summary, we propose the framework ImageNet-UA to measure robustness to a diverse set ofattacks, made possible by our four new adversarial attacks. Since existing defenses scale poorly tomultiple attacks [30, 53], finding defense techniques which generalize to unforeseen attacks is crucialto create robust models. We suggest ImageNet-UA as a way to measure progress towards this goal.
2 Related Work
Adversarial robustness is notoriously difficult to correctly evaluate [39, 2]. To that end, Carliniet al. [7] provide extensive guidance for sound adversarial robustness evaluation. By measuringattack success rates across several distortion sizes and using a broader threat model with diversedifferentiable attacks, ImageNet-UA has several of their recommendations built-in. Previous work on
2
Randomly InitializedJPEG
AdversariallyOptimized JPEG
Randomly InitializedFog
AdversariallyOptimized Fog
Otter (100.0%) Basketball (100.0%) Otter (100.0%) Titi Monkey (100.0%)
Randomly InitializedSnow
AdversariallyOptimized Snow
Randomly InitializedGabor
AdversariallyOptimized Gabor
Otter (100.0%) Loafer (98.0%) Otter (100.0%) Zebra (100.0%)Figure 2: Randomly sampled distortions and adversarially optimized distortions from our new attacks.Attacks are targeted to the target class in red. Stochastic average-case versions of our attacks affectclassifiers minimally, while adversarial versions are optimized to reveal high-confidence errors. Thesnowflakes in Snow decrease in intensity after optimization, demonstrating that lighter adversarialsnowflakes are more effective than heavy random snowfall at uncovering model weaknesses.
evaluation considers small sets of fixed attacks. DeepFool [35] and CLEVER [55] estimate empiricalrobustness, the expected minimum ε needed to successfully attack an image. They apply only toattacks optimizing over an Lp-ball of radius ε, and CLEVER is susceptible to gradient masking [20].Wu et al. [56] evaluate against physically-realizable attacks from Evtimov et al. [15] and Sharif et al.[48], thus using a threat model restricted to occlusion attacks on narrow datasets.
Prior attacks outside the Lp threat model exist, but lack the general applicability and fast optimizationof ours. Song et al. [51] attack using variational autoencoders, yet the attacks are weak and requiresimple image distributions suitable for VAEs. Qiu et al. [42] create adversarial images with aStarGAN, which is subject to GAN instabilities. Engstrom et al. [14] apply Euclidean transformationsdetermined by brute-force search. Zhao et al. [61] use perceptual color distances to align humanperception and L2 perturbations. Evtimov et al. [15] and Sharif et al. [48] attack stop signs and face-recognition systems with carefully placed patches or modified eyeglass frames, requiring physicalobject creation and applying only to specific image types. In contrast, our attacks are fast by virtue ofdifferentiability, broadly applicable, and independent of auxiliary generative models.
3 New Attacks for a Broader Threat Model
There are few diverse, easily optimizable, plug-and-play adversarial attacks in the current literature;outside of Elastic [57], most are Lp attacks such as L∞ [22], L2 [52, 6], L1 [9]. We rectify thisdeficiency with four novel adversarial attacks: JPEG, Fog, Snow, and Gabor. Our attacks aredifferentiable and fast, while optimizing over enough parameters to be strong. We show exampleadversarial images in Figure 1 and compare stochastic and adversarial distortions in Figure 2.
Our novel attacks provide a broad range of test-time adversaries distinct from L∞ or L2 attacks. Theyare intended as unforeseen attacks not used during training, allowing them to evaluate whether a de-fense can generalize from L∞ or L2 to a much more varied set of distortions than current evaluations.Though our attacks are not exhaustive, performing well against them already demonstrates robustnessto occlusion, spatial similarity, and simulated weather, all of which are absent from previous attacks.
Our attacks create an adversarial image x′ from a clean image x with true label y. Let model f mapimages to a softmax distribution, and let `(f(x), y) be the cross-entropy loss. Given a target classy′ 6= y, our attacks attempt to find a valid image x′ such that (1) the attacked image x′ is obtained
3
by applying a distortion (of size controlled by a parameter ε) to x, and (2) the loss `(f(x′), y′) isminimized. An unforeseen adversarial attack is a white- or black-box adversarial attack unknown tothe defense designer which does not change the true label of x according to an oracle or human.
3.1 Four New Unforeseen Attacks
JPEG. JPEG applies perturbations in a JPEG-encoded space of compressed images rather thanraw pixel space. After color-space conversion, JPEG encodes small image patches using the discretecosine transform. It then uses projected gradient descent to find an L∞-constrained adversarialperturbation in the resulting frequency space. The perturbed frequency coefficients are quantized andreverse-transformed to obtain the image in pixel space. We use ideas from Shin and Song [50] tomake this differentiable. The resulting attack is conspicuously distinct from Lp attacks.
Fog. Fog simulates worst-case weather conditions. Robustness to adverse weather is a safetycritical priority for autonomous vehicles, and Figure 2 shows Fog provides a more rigorous stress-testthan stochastic fog [25]. Fog creates adversarial fog-like occlusions by adversarially optimizingparameters in the diamond-square algorithm [16] typically used to render stochastic fog effects.
Snow. Snow simulates snowfall with occlusions of randomly located small image regions rep-resenting snowflakes. It adversarially optimizes their intensity and direction. Making Snow fastand differentiable is non-trivial and hinges on the use of an exponential distribution for snowflakeintensities. Compared to synthetic stochastic snow [25], our adversarial snow is faster and includessnowflakes at differing angles instead of one fixed angle. Figure 2 shows adversarial snow exposesmodel weaknesses more effectively than the easier stochastic, average-case snow.
Gabor. Gabor spatially occludes the image with visually diverse Gabor noise [31]. Gaboradversarially optimizes semantically meaningful parameters (orientation, bandwidth, etc.) to createdifferent Gabor kernels used in Gabor noise. While rendering Gabor noise, we use spectral variancenormalization [10] and initialize our optimization parameters with a sparse random matrix.
3.2 Improving Existing Attacks
Elastic modifies the attack of Xiao et al. [57]; it warps the image by distortions x′ = Flow(x, V ),where V : {1, . . . , 224}2 → R2 is a vector field on pixel space, and Flow sets the value of pixel (i, j)to the bilinearly interpolated original value at (i, j) + V (i, j). We construct V by smoothing a vectorfield W by a Gaussian kernel (size 25× 25, σ ≈ 3 for a 224× 224 image) and optimize W under‖W (i, j)‖∞ ≤ ε for all i, j. The resulting attack is suitable for large-scale images.
The other three attacks are L1, L2, L∞ attacks, but we improve the L1 attack. For L∞ and L2
constraints, we use randomly-initialized projected gradient descent (PGD), which applies gradientdescent and projection to the L∞ and L2 balls [33]. Projection is difficult for L1, and previous L1
attacks resort to heuristics [9, 53]. We replace PGD with the Frank-Wolfe algorithm [17], whichoptimizes a linear function instead of projecting at each step (pseudocode in Appendix D). Thismakes our L1 attack more principled than previous implementations.
4 ImageNet-UA: Measuring Robustness to Unforeseen Attacks
We propose the framework ImageNet-UA and its CIFAR-10 analogue CIFAR-10-UA to measure andsummarize model robustness while fulfilling the following desiderata:
• Defenses should be evaluated against a broad threat model through a diverse set of attacks.
• Defenses should exhibit generalization to attacks not exactly identical to train-time attacks.
• The range of distortion sizes used for an attack must be wide enough to avoid misleadingconclusions caused by overly weak or strong versions of that attack (Figure 3).
The ImageNet-UA evaluation framework aggregates robustness information into a single measure,the mean Unforeseen Adversarial Robustness (mUAR). The mUAR is an average over six differentattacks of the Unforeseen Adversarial Robustness (UAR), a metric which assesses the robustness of adefense against a specific attack by using a wide range of distortion sizes. UAR is normalized using ameasure of attack strength, the ATA, which we now define.
4
Measuring Robustness Requires a Range of Distortion Sizes
0 2000 4000L2 distortion size
25
50A
ccur
acy
(a) L2 vs. L2-training
0 5 10 15Elastic distortion size
0
50
Acc
urac
y
(b) Elastic vs. L2-training
Figure 3: Accuracies of L2 and Elastic attacks at different distortion sizes against a ResNet-50model adversarially trained against L2 at ε = 9600 on ImageNet-100. At small distortion sizes, themodel appears to defend well against Elastic, but large distortion sizes reveal that robustness does nottransfer from L2 to Elastic.
Adversarial Training Accuracy (ATA). The Adversarial Training Accuracy ATA(A, ε) estimatesthe strength of an attack A against adversarial training [33], one of the strongest currently knowndefense methods. For a distortion size ε, it is the best adversarial test accuracy against A achieved byadversarial training against A. We allow a possibly different distortion size ε′ during training, sincethis sometimes improves accuracy, and we choose a fixed architecture for each dataset.
For ImageNet-100, we choose ResNet-50 for the architecture, and for CIFAR-10 we choose ResNet-56. When evaluating a defense with architecture other than ResNet-50 or ResNet-56, we recommendusing ATA values computed with these architectures to enable consistent comparison. To estimateATA(A, ε) in practice, we evaluate models adversarially trained against distortion size ε′ for ε′ in alarge range (we describe this range at this section’s end).
UAR: Robustness Against a Single Attack. The UAR, a building block for the mUAR, averages amodel’s robustness to a single attack over six distortion sizes ε1, . . . , ε6 chosen for each attack (wedescribe the selection procedure at the end of this section). It is defined as
UAR(A) := 100×∑6
k=1 Acc(A, εk,M)∑6k=1 ATA(A, εk)
, (1)
where Acc(A, εk,M) is the accuracy Acc(A, εk,M) of a model M after attack A at distortion sizeεk. The normalization in (1) makes attacks of different strengths more commensurable in a stable way.We give values of ATA(A, εk) and εk for our attacks on ImageNet-100 and CIFAR-10 in Tables 4and 5 (Appendix B), allowing computation of UAR of a defense against a single attack with sixadversarial evaluations and no adversarial training.
mUAR: Mean Unforeseen Attack Robustness. We summarize a defense’s performance onImageNet-UA with the mean Unforeseen Attack Robustness (mUAR), an average of UAR scores forthe L1, Elastic, JPEG, Fog, Snow, and Gabor attacks:
mUAR :=1
6
[UAR(L1)+UAR(Elastic)+UAR(JPEG)+UAR(Fog)+UAR(Snow)+UAR(Gabor)
].
Our measure mUAR estimates robustness to a broad threat model containing six unforeseen attacksat six distortion sizes each, meaning high mUAR requires generalization to several held-out attacks.In particular, it cannot be achieved by the common practice of engineering defenses to a single attack,which Figure 4 shows does not necessarily provide robustness to different attacks.
Our four novel attacks play a crucial role in mUAR by allowing us to estimate robustness to asufficiently large set of adversarial attacks. As is customary when studying train-test mismatches anddistributional shift, we advise against adversarially training with these six attacks when evaluatingImageNet-UA to preserve the validity of mUAR, though we encourage training with other attacks.
Distortion Size Choices. We explain the ε′ values used to estimate ATA and the choice of ε1, . . . , ε6used to define UAR. This calibration of distortion sizes adjusts for the fact (Figure 3) that adversarialrobustness against an attack may vary drastically with distortion size. Further, the relation between
5
L∞ L2
L1
JPE
GE
last
ic
Fog
Snow
Gab
orAdversarial Attack
None
L∞
L2
L1
JPEG
Elastic
Fog
Snow
Gabor
Ad
vers
aria
llyT
rain
edD
efen
se
7 17 22 0 31 16 10 5
88 42 15 14 49 20 37 55
80 88 79 67 48 18 38 53
62 71 89 56 43 18 31 47
65 70 54 92 40 19 31 52
23 25 11 1 91 25 40 41
1 3 8 0 28 91 43 54
13 15 9 1 39 37 93 60
12 19 14 0 39 29 40 82
Defense Robustness Under Different Attacks
Figure 4: UAR for adversarially traineddefenses (row) against attacks (col) onImageNet-100. Defenses from L∞ to Ga-bor were trained with ε = 32, 4800, 612000,2, 16, 8192, 8, and 1600, respectively.
StandardTraining
ε = 1 ε = 2 ε = 4 ε = 8 ε = 16 ε = 320
20
40
60
80
Performance of Defenses Adversarially Trained Against L∞
UAR(L∞)
mUAR
Figure 5: UAR(L∞) and mUAR for L∞-trained modelsat different distortion sizes. Increasing distortion size inL∞-training improves UAR(L∞) but hurts the mUAR,suggesting models heavily fit L∞ at the cost of general-ization.
distortion size and attack strength varies between attacks, so too many or too few εk values in acertain range may cause an attack to appear artificially strong or weak according to UAR.
We choose distortion sizes between minimum and maximum values εmin and εmax defined as follows:
1. The minimum distortion size εmin is the largest ε for which the adversarial accuracy of anadversarially trained model at distortion size ε is comparable to that of a model trained andevaluated on unattacked data (for ImageNet-100, within 3 of 87).
2. The maximum distortion size εmax is the smallest ε which either reduces the adversarialaccuracy of an adversarially trained model at distortion size ε below 25 or yields imageswhich confuse humans (adversarial accuracy can remain non-zero in this case).
As is typical in recent work on adversarial examples [3, 15, 13, 41], our attacks are perceptible atlarge distortion sizes, reflecting the perceptibility of attacks in real world threat models suggested byGilmer et al. [19].
For ATA, we evaluate against models adversarially trained with ε′ increasing geometrically fromεmin to εmax by factors of 2. We then choose εk as follows: We compute ATA at ε increasinggeometrically from εmin to εmax by factors of 2 and take the size-6 subset whose ATA values haveminimum `1-distance to the ATA values of the L∞ attack in Table 4 (Appendix B.1). For example,for Gabor, (εmin, εmax) = (6.25, 3200), so we compute ATAs at the 10 values ε = 6.25, . . . , 3200.Viewing size-6 subsets of the ATAs as vectors with decreasing coordinates, we select εk for Gaborcorresponding to the vector with minimum `1-distance to the ATA vector for L∞.
5 New Insights From ImageNet-UA
We use ImageNet-UA to assess existing methods for adversarial defense and evaluation. First,ImageNet-UA reveals that L∞ trained defenses fail to generalize to different attacks, indicatingsubstantial weakness in current L∞ adversarial robustness evaluation. We establish a baselinefor ImageNet-UA using L2 adversarial training which is difficult to improve upon by adversarialtraining alone. Finally, we show non-adversarially trained models can still improve robustness onImageNet-UA over standard models and suggest this as a direction for further inquiry.
5.1 Experimental Setup
We adversarially train 48 models against the 8 attacks from Section 3 and evaluate against targetedattacks. We use the CIFAR-10 and ImageNet-100 datasets for ImageNet-UA and CIFAR-10-UA.ImageNet-100 is a 100-class subset of ImageNet-1K [12] containing every tenth class by WordNetID order; we use a subset of ImageNet-1K due to the high compute cost of adversarial training
6
Table 1: Clean Accuracy, UAR, and mUAR scores for models adversarially trained against L∞ and L2
attacks. L∞ training, the most popular defense, provides less robustness than L2 training. Comparingthe highest mUAR achieved to individual UAR values in Figure 4 indicates a large robustness gap.
Clean Acc. L∞ L2 mUAR Clean Acc. L∞ L2 mUAR
Normal Training 86.7 7.3 17.2 14.0 Normal Training 86.7 7.3 17.2 14.0L∞ ε = 1 86.2 46.4 54.2 30.7 L2 ε = 150 86.6 38.0 49.4 27.1L∞ ε = 2 85.5 59.8 64.4 36.9 L2 ε = 300 85.9 49.7 60.1 33.3L∞ ε = 4 83.9 72.1 73.6 42.3 L2 ε = 600 84.7 61.9 71.6 40.0L∞ ε = 8 79.8 82.6 72.0 42.2 L2 ε = 1200 82.3 72.9 82.0 46.8L∞ ε = 16 74.5 89.1 60.0 37.5 L2 ε = 2400 76.8 79.6 88.5 50.7L∞ ε = 32 70.8 88.1 41.9 31.8 L2 ε = 4800 68.3 80.4 87.7 50.5
Table 2: Clean Accuracy, UAR, and mUAR scores for models jointly trained against (L∞, L2). Jointtraining does not provide much additional robustness.
Clean Acc. L∞ L2 mUARL∞ ε = 1, L2 ε = 300 86.1 50.3 60.2 33.6L∞ ε = 2, L2 ε = 600 85.1 62.8 72.5 41.0L∞ ε = 4, L2 ε = 1200 81.3 72.9 81.2 46.9L∞ ε = 8, L2 ε = 2400 76.5 80.0 87.3 50.8L∞ ε = 16, L2 ε = 4800 68.4 81.5 87.9 50.9
on large-scale images. We use ResNet-56 for CIFAR-10 and ResNet-50 from torchvision forImageNet-100 [24]. We provide training hyperparameters in Appendix A.
To adversarially train [33] against attack A, at each mini-batch we select a uniform random (incorrect)target class for each training image. For maximum distortion size ε, we apply targeted attack A to thecurrent model with distortion size ε′ ∼ Uniform(0, ε) and take a SGD step using only the attackedimages. Randomly scaling ε′ improves performance against smaller distortions.
We train on 10-step attacks for attacks other than Elastic, where we use 30 steps due to a harderoptimization. For Lp, JPEG, and Elastic, we use step size ε/
√steps; for Fog, Gabor, and Snow, we
use step size√
0.001/steps because the latent space is independent of ε. These choices have optimalrates for non-smooth convex functions [36, 37]. We evaluate on 200-step targeted attacks withuniform random (incorrect) target, using more steps for evaluation than training per best practices [8].
Figure 4 summarizes ImageNet-100 results. Full results for ImageNet-100 and CIFAR-10 are inAppendix E and robustness checks to random seed and attack iterations are in Appendix F.
5.2 ImageNet-UA Reveals Weaknessess in L∞ Training and Testing
We use ImageNet-UA to reveal weaknesses in the common practices of L∞ robustness evaluationand L∞ adversarial training. We compute the mUAR and UAR(L∞) for models trained against theL∞ attack with distortion size ε and show results in Figure 5. For small ε ≤ 4, mUAR and UAR(L∞)increase together with ε. For larger ε ≥ 8, UAR(L∞) continues to increase with ε, but the mUARdecreases, a fact which is not apparent from L∞ evaluation.
The decrease in mUAR while UAR(L∞) increases suggests that L∞ adversarial training begins toheavily fit L∞ distortions at the expense of generalization at larger distortion sizes. Thus, while it isthe most commonly used defense procedure, L∞ training may not lead to improvements on otherattacks or to real-world robustness.
Worse, L∞ evaluation againstL∞ adversarial training at higher distortions indicates higher robustness.In contrast, mUAR reveals that L∞ adversarial training at higher distortions in fact hurts robustnessagainst a more diverse set of attacks. Thus, L∞ evaluation gives a misleading picture of robustness.This is particularly important because L∞ evaluation is the most ubiquitous measure of robustness indeep learning [22, 33, 58].
7
Table 3: Non-adversarial defenses can noticeably improve ImageNet-UA performance. ResNeXt-101 (32×8d) + WSL is a ResNeXt-101 trained on approximately 1 billion images [34]. StylizedImageNet is trained on a modification of ImageNet using style transfer [18]. Patch Gaussianaugments using Gaussian distortions on small portions of the image [32]. AugMix mixes simplerandom augmentations of the image [27]. These results suggest a complementary avenue towardImageNet-UA performance may be through non-adversarial defenses.
Clean Acc.L∞ L2 L1 Elastic JPEG Fog Snow Gabor mUARSqueezeNet 84.1 5.2 11.2 14.9 25.9 1.9 20.1 9.8 4.4 12.8ResNeXt-101 (32×8d) 95.9 2.5 5.5 20.7 26.5 1.8 14.1 12.4 5.3 13.4ResNeXt-101 (32×8d) + WSL 97.1 3.0 5.7 28.3 29.4 1.9 26.2 20.3 8.0 19.0ResNet-18 91.6 2.7 8.2 13.5 22.6 1.8 20.3 9.5 4.2 12.0ResNet-50 94.2 2.7 6.6 20.1 24.9 1.8 15.8 11.9 4.9 13.2ResNet-50 + Stylized ImageNet 94.6 2.9 7.4 22.8 26.0 1.8 16.2 12.5 8.1 14.6ResNet-50 + Patch Gaussian 93.6 4.5 10.9 27.4 28.2 1.8 23.9 10.5 5.2 16.2ResNet-50 + AugMix 95.1 6.1 13.4 34.3 38.8 1.8 28.6 24.7 11.1 23.2
5.3 Limits of Adversarial Training for ImageNet-UA
We establish a baseline on ImageNet-UA using L2 adversarial training but show a significant perfor-mance gap even for more sophisticated existing adversarial training methods. To do so, we evaluateseveral adversarial training methods on ImageNet-UA and show results in Table 1.
Our results show that L2 trained models outperform L∞ trained models and have significantlyimproved absolute performance, increasing mUAR from 14.0 to 50.7 compared to an undefendedmodel. The individual UAR values in Figure 7 (Appendix E.1) improve substantially against allattacks other than Fog, including several (Elastic, Gabor, Snow) of extremely different nature to L2.
This result suggests pushing adversarial training further by training against multiple attacks simulta-neously via joint adversarial training [30, 53] detailed in Appendix C. Table 2 shows that, despiteusing twice the compute of L2 training, (L∞, L2) joint training only improves the mUAR from 50.7to 50.9. We thus recommend L2 training as a baseline for ImageNet-UA, though there is substantialroom for improvement compared to the highest UARs against individual attacks in Figure 4, whichare all above 80 and often above 90.
5.4 ImageNet-UA Robustness through Non-Adversarial Defenses
We find that methods can improve robustness to unforeseen attacks without adversarial training. Table3 shows mUAR for diverse architectures including SqueezeNet [28], ResNeXts [59], and ResNets.For ImageNet-1K models, we predict ImageNet-100 classes by masking 900 logits.
A popular defense against average case distortions [25] is Stylized ImageNet [18], which modifiestraining images using image style transfer in hopes of making networks rely less on textural features.Table 3 shows it provides some improvement on ImageNet-UA. More recently, Lopes et al. [32]propose to train against Gaussian noise applied to small image patches, improving the mUAR by3% over the ResNet-50 baseline. The second largest mUAR improvement comes from training aResNeXt on approximately 1 billion images [34]. This three orders of magnitude increase in trainingdata yields a 5.4% mUAR increase over a vanilla ResNeXt baseline. Finally, Hendrycks et al. [27]create AugMix, which randomly mixes stochastically generated augmentations. Although AugMixdid not use random nor adversarial noise, it improves robustness to unforeseen attacks by 10%.
These results imply that defenses not relying on adversarial examples can improve ImageNet-UAperformance. They indicate that training on more data only somewhat increases robustness onImageNet-UA, quite unlike many other robustness benchmarks [25, 26] where more data helpstremendously [38]. While models with lower clean accuracy including SqueezeNet and ResNet-18oddly have higher UAR(L∞) and UAR(L2) than many other models, there is no clear differencein mUAR. Last, these non-adversarial defenses do not come at a large cost to accuracy on cleanexamples, unlike adversarial defenses. Much remains to explore, and we hope non-adversarialdefenses will be a promising avenue toward adversarial robustness.
8
6 Conclusion
This work proposes a framework ImageNet-UA to evaluate robustness of a defense against unforeseenattacks. Because existing adversarial defense techniques do not scale to multiple attacks, developingmodels which can defend against attacks not seen at train-time is essential for robustness. Our resultsusing ImageNet-UA show that the common practice of L∞ training and evaluation fails to achieve ormeasure this broader form of robustness. As a result, it can provide a misleading sense of robustness.By incorporating our 4 novel and strong adversarial attacks, ImageNet-UA enables evaluation onthe diverse held-out attacks necessary to measure progress towards robustness more broadly.
Acknowledgements
D. K., Y. S., and J. S. were supported by a grant from the Open Philanthropy Project. D. K. wassupported by NSF Grant DGE-1656518. Y. S. was supported by a Junior Fellow award from theSimons Foundation and NSF Grant DMS-1701654. D. H. was supported by NSF Frontier Award1804794. Work by D. K. and Y. S. was partially done at OpenAI.
References
[1] A. Athalye, L. Engstrom, A. Ilyas, and K. Kwok. Synthesizing robust adversarial examples. CoRR,abs/1707.07397, 2017. URL http://arxiv.org/abs/1707.07397.
[2] A. Athalye, N. Carlini, and D. Wagner. Obfuscated gradients give a false sense of security: Circumventingdefenses to adversarial examples. arXiv preprint arXiv:1802.00420, 2018.
[3] A. Athalye, L. Engstrom, A. Ilyas, and K. Kwok. Synthesizing robust adversarial examples. In J. Dy andA. Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80of Proceedings of Machine Learning Research, pages 284–293, Stockholmsmässan, Stockholm Sweden,10–15 Jul 2018. PMLR. URL http://proceedings.mlr.press/v80/athalye18b.html.
[4] W. Brendel, J. Rauber, and M. Bethge. Decision-based adversarial attacks: Reliable attacks againstblack-box machine learning models. arXiv preprint arXiv:1712.04248, 2017.
[5] T. B. Brown, D. Mané, A. Roy, M. Abadi, and J. Gilmer. Adversarial patch. CoRR, abs/1712.09665, 2017.URL http://arxiv.org/abs/1712.09665.
[6] N. Carlini and D. Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposiumon Security and Privacy (SP), pages 39–57. IEEE, 2017.
[7] N. Carlini, A. Athalye, N. Papernot, W. Brendel, J. Rauber, D. Tsipras, I. G. Goodfellow, and A. Madry.On evaluating adversarial robustness: Principles of rigorous evaluations. 2019.
[8] N. Carlini, A. Athalye, N. Papernot, W. Brendel, J. Rauber, D. Tsipras, I. J. Goodfellow, A. Madry,and A. Kurakin. On evaluating adversarial robustness. CoRR, abs/1902.06705, 2019. URL http://arxiv.org/abs/1902.06705.
[9] P.-Y. Chen, Y. Sharma, H. Zhang, J. Yi, and C.-J. Hsieh. EAD: Elastic-net attacks to deep neural networksvia adversarial examples. In Thirty-second AAAI conference on artificial intelligence, 2018.
[10] K. T. Co, L. Muñoz-González, and E. C. Lupu. Sensitivity of deep convolutional networks to Gabor noise.CoRR, abs/1906.03455, 2019. URL http://arxiv.org/abs/1906.03455.
[11] J. M. Cohen, E. Rosenfeld, and J. Z. Kolter. Certified adversarial robustness via randomized smoothing.CoRR, abs/1902.02918, 2019. URL http://arxiv.org/abs/1902.02918.
[12] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A large-scale hierarchical imagedatabase. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. IEEE,2009.
[13] Y. Dong, T. Pang, H. Su, and J. Zhu. Evading defenses to transferable adversarial examples by translation-invariant attacks. In Proceedings of the IEEE Computer Society Conference on Computer Vision andPattern Recognition, 2019.
[14] L. Engstrom, B. Tran, D. Tsipras, L. Schmidt, and A. Madry. A rotation and a translation suffice: FoolingCNNs with simple transformations. arXiv preprint arXiv:1712.02779, 2017.
9
[15] I. Evtimov, K. Eykholt, E. Fernandes, T. Kohno, B. Li, A. Prakash, A. Rahmati, and D. X. Song. Robustphysical-world attacks on deep learning models. 2017.
[16] A. Fournier, D. Fussell, and L. Carpenter. Computer rendering of stochastic models. Commun. ACM, 25(6):371–384, June 1982. ISSN 0001-0782. doi: 10.1145/358523.358553. URL http://doi.acm.org/10.1145/358523.358553.
[17] M. Frank and P. Wolfe. An algorithm for quadratic programming. Naval research logistics quarterly, 3(1-2):95–110, 1956.
[18] R. Geirhos, P. Rubisch, C. Michaelis, M. Bethge, F. A. Wichmann, and W. Brendel. Imagenet-trained CNNsare biased towards texture; increasing shape bias improves accuracy and robustness. In International Con-ference on Learning Representations, 2019. URL https://openreview.net/forum?id=Bygh9j09KX.
[19] J. Gilmer, R. P. Adams, I. J. Goodfellow, D. Andersen, and G. E. Dahl. Motivating the rules of the gamefor adversarial example research. ArXiv, abs/1807.06732, 2018.
[20] I. Goodfellow. Gradient masking causes CLEVER to overestimate adversarial perturbation size. arXivpreprint arXiv:1804.07870, 2018.
[21] I. J. Goodfellow. A research agenda: Dynamic models to defend against correlated attacks. ArXiv,abs/1903.06293, 2019.
[22] I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. arXivpreprint arXiv:1412.6572, 2014.
[23] P. Goyal, P. Dollár, R. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia, and K. He.Accurate, large minibatch SGD: Training ImageNet in 1 hour. arXiv preprint arXiv:1706.02677, 2017.
[24] K. He, X. Zhang, S. Ren, and J. Sun. Identity mappings in deep residual networks. In European conferenceon computer vision, pages 630–645. Springer, 2016.
[25] D. Hendrycks and T. Dietterich. Benchmarking neural network robustness to common corruptions andperturbations. In International Conference on Learning Representations, 2019.
[26] D. Hendrycks, K. Zhao, S. Basart, J. Steinhardt, and D. Song. Natural adversarial examples. arXiv preprintarXiv:1907.07174, 2019.
[27] D. Hendrycks, N. Mu, E. D. Cubuk, B. Zoph, J. Gilmer, and B. Lakshminarayanan. AugMix: A simple dataprocessing method to improve robustness and uncertainty. Proceedings of the International Conference onLearning Representations (ICLR), 2020.
[28] F. N. Iandola, M. W. Moskewicz, K. Ashraf, S. Han, W. J. Dally, and K. Keutzer. Squeezenet: AlexNet-levelaccuracy with 50x fewer parameters and <1mb model size. ArXiv, abs/1602.07360, 2017.
[29] J.-H. Jacobsen, J. Behrmannn, N. Carlini, F. Tramèr, and N. Papernot. Exploiting excessive invariancecaused by norm-bounded adversarial robustness, 2019.
[30] M. Jordan, N. Manoj, S. Goel, and A. G. Dimakis. Quantifying perceptual distortion of adversarialexamples. arXiv e-prints, art. arXiv:1902.08265, Feb 2019.
[31] A. Lagae, S. Lefebvre, G. Drettakis, and P. Dutré. Procedural noise using sparse Gabor convolution. ACMTrans. Graph., 28(3):54:1–54:10, July 2009. ISSN 0730-0301. doi: 10.1145/1531326.1531360. URLhttp://doi.acm.org/10.1145/1531326.1531360.
[32] R. G. Lopes, D. Yin, B. Poole, J. Gilmer, and E. D. Cubuk. Improving robustness without sacrificingaccuracy with patch gaussian augmentation. ArXiv, abs/1906.02611, 2019.
[33] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu. Towards deep learning models resistant toadversarial attacks. arXiv preprint arXiv:1706.06083, 2017.
[34] D. Mahajan, R. Girshick, V. Ramanathan, K. He, M. Paluri, Y. Li, A. Bharambe, and L. van der Maaten.Exploring the limits of weakly supervised pretraining. In V. Ferrari, M. Hebert, C. Sminchisescu, andY. Weiss, editors, Computer Vision – ECCV 2018, pages 185–201, Cham, 2018. Springer InternationalPublishing. ISBN 978-3-030-01216-8.
[35] S.-M. Moosavi-Dezfooli, A. Fawzi, and P. Frossard. DeepFool: a simple and accurate method to fool deepneural networks. arXiv preprint arXiv:1511.04599, 2015.
10
[36] A. Nemirovski and D. Yudin. On Cezari’s convergence of the steepest descent method for approximatingsaddle point of convex-concave functions. In Soviet Math. Dokl, volume 19, pages 258–269, 1978.
[37] A. Nemirovski and D. Yudin. Problem Complexity and Method Efficiency in Optimization. Intersci. Ser.Discrete Math. Wiley, New York, 1983.
[38] A. E. Orhan. Robustness properties of Facebook’s ResNeXt WSL models. ArXiv, abs/1907.07640, 2019.
[39] N. Papernot, P. McDaniel, I. Goodfellow, S. Jha, Z. B. Celik, and A. Swami. Practical black-boxattacks against machine learning. In Proceedings of the 2017 ACM on Asia conference on computer andcommunications security, pages 506–519. ACM, 2017.
[40] H. Qian and M. N. Wegman. L2-nonexpansive neural networks. In International Conference on LearningRepresentations (ICLR), 2019. URL https://openreview.net/forum?id=ByxGSsR9FQ.
[41] C. Qin, J. Martens, S. Gowal, D. Krishnan, K. Dvijotham, A. Fawzi, S. De, R. Stanforth, and P. Kohli.Adversarial robustness through local linearization, 2019.
[42] H. Qiu, C. Xiao, L. Yang, X. Yan, H. Lee, and B. Li. Semanticadv: Generating adversarial examples viaattribute-conditional image editing. ArXiv, abs/1906.07927, 2019.
[43] E. Raff, J. Sylvester, S. Forsyth, and M. McLean. Barrage of random transforms for adversarially robustdefense. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages6528–6537, 2019.
[44] P. Rajpurkar, R. Jia, and P. Liang. Know what you don’t know: Unanswerable questions for SQuAD. InAssociation for Computational Linguistics (ACL), 2018.
[45] B. Recht, R. Roelofs, L. Schmidt, and V. Shankar. Do imagenet classifiers generalize to imagenet? InICML, 2019.
[46] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. S.Bernstein, A. C. Berg, and F.-F. Li. ImageNet large scale visual recognition challenge. InternationalJournal of Computer Vision, 115:211–252, 2014.
[47] L. Schott, J. Rauber, W. Brendel, and M. Bethge. Towards the first adversarially robust neural networkmodel on MNIST. May 2019. URL https://arxiv.org/pdf/1805.09190.pdf.
[48] M. Sharif, S. Bhagavatula, L. Bauer, and M. K. Reiter. Accessorize to a crime: Real and stealthy attackson state-of-the-art face recognition. In Proceedings of the 23rd ACM SIGSAC Conference on Computerand Communications Security, 2016.
[49] M. Sharif, S. Bhagavatula, L. Bauer, and M. K. Reiter. A general framework for adversarial examples withobjectives. ACM Transactions on Privacy and Security (TOPS), 22(3):1–30, 2019.
[50] R. Shin and D. Song. JPEG-resistant adversarial images. In NIPS 2017 Workshop on Machine Learningand Computer Security, 2017.
[51] Y. Song, R. Shu, N. Kushman, and S. Ermon. Constructing unrestricted adversarial examples withgenerative models. In NeurIPS, 2018.
[52] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus. Intriguingproperties of neural networks. arXiv preprint arXiv:1312.6199, 2013.
[53] F. Tramèr and D. Boneh. Adversarial training and robustness for multiple perturbations. arXiv e-prints, art.arXiv:1904.13000, Apr 2019.
[54] F. Tramèr, P. Dupré, G. Rusak, G. Pellegrino, and D. Boneh. Ad-versarial: Defeating perceptual ad-blocking.CoRR, abs/1811.03194, 2018. URL http://arxiv.org/abs/1811.03194.
[55] T.-W. Weng, H. Zhang, P.-Y. Chen, J. Yi, D. Su, Y. Gao, C.-J. Hsieh, and L. Daniel. Evaluating therobustness of neural networks: An extreme value theory approach. arXiv preprint arXiv:1801.10578, 2018.
[56] T. Wu, L. Tong, and Y. Vorobeychik. Defending against physically realizable attacks on image classification.In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=H1xscnEKDr.
[57] C. Xiao, J.-Y. Zhu, B. Li, W. He, M. Liu, and D. Song. Spatially transformed adversarial examples. arXivpreprint arXiv:1801.02612, 2018.
11
[58] C. Xie, Y. Wu, L. v. d. Maaten, A. Yuille, and K. He. Feature denoising for improving adversarialrobustness. arXiv preprint arXiv:1812.03411, 2018.
[59] S. Xie, R. B. Girshick, P. Dollár, Z. Tu, and K. He. Aggregated residual transformations for deep neuralnetworks. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5987–5995,2016.
[60] H. Zhang, Y. Yu, J. Jiao, E. Xing, L. E. Ghaoui, and M. Jordan. Theoretically principled trade-offbetween robustness and accuracy. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36thInternational Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research,pages 7472–7482, Long Beach, California, USA, 09–15 Jun 2019. PMLR. URL http://proceedings.mlr.press/v97/zhang19p.html.
[61] Z. Zhao, Z. Liu, and M. Larson. Towards large yet imperceptible adversarial image perturbations withperceptual color distance. ArXiv, abs/1911.02466, 2019.
12
A Training hyperparameters
For ImageNet-100, we trained on machines with 8 NVIDIA V100 GPUs using standard data aug-mentation [24]. Following best practices for multi-GPU training [23], we ran synchronized SGD for90 epochs with batch size 32×8 and a learning rate schedule with 5 “warm-up” epochs and a decayat epochs 30, 60, and 80 by a factor of 10. Initial learning rate after warm-up was 0.1, momentumwas 0.9, and weight decay was 10−4. For CIFAR-10, we trained on a single NVIDIA V100 GPU for200 epochs with batch size 32, initial learning rate 0.1, momentum 0.9, and weight decay 10−4. Wedecayed the learning rate at epochs 100 and 150.
B Calibration of ImageNet-UA and CIFAR-10-UA
B.1 Calibration for ImageNet-UA
Calibrated distortion sizes and ATA values are in Table 4.
B.2 Calibration for CIFAR-10-UA
The ε calibration procedure for CIFAR-10 was similar to that used for ImageNet-100. We startedwith small εmin values and increased ε geometrically with ratio 2 until adversarial accuracy of anadversarially trained model dropped below 40. Note that this threshold is higher for CIFAR-10 thanImageNet-100 because there are fewer classes. The resulting ATA values for CIFAR-10 are shown inTable 5.
C Joint adversarial training
Our joint adversarial training procedure for two attacks A and A′ is as follows. At each training step,we compute the attacked image under both A and A′ and backpropagate with respect to gradientsinduced by the image with greater loss. This corresponds to the “max” loss of [53]. We trainResNet-50 models for (L∞, L2), (L∞, L1), and (L∞,Elastic) on ImageNet-100.
Table 6 shows training against (L∞, L1) is worse than training against L1 at the same distortionsize and performs particularly poorly at large distortion sizes. Table 7 shows joint training against(L∞,Elastic) also performs poorly, never matching the UAR score of training against Elastic atmoderate distortion size (ε = 2).
Table 4: Calibrated distortion sizes and ATA values for different distortion types on ImageNet-100.Attack ε1 ε2 ε3 ε4 ε5 ε6 ATA1 ATA2 ATA3 ATA4 ATA5 ATA6
L∞ 1 2 4 8 16 32 84.6 82.1 76.2 66.9 40.1 12.9L2 150 300 600 1200 2400 4800 85.0 83.5 79.6 72.6 59.1 19.9L1 9562.5 19125 76500 153000 306000 612000 84.4 82.7 76.3 68.9 56.4 36.1Elastic 0.25 0.5 2 4 8 16 85.9 83.2 78.1 75.6 57.0 22.5JPEG 0.062 0.125 0.250 0.500 1 2 85.0 83.2 79.3 72.8 34.8 1.1Fog 128 256 512 2048 4096 8192 85.8 83.8 79.0 68.4 67.9 64.7Snow 0.0625 0.125 0.25 2 4 8 84.0 81.1 77.7 65.6 59.5 41.2Gabor 6.25 12.5 25 400 800 1600 84.0 79.8 79.8 66.2 44.7 14.6
Table 5: Calibrated distortion sizes and ATA values for ResNet-56 on CIFAR-10
Attack ε1 ε2 ε3 ε4 ε5 ε6 ATA1 ATA2 ATA3 ATA4 ATA5 ATA6
L∞ 1 2 4 8 16 32 91.0 87.8 81.6 71.3 46.5 23.1L2 40 80 160 320 640 2560 90.1 86.4 79.6 67.3 49.9 17.3L1 195 390 780 1560 6240 24960 92.2 90.0 83.2 73.8 47.4 35.3JPEG 0.03125 0.0625 0.125 0.25 0.5 1 89.7 87.0 83.1 78.6 69.7 35.4Elastic 0.125 0.25 0.5 1 2 8 87.4 81.3 72.1 58.2 45.4 27.8
13
Table 6: UAR scores for L1-trained models and (L∞, L1)-jointly trained models. At each distortionsize, L1-training performs better than joint training.
UARL∞ UARL1
L∞ ε = 2, L1 ε = 76500 48 66L∞ ε = 4, L1 ε = 153000 51 72L∞ ε = 8, L1 ε = 306000 44 62L1 ε = 76500 50 70L1 ε = 153000 54 81L1 ε = 306000 59 87
Table 7: UAR scores for L∞- and Elastic-trained models and (L∞, Elastic)-jointly trained models.No jointly trained model matches a Elastic-trained model on UAR vs. Elastic.
UARL∞ UARElasticL∞ ε = 4, Elastic ε = 2 68 63L∞ ε = 8, Elastic ε = 4 35 65L∞ ε = 16, Elastic ε = 8 69 43Elastic ε = 2 37 68Elastic ε = 4 36 81Elastic ε = 8 31 91
D The Frank-Wolfe Algorithm
We chose to use the Frank-Wolfe algorithm for optimizing the L1 attack, as Projected GradientDescent would require projecting onto a truncated L1 ball, which is a complicated operation. Incontrast, Frank-Wolfe only requires optimizing linear functions g>x over a truncated L1 ball; thiscan be done by sorting coordinates by the magnitude of g and moving the top k coordinates to theboundary of their range (with k chosen by binary search). This is detailed in Algorithm 1.
E Full evaluation results
E.1 Full evaluation results and analysis for ImageNet-100
We show the full results of all adversarial attacks against all adversarial defenses for ImageNet-100in Figure 6. These results also include L1-JPEG and L2-JPEG attacks, which are modifications ofthe JPEG attack applying Lp-constraints in the compressed JPEG space instead of L∞ constraints.Full UAR scores are provided for ImageNet-100 in Figure 7.
E.2 Full evaluation results and analysis for CIFAR-10
We show the results of adversarial attacks and defenses for CIFAR-10 in Figure 8. We experienceddifficulty training the L2 and L1 attacks at distortion sizes greater than those shown and have omittedthose runs, which we believe may be related to the small size of CIFAR-10 images. Full UAR valuesfor CIFAR-10 are shown in Figure 9.
F Robustness of our results
F.1 Replication
We replicated our results for the first three rows of Figure 6 with different random seeds to see thevariation in our results. As shown in Figure 10, deviations in results are minor.
F.2 Convergence
We replicated the results in Figure 6 with 50 instead of 200 steps to see how the results changedbased on the number of steps in the attack. As shown in Figure 11, the deviations are minor.
14
No atta
ckL
=1L
=2L
=4L
=8L
=16L
=32 L2 =15
0L2
=300
L2 =60
0L2
=1200
L2 =24
00L2
=4800
L1 =95
62.44
L1 =19
125
L1 =38
250.1
L1 =76
500
L1 =15
3000
L1 =30
6000
L1 =61
2000
L-JP
EG =0.0
3125
L-JP
EG =0.0
625
L-JP
EG =0.1
25
L-JP
EG =0.2
5
L-JP
EG =0.5
L-JP
EG =1
L-JP
EG =2
L2-JP
EG =2
L2-JP
EG =4
L2-JP
EG =8
L2-JP
EG =16
L2-JP
EG =32
L2-JP
EG =64
L2-JP
EG =12
8
L2-JP
EG =25
6L1
-JPEG
=128
L1-JP
EG =25
6
L1-JP
EG =51
2
L1-JP
EG =10
24
L1-JP
EG =20
48
L1-JP
EG =40
96
L1-JP
EG =81
92
L1-JP
EG =16
384
L1-JP
EG =32
768
L1-JP
EG =65
536
L1-JP
EG =13
1072
Elastic
=0.25
Elastic
=0.5
Elastic
=1
Elastic
=2
Elastic
=4
Elastic
=8
Elastic
=16 Fog =12
8
Fog =25
6
Fog =51
2
Fog =10
24
Fog =20
48
Fog =40
96
Fog =81
92
Fog =16
384
Fog =32
768
Fog =65
536
Gabor
=6.25
Gabor
=12.5
Gabor
=25
Gabor
=50
Gabor
=100
Gabor
=200
Gabor
=400
Gabor
=800
Gabor
=1600
Gabor
=3200
Snow
=0.031
25
Snow
=0.062
5
Snow
=0.125
Snow
=0.25
Snow
=0.5Sn
ow =1
Snow
=2Sn
ow =4
Snow
=8
Snow
=16
Norm
al tr
aini
ng
L
=1
L
=2
L
=4
L
=8
L
=16
L
=32
L 2
=15
0L 2
=
300
L 2
=60
0L 2
=
1200
L 2
=24
00L 2
=
4800
L 1
=95
62.4
4L 1
=
1912
5L 1
=
3825
0.1
L 1
=76
500
L 1
=15
3000
L 1
=30
6000
L 1
=61
2000
L-JP
EG
=0.
0312
5L
-JPEG
=
0.06
25L
-JPEG
=
0.12
5L
-JPEG
=
0.25
L-JP
EG
=0.
5L
-JPEG
=
1L
-JPEG
=
2
L 2-JP
EG
=2
L 2-JP
EG
=4
L 2-JP
EG
=8
L 2-JP
EG
=16
L 2-JP
EG
=32
L 2-JP
EG
=64
L 2-JP
EG
=12
8L 2
-JPEG
=
256
L 1-JP
EG
=12
8L 1
-JPEG
=
256
L 1-JP
EG
=51
2L 1
-JPEG
=
1024
L 1-JP
EG
=20
48L 1
-JPEG
=
4096
L 1-JP
EG
=81
92L 1
-JPEG
=
1638
4L 1
-JPEG
=
3276
8L 1
-JPEG
=
6553
6L 1
-JPEG
=
1310
72
Elas
tic
=0.
25El
astic
=
0.5
Elas
tic
=1
Elas
tic
=2
Elas
tic
=4
Elas
tic
=8
Elas
tic
=16
Fog
=12
8Fo
g =
256
Fog
=51
2Fo
g =
1024
Fog
=20
48Fo
g =
4096
Fog
=81
92Fo
g =
1638
4Fo
g =
3276
8Fo
g =
6553
6
Gabo
r =
6.25
Gabo
r =
12.5
Gabo
r =
25Ga
bor
=50
Gabo
r =
100
Gabo
r =
200
Gabo
r =
400
Gabo
r =
800
Gabo
r =
1600
Gabo
r =
3200
Snow
=
0.03
125
Snow
=
0.06
25Sn
ow
=0.
125
Snow
=
0.25
Snow
=
0.5
Snow
=
1Sn
ow
=2
Snow
=
4Sn
ow
=8
Snow
=
16
8725
1 0
0 0
057
11 0
0 0
061
29 5
0 0
0 0
20 1
0 0
0 0
070
25 1
0 0
0 0
050
20 3
0 0
0 0
0 0
0 0
7947
6 0
0 0
054
15 1
0 0
0 0
0 0
012
3 1
0 0
0 0
0 0
064
32 7
1 0
0 0
0 0
0
8684
7014
0 0
086
8148
2 0
080
6635
5 0
0 0
8471
13 0
0 0
086
8466
10 0
0 0
076
6028
5 0
0 0
0 0
0 0
8475
36 3
0 0
074
47 9
0 0
0 0
0 0
074
28 4
1 0
0 0
0 0
080
6829
4 0
0 0
0 0
085
8581
50 2
0 0
8583
7118
0 0
8172
5218
1 0
084
8148
1 0
0 0
8584
7734
1 0
0 0
7969
4616
2 0
0 0
0 0
084
7852
7 0
0 0
7347
10 0
0 0
0 0
0 0
8159
12 1
0 0
0 0
0 0
7973
44 9
1 0
0 0
0 0
8483
8274
22 0
084
8379
48 2
080
7561
32 6
0 0
8482
6910
0 0
084
8379
54 5
0 0
079
7358
31 7
1 0
0 0
0 0
8379
6215
1 0
070
43 9
1 0
0 0
0 0
082
7537
3 0
0 0
0 0
079
7456
18 3
1 0
0 0
080
7979
7659
6 0
7978
7350
7 0
7264
4927
6 1
079
7759
12 0
0 0
7978
7043
6 0
0 0
7467
5433
11 2
0 0
0 0
079
7666
31 2
0 0
6333
6 1
1 0
0 0
1 0
7977
6417
1 0
0 0
0 0
7472
6231
9 2
0 0
0 0
7574
7473
6734
173
7163
30 3
058
4324
8 1
0 0
7367
37 3
0 0
073
7059
26 3
0 0
064
5642
2511
3 0
0 0
0 0
7371
6642
11 1
055
25 4
1 1
0 0
0 0
074
7368
44 4
0 0
0 0
068
6659
4017
6 1
0 0
071
7170
6962
40 8
6960
33 5
0 0
3721
8 2
0 0
065
42 8
0 0
0 0
6861
34 7
1 0
0 0
5647
3623
11 4
1 0
0 0
070
6862
4415
2 0
5529
5 1
0 0
0 0
0 0
7069
6349
16 2
2 1
0 0
6360
5437
17 5
1 0
0 0
8782
53 3
0 0
085
7834
1 0
080
6936
5 0
0 0
8151
2 0
0 0
085
8148
2 0
0 0
073
4815
2 0
0 0
0 0
0 0
8371
28 1
0 0
073
4610
0 0
0 0
0 0
062
15 3
0 0
0 0
0 0
078
6020
3 0
0 0
0 0
085
8474
22 0
0 0
8582
65 8
0 0
8276
5718
1 0
084
7522
0 0
0 0
8583
7317
0 0
0 0
7866
35 7
0 0
0 0
0 0
083
7643
3 0
0 0
7447
11 0
0 0
0 0
0 0
7739
6 1
0 0
0 0
0 0
7967
33 5
0 0
0 0
0 0
8484
8156
4 0
084
8377
40 1
083
8071
44 9
0 0
8481
60 3
0 0
085
8480
49 2
0 0
082
7559
24 3
0 0
0 0
0 0
8379
56 9
0 0
072
4511
1 0
0 0
0 0
081
6517
2 0
0 0
0 0
078
7246
11 1
0 0
0 0
082
8281
7428
0 0
8282
8069
15 0
8280
7765
32 4
082
8176
34 0
0 0
8282
8174
25 0
0 0
8180
7356
24 4
0 0
0 0
081
7866
23 1
0 0
6740
8 1
0 0
0 0
0 0
8175
42 5
0 0
0 0
0 0
7672
5722
3 1
0 0
0 0
7777
7674
56 6
077
7776
7350
277
7675
7157
23 1
7776
7563
9 0
077
7676
7457
7 0
077
7673
6852
25 5
1 0
0 0
7674
6838
4 0
059
30 6
1 1
1 0
1 0
076
7461
16 1
0 0
0 0
070
6858
3510
2 0
0 0
068
6868
6761
28 1
6868
6867
5920
6968
6866
6144
1368
6867
6437
2 0
6868
6867
6233
1 0
6868
6866
6149
3012
4 2
368
6663
4711
1 0
4923
5 1
1 1
0 1
0 0
6867
6130
3 1
1 0
0 0
6058
5440
17 5
2 1
0 0
8671
24 1
0 0
082
6414
0 0
083
7753
13 0
0 0
6718
0 0
0 0
084
7122
1 0
0 0
071
4411
1 0
0 0
0 0
0 0
8266
19 1
0 0
068
33 3
0 0
0 0
0 0
041
9 2
0 0
0 0
0 0
073
5014
2 0
0 0
0 0
086
7841
3 0
0 0
8474
32 1
0 0
8481
6832
3 0
076
41 2
0 0
0 0
8479
48 3
0 0
0 0
7863
28 5
0 0
0 0
0 0
083
7128
1 0
0 0
6936
4 0
0 0
0 0
0 0
5617
3 0
0 0
0 0
0 0
7559
22 2
0 0
0 0
0 0
8581
6211
0 0
084
8055
6 0
084
8277
5412
0 0
8164
14 0
0 0
084
8268
20 0
0 0
081
7551
18 3
0 0
0 0
0 0
8275
41 3
0 0
070
39 6
0 0
0 0
0 0
071
32 5
1 0
0 0
0 0
077
6430
4 0
0 0
0 0
084
8271
28 1
0 0
8381
6720
0 0
8483
8172
40 4
082
7545
3 0
0 0
8482
7648
4 0
0 0
8380
7247
14 2
0 0
0 0
082
7752
6 0
0 0
6837
5 0
0 0
0 0
0 0
7750
12 1
0 0
0 0
0 0
7667
34 8
1 0
0 0
0 0
8179
7243
3 0
080
7869
35 2
081
8079
7664
26 1
7974
6017
0 0
080
7976
6216
0 0
080
8078
7145
12 1
0 0
0 0
7975
5712
0 0
065
36 5
0 0
0 0
0 0
077
6123
3 0
0 0
0 0
073
6644
12 1
0 0
0 0
079
7772
5310
0 0
7877
7146
6 0
7978
7875
6945
777
7466
40 3
0 0
7877
7568
41 2
0 0
7878
7774
6436
7 1
0 0
077
7362
21 1
0 0
6132
4 0
0 0
0 0
0 0
7665
35 6
0 0
0 0
0 0
7065
4920
4 1
0 0
0 0
7271
6959
24 1
072
7169
5518
072
7271
7067
5524
7169
6551
15 0
071
7170
6753
13 0
071
7170
6964
5224
6 1
0 0
7068
6133
3 0
050
24 5
0 0
0 0
0 0
069
6242
12 1
0 0
0 0
060
5646
24 8
2 0
0 0
1
8775
28 1
0 0
083
58 7
0 0
075
5014
1 0
0 0
8683
54 2
0 0
086
8682
55 3
0 0
083
8067
36 7
0 0
0 0
0 0
8365
14 0
0 0
067
28 2
0 0
0 0
0 0
034
6 1
0 0
0 0
0 0
072
4411
1 0
0 0
0 0
087
8047
3 0
0 0
8471
19 0
0 0
7760
24 2
0 0
086
8475
17 0
0 0
8786
8474
18 0
0 0
8582
7657
21 2
0 0
0 0
083
7020
1 0
0 0
6933
3 0
0 0
0 0
0 0
5111
2 0
0 0
0 0
0 0
7551
13 1
0 0
0 0
0 0
8683
6814
0 0
084
7943
2 0
080
6737
7 0
0 0
8585
8357
1 0
086
8685
8151
1 0
085
8381
7247
14 1
0 0
0 0
8373
28 1
0 0
066
30 3
0 0
0 0
0 0
069
27 4
1 0
0 0
0 0
077
5920
3 0
0 0
0 0
084
8377
42 3
0 0
8381
6613
0 0
8073
5318
2 0
084
8483
7714
0 0
8484
8482
7317
0 0
8483
8278
6843
13 2
0 0
082
7540
3 0
0 0
6530
3 0
0 0
0 0
0 0
7852
13 1
0 0
0 0
0 0
7664
29 5
1 0
0 0
0 0
8180
7866
17 1
080
7974
41 3
078
7462
35 7
0 0
8181
8079
64 0
081
8181
8077
57 2
080
8080
7976
6952
28 9
2 1
8076
52 7
0 0
063
30 3
0 0
0 0
0 0
079
6935
6 1
0 0
0 0
074
6746
11 2
0 0
0 0
079
7977
6827
1 0
7978
7450
7 0
7775
6850
21 3
080
7979
7773
34 0
8079
7979
7668
32 0
8079
7979
7876
7368
6148
4878
7558
12 0
0 0
6129
3 0
0 0
0 0
0 0
7871
4712
1 0
0 0
0 0
7167
4817
4 1
0 0
0 0
7877
7663
19 1
078
7773
47 5
077
7568
4817
2 0
7878
7775
6235
178
7878
7876
6745
1778
7878
7777
7573
6965
5547
7774
5711
0 0
060
29 4
0 0
0 0
0 0
077
7044
14 2
0 0
0 0
070
6545
16 4
1 0
0 0
1
8764
12 0
0 0
080
44 2
0 0
072
4310
1 0
0 0
8571
15 0
0 0
086
8574
23 0
0 0
083
7654
17 1
0 0
0 0
0 0
8259
10 0
0 0
064
24 2
0 0
0 0
0 0
022
4 1
0 0
0 0
0 0
070
40 9
1 0
0 0
0 0
087
7526
1 0
0 0
8359
8 0
0 0
7552
16 1
0 0
086
8249
1 0
0 0
8686
8256
3 0
0 0
8481
7041
8 0
0 0
0 0
082
6515
0 0
0 0
6727
2 0
0 0
0 0
0 0
33 6
1 0
0 0
0 0
0 0
7444
10 1
0 0
0 0
0 0
8681
50 4
0 0
084
7424
1 0
079
6428
3 0
0 0
8685
7415
0 0
086
8684
7624
0 0
085
8379
6428
3 0
0 0
0 0
8369
19 1
0 0
068
33 3
0 0
0 0
0 0
050
11 2
0 0
0 0
0 0
075
5214
1 0
0 0
0 0
085
8370
15 0
0 0
8480
50 3
0 0
8173
4610
0 0
086
8582
56 1
0 0
8685
8582
61 3
0 0
8584
8378
6024
2 0
0 0
083
7330
1 0
0 0
6935
4 0
0 0
0 0
0 0
6825
4 0
0 0
0 0
0 0
7759
20 3
0 0
0 0
0 0
8483
7844
3 0
083
8270
18 0
081
7762
25 2
0 0
8484
8375
14 0
084
8484
8276
29 0
084
8383
8175
5722
3 0
0 0
8276
42 3
0 0
068
35 4
0 0
0 0
0 0
077
4710
1 0
0 0
0 0
077
6732
5 1
0 0
0 0
081
8179
6717
0 0
8180
7649
4 0
8077
7045
11 1
081
8181
7958
0 0
8181
8180
7965
4 0
8181
8180
7873
6034
10 2
180
7654
9 0
0 0
6634
5 0
0 0
0 0
0 0
7968
31 4
0 0
0 0
0 0
7569
4513
2 0
0 0
0 0
7778
7672
40 3
078
7775
6316
077
7571
5726
3 0
7877
7776
7119
078
7877
7776
7239
077
7777
7776
7572
6755
3832
7775
6219
1 0
061
29 4
0 0
0 0
0 0
076
7248
11 1
0 0
0 0
071
6855
23 6
1 0
0 0
177
7776
7136
2 0
7777
7562
15 0
7776
7159
30 5
078
7777
7671
40 0
7878
7877
7672
47 3
7878
7777
7775
7470
6356
5777
7564
19 1
0 0
6029
4 0
0 0
0 0
0 0
7671
4713
1 0
0 0
0 0
6966
5325
7 1
0 0
0 1
8766
15 0
0 0
081
51 5
0 0
077
5517
1 0
0 0
8369
16 0
0 0
086
8579
40 1
0 0
085
8375
4812
0 0
0 0
0 0
8158
10 0
0 0
064
24 1
0 0
0 0
0 0
024
4 1
0 0
0 0
0 0
071
40 8
1 0
0 0
0 0
086
7427
1 0
0 0
8363
11 0
0 0
7963
26 2
0 0
085
7838
1 0
0 0
8686
8261
6 0
0 0
8584
8063
26 2
0 0
0 0
081
6313
0 0
0 0
6830
2 0
0 0
0 0
0 0
34 7
1 0
0 0
0 0
0 0
7443
10 1
0 0
0 0
0 0
8679
47 3
0 0
084
7426
1 0
081
7138
5 0
0 0
8582
64 8
0 0
086
8584
7527
0 0
085
8482
7549
10 0
0 0
0 0
8267
18 0
0 0
068
30 3
0 0
0 0
0 0
048
11 2
0 0
0 0
0 0
076
5214
2 0
0 0
0 0
086
8366
12 0
0 0
8480
51 3
0 0
8275
5313
1 0
085
8477
36 0
0 0
8686
8581
57 3
0 0
8685
8480
6634
4 0
0 0
083
7329
2 0
0 0
6934
4 0
0 0
0 0
0 0
6321
3 0
0 0
0 0
0 0
7861
19 2
0 0
0 0
0 0
8482
7534
1 0
084
8167
13 0
082
7863
27 2
0 0
8484
8166
6 0
085
8484
8273
22 0
084
8483
8175
5722
2 0
0 0
8275
39 3
0 0
069
35 5
0 0
0 0
0 0
074
39 7
1 0
0 0
0 0
078
6631
4 0
0 0
0 0
083
8279
55 5
0 0
8381
7534
1 0
8178
6939
6 0
083
8382
7632
0 0
8383
8382
7850
1 0
8383
8382
7971
4918
3 0
081
7650
6 0
0 0
6634
5 0
0 0
0 0
0 0
7956
16 2
0 0
0 0
0 0
7668
38 7
1 0
0 0
0 0
8180
7864
14 0
080
8076
48 3
080
7770
4812
1 0
8180
8077
54 1
081
8180
8079
64 6
080
8180
8079
7567
4517
4 4
7975
55 9
0 0
065
33 4
0 0
0 0
0 0
078
6726
3 0
0 0
0 0
074
6947
12 2
0 0
0 0
080
7978
7026
1 0
7979
7658
8 0
7876
7153
18 1
080
7979
7766
3 0
8079
7979
7870
17 0
7979
7979
7876
7362
4119
1779
7658
12 0
0 0
6228
3 0
0 0
0 0
0 0
7871
39 6
0 0
0 0
0 0
7369
5017
3 1
0 0
0 0
7777
7771
33 1
078
7775
6011
077
7570
5622
2 0
7878
7777
68 6
078
7878
7776
7022
078
7777
7777
7573
6551
3126
7774
6015
1 0
059
27 3
0 0
0 0
0 0
077
7244
9 1
0 0
0 0
071
6750
21 4
1 0
0 0
076
7574
6937
2 0
7575
7359
13 0
7472
6854
22 2
075
7575
7468
9 0
7575
7574
7468
24 0
7575
7475
7473
7267
5946
4274
7157
16 1
0 0
5222
2 0
0 0
0 0
0 0
7570
5214
1 0
0 0
0 0
6663
4921
5 1
0 0
0 1
7273
7165
29 2
073
7269
5410
072
7065
5019
2 0
7373
7272
64 5
073
7373
7271
6418
073
7373
7272
7169
6455
4137
7269
5413
1 0
047
17 2
0 0
0 0
0 0
072
6744
11 1
0 0
0 0
063
5943
16 3
1 0
0 0
1
8763
14 0
0 0
079
44 4
0 0
072
4814
2 0
0 0
6414
0 0
0 0
082
5811
0 0
0 0
062
33 9
1 0
0 0
0 0
0 0
8578
45 2
0 0
070
35 4
0 0
0 0
0 0
039
10 2
1 0
0 0
0 0
075
5220
3 0
0 0
0 0
087
7325
1 0
0 0
8258
10 0
0 0
7658
25 4
0 0
076
34 1
0 0
0 0
8471
25 1
0 0
0 0
6743
17 3
0 0
0 0
0 0
086
8369
15 0
0 0
7242
8 0
0 0
0 0
0 0
5621
5 1
0 0
0 0
0 0
7661
30 8
1 0
0 0
0 0
8577
40 3
0 0
082
6721
1 0
077
6335
9 1
0 0
8054
9 0
0 0
083
7641
4 0
0 0
067
4722
6 1
0 0
0 0
0 0
8483
7851
3 0
072
4711
0 0
0 0
0 0
071
4214
3 0
0 0
0 0
076
6843
16 4
1 0
0 0
084
7849
7 0
0 0
8171
30 2
0 0
7562
3711
1 0
079
5611
0 0
0 0
8276
45 8
0 0
0 0
6852
29 9
2 0
0 0
0 0
084
8381
7332
1 0
7350
15 1
0 0
0 0
0 0
7655
22 5
1 0
0 0
0 0
7669
5126
9 2
1 1
0 0
8174
47 8
0 0
078
6727
2 0
066
4925
7 1
0 0
7036
4 0
0 0
078
6831
4 0
0 0
061
4221
7 1
0 0
0 0
0 0
8180
7978
7117
068
4613
1 0
0 0
0 0
075
5932
9 2
1 1
1 0
072
6855
3415
5 3
1 1
178
6939
5 0
0 0
7354
15 1
0 0
4728
10 2
0 0
053
13 1
0 0
0 0
7046
12 1
0 0
0 0
4527
11 3
0 0
0 0
0 0
078
7676
7576
57 4
6342
13 1
0 0
0 0
0 0
7159
3512
3 1
1 1
0 0
6663
5436
19 9
5 2
1 1
7458
22 2
0 0
063
32 5
0 0
031
14 4
1 0
0 0
27 3
0 0
0 0
055
22 2
0 0
0 0
032
16 4
1 0
0 0
0 0
0 0
7473
7271
7057
2160
3812
1 0
0 0
0 0
068
5229
10 3
1 1
1 0
063
6052
3824
14 9
4 1
1
8740
2 0
0 0
069
16 0
0 0
062
26 4
0 0
0 0
23 1
0 0
0 0
074
27 1
0 0
0 0
049
18 2
0 0
0 0
0 0
0 0
8154
9 0
0 0
083
6933
3 0
0 0
0 0
019
3 1
0 0
0 0
0 0
077
4712
1 0
0 0
0 0
088
40 2
0 0
0 0
6817
0 0
0 0
6024
3 0
0 0
017
1 0
0 0
0 0
7122
1 0
0 0
0 0
4515
2 0
0 0
0 0
0 0
082
5812
0 0
0 0
8578
5921
2 0
0 0
0 0
25 5
1 0
0 0
0 0
0 0
8057
19 2
0 0
0 0
0 0
8729
1 0
0 0
059
12 0
0 0
054
19 2
0 0
0 0
9 0
0 0
0 0
057
11 0
0 0
0 0
030
6 0
0 0
0 0
0 0
0 0
8261
14 1
0 0
086
8274
5116
2 1
1 1
131
6 1
0 0
0 0
0 0
081
6527
4 0
0 0
0 0
086
23 1
0 0
0 0
50 9
0 0
0 0
4916
2 0
0 0
0 6
0 0
0 0
0 0
45 8
0 0
0 0
0 0
29 7
1 0
0 0
0 0
0 0
082
6316
1 0
0 0
8684
7867
4316
5 2
1 1
40 9
2 0
0 0
0 0
0 0
8268
39 8
1 0
0 0
0 0
8517
1 0
0 0
042
6 0
0 0
046
16 2
0 0
0 0
4 0
0 0
0 0
035
5 0
0 0
0 0
027
7 1
0 0
0 0
0 0
0 0
8163
19 1
0 0
084
8379
7156
3416
7 3
152
16 3
1 0
0 0
0 0
080
7244
11 2
0 0
0 0
078
8 0
0 0
0 0
24 2
0 0
0 0
26 5
0 0
0 0
0 2
0 0
0 0
0 0
23 2
0 0
0 0
0 0
15 3
0 0
0 0
0 0
0 0
074
6019
1 0
0 0
7978
7672
6758
4629
10 3
6234
9 2
0 0
0 0
0 0
7569
5019
4 1
0 0
0 0
69 4
0 0
0 0
012
1 0
0 0
025
8 2
0 0
0 0
4 0
0 0
0 0
012
2 0
0 0
0 0
0 5
1 0
0 0
0 0
0 0
0 0
6549
11 0
0 0
070
7070
6868
6865
5739
1768
6663
4815
2 3
1 0
067
6560
4930
12 2
0 0
062
3 0
0 0
0 0
9 1
0 0
0 0
18 6
1 0
0 0
0 3
0 0
0 0
0 0
8 1
0 0
0 0
0 0
4 1
0 0
0 0
0 0
0 0
056
38 6
0 0
0 0
6465
6565
6464
6462
5333
6261
5952
27 6
3422
10 2
6159
5648
3619
6 1
0 0
51 3
0 0
0 0
0 8
1 0
0 0
022
9 2
0 0
0 0
4 0
0 0
0 0
0 8
1 0
0 0
0 0
0 5
2 0
0 0
0 0
0 0
0 0
4726
3 0
0 0
056
5757
5757
5755
5141
2354
5452
4521
427
1910
353
5146
3928
14 3
1 0
042
5 1
0 0
0 0
10 2
0 0
0 0
20 9
3 1
0 0
0 6
1 0
0 0
0 0
12 3
0 0
0 0
0 0
7 3
1 0
0 0
0 0
0 0
041
26 5
0 0
0 0
5150
5150
5049
4845
3928
4646
4538
19 4
2618
8 2
4744
4034
2211
3 0
0 0
8655
9 0
0 0
075
33 2
0 0
066
3911
1 0
0 0
35 2
0 0
0 0
064
16 1
0 0
0 0
030
8 1
0 0
0 0
0 0
0 0
8373
34 2
0 0
074
4912
0 0
0 0
0 0
082
6622
4 0
0 0
0 0
080
7141
9 1
0 0
0 0
085
37 3
0 0
0 0
6417
1 0
0 0
6029
6 1
0 0
016
1 0
0 0
0 0
42 5
0 0
0 0
0 0
15 3
0 0
0 0
0 0
0 0
083
7438
2 0
0 0
7552
17 1
0 0
0 0
0 0
8479
5614
1 0
0 0
0 0
8075
5418
3 1
0 0
0 0
8524
2 0
0 0
051
10 0
0 0
050
22 5
1 0
0 0
8 0
0 0
0 0
032
3 0
0 0
0 0
012
3 0
0 0
0 0
0 0
0 0
8273
37 3
0 0
072
4912
0 0
0 0
0 0
083
8072
41 7
0 0
0 0
079
7457
29 6
1 0
0 0
084
21 2
0 0
0 0
48 9
0 0
0 0
4919
4 1
0 0
0 8
0 0
0 0
0 0
31 3
0 0
0 0
0 0
12 3
0 0
0 0
0 0
0 0
081
7440
3 0
0 0
7144
9 0
0 0
0 0
0 0
8280
8064
23 2
1 1
0 0
7871
5730
11 2
1 0
0 0
8323
2 0
0 0
050
9 0
0 0
045
18 3
0 0
0 0
9 0
0 0
0 0
038
5 0
0 0
0 0
017
4 1
0 0
0 0
0 0
0 0
8072
39 4
0 0
070
43 6
0 0
0 0
0 0
081
7773
6961
12 9
2 0
076
7052
2710
2 0
0 0
083
33 3
0 0
0 0
5713
1 0
0 0
4517
3 0
0 0
014
1 0
0 0
0 0
5112
1 0
0 0
0 0
27 9
2 0
0 0
0 0
0 0
080
7238
4 0
0 0
6839
5 0
0 0
0 0
0 0
8076
7066
6253
21 2
0 0
7670
5225
6 1
0 0
0 0
8234
4 0
0 0
058
15 1
0 0
041
15 2
0 0
0 0
16 1
0 0
0 0
058
16 1
0 0
0 0
029
8 1
0 0
0 0
0 0
0 0
7971
42 6
0 0
068
40 6
0 0
0 0
0 0
077
7367
6157
4729
3 0
075
6951
25 7
1 0
0 0
081
37 4
0 0
0 0
5915
1 0
0 0
4015
2 0
0 0
017
1 0
0 0
0 0
5916
1 0
0 0
0 0
3210
2 0
0 0
0 0
0 0
078
7043
9 0
0 0
6943
9 0
0 0
0 0
0 0
7872
6457
5250
46 9
0 0
7571
5525
7 2
0 0
0 0
7938
5 0
0 0
060
16 1
0 0
041
14 2
0 0
0 0
22 1
0 0
0 0
063
21 1
0 0
0 0
031
10 2
0 0
0 0
0 0
0 0
7769
4510
0 0
069
4713
1 0
0 0
0 0
076
7162
5447
4961
28 3
074
7160
3210
3 0
0 0
077
35 5
0 0
0 0
5615
1 0
0 0
3612
2 0
0 0
021
1 0
0 0
0 0
5920
1 0
0 0
0 0
3110
2 0
0 0
0 0
0 0
075
6847
13 1
0 0
6950
21 4
1 0
0 0
0 0
7469
6153
4651
6645
15 1
7473
6750
23 8
2 0
0 0
8246
5 0
0 0
068
24 1
0 0
058
29 7
1 0
0 0
27 1
0 0
0 0
068
27 1
0 0
0 0
041
15 2
0 0
0 0
0 0
0 0
7552
13 1
0 0
074
5113
0 0
0 0
0 0
032
7 2
0 0
0 0
0 0
085
7633
4 0
0 0
0 0
081
47 6
0 0
0 0
6726
1 0
0 0
5830
7 0
0 0
026
1 0
0 0
0 0
6524
1 0
0 0
0 0
4618
3 0
0 0
0 0
0 0
074
5516
1 0
0 0
7659
22 2
0 0
0 0
0 0
5114
3 1
0 0
0 0
0 0
8683
6316
1 0
0 0
0 0
7943
7 0
0 0
062
23 2
0 0
053
26 6
0 0
0 0
20 1
0 0
0 0
054
16 1
0 0
0 0
033
11 1
0 0
0 0
0 0
0 0
7357
23 2
0 0
075
6028
4 0
0 0
0 0
065
30 6
1 0
0 0
0 0
085
8479
5110
1 0
0 0
078
31 4
0 0
0 0
5114
1 0
0 0
4418
4 1
0 0
014
1 0
0 0
0 0
39 8
1 0
0 0
0 0
22 6
1 0
0 0
0 0
0 0
074
6128
3 0
0 0
7563
3610
1 0
0 0
0 0
7353
17 3
0 0
0 0
0 0
8584
8174
40 9
1 0
0 0
7627
4 0
0 0
043
11 1
0 0
036
15 4
1 0
0 0
13 1
0 0
0 0
036
8 1
0 0
0 0
019
6 1
0 0
0 0
0 0
0 0
7263
32 4
0 0
074
6440
14 2
0 0
0 0
074
6535
8 2
0 0
0 0
082
8281
7869
3913
2 0
074
22 3
0 0
0 0
34 7
1 0
0 0
26 9
2 0
0 0
0 7
0 0
0 0
0 0
25 5
0 0
0 0
0 0
19 6
1 0
0 0
0 0
0 0
069
5831
4 0
0 0
7260
35 8
1 0
0 0
0 0
6958
29 6
1 0
0 0
0 0
8080
7769
4926
10 2
0 0
7125
4 0
0 0
028
6 0
0 0
012
3 1
0 0
0 0
9 1
0 0
0 0
024
4 0
0 0
0 0
016
5 1
0 0
0 0
0 0
0 0
6860
38 9
1 0
070
6036
12 2
0 0
0 0
070
6753
24 5
1 1
1 0
079
7876
7465
5140
20 4
067
24 5
1 0
0 0
25 6
1 0
0 0
11 4
1 0
0 0
0 7
1 0
0 0
0 0
20 4
1 0
0 0
0 0
15 6
1 0
0 0
0 0
0 0
063
5942
14 1
0 0
6655
3413
4 1
0 0
0 0
6462
5329
9 2
2 2
1 0
7575
7472
6761
4834
16 3
6634
10 2
0 0
041
14 2
0 0
025
11 3
1 0
0 0
15 3
0 0
0 0
029
8 2
0 0
0 0
020
8 3
1 0
0 0
0 0
0 0
6462
5327
3 0
065
5637
17 7
2 1
0 1
065
6563
5535
1413
9 6
272
7271
7170
6966
6041
861
3413
3 0
0 0
4219
4 1
0 0
3016
6 2
0 0
027
8 1
0 0
0 0
4017
4 1
0 0
0 0
2211
4 1
0 0
0 0
0 0
060
5852
31 6
1 0
6153
3519
7 3
1 1
1 1
6161
5954
4221
1914
8 3
6565
6464
6362
5954
36 9
0.0
0.2
0.4
0.6
0.8
1.0
Adversarial accuracy
Figu
re6:
Acc
urac
yof
adve
rsar
iala
ttack
(col
umn)
agai
nsta
dver
sari
ally
trai
ned
mod
el(r
ow)o
nIm
ageN
et-1
00.
15
Algorithm 1 Pseudocode for the Frank-Wolfe algorithm for the L1 attack.
1: Input: function f , initial input x ∈ [0, 1]d, L1 radius ρ, number of steps T .2: Output: approximate maximizer x̄ of f over the truncated L1 ball B1(ρ;x) ∩ [0, 1]d centered atx.
3:4: x(0) ← RandomInit(x) {Random initialization}5: for t = 1, . . . , T do6: g ← ∇f(x(t−1)) {Obtain gradient}7: for k = 1, . . . , d do8: sk ← index of the coordinate of g by with kth largest norm9: end for
10: Sk ← {s1, . . . , sk}.11:12: {Compute move to boundary of [0, 1] for each coordinate.}13: for i = 1, . . . , d do14: if gi > 0 then15: bi ← 1− xi16: else17: bi ← −xi18: end if19: end for20: Mk ←
∑i∈Sk|bi| {Compute L1-perturbation of moving k largest coordinates.}
21: k∗ ← max{k |Mk ≤ ρ} {Choose largest k satisfying L1 constraint.}22:23: {Compute x̂ maximizing g>x over the L1 ball.}24: for i = 1, . . . , d do25: if i ∈ Sk∗ then26: x̂i ← xi + bi27: else if i = sk∗+1 then28: x̂i ← xi + (ρ−Mk∗) sign(gi)29: else30: x̂i ← xi31: end if32: end for33: x(t) ← (1− 1
t )x(t−1) + 1t x̂ {Average x̂ with previous iterates}
34: end for35: x̄← x(T )
16
L∞ L2
L1
L∞
-JP
EG
L2-J
PE
GE
last
ic
Fog
Gab
orSn
ow
Normal Training
L∞ ε = 1
L∞ ε = 2
L∞ ε = 4
L∞ ε = 8
L∞ ε = 16
L∞ ε = 32
L∞-JPEG ε = 0.0625
L∞-JPEG ε = 0.125
L∞-JPEG ε = 0.25
L∞-JPEG ε = 0.5
L∞-JPEG ε = 1
L∞-JPEG ε = 2
Fog ε = 128
Fog ε = 256
Fog ε = 512
Fog ε = 2048
Fog ε = 4096
Fog ε = 8192
7 17 22 0 0 31 16 5 10
110 110 110 110 110 110 110 110 110
46 54 37 24 21 40 29 29 25
60 64 42 36 30 42 29 41 31
72 74 48 45 37 44 27 53 37
83 72 42 42 32 47 23 60 41
89 60 27 30 24 49 19 58 41
88 42 15 14 11 49 20 55 37
110 110 110 110 110 110 110 110 110
36 44 34 49 48 38 23 17 16
46 52 38 63 59 39 22 27 20
56 61 43 73 69 40 22 39 24
67 69 48 85 80 41 21 50 30
69 72 56 96 91 41 21 53 32
65 70 54 92 98 40 19 52 31
110 110 110 110 110 110 110 110 110
12 21 22 0 0 34 41 6 15
11 22 21 0 0 35 50 8 19
8 18 18 0 0 36 58 10 23
5 12 15 0 0 36 78 20 31
2 7 8 0 0 34 90 29 34
1 3 8 0 0 28 91 54 43
L∞ L2
L1
L∞
-JP
EG
L2-J
PE
GE
last
ic
Fog
Gab
orSn
ow
Normal Training
L2 ε = 150
L2 ε = 300
L2 ε = 600
L2 ε = 1200
L2 ε = 2400
L2 ε = 4800
L2-JPEG ε = 8
L2-JPEG ε = 16
L2-JPEG ε = 32
L2-JPEG ε = 64
L2-JPEG ε = 128
L2-JPEG ε = 256
Gabor ε = 6.25
Gabor ε = 12.5
Gabor ε = 25
Gabor ε = 400
Gabor ε = 800
Gabor ε = 1600
7 17 22 0 0 31 16 5 10
110 110 110 110 110 110 110 110 110
38 49 38 15 13 39 29 22 20
50 60 44 27 24 40 29 33 26
62 72 53 40 36 42 28 44 31
73 82 65 54 49 46 26 54 37
80 88 75 63 58 48 22 57 40
80 88 79 67 63 48 18 53 38
110 110 110 110 110 110 110 110 110
37 46 36 49 50 38 23 17 17
47 55 41 63 62 39 24 26 20
57 63 46 72 74 40 24 36 25
67 73 53 84 84 41 23 48 31
74 77 59 90 93 43 21 53 36
72 76 61 96 96 43 21 53 36
110 110 110 110 110 110 110 110 110
17 28 26 1 0 39 30 46 30
11 20 22 0 0 40 32 59 36
7 15 18 0 0 39 29 64 39
10 18 14 0 0 39 25 68 36
11 19 14 0 0 39 27 73 37
12 19 14 0 0 39 29 82 40
L∞ L2
L1
L∞
-JP
EG
L2-J
PE
GE
last
ic
Fog
Gab
orSn
ow
Normal Training
L1 ε = 9562
L1 ε = 19125
L1 ε = 76500
L1 ε = 153000
L1 ε = 306000
L1 ε = 612000
Elastic ε = 0.25
Elastic ε = 0.5
Elastic ε = 2
Elastic ε = 4
Elastic ε = 8
Elastic ε = 16
Snow ε = 0.0625
Snow ε = 0.125
Snow ε = 0.25
Snow ε = 2
Snow ε = 4
Snow ε = 8
7 17 22 0 0 31 16 5 10
110 110 110 110 110 110 110 110 110
26 40 43 5 6 37 22 14 16
33 47 49 12 14 39 23 21 20
50 63 70 34 35 41 24 38 27
54 66 81 42 42 41 24 44 30
59 70 87 51 50 43 21 48 33
62 71 89 56 55 43 18 47 31
110 110 110 110 110 110 110 110 110
21 32 30 4 3 41 24 14 18
27 38 34 10 7 46 27 22 24
37 46 37 19 15 68 30 42 36
36 44 30 11 9 81 29 46 40
31 36 19 4 3 91 26 45 39
23 25 11 1 1 91 25 41 40
110 110 110 110 110 110 110 110 110
15 24 22 0 0 32 35 18 39
14 22 20 0 0 33 36 28 52
10 16 16 0 0 34 39 39 59
8 9 4 0 0 34 38 52 71
8 8 4 0 0 34 36 50 78
13 15 9 1 0 39 37 60 93
Figure 7: UAR scores for adv. trained defenses (rows) against distortion types (columns) forImageNet-100.
17
No atta
ckL
=1 L =2 L =4 L =8
L =16 L =32
L2 =10 L2 =20 L2 =40 L2 =80
L2 =16
0L2
=320
L2 =64
0L2
=1280
L2 =25
60L2
=5120 L1
=195
L1 =39
0L1
=780
L1 =15
60L1
=3120
L1 =62
40
L1 =12
480
L1 =24
960
L1 =49
920
L-JP
EG =0.0
3125
L-JP
EG =0.0
625
L-JP
EG =0.1
25
L-JP
EG =0.2
5
L-JP
EG =0.5
L-JP
EG =1
L2-JP
EG =0.0
625
L2-JP
EG =0.1
25
L2-JP
EG =0.2
5
L2-JP
EG =0.5
L2-JP
EG =1
L2-JP
EG =2
L2-JP
EG =4
L2-JP
EG =8
L1-JP
EG =1
L1-JP
EG =2
L1-JP
EG =4
L1-JP
EG =8
L1-JP
EG =16
L1-JP
EG =32
L1-JP
EG =64
L1-JP
EG =12
8
L1-JP
EG =25
6
L1-JP
EG =51
2
L1-JP
EG =10
24Ela
stic =0.1
25
Elastic
=0.25
Elastic
=0.5Ela
stic =1
Elastic
=2Ela
stic =4
Elastic
=8
Elastic
=16
Norm
al tr
aini
ng
L
=1
L
=2
L
=4
L
=8
L
=16
L
=32
L 2
=10
L 2
=20
L 2
=40
L 2
=80
L 2
=16
0L 2
=
320
L 2
=64
0L 2
=
1280
L 2
=25
60L 2
=
5120
L 1
=19
5L 1
=
390
L 1
=78
0L 1
=
1560
L 1
=31
20L 1
=
6240
L 1
=12
480
L 1
=24
960
L 1
=49
920
L-JP
EG
=0.
0312
5L
-JPEG
=
0.06
25L
-JPEG
=
0.12
5L
-JPEG
=
0.25
L-JP
EG
=0.
5L
-JPEG
=
1
L 2-JP
EG
=0.
0625
L 2-JP
EG
=0.
125
L 2-JP
EG
=0.
25L 2
-JPEG
=
0.5
L 2-JP
EG
=1
L 2-JP
EG
=2
L 2-JP
EG
=4
L 2-JP
EG
=8
L 1-JP
EG
=1
L 1-JP
EG
=2
L 1-JP
EG
=4
L 1-JP
EG
=8
L 1-JP
EG
=16
L 1-JP
EG
=32
L 1-JP
EG
=64
L 1-JP
EG
=12
8L 1
-JPEG
=
256
L 1-JP
EG
=51
2L 1
-JPEG
=
1024
Elas
tic
=0.
125
Elas
tic
=0.
25El
astic
=
0.5
Elas
tic
=1
Elas
tic
=2
Elas
tic
=4
Elas
tic
=8
Elas
tic
=16
9359
9 0
0 0
091
8349
5 1
3 4
3 3
185
7139
9 0
0 0
0 0
21 0
0 0
0 0
9291
8350
5 0
2 2
8982
6124
2 0
0 0
0 0
0 9
0 0
0 0
0 0
0
9389
7937
1 0
092
9289
7425
0 2
2 2
190
8673
44 9
0 0
0 0
8350
3 0
0 0
9292
9188
7222
0 0
9190
8466
28 3
0 0
0 0
067
11 0
0 0
0 0
093
9186
6412
0 0
9292
9083
50 4
0 2
2 1
9087
7854
17 1
0 0
087
6914
0 0
092
9291
9081
41 1
092
9186
7339
7 0
0 0
0 0
7826
1 0
0 0
0 1
9190
8878
40 1
091
9190
8567
17 0
0 1
090
8880
6228
3 0
0 0
8879
40 1
0 0
9190
9089
8459
8 0
9089
8677
5216
1 0
0 0
083
50 3
0 0
0 0
089
8887
8263
13 0
8888
8785
7433
0 0
0 0
8786
8065
35 5
0 0
086
8053
4 0
088
8888
8783
6616
088
8785
7757
24 3
0 0
0 0
8366
12 0
0 0
0 0
8383
8280
7139
283
8382
8175
49 4
0 0
083
8178
6948
16 1
0 0
8178
6420
0 0
8383
8382
8071
33 1
8382
8176
6539
11 1
0 0
080
7234
3 0
0 0
184
8382
7766
4723
8484
8280
6943
10 0
0 0
8178
7363
4827
8 1
082
7762
29 6
184
8484
8379
6636
884
8381
7666
4932
18 9
4 1
8066
22 2
0 0
0 1
9281
48 4
0 0
091
8980
42 3
3 5
5 3
189
8263
27 2
0 0
0 0
5910
0 0
0 0
9291
8979
37 1
2 3
9087
7442
8 0
0 0
0 0
038
1 0
0 0
0 0
094
8867
13 0
0 0
9392
8762
8 0
3 3
2 1
9185
7035
4 0
0 0
074
22 0
0 0
093
9391
8655
4 0
192
8980
5312
0 0
0 0
0 0
51 3
0 0
0 0
0 0
9390
8142
1 0
093
9290
7832
1 3
3 2
192
8879
5315
0 0
0 0
8555
4 0
0 0
9392
9289
7626
0 1
9290
8569
30 3
0 0
0 0
069
13 0
0 0
0 0
092
9086
6613
0 0
9291
9085
61 9
0 3
2 1
9189
8468
33 4
0 0
088
7730
0 0
092
9291
9085
58 6
091
9087
7851
14 1
0 0
0 0
7932
1 0
0 0
0 0
9089
8777
38 1
090
9089
8675
33 0
0 1
089
8886
7752
16 1
0 0
8884
60 7
0 0
9090
9089
8775
25 0
9089
8782
6431
5 0
0 0
082
55 5
0 0
0 0
087
8685
8060
11 0
8786
8685
8057
8 0
0 0
8686
8480
6736
5 0
085
8476
37 1
086
8686
8685
8156
686
8685
8376
5824
3 0
0 0
8266
18 1
0 0
0 0
8079
7977
6834
180
8080
7977
6732
0 0
080
7979
7773
5724
1 0
7979
7562
14 0
7979
8079
7977
6930
7979
7978
7669
5123
4 0
077
7038
6 1
0 0
173
7373
7166
4914
7373
7373
7166
5015
0 0
7373
7271
6962
4721
173
7270
6447
1473
7373
7372
7267
5473
7373
7271
6861
5037
2516
7267
5123
6 2
2 3
6969
6867
6349
3369
6868
6867
6250
3217
468
6868
6765
5846
3424
6868
6661
4834
6969
6868
6867
6452
6868
6868
6764
5849
4035
3268
6452
3421
11 6
777
7674
6855
30 7
7776
7574
6754
24 3
0 0
7676
7471
6450
3116
576
7469
5216
177
7776
7675
7261
2676
7676
7471
6447
2711
4 2
7364
33 4
1 0
2 4
9384
55 6
0 0
092
9084
52 5
1 4
4 3
191
8875
44 8
0 0
0 0
6817
0 0
0 0
9392
9184
50 4
0 2
9189
8155
15 1
0 0
0 0
041
2 0
0 0
0 0
094
8662
13 0
0 0
9392
8560
10 0
3 3
3 1
9290
8363
28 3
0 0
074
29 1
0 0
093
9391
8662
11 0
192
9187
7338
7 0
0 0
0 0
47 3
0 0
0 0
0 0
9373
31 2
0 0
092
8770
25 1
1 2
3 2
192
9083
6631
3 0
0 0
5510
0 0
0 0
9392
8976
35 2
0 1
9290
8467
29 3
0 0
0 0
034
1 0
0 0
0 0
094
5614
1 0
0 0
9182
48 8
0 0
1 2
2 1
9289
8368
37 8
0 0
040
4 0
0 0
092
9186
6419
1 0
191
8983
6943
11 0
0 0
0 0
25 0
0 0
0 0
0 0
9449
15 1
0 0
088
7440
8 0
0 1
2 2
190
8781
7045
14 0
0 0
38 5
0 0
0 0
9291
8561
20 1
0 0
9187
7758
28 6
0 0
0 0
026
1 0
0 0
0 0
093
5413
1 0
0 0
8978
43 7
0 0
0 1
2 1
9086
7962
35 9
1 0
039
6 0
0 0
092
9084
6119
1 0
090
8779
6338
14 2
0 0
0 0
35 1
0 0
0 0
0 0
9266
20 1
0 0
090
8255
12 0
0 0
0 1
190
8782
7045
15 1
0 0
44 9
0 0
0 0
9190
8462
22 2
0 0
8986
7757
3010
2 0
0 0
046
3 0
0 0
0 0
089
7748
12 1
0 0
8884
7139
8 0
0 0
1 0
8886
8374
5527
4 0
067
34 7
1 0
089
8885
7853
17 2
088
8682
7152
2810
2 0
0 0
6319
1 0
0 0
0 0
4646
4744
3720
245
4546
4642
3416
2 0
046
4646
4747
4742
26 4
4849
4539
25 3
4748
4849
4948
4535
5052
5457
5756
5446
3014
640
3423
8 1
0 0
1
9288
7531
1 0
091
9086
6414
0 3
3 2
188
8162
27 3
0 0
0 0
9083
50 2
0 0
9191
9190
8770
17 0
9190
8882
6329
5 0
0 0
057
5 0
0 0
0 0
191
8879
46 3
0 0
9089
8774
30 1
1 2
2 1
8883
6838
7 0
0 0
090
8776
28 0
090
9090
9089
8456
590
9089
8780
6130
5 0
0 0
6611
0 0
0 0
0 0
8987
8159
10 0
088
8886
7844
4 0
2 1
187
8373
4813
1 0
0 0
8887
8363
5 0
8989
8988
8886
7737
8988
8887
8578
6332
5 0
071
21 0
0 0
0 0
086
8480
6522
1 0
8585
8378
5612
0 1
1 0
8481
7456
23 2
0 0
085
8583
7530
086
8686
8585
8480
6386
8585
8584
8173
5424
4 0
7233
1 0
0 0
0 0
8381
7970
39 4
082
8281
7864
27 1
0 0
082
8075
6337
7 0
0 0
8282
8279
61 3
8383
8382
8282
8074
8383
8282
8280
7769
5228
1173
47 5
0 0
0 0
080
7977
6944
6 0
7979
7875
6530
2 0
0 0
7978
7462
38 9
0 0
079
7978
7770
3580
8080
7979
7977
7380
8080
7979
7876
7469
6253
7251
8 0
0 0
0 0
9474
23 0
0 0
092
8763
11 0
2 2
2 2
186
7035
5 0
0 0
0 0
50 4
0 0
0 0
9393
9075
25 0
1 1
9186
7136
4 0
0 0
0 0
014
0 0
0 0
0 0
193
7730
0 0
0 0
9288
7018
0 2
3 3
2 0
8672
40 7
0 0
0 0
069
15 0
0 0
093
9391
8448
2 0
291
8878
5111
0 0
0 0
0 0
21 0
0 0
0 0
0 1
9386
58 7
0 0
093
9181
42 2
1 2
2 2
188
7852
15 0
0 0
0 0
8348
2 0
0 0
9393
9289
7422
0 1
9290
8567
28 3
0 0
0 0
037
1 0
0 0
0 0
093
8871
20 0
0 0
9291
8661
9 0
2 2
2 1
8982
6325
2 0
0 0
089
7627
0 0
093
9392
9187
60 6
092
9189
8260
22 2
0 0
0 0
53 2
0 0
0 0
0 1
9186
6921
0 0
091
8985
6313
0 2
2 2
188
8266
31 3
0 0
0 0
9085
62 7
0 0
9291
9191
8880
39 1
9191
9087
7854
17 1
0 0
058
5 0
0 0
0 0
090
8780
51 5
0 0
8988
8676
38 2
1 2
2 1
8784
7346
11 0
0 0
089
8779
44 1
090
8989
8988
8572
2090
8989
8885
7755
19 1
0 0
6714
0 0
0 0
0 0
8886
8161
15 0
088
8785
7952
7 0
1 2
086
8477
5622
2 0
0 0
8787
8470
15 0
8888
8888
8786
8156
8888
8887
8682
7349
15 1
071
26 0
0 0
0 0
085
8380
6830
2 0
8584
8378
6219
0 0
1 0
8482
7763
34 5
0 0
085
8483
7848
085
8585
8584
8481
7285
8585
8483
8279
6947
16 3
7340
2 0
0 0
0 0
9379
37 1
0 0
092
8974
26 1
1 3
3 2
187
7750
13 0
0 0
0 0
7121
0 0
0 0
9393
9186
58 6
0 1
9290
8566
26 2
0 0
0 0
024
0 0
0 0
0 0
093
8560
10 0
0 0
9290
8350
5 1
4 4
3 1
8982
6326
2 0
0 0
083
52 5
0 0
093
9291
8979
35 1
292
9189
8157
17 1
0 0
0 0
43 2
0 0
0 0
0 0
9178
44 4
0 0
090
8777
40 2
0 2
2 2
187
8060
21 1
0 0
0 0
8875
29 0
0 0
9292
9291
8667
14 0
9291
9087
7645
9 0
0 0
043
1 0
0 0
0 0
091
8775
34 1
0 0
9089
8670
23 0
1 2
1 1
8885
7242
8 0
0 0
089
8359
9 0
091
9190
9088
8047
291
9090
8883
6937
5 0
0 0
60 8
0 0
0 0
0 0
8987
7947
4 0
089
8886
7637
1 1
3 2
188
8576
5114
0 0
0 0
8886
7534
0 0
8989
8989
8884
6921
8989
8988
8679
6022
1 0
066
13 0
0 0
0 0
088
8579
54 8
0 0
8787
8577
46 4
0 2
1 1
8684
7756
20 1
0 0
087
8580
55 5
088
8888
8787
8577
4388
8787
8785
8271
4712
0 0
6818
0 0
0 0
0 0
8886
8160
14 0
087
8785
7953
7 0
2 2
186
8478
6025
2 0
0 0
8786
8367
16 0
8888
8887
8785
8058
8888
8787
8683
7659
27 3
071
25 0
0 0
0 0
086
8481
6420
0 0
8685
8479
5812
0 0
1 0
8583
7863
30 3
0 0
086
8582
7232
086
8686
8685
8581
6686
8686
8584
8379
6946
15 2
7231
1 0
0 0
0 0
8584
8167
27 1
085
8483
7962
18 0
0 1
084
8378
6534
5 0
0 0
8584
8275
45 1
8585
8585
8584
8170
8585
8585
8483
8073
5936
1372
38 2
0 0
0 0
084
8380
7138
2 0
8483
8279
6728
0 0
0 0
8382
7868
43 9
0 0
084
8382
7756
484
8484
8484
8381
7484
8484
8483
8280
7666
4926
7548
4 0
0 0
0 0
8281
7971
40 3
082
8281
7867
31 1
0 0
081
8077
6744
11 0
0 0
8282
8176
59 9
8282
8282
8281
7973
8282
8282
8281
7975
6956
3874
50 6
0 0
0 0
0
9385
6013
0 0
092
9082
50 8
0 2
2 2
189
8264
32 6
0 0
0 0
6519
1 0
0 0
9291
8769
24 2
0 0
8879
5723
3 0
0 0
0 0
086
60 5
0 0
0 0
091
8462
17 1
0 0
9088
8154
12 0
0 0
0 0
8782
6738
10 1
0 0
070
29 2
0 0
090
8986
7132
3 0
086
7958
26 4
0 0
0 0
0 0
8778
33 0
0 0
0 0
8781
6525
2 0
087
8579
5718
1 0
0 0
084
7966
4215
2 0
0 0
7343
8 0
0 0
8786
8474
45 9
0 0
8477
6135
10 1
0 0
0 0
085
8162
8 0
0 0
082
7867
36 6
0 0
8280
7659
23 3
0 0
0 0
7770
5636
15 3
0 0
073
5219
3 0
081
8180
7454
20 2
079
7464
4520
6 1
0 0
0 0
8179
7240
7 1
0 0
8175
6542
15 1
080
7870
4817
2 0
0 0
070
5945
3118
9 3
1 0
6847
20 4
0 0
7878
7770
5020
3 0
7773
6447
2610
3 1
0 0
079
7671
5837
1911
776
7164
4519
2 0
7573
6647
18 3
0 0
0 0
6354
3925
15 8
3 0
064
4619
4 1
074
7473
6649
21 4
072
6961
4627
11 4
1 0
0 0
7472
6858
4533
2310
6865
5941
14 1
068
6659
4217
3 0
0 0
054
4533
2010
3 1
0 0
5841
16 3
0 0
6867
6559
4319
4 0
6562
5440
2411
4 1
0 0
068
6763
5342
3428
1561
5853
3813
1 0
6059
5339
15 2
0 0
0 0
5041
3119
9 3
0 0
053
3714
2 0
060
5958
5339
17 3
058
5549
3721
8 2
1 0
0 0
6160
5849
3929
2318
0.0
0.2
0.4
0.6
0.8
1.0
Adversarial accuracy
Figu
re8:
Acc
urac
yof
adve
rsar
iala
ttack
(col
umn)
agai
nsta
dver
sari
ally
trai
ned
mod
el(r
ow)o
nC
IFA
R-1
0.
18
L∞ L2
L1
L∞
-JP
EG
L1-J
PE
GE
last
ic
Normal Training
L∞ ε = 1
L∞ ε = 2
L∞ ε = 4
L∞ ε = 8
L∞ ε = 16
L∞ ε = 32
L∞-JPEG ε = 0.03125
L∞-JPEG ε = 0.0625
L∞-JPEG ε = 0.125
L∞-JPEG ε = 0.25
L∞-JPEG ε = 0.5
L∞-JPEG ε = 1
17 16 48 5 25 3
110110110110110110
51 49 69 31 37 21
63 59 73 38 39 28
74 67 76 47 40 36
83 72 77 50 39 43
89 75 78 55 40 51
94 72 76 58 48 45
110110110110110110
48 43 61 51 42 17
54 50 66 63 49 21
59 55 69 74 58 25
63 59 71 81 64 29
68 64 73 88 79 34
68 64 71 94 99 35
L∞ L2
L1
L∞
-JP
EG
L1-J
PE
GE
last
ic
Normal Training
L2 ε = 40
L2 ε = 80
L2 ε = 160
L2 ε = 320
L2 ε = 640
L2 ε = 2560
L1-JPEG ε = 2
L1-JPEG ε = 8
L1-JPEG ε = 64
L1-JPEG ε = 256
L1-JPEG ε = 512
L1-JPEG ε = 1024
17 16 48 5 25 3
110110110110110110
53 53 74 32 38 22
64 63 80 44 40 30
73 73 84 54 41 38
80 81 88 64 46 45
84 86 88 70 51 52
87 85 86 78 71 66
110110110110110110
39 37 62 32 41 12
49 47 68 54 51 18
60 58 73 76 66 26
65 62 75 84 85 30
68 66 76 87 92 34
68 66 75 88 96 35
L∞ L2
L1
L∞
-JP
EG
L1-J
PE
GE
last
ic
Normal Training
L1 ε = 195
L1 ε = 390
L1 ε = 780
L1 ε = 1560
L1 ε = 6240
L1 ε = 49920
Elastic ε = 0.125
Elastic ε = 0.25
Elastic ε = 0.5
Elastic ε = 1
Elastic ε = 2
Elastic ε = 8
17 16 48 5 25 3
110110110110110110
36 38 70 19 34 12
40 41 79 24 39 13
26 26 79 15 37 9
18 15 80 10 37 7
17 13 77 10 36 10
49 47 61 47 51 29
110110110110110110
40 37 63 19 24 41
41 38 65 23 25 53
43 40 65 28 27 64
47 41 57 33 29 75
49 35 51 32 29 89
45 31 37 27 25 86
Figure 9: UAR scores on CIFAR-10. Displayed UAR scores are multiplied by 100 for clarity.
No
atta
ckL∞ε
=1
L∞ε
=2
L∞ε
=4
L∞ε
=8
L∞ε
=16
L∞ε
=32
L2ε
=15
0L
2ε
=30
0L
2ε
=60
0L
2ε
=12
00L
2ε
=24
00L
2ε
=48
00L
1ε
=95
62.4
4
L1ε
=19
125
L1ε
=76
500
L1ε
=15
3000
L1ε
=30
6000
L1ε
=61
2000
L∞
-JP
EGε
=0.
0312
5
L∞
-JP
EGε
=0.
125
L∞
-JP
EGε
=0.
25
L∞
-JP
EGε
=0.
5
L∞
-JP
EGε
=1
L2-J
PE
Gε
=2
L2-J
PE
Gε
=16
L2-J
PE
Gε
=32
L2-J
PE
Gε
=64
L2-J
PE
Gε
=12
8
L2-J
PE
Gε
=25
6E
last
icε
=0.
25E
last
icε
=1
Ela
sticε
=2
Ela
sticε
=4
Ela
sticε
=8
Ela
sticε
=16
Attack (evaluation)
L∞ ε = 1
L∞ ε = 2
L∞ ε = 4
L∞ ε = 8
L∞ ε = 16
L∞ ε = 32
L2 ε = 150
L2 ε = 300
L2 ε = 600
L2 ε = 1200
L2 ε = 2400
L2 ε = 4800
L1 ε = 9562.44
L1 ε = 19125
L1 ε = 38250.1
L1 ε = 76500
L1 ε = 153000
L1 ε = 306000
L1 ε = 612000
Att
ack
(adv
ersa
rial
trai
ning
)
87 84 70 13 0 0 0 85 81 47 2 0 0 80 66 6 0 0 0 84 13 0 0 0 86 10 0 0 0 0 85 38 3 0 0 0
85 85 81 50 2 0 0 85 83 71 18 0 0 81 72 18 1 0 0 84 50 1 0 0 85 33 1 0 0 0 84 52 7 0 0 0
84 83 82 74 23 0 0 84 83 78 47 2 0 80 74 29 5 0 0 83 67 7 0 0 84 49 3 0 0 0 83 62 16 1 0 0
80 80 79 77 59 6 0 80 78 73 46 5 0 72 62 23 6 0 0 79 54 8 0 0 79 41 5 0 0 0 79 66 30 3 0 0
74 74 74 73 67 34 1 74 72 64 35 3 0 61 48 10 2 0 0 73 41 4 0 0 73 29 4 0 0 0 74 67 43 10 1 0
71 71 70 69 63 40 8 69 62 35 6 0 0 40 22 2 0 0 0 65 9 0 0 0 69 8 1 0 0 0 70 64 46 15 1 0
87 82 54 3 0 0 0 85 79 33 0 0 0 81 68 5 0 0 0 82 2 0 0 0 85 2 0 0 0 0 84 31 2 0 0 0
86 84 74 21 0 0 0 85 82 65 8 0 0 82 76 19 1 0 0 84 20 0 0 0 85 14 0 0 0 0 83 44 4 0 0 0
85 84 80 56 3 0 0 84 83 78 41 1 0 83 79 44 8 0 0 84 59 3 0 0 84 51 2 0 0 0 83 57 10 0 0 0
82 82 80 73 28 0 0 82 81 79 68 15 0 81 80 65 32 4 0 82 75 33 0 0 82 73 24 0 0 0 81 65 21 1 0 0
77 77 76 74 56 6 0 77 77 76 73 48 2 77 76 71 56 22 1 77 75 62 9 0 77 75 57 6 0 0 77 68 37 3 0 0
69 69 68 67 61 27 1 69 69 69 67 60 20 69 69 66 61 45 14 69 68 63 38 2 69 68 63 34 2 0 69 63 48 12 1 0
87 70 22 0 0 0 0 82 63 13 0 0 0 84 77 13 1 0 0 66 0 0 0 0 84 1 0 0 0 0 82 18 0 0 0 0
86 77 43 3 0 0 0 84 74 33 1 0 0 84 81 33 3 0 0 76 2 0 0 0 84 3 0 0 0 0 83 29 1 0 0 0
85 80 60 10 0 0 0 84 79 54 5 0 0 84 83 53 12 0 0 81 13 0 0 0 84 17 0 0 0 0 83 40 2 0 0 0
84 81 70 29 1 0 0 83 80 66 21 0 0 83 83 71 41 5 0 81 42 3 0 0 83 46 3 0 0 0 82 52 6 0 0 0
82 80 75 47 4 0 0 81 79 72 40 2 0 81 81 76 63 24 1 80 63 20 0 0 81 64 20 0 0 0 80 59 13 0 0 0
78 77 72 51 10 0 0 78 76 70 45 6 0 78 78 75 67 40 4 77 65 35 2 0 78 67 35 2 0 0 77 62 22 1 0 0
74 73 68 52 15 0 0 73 72 67 46 9 0 73 73 71 66 47 12 72 62 44 9 0 73 63 45 8 0 0 72 59 26 2 0 00.0
0.2
0.4
0.6
0.8
1.0
Adversarial
accuracy
Figure 10: Replica of the first three block rows of Figure 6 with different random seeds. Deviationsin results are minor.
19
No atta
ckL
=1L
=2L
=4L
=8L
=16L
=32 L2 =15
0L2
=300
L2 =60
0L2
=1200
L2 =24
00L2
=4800
L1 =95
62.44
L1 =19
125
L1 =38
250.1
L1 =76
500
L1 =15
3000
L1 =30
6000
L1 =61
2000
L-JP
EG =0.0
3125
L-JP
EG =0.0
625
L-JP
EG =0.1
25
L-JP
EG =0.2
5
L-JP
EG =0.5
L-JP
EG =1
L-JP
EG =2
L2-JP
EG =2
L2-JP
EG =4
L2-JP
EG =8
L2-JP
EG =16
L2-JP
EG =32
L2-JP
EG =64
L2-JP
EG =12
8
L2-JP
EG =25
6L1
-JPEG
=128
L1-JP
EG =25
6
L1-JP
EG =51
2
L1-JP
EG =10
24
L1-JP
EG =20
48
L1-JP
EG =40
96
L1-JP
EG =81
92
L1-JP
EG =16
384
L1-JP
EG =32
768
L1-JP
EG =65
536
L1-JP
EG =13
1072
Elastic
=0.25
Elastic
=0.5
Elastic
=1
Elastic
=2
Elastic
=4
Elastic
=8
Elastic
=16 Fog =12
8
Fog =25
6
Fog =51
2
Fog =10
24
Fog =20
48
Fog =40
96
Fog =81
92
Fog =16
384
Fog =32
768
Fog =65
536
Gabor
=6.25
Gabor
=12.5
Gabor
=25
Gabor
=50
Gabor
=100
Gabor
=200
Gabor
=400
Gabor
=800
Gabor
=1600
Gabor
=3200
Snow
=0.031
25
Snow
=0.062
5
Snow
=0.125
Snow
=0.25
Snow
=0.5Sn
ow =1
Snow
=2Sn
ow =4
Snow
=8
Snow
=16
Norm
al tr
aini
ng
L
=1
L
=2
L
=4
L
=8
L
=16
L
=32
L 2
=15
0L 2
=
300
L 2
=60
0L 2
=
1200
L 2
=24
00L 2
=
4800
L 1
=95
62.4
4L 1
=
1912
5L 1
=
3825
0.1
L 1
=76
500
L 1
=15
3000
L 1
=30
6000
L 1
=61
2000
L-JP
EG
=0.
0312
5L
-JPEG
=
0.06
25L
-JPEG
=
0.12
5L
-JPEG
=
0.25
L-JP
EG
=0.
5L
-JPEG
=
1L
-JPEG
=
2
L 2-JP
EG
=2
L 2-JP
EG
=4
L 2-JP
EG
=8
L 2-JP
EG
=16
L 2-JP
EG
=32
L 2-JP
EG
=64
L 2-JP
EG
=12
8L 2
-JPEG
=
256
L 1-JP
EG
=12
8L 1
-JPEG
=
256
L 1-JP
EG
=51
2L 1
-JPEG
=
1024
L 1-JP
EG
=20
48L 1
-JPEG
=
4096
L 1-JP
EG
=81
92L 1
-JPEG
=
1638
4L 1
-JPEG
=
3276
8L 1
-JPEG
=
6553
6L 1
-JPEG
=
1310
72
Elas
tic
=0.
25El
astic
=
0.5
Elas
tic
=1
Elas
tic
=2
Elas
tic
=4
Elas
tic
=8
Elas
tic
=16
Fog
=12
8Fo
g =
256
Fog
=51
2Fo
g =
1024
Fog
=20
48Fo
g =
4096
Fog
=81
92Fo
g =
1638
4Fo
g =
3276
8Fo
g =
6553
6
Gabo
r =
6.25
Gabo
r =
12.5
Gabo
r =
25Ga
bor
=50
Gabo
r =
100
Gabo
r =
200
Gabo
r =
400
Gabo
r =
800
Gabo
r =
1600
Gabo
r =
3200
Snow
=
0.03
125
Snow
=
0.06
25Sn
ow
=0.
125
Snow
=
0.25
Snow
=
0.5
Snow
=
1Sn
ow
=2
Snow
=
4Sn
ow
=8
Snow
=
16
8728
2 0
0 0
058
13 0
0 0
070
4413
2 0
0 0
22 1
0 0
0 0
072
28 2
0 0
0 0
059
32 9
1 0
0 0
0 0
0 0
8050
9 0
0 0
070
37 6
0 0
0 0
0 0
018
5 2
1 0
0 0
0 0
064
3710
1 0
0 0
0 0
0
8684
7015
0 0
085
8250
3 0
081
7043
10 1
0 0
8472
16 0
0 0
086
8468
13 0
0 0
077
6436
10 1
0 0
0 0
0 0
8475
39 4
0 0
078
6023
3 0
0 0
0 0
075
34 6
1 0
0 0
0 0
080
6733
7 1
0 0
0 0
085
8580
51 3
0 0
8583
7121
0 0
8174
5724
3 0
084
8149
2 0
0 0
8584
7738
1 0
0 0
7971
5222
4 0
0 0
0 0
084
7953
10 0
0 0
7656
19 2
1 1
0 1
0 0
8162
16 2
0 0
0 0
0 0
8073
4714
2 0
0 0
0 0
8484
8274
24 0
084
8379
50 3
081
7664
39 9
1 0
8382
7013
0 0
084
8380
56 6
0 0
079
7462
3811
1 0
0 0
0 0
8379
6218
1 0
073
5015
2 1
1 1
1 1
082
7741
5 0
0 0
0 0
079
7659
23 4
1 1
0 0
080
7979
7659
7 0
7978
7352
9 0
7365
5331
9 1
080
7760
15 0
0 0
7978
7146
8 0
0 0
7470
5939
16 4
1 0
0 0
079
7766
34 4
0 0
6640
9 1
1 1
1 1
1 1
7978
6518
1 0
0 0
0 0
7573
6538
11 3
1 1
1 1
7574
7473
6735
173
7164
32 4
059
4627
10 2
0 0
7368
39 5
0 0
074
7058
29 4
0 0
064
5846
3115
5 1
0 0
0 0
7371
6544
12 1
058
29 6
1 1
1 1
1 1
074
7369
46 5
1 1
1 0
068
6661
4421
8 2
1 0
071
7170
6963
4314
6860
35 7
0 0
3924
9 3
0 0
065
4511
1 0
0 1
6860
35 9
2 0
1 1
5951
4127
15 7
2 0
0 0
070
6863
4616
2 0
5833
9 1
1 1
1 1
1 0
7069
6452
18 3
3 2
1 1
6361
5641
21 7
2 1
0 0
8782
53 4
0 0
085
7836
1 0
080
7143
10 1
0 0
8153
3 0
0 0
085
8151
3 0
0 0
075
5523
4 0
0 0
0 0
0 0
8472
32 2
0 0
077
5619
2 0
0 0
0 0
065
21 4
1 0
0 0
0 0
078
6124
4 0
0 0
0 0
085
8473
23 0
0 0
8582
6610
0 0
8377
6126
3 0
084
7625
0 0
0 0
8584
7320
0 0
0 0
7969
4211
1 0
0 0
0 0
084
7645
5 0
0 0
7655
20 2
0 0
0 0
0 0
7845
8 1
0 0
0 0
0 0
7969
35 7
1 0
0 0
0 0
8484
8158
4 0
084
8378
41 1
083
8073
5013
0 0
8481
61 4
0 0
084
8480
52 3
0 0
081
7661
30 6
0 0
0 0
0 0
8379
5812
0 0
075
5216
2 0
0 0
0 0
081
6720
2 0
0 0
0 0
079
7350
14 2
0 0
0 0
082
8281
7429
0 0
8282
8068
17 0
8180
7767
38 6
082
8176
37 0
0 0
8282
8174
28 0
0 0
8180
7459
28 6
0 0
0 0
081
7867
24 2
0 0
6945
12 1
0 1
1 1
0 1
8176
45 6
0 0
1 0
0 0
7672
6026
5 1
0 0
1 0
7777
7673
57 7
077
7676
7249
377
7675
7158
26 3
7676
7564
13 0
077
7776
7458
8 0
077
7674
6956
29 8
1 0
0 0
7674
6839
5 1
061
34 8
1 1
0 0
1 0
176
7461
19 1
0 1
1 0
069
6861
3812
3 1
1 1
168
6868
6761
28 1
6868
6867
6020
6868
6866
6146
1668
6867
6438
2 0
6868
6867
6336
2 0
6868
6866
6251
3315
6 3
368
6763
4912
1 0
5127
6 1
1 1
1 1
1 1
6867
6131
3 1
1 1
1 1
6059
5542
19 6
2 1
1 1
8671
26 1
0 0
082
6516
0 0
083
7961
21 1
0 0
6920
1 0
0 0
084
7225
1 0
0 0
074
5320
3 0
0 0
0 0
0 0
8267
22 1
0 0
074
4810
1 0
0 0
0 0
047
13 3
1 0
0 0
0 0
073
5318
2 0
0 0
0 0
086
7843
3 0
0 0
8475
34 1
0 0
8481
7241
5 0
077
43 3
0 0
0 0
8479
49 5
0 0
0 0
8068
4010
1 0
0 0
0 0
083
7132
2 0
0 0
7449
12 1
0 0
0 0
0 0
6122
4 1
0 0
0 0
0 0
7558
23 3
1 0
0 0
0 0
8581
6313
0 0
084
8057
7 0
085
8378
5818
1 0
8166
19 1
0 0
084
8269
23 1
0 0
082
7759
26 5
0 0
0 0
0 0
8375
43 4
0 0
074
5113
1 0
0 0
0 0
073
38 7
1 0
0 0
0 0
076
6532
6 1
0 0
0 0
084
8271
31 1
0 0
8381
6923
0 0
8483
8173
43 5
082
7548
5 0
0 0
8482
7751
5 0
0 0
8381
7453
19 3
0 0
0 0
082
7853
8 0
0 0
7248
12 1
0 0
0 0
0 0
7855
16 2
0 0
0 0
0 0
7667
39 9
1 0
0 0
0 0
8179
7346
4 0
080
7970
39 2
081
8079
7764
24 1
7976
6320
1 0
080
7977
6420
1 0
080
7978
7044
13 1
0 0
0 0
7975
5814
0 0
069
4511
0 0
0 0
0 0
078
6327
4 0
0 0
0 0
072
6646
15 2
0 0
0 1
179
7773
5412
0 0
7877
7249
8 0
7979
7876
7149
877
7569
47 4
0 0
7877
7569
45 4
0 0
7878
7774
6638
9 1
0 0
077
7462
23 1
0 0
6439
9 1
0 0
0 0
0 0
7567
39 8
1 0
0 0
0 0
6964
4921
5 1
1 0
1 1
7271
6960
28 1
072
7168
5722
172
7271
7067
5724
7170
6654
18 0
072
7170
6756
17 0
071
7170
6965
5025
7 1
1 0
7068
6135
4 0
053
30 8
1 1
0 1
0 0
069
6243
14 2
1 1
0 0
060
5646
25 9
2 1
1 1
1
8775
29 1
0 0
083
60 9
0 0
077
5825
4 0
0 0
8683
57 3
0 0
086
8682
57 4
0 0
084
8171
4613
1 0
0 0
0 0
8266
18 1
0 0
075
4811
1 0
0 0
0 0
041
9 2
1 0
0 0
0 0
073
4713
2 0
0 0
0 0
087
8148
3 0
0 0
8471
21 0
0 0
7965
33 6
0 0
086
8576
20 0
0 0
8786
8474
21 0
0 0
8583
7862
30 5
0 0
0 0
083
7122
1 0
0 0
7551
13 1
0 0
0 0
0 0
5515
3 1
0 0
0 0
0 0
7451
18 3
0 0
0 0
0 0
8683
6816
0 0
084
7844
3 0
080
7144
11 1
0 0
8685
8360
2 0
086
8685
8155
2 0
085
8381
7451
20 3
0 0
0 0
8373
31 2
0 0
075
5113
1 0
0 0
0 0
071
33 6
1 0
0 0
0 0
076
5823
4 1
0 0
0 0
084
8377
43 3
0 0
8381
6616
0 0
8074
5624
3 0
084
8483
7720
0 0
8484
8482
7321
0 0
8483
8279
7049
18 4
1 0
083
7543
4 0
0 0
7349
13 1
0 0
0 0
0 0
7856
18 3
0 0
0 0
0 0
7665
34 6
1 0
0 0
0 1
8180
7866
19 1
080
7974
42 3
079
7565
3910
1 0
8181
8079
66 1
081
8180
8077
59 3
081
8080
7976
7157
3513
3 2
7975
5310
0 0
070
4612
1 0
0 0
0 0
079
7241
8 1
0 0
0 0
074
6848
15 2
1 0
0 0
079
7977
6830
2 0
7978
7453
9 0
7775
6954
24 4
080
7979
7773
41 0
8080
8079
7669
39 1
8079
7978
7776
7469
6253
4879
7559
14 1
0 0
6744
11 1
0 0
0 0
0 0
7873
5016
3 1
1 0
0 0
7168
5321
5 1
1 0
0 1
7877
7664
20 1
078
7773
50 8
077
7569
5222
4 0
7878
7876
7061
1378
7878
7877
7265
5378
7878
7877
7674
7269
6560
7774
5713
1 0
065
4210
1 0
0 0
0 0
077
7149
18 4
1 1
1 0
070
6549
20 6
2 1
0 1
1
8764
14 0
0 0
080
45 3
0 0
075
5319
2 0
0 0
8472
17 0
0 0
086
8576
26 0
0 0
083
7861
27 5
0 0
0 0
0 0
8161
14 1
0 0
075
5012
1 0
0 0
0 0
028
6 2
0 0
0 0
0 0
071
4413
2 0
0 0
0 0
087
7528
1 0
0 0
8260
9 0
0 0
7760
25 4
0 0
086
8151
2 0
0 0
8786
8259
4 0
0 0
8482
7450
15 1
0 0
0 0
082
6618
1 0
0 0
7548
11 0
0 0
0 0
0 0
38 9
2 1
0 0
0 0
0 0
7445
14 2
0 0
0 0
0 0
8681
50 4
0 0
084
7426
1 0
080
6737
7 0
0 0
8684
7519
0 0
086
8684
7727
0 0
085
8380
6838
7 0
0 0
0 0
8370
23 1
0 0
075
5013
1 0
0 0
0 0
054
15 3
1 0
0 0
0 0
075
5519
2 0
0 0
0 0
085
8370
17 0
0 0
8481
52 4
0 0
8175
5216
1 0
085
8582
57 1
0 0
8585
8582
63 4
0 0
8584
8279
6330
5 0
0 0
083
7433
2 0
0 0
7551
14 1
0 0
0 0
0 0
7030
6 1
0 0
0 0
0 0
7660
25 4
0 0
0 0
0 1
8482
7846
3 0
083
8170
20 0
081
7764
30 4
0 0
8484
8376
17 0
084
8484
8377
33 0
084
8383
8175
6028
6 1
0 0
8275
44 4
0 0
074
4813
1 0
0 0
0 0
078
5214
2 0
0 0
0 0
077
6636
7 1
0 0
0 0
081
8179
6719
0 0
8180
7650
4 0
7978
7049
15 1
081
8181
7960
1 0
8181
8180
7966
6 0
8181
8080
7874
6240
15 3
180
7656
10 1
0 0
7044
11 1
0 0
0 0
0 0
8069
35 6
1 1
0 0
0 0
7570
4915
2 1
0 0
1 1
7777
7671
40 3
077
7776
6418
076
7572
6030
5 0
7878
7776
7226
078
7777
7776
7244
177
7777
7776
7572
6857
4133
7774
6221
2 0
065
38 9
1 0
0 0
0 0
077
7351
13 2
1 1
1 0
070
6856
27 7
2 1
0 1
177
7776
7138
2 0
7777
7561
16 0
7775
7260
34 7
078
7777
7672
48 0
7878
7777
7673
5514
7877
7777
7675
7371
6458
5577
7564
22 1
0 0
6438
10 1
0 1
0 1
0 0
7672
5015
2 1
0 0
0 0
7067
5730
9 2
1 0
1 1
8767
17 1
0 0
080
53 6
0 0
078
6327
3 0
0 0
8470
19 0
0 0
086
8579
43 2
0 0
086
8476
5618
2 0
0 0
0 0
8160
12 1
0 0
073
44 8
0 0
0 0
0 0
031
7 2
0 0
0 0
0 0
070
4410
2 0
0 0
0 0
086
7430
1 0
0 0
8364
13 0
0 0
8168
36 5
0 0
085
7841
2 0
0 0
8685
8363
8 0
0 0
8584
8168
35 5
0 0
0 0
082
6316
0 0
0 0
7448
10 0
0 0
0 0
0 0
4110
2 0
0 0
0 0
0 0
7349
13 2
0 0
0 0
0 0
8679
49 4
0 0
083
7429
1 0
081
7245
9 1
0 0
8582
6510
0 0
086
8584
7631
0 0
085
8583
7554
17 1
0 0
0 0
8268
22 1
0 0
075
4811
1 0
0 0
0 0
054
16 3
1 0
0 0
0 0
074
5316
2 0
0 0
0 0
086
8267
13 0
0 0
8580
50 4
0 0
8377
5719
1 0
086
8478
39 1
0 0
8685
8582
59 4
0 0
8685
8480
7040
7 1
0 0
082
7332
2 0
0 0
7550
14 1
0 0
0 0
0 0
6727
5 1
0 0
0 0
0 0
7662
23 4
1 0
0 0
0 0
8483
7534
1 0
084
8167
14 0
082
7866
33 4
0 0
8484
8166
8 0
085
8584
8274
25 0
084
8483
8276
6128
5 1
0 0
8275
41 4
0 0
074
4914
1 0
0 0
0 0
075
43 9
1 0
0 0
0 0
077
6634
6 1
0 0
0 0
083
8279
56 6
0 0
8382
7535
1 0
8178
7044
9 0
083
8382
7636
0 0
8383
8382
7952
1 0
8383
8282
7972
5322
4 1
081
7652
8 0
0 0
7145
13 1
0 0
0 0
0 0
7960
20 2
0 0
0 0
0 0
7669
4310
1 0
0 0
0 0
8180
7866
15 0
080
8075
49 3
079
7772
5216
1 0
8181
8078
58 1
081
8180
8078
65 9
081
8080
8079
7668
4922
6 3
7975
5511
0 0
069
4311
1 0
0 0
0 0
078
6832
5 1
0 0
0 0
075
7050
16 3
1 0
0 0
180
7978
7027
1 0
7979
7658
9 0
7877
7255
23 2
080
7979
7868
7 0
8079
7979
7871
25 0
8079
7979
7877
7364
4423
1878
7559
14 1
0 0
6640
9 1
1 0
0 0
0 0
7872
43 9
1 0
0 0
0 0
7469
5221
4 1
1 0
1 1
7777
7671
34 2
077
7775
6113
077
7671
5826
3 0
7878
7877
7012
078
7878
7776
7132
078
7777
7777
7573
6754
3628
7774
6017
1 0
064
37 7
1 0
1 0
0 0
177
7348
12 1
0 0
0 0
070
6853
25 6
2 1
1 0
176
7574
6938
2 0
7574
7360
15 0
7473
6857
28 4
075
7575
7469
22 0
7575
7575
7469
40 1
7575
7575
7474
7268
6150
4574
7158
17 1
0 0
5831
6 1
1 1
0 1
1 0
7572
5619
2 1
1 0
0 0
6764
5224
7 2
1 1
1 1
7273
7164
31 2
073
7269
5411
072
7166
5224
4 0
7373
7372
6617
073
7373
7371
6636
173
7373
7372
7169
6557
4642
7269
5414
1 0
052
25 5
1 1
1 0
1 1
172
6849
15 2
1 1
0 0
064
6047
21 5
1 1
0 1
1
8763
15 0
0 0
079
45 4
0 0
075
5826
5 0
0 0
6516
1 0
0 0
082
6013
0 0
0 0
068
4417
3 0
0 0
0 0
0 0
8578
47 4
0 0
077
5315
1 0
0 0
0 0
045
15 4
1 0
0 0
0 0
074
5526
5 1
0 0
0 0
187
7428
1 0
0 0
8260
12 0
0 0
7864
3711
1 0
077
36 2
0 0
0 0
8472
28 2
0 0
0 0
7052
26 7
1 0
0 0
0 0
086
8369
19 0
0 0
7757
22 2
0 0
0 0
0 0
6025
7 2
0 0
0 0
0 0
7562
3411
2 1
0 0
0 0
8577
42 4
0 0
082
6824
1 0
077
6846
19 3
0 0
8057
11 0
0 0
083
7644
6 0
0 0
071
5632
10 2
0 0
0 0
0 0
8483
7852
4 0
076
5824
3 0
0 0
0 0
073
4818
4 1
0 0
0 0
076
6849
21 6
2 1
0 0
084
7851
9 0
0 0
8272
34 3
0 0
7666
4619
4 0
079
5814
0 0
0 0
8276
4910
1 0
0 0
7159
3817
3 0
0 0
0 0
084
8381
7430
1 0
7559
25 4
0 0
0 0
0 0
7659
29 7
1 1
1 1
0 0
7669
5429
10 4
1 1
1 1
8175
4910
0 0
078
6729
3 0
068
5432
12 2
0 0
7039
5 0
0 0
078
6834
5 0
0 0
064
5030
11 2
0 0
0 0
0 0
8180
7978
6714
172
5322
3 0
0 0
0 0
075
6338
13 3
1 1
1 1
172
6855
3515
6 2
1 1
178
6942
6 0
0 0
7356
17 1
0 0
5235
16 5
1 0
055
16 1
0 0
0 0
7148
13 1
0 0
0 0
5134
18 5
1 0
0 0
0 0
077
7776
7675
5415
6749
21 4
0 0
0 0
0 0
7262
4217
4 1
1 1
1 1
6663
5438
20 9
4 2
1 1
7459
24 2
0 0
064
35 6
0 0
036
20 7
2 0
0 0
30 4
0 0
0 0
056
24 3
0 0
0 0
039
23 8
2 0
0 0
0 0
0 0
7473
7272
7063
5064
4419
4 1
0 0
0 0
069
5737
18 6
2 1
1 1
163
6052
3823
12 5
2 1
1
8742
2 0
0 0
070
19 1
0 0
068
4011
2 0
0 0
25 1
0 0
0 0
074
31 1
0 0
0 0
058
29 7
1 0
0 0
0 0
0 0
8157
12 1
0 0
085
7852
12 1
0 0
0 0
024
7 2
1 0
0 0
0 0
076
4917
2 0
0 0
0 0
088
41 3
0 0
0 0
6919
1 0
0 0
6637
9 1
0 0
020
0 0
0 0
0 0
7225
1 0
0 0
0 0
5625
5 0
0 0
0 0
0 0
082
6116
1 0
0 0
8782
7140
6 0
0 0
1 0
32 8
2 1
0 0
0 0
0 0
7959
23 3
0 0
0 0
0 0
8731
2 0
0 0
062
14 0
0 0
064
33 7
1 0
0 0
10 0
0 0
0 0
059
13 0
0 0
0 0
042
13 2
0 0
0 0
0 0
0 0
8262
19 1
0 0
086
8480
6632
6 1
1 1
137
9 2
0 0
0 0
0 0
081
6430
6 0
0 0
0 0
086
25 1
0 0
0 0
5310
0 0
0 0
6131
7 0
0 0
0 7
0 0
0 0
0 0
48 9
0 0
0 0
0 0
4315
2 0
0 0
0 0
0 0
082
6421
2 0
0 0
8685
8276
6030
8 2
1 1
4814
4 1
0 0
0 0
0 0
8168
39 9
1 0
0 0
0 0
8519
1 0
0 0
044
8 0
0 0
058
30 6
1 0
0 0
5 0
0 0
0 0
038
5 0
0 0
0 0
041
14 2
0 0
0 0
0 0
0 0
8166
23 2
0 0
084
8482
7870
5430
12 3
160
25 6
1 0
0 0
0 0
081
7244
11 1
0 0
0 0
078
9 0
0 0
0 0
26 3
0 0
0 0
3712
1 0
0 0
0 2
0 0
0 0
0 0
25 3
0 0
0 0
0 0
25 7
1 0
0 0
0 0
0 0
074
6023
2 0
0 0
7979
7876
7368
5838
12 3
6746
18 4
1 0
0 0
0 0
7570
5020
4 0
0 0
0 0
69 4
0 0
0 0
014
2 0
0 0
036
16 4
1 0
0 0
5 0
0 0
0 0
014
2 0
0 0
0 0
0 9
2 0
0 0
0 0
0 0
0 0
6450
12 1
0 0
070
7070
7069
6967
6145
2468
6764
5323
6 8
6 4
166
6558
4421
4 1
0 0
062
3 0
0 0
0 0
10 1
0 0
0 0
2712
3 0
0 0
0 3
0 0
0 0
0 0
9 1
0 0
0 0
0 0
8 2
0 0
0 0
0 0
0 0
056
38 7
0 0
0 0
6565
6666
6666
6664
5535
6261
6055
3618
4439
2915
6160
5444
2810
1 0
0 1
51 3
1 0
0 0
0 9
1 0
0 0
029
14 4
1 0
0 0
4 0
0 0
0 0
0 8
1 0
0 0
0 0
0 8
3 1
0 0
0 0
0 0
0 0
4627
3 0
0 0
057
5859
5859
5957
5343
2454
5453
4728
1235
3224
1453
5146
3618
5 1
0 0
042
6 1
0 0
0 0
10 2
0 0
0 0
2614
5 1
0 0
0 7
1 0
0 0
0 0
12 3
1 0
0 0
0 0
12 4
1 0
0 0
0 0
0 0
041
26 5
0 0
0 0
5151
5251
5151
4947
4029
4747
4641
2412
3530
2212
4745
4030
16 4
1 0
0 0
8656
10 0
0 0
075
35 3
0 0
070
5122
4 0
0 0
37 3
0 0
0 0
065
20 1
0 0
0 0
043
16 3
0 0
0 0
0 0
0 0
8373
37 3
0 0
078
6125
3 0
0 0
0 0
083
7132
7 1
0 0
0 0
080
7144
12 2
0 0
0 0
085
39 4
0 0
0 0
6419
1 0
0 0
6642
15 3
0 0
018
1 0
0 0
0 0
46 6
0 0
0 0
0 0
27 7
1 0
0 0
0 0
0 0
082
7441
4 0
0 0
7863
31 7
1 0
0 0
0 0
8481
6525
4 1
1 1
0 0
8075
5823
4 1
0 0
0 0
8526
2 0
0 0
053
12 1
0 0
060
3512
2 0
0 0
10 0
0 0
0 0
034
4 0
0 0
0 0
021
6 1
0 0
0 0
0 0
0 0
8274
41 4
0 0
076
5823
3 0
0 0
0 0
083
8279
5413
1 1
1 1
079
7561
33 8
2 0
0 0
084
23 2
0 0
0 0
5010
1 0
0 0
5834
11 2
0 0
010
1 0
0 0
0 0
34 4
0 0
0 0
0 0
22 6
1 0
0 0
0 0
0 0
082
7343
5 0
0 0
7557
21 2
0 0
0 0
0 0
8382
8178
48 5
4 3
2 0
7874
6239
15 4
1 0
0 0
8326
2 0
0 0
052
10 1
0 0
057
30 9
1 0
0 0
10 0
0 0
0 0
040
6 0
0 0
0 0
028
9 2
0 0
0 0
0 0
0 0
8073
41 5
0 0
075
5619
1 0
0 0
0 0
081
8079
8078
4228
20 8
277
7157
3513
4 1
0 0
083
34 4
0 0
0 0
5715
1 0
0 0
5329
9 1
0 0
017
1 0
0 0
0 0
5314
1 0
0 0
0 0
3916
4 1
0 0
0 0
0 0
080
7242
7 0
0 0
7353
17 1
0 0
0 0
0 0
8179
7879
8078
6436
8 1
7771
5531
10 2
0 0
0 0
8236
4 0
0 0
058
16 1
0 0
050
24 7
1 0
0 0
18 1
0 0
0 0
059
18 1
0 0
0 0
039
17 4
1 0
0 0
0 0
0 0
7971
44 8
0 0
072
5418
1 0
0 0
0 0
079
7776
7676
7873
4511
176
7056
32 9
3 1
0 0
181
38 5
0 0
0 0
6017
1 0
0 0
4923
6 1
0 0
019
1 0
0 0
0 0
6018
1 0
0 0
0 0
4220
5 1
0 0
0 0
0 0
078
7145
11 0
0 0
7355
21 2
0 0
0 0
0 0
7976
7473
7577
7762
24 3
7571
5830
10 3
1 0
0 0
7940
6 0
0 0
060
18 1
0 0
049
24 6
0 0
0 0
24 1
0 0
0 0
063
23 2
0 0
0 0
041
18 5
1 0
0 0
0 0
0 0
7670
4713
0 0
072
5525
5 1
0 0
0 0
077
7471
7072
7679
7349
1275
7162
3715
5 2
0 0
077
36 6
0 0
0 0
5717
1 0
0 0
4419
4 1
0 0
023
1 0
0 0
0 0
6021
2 0
0 0
0 0
4019
5 1
0 0
0 0
0 0
074
6848
16 1
0 0
7158
3110
2 1
1 1
0 0
7572
7069
6975
7875
6536
7473
6952
2610
4 1
0 0
8247
6 0
0 0
069
26 1
0 0
062
4114
3 0
0 0
29 2
0 0
0 0
068
29 2
0 0
0 0
048
24 5
1 0
0 0
0 0
0 0
7454
17 2
0 0
078
6532
5 0
0 0
0 0
037
11 3
1 0
0 0
0 0
085
7537
6 1
0 0
0 0
081
48 8
0 0
0 0
6829
2 0
0 0
6339
14 2
0 0
028
1 0
0 0
0 0
6527
2 0
0 0
0 0
5228
7 1
0 0
0 0
0 0
075
5520
3 0
0 0
7969
39 8
0 0
0 0
0 0
5620
5 1
0 0
0 0
0 0
8682
6518
2 0
0 0
0 0
7944
8 0
0 0
063
25 2
0 0
058
3613
2 0
0 0
22 2
0 0
0 0
056
18 1
0 0
0 0
041
17 4
1 0
0 0
0 0
0 0
7458
28 4
0 0
078
6842
12 2
0 0
0 0
067
3810
3 1
0 0
0 0
085
8480
53 9
1 0
0 0
078
33 5
0 0
0 0
5316
1 0
0 0
5129
10 2
0 0
015
1 0
0 0
0 0
4110
1 0
0 0
0 0
3012
3 0
0 0
0 0
0 0
074
6131
5 0
0 0
7869
4518
4 1
0 0
0 0
7560
25 7
1 0
1 0
0 0
8584
8376
44 8
1 0
0 0
7627
5 0
0 0
044
12 1
0 0
043
23 7
2 0
0 0
16 2
0 0
0 0
038
9 1
0 0
0 0
028
12 3
1 0
0 0
0 0
0 0
7363
36 6
0 0
076
6948
22 6
1 1
1 0
074
6844
14 3
1 1
1 0
082
8281
7971
4410
1 0
074
23 4
0 0
0 0
35 8
1 0
0 0
3414
4 1
0 0
0 8
1 0
0 0
0 0
27 5
0 0
0 0
0 0
2812
3 0
0 0
0 0
0 0
069
6033
6 0
0 0
7466
4518
4 1
0 0
0 0
7061
3914
4 2
2 1
1 0
8080
7769
5130
14 2
0 0
7127
5 0
0 0
030
6 1
0 0
018
6 2
0 0
0 0
9 1
0 0
0 0
025
5 1
0 0
0 0
025
11 3
1 0
0 0
0 0
0 0
6860
4011
1 0
072
6445
21 6
1 1
1 0
070
6859
35 9
3 3
2 2
178
7878
7564
5039
19 3
167
26 6
1 0
0 0
27 6
1 0
0 0
17 6
2 0
0 0
0 9
1 0
0 0
0 0
21 5
1 0
0 0
0 0
2511
3 1
0 0
0 0
0 0
064
5842
16 2
0 0
6760
4119
7 2
1 0
1 1
6562
5741
18 8
8 5
3 2
7575
7472
6860
5139
14 2
6636
11 2
0 0
041
16 3
0 0
032
17 6
1 0
0 0
16 3
0 0
0 0
030
9 2
0 0
0 0
029
15 6
2 0
0 0
0 0
0 0
6461
5429
4 1
066
6043
22 9
3 1
1 1
165
6563
5842
2523
2016
1172
7272
7170
6967
5934
961
3514
3 1
0 0
4320
5 1
0 0
3622
10 4
1 0
029
9 2
0 0
0 0
4118
4 1
0 0
0 0
3017
7 2
1 0
0 0
0 0
060
5851
32 7
1 0
6256
3922
9 4
1 1
1 1
6161
5955
4633
3128
2417
6565
6464
6362
5951
34 8
0.0
0.2
0.4
0.6
0.8
1.0
Adversarial accuracy
Figu
re11
:Rep
lica
ofFi
gure
6w
ith50
step
sin
stea
dof
200
atev
alua
tion
time.
Dev
iatio
nsin
resu
ltsar
em
inor
.
20
Related Documents
