Pattern Recognition and Applications Group University of Cagliari, Italy Department of Electrical and Electronic Engineering R A P Adversarial pattern classification Adversarial pattern classification using multiple classifiers and randomization using multiple classifiers and randomization Battista Biggio, Giorgio Fumera, Fabio Roli S+SSPR 2008,Orlando, Florida, December 4th, 2008
17
Embed
Adversarial Pattern Classification Using Multiple Classifiers and Randomisation
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Pattern Recognition and Applications GroupUniversity of Cagliari, Italy
Department of Electrical and Electronic Engineering
using multiple classifiers and randomizationusing multiple classifiers and randomization
Battista Biggio, Giorgio Fumera, Fabio Roli
S+SSPR 2008,Orlando, Florida, December 4th, 2008
physicalprocess
acquisition/measurement
pattern(image, textdocument, ...)
x1x2...xn
featurevector
learningalgorithm
classifier
randomnoise
ed by sets of coupleds for formal neuronsation of essentialsfeat
Example: OCR
But many security applications, such as spam filtering, do not fit well with theabove model:
noise is not random, but adversarial. Malicious errors.false negatives are not random, they are created to evade the classifiertraining data can be “tainted” by the attackeran important classifier’s feature is its “hardness of evasion”, that is, the effort thatthe attacker has to do for evading the classifier
Standard pattern Standard pattern classification classification modelmodel
It’s a game with two players: the classifier and the adversaryThe adversary camouflages illegitimate patterns in adversarial way to evade the classifierThe classifier should be adversary-aware to handle the adversarial noise and toimplement defence strategies
Results reported in this paper showed that classifier performancesignificantly degrades if the adversarial nature of the task is not takeninto account, while an adversary-aware classifier can performsignificantly better
By anticipating the adversary strategy, we can defeat it.
““If you know the enemy and know yourself, you need not fearIf you know the enemy and know yourself, you need not fearthe result of a hundred battlesthe result of a hundred battles””((Sun Tzu, 500 BC)Sun Tzu, 500 BC)
Real anti-spam filters should be adversary-aware, which means thatthey should adapt to and anticipate adversary’s moves: exploiting thefeedback of the user, changing their operation, etc.
Real anti-spam filters can be re-trained by the feedback of the userswhich can provide correct labels for the mislabelled mails. In the modelof Dalvi et al., this corresponds to the assumption of perfect knowledgeof the adversary’s strategy function A(x)
Defence strategies in adversarial classificationDefence strategies in adversarial classification
Beyond retraining, are there other defence strategies thatBeyond retraining, are there other defence strategies thatwe can implement?we can implement?
A defence strategy: hiding information by randomizationA defence strategy: hiding information by randomization
““Keep the adversary guessing. If your strategy is a mystery, it cannot beKeep the adversary guessing. If your strategy is a mystery, it cannot becounteracted. This gives you a significant advantagecounteracted. This gives you a significant advantage””(Sun Tzu, 500 BC)(Sun Tzu, 500 BC)
Am I evading it? X2
X1
+y1(x)
-x
x’
y2(x)
- +Two randomrealizations of theboundary yc(x)
An intuitive strategy for making a classifier harder to evade is to hideinformation about it to the adversary
A possible implementation of this strategy is to introduce some randomness inthe placement of the classification boundary
A defence strategy: hiding information by randomizationA defence strategy: hiding information by randomization
Am I evading it? X2
X1
+y1(x)
-x
x’A(x)=x’
y2(x)
- +Two randomrealizations of yc(x)A(x)=x’ does notevade y1(x) !
Consider a randomized classifier yc(x, T), where the random variable is the training set T
Example: assume that UA(-,+)=5, UA(+,+)=0, W(x’, x)=3
Case 1: the adversary knows the actual boundary y2(x)The adversary’s gain if the pattern x is changed into x’ is UA(x’, x) - W(x’, x)= 5 - 3 = 2,then the adversary does the transformation ad evades the classifier.
Case 2: two random boundaries with P(y1(x))=P(y1(x))=0.5The expected gain is: [UA(x’, x) * P(y1(x)) + UA(x’, x) * P(y2(x))] - W(x’, x)= [0 * 0.5 - 5 * 0.5] - 3 = 2.5 - 3 < 0,then the adversary does not move, even if such move would allow evading the classifier.
A defence strategy: hiding information by randomizationA defence strategy: hiding information by randomization
Am I evading it? X2
X1
+y1(x)
-x
x’A(x)=x’
y2(x)
- +
Why is a randomized classifier harder to evade?Why is a randomized classifier harder to evade?
In the Proceedings paper we show that adversary’s strategy A(x) becomessuboptimal. Adversary does not camouflage malicious patterns that would allowevading the classifier, or camouflage malicious patterns which are misclassifiedby the classifier.
�
EYC
A(x){ } = argmaxx '!X
EYC
UA(y
c(x' ),+){ }"W (x, x' )[ ]
EYC
A(x){ }# A(x /yc (x' )) = Aopt(x)
Key points:
yc(x) becomes a random variable: YcThe adversary has to compute theexpected value of A(x) by averagingover possible realizations of yc(x)
Black/White List
URL Filter
Signature Filter
Header Analysis
Content Analysis
Σ… Assigned class
legitimate
spam
Evade hard MCS with randomizationEvade hard MCS with randomization
http://spamassassin.apache.org
The defence strategy based on “randomization” can be implemented in severalwaysWe implemented it using the multiple classifiers approach, by the randomisationof the combination functionFor our experiments, we used the SpamAssassin filter that is basically a linearlyweighted combination of classifiers, and randomized the weights by training setbootstrapping
ExperimentsExperiments with multiple classifiers and randomizationwith multiple classifiers and randomization
E-mail data set: TREC 200775,419 real e-mail messagesreceived between Apr.-July 200725,220 ham, 50,199 spam
SpamAssassin architecture
Experimental set upWe used SpamAssassin filter with aweighted sum as combination function(a SVM with linear kernel)
Randomization of the combinationfunction by bootstrap. The adversary“sees” 100 different sets of weightswith identical probability.
Key point: the adversary does not know theactual set of weights deployed for combiningmultiple classifiers (filtering rules). So it candevise only a suboptimal strategy A(x).
11.2119.551.461.300.560.98
�
A
rnd
U
�
C
rnd
U
�
C
det
U
�
detFN (%)
�
rnd
FN (%)
�
A
det
U
The average false negative rate decreasesfrom 19.55% to 11.25% when the classifieruses randomizationThis is confirmed by the decrease ofadversary’s utility and the increase ofclassifier’s utility
Assume that the adversary can make anymodification which reduces the score of a rule