Pa#ern Recogni-on and Applica-ons Lab University of Cagliari, Italy Department of Electrical and Electronic Engineering Machine Learning Under Attack: Vulnerability Exploitation and Security Measures BaAsta Biggio [email protected]Dept. Of Electrical and Electronic Engineering University of Cagliari, Italy Vigo, Spain, June 21, 2016 IH&MMSec
55
Embed
Machine Learning under Attack: Vulnerability Exploitation and Security Measures
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Pa#ernRecogni-onandApplica-onsLab
University
ofCagliari,Italy
DepartmentofElectricalandElectronic
Engineering
Machine Learning Under Attack: Vulnerability Exploitation and Security Measures
• Machine Learning (ML) and Pattern Recognition (PR) increasingly used in Personal and Consumer applications
2
http://pralab.diee.unica.it
iPhone 5s with Fingerprint Recognition…
3
http://pralab.diee.unica.it
… Cracked a Few Days After Its Release
4
EU FP7 Project: TABULA RASA
http://pralab.diee.unica.it
Smart Fridge Caught Sending Spam
• Jan., 2014: A fridge has been caught sending spam after a web attack managed to compromise smart gadgets
• The fridge was one of the 100,000 compromised devices used in the spam campaign
5http://www.bbc.com/news/technology-25780908
http://pralab.diee.unica.it
New Challenges for ML/PR
• We are living exciting time for ML/PR technologies – Our work feeds a lot of consumer technologies
for personal applications
• This opens up new big possibilities but also new security risks
• Proliferation and sophistication
of attacks and cyberthreats – Skilled / economically-motivated
attackers (e.g., ransomware)
• Several security systems use machine learning to detect attacks – but … is machine learning secure enough?
6
http://pralab.diee.unica.it
Are we ready for this?
Can we use classical pattern recognition and machine learning techniques under attack? No, we cannot. We are facing an adversarial setting… We should learn to find secure patterns
7
http://pralab.diee.unica.it
Secure Patterns in Nature
• Learning of secure patterns is a well-known problem in nature – Mimicry and camouflage – Arms race between predators and preys
8
http://pralab.diee.unica.it
Secure Patterns in Computer Security
• Similar phenomenon in machine learning and computer security – Obfuscation and polymorphism to hide malicious content
Spam emails Malware
Start 2007 with a bang! Make WBFS YOUR PORTFOLIO’s first winner of the year ...
...var PGuDO0uq19+PGuDO0uq20; EbphZcei=PVqIW5sV.replace(/jTUZZ/g,"%"); var eWfleJqh=unescape;Var NxfaGVHq=“pqXdQ23KZril30”;q9124=this; var SkuyuppD= q9124["WYd1GoGYc2uG1mYGe2YnltY".replace(/[Y12WlG\:]/g, "")];SkuyuppD.write(eWfleJqh(EbphZcei));...
9
http://pralab.diee.unica.it
• Adaptation/evolution is crucial to survive!
Arms Race
Attackers Evasion techniques
System designers Design of effective countermeasures
– maximum number of samples that can be added to the training data • the attacker usually controls only a small fraction of the training samples
– maximum amount of modifications • application-specific constraints
in feature space • e.g., max. number of words that
are modified in spam emails
Adversary’s Capability
d(x, !x ) ≤ dmax
x2
x1
f(x)
x
Feasible domain
x '
20
http://pralab.diee.unica.it
Main Attack Scenarios
• Evasion attacks – Goal: integrity violation, indiscriminate attack – Knowledge: perfect / limited – Capability: manipulating test samples e.g., manipulation of spam emails at test time to evade detection
• Poisoning attacks – Goal: availability violation, indiscriminate attack – Knowledge: perfect / limited – Capability: injecting samples into the training data e.g., send spam with some ‘good words’ to poison the anti-spam filter, which may subsequently misclassify legitimate emails containing such ‘good words’
21
http://pralab.diee.unica.it
Targeted classifier: SVM
• Maximum-margin linear classifier f (x) = sign(g(x)), g(x) = wT x + b
• Kernel functions for nonlinear classification – e.g., RBF Kernel
Kernels and Nonlinearity
w = αi yixii∑ → g(x) = αi yi x, xi
i∑ + b
support vectors
k(x, xi ) = exp −γ x − xi2( )−2−1.5−1−0.500.511.5
23
http://pralab.diee.unica.it
EvasionA>acks
24
1. B. Biggio, I. Corona, D. Maiorca, B. Nelson, N. Srndic, P. Laskov, G. Giacinto, and F. Roli. Evasion attacks against machine learning at test time. ECML PKDD, 2013.
2. B. Biggio et al., Security evaluation of SVMs. SVM applications. Springer, 2014 3. F. Zhang et al., Adversarial feature selection against evasion attacks, IEEE TCYB 2016.
http://pralab.diee.unica.it
A Simple Example
• Problem: how to evade a linear (trained) classifier? – We have seen this already…
• But… what if the classifier is nonlinear? – Decision functions can be arbitrarily complicated, with no clear
relationship between features (x) and classifier parameters (w)
St4rt 2007 with a b4ng! Make WBFS YOUR PORTFOLIO’s first winner of the year ... campus
startbang portfolio winneryear ... university campus
0 0111...01
startbang portfolio winneryear ... university campus
Number of added keywords to each PDF Number of added keywords to each PDF
31
http://pralab.diee.unica.it
Security Measures against Evasion Attacks
• Multiple Classifier Systems (MCSs) – Feature Equalization
[Kolcz and Teo, CEAS 2009; Biggio et al., IJMLC 2010]
– 1.5-class classification [Biggio et al., MCS 2015]
• Adversarial Feature Selection [Zhang et al., IEEE TCYB 2016]
• Learning with Invariances: Nightmare at Test Time (InvarSVM) – Robust optimization (zero-sum games)
[Globerson and Teo, ICML 2006]
• Game Theory (NashSVM)
– Classifier vs. Adversary (non-zero-sum games) [Brueckner et al., JMLR 2012]
32
http://pralab.diee.unica.it
MCSs for Feature Equalization
• Rationale: more uniform feature weight distributions require the attacker to modify more features to evade detection
331. Kolcz and C. H. Teo. Feature weighting for improved classifier robustness, CEAS 2009. 2. B. Biggio, G. Fumera, and F. Roli. Multiple classifier systems for robust classifier
design in adversarial environments. Int’l J. Mach. Learn. Cyb., 1(1):27–41, 2010.
• Rationale: feature selection based on accuracy and security – wrapper-based backward/forward feature selection – main limitation: computational complexity
34
0 5 10 15 200
0.2
0.4
0.6
0.8
1Feature set size: 100
TP a
t FP=
1%
Traditional (PK)WAFS (PK)
0 5 10 15 200
0.2
0.4
0.6
0.8
1
TP a
t FP=
1%
max. num. of modified words
Traditional (LK)WAFS (LK)
0 5 10 15 200
0.2
0.4
0.6
0.8
1Feature set size: 200
TP a
t FP=
1%
Traditional (PK)WAFS (PK)
0 5 10 15 200
0.2
0.4
0.6
0.8
1
TP a
t FP=
1%
max. num. of modified words
Traditional (LK)WAFS (LK)
0 5 10 15 200
0.2
0.4
0.6
0.8
1Feature set size: 300
TP a
t FP=
1%
Traditional (PK)WAFS (PK
0 5 10 15 200
0.2
0.4
0.6
0.8
1
TP a
t FP=
1%
max. num. of modified words
Traditional (LK)WAFS (LK)
0 5 10 15 200
0.2
0.4
0.6
0.8
1Feature set size: 400
TP a
t FP=
1%
Traditional (PK)WAFS (PK)
0 5 10 15 200
0.2
0.4
0.6
0.8
1
TP a
t FP=
1%
max. num. of modified words
Traditional (LK)WAFS (LK)
0 5 10 15 200
0.2
0.4
0.6
0.8
1Feature set size: 100
TP a
t FP=
1%
Traditional (PK)WAFS (PK)
0 5 10 15 200
0.2
0.4
0.6
0.8
1
TP a
t FP=
1%
max. num. of modified words
Traditional (LK)WAFS (LK)
0 5 10 15 200
0.2
0.4
0.6
0.8
1Feature set size: 200
TP a
t FP=
1%
Traditional (PK)WAFS (PK)
0 5 10 15 200
0.2
0.4
0.6
0.8
1
TP a
t FP=
1%
max. num. of modified words
Traditional (LK)WAFS (LK)
0 5 10 15 200
0.2
0.4
0.6
0.8
1Feature set size: 300
TP a
t FP=
1%
Traditional (PK)WAFS (PK
0 5 10 15 200
0.2
0.4
0.6
0.8
1
TP a
t FP=
1%
max. num. of modified words
Traditional (LK)WAFS (LK)
0 5 10 15 200
0.2
0.4
0.6
0.8
1Feature set size: 400
TP a
t FP=
1%
Traditional (PK)WAFS (PK)
0 5 10 15 200
0.2
0.4
0.6
0.8
1
TP a
t FP=
1%
max. num. of modified words
Traditional (LK)WAFS (LK)
Experimental results on spam filtering (linear SVM)
http://pralab.diee.unica.it
1.5-class Classification Underlying rationale
35
2−class classification
−5 0 5−5
0
5
1−class classification (legitimate)
−5 0 5−5
0
5
• 2-class classification is usually more accurate in the absence of attack • … but potentially more vulnerable under attack (not enclosing legitimate data)
1.5C classification (MCS)
−5 0 5−5
0
5
1.5-class classification aims at retaining high accuracy and security under attack
http://pralab.diee.unica.it
data 1C Classifier (malicious)
Feature Extraction
malicious
1C Classifier (legitimate)
2C Classifier
1C Classifier (legitimate)
legitimate
x
g1(x)
g2(x)
g3(x)
g(x) ≥ t g(x)
true
false
Secure 1.5-class Classification with MCSs
• Heuristic approach to 1.5-class classification
36
0 5 10 15 20 25 300
0.2
0.4
0.6
0.8
1
maximum number of modified words
AUC 1%
(PK)
2C SVM1C SVM (L)1C SVM (M)1.5C MCS
0 5 10 15 20 25 300
0.2
0.4
0.6
0.8
1
maximum number of modified words
AUC 1%
(LK)
2C SVM1C SVM (L)1C SVM (M)1.5C MCS
Spam
filte
ring
http://pralab.diee.unica.it
PoisoningMachineLearning
37
1. B. Biggio, B. Nelson, P. Laskov. Poisoning attacks against SVMs. ICML, 2012 2. B. Biggio et al., Security evaluation of SVMs. SVM applications. Springer, 2014 3. H. Xiao et al., Is feature selection secure against training data poisoning? ICML, 2015
http://pralab.diee.unica.it
Classifier
WebServer
Tr
HTTPrequests
Poisoning Attacks against SVMs [B. Biggio, B. Nelson, P. Laskov, ICML 2012]
• Adversary model – Goal: to maximize classification error (availiability, indiscriminate) – Knowledge: perfect knowledge (trained SVM and TR set are known) – Capability: injecting samples into TR
• Attack strategy – optimal attack point xc in TR that maximizes classification error
xc
classifica-onerror=0.039classifica-onerror=0.022
Adversary Model and Attack Strategy
40
http://pralab.diee.unica.it
• Adversary model – Goal: to maximize classification error (availiability, indiscriminate) – Knowledge: perfect knowledge (trained SVM and TR set are known) – Capability: injecting samples into TR
• Attack strategy – optimal attack point xc in TR that maximizes classification error
classifica-onerror=0.022
xc
classifica-onerrorasafunc-onofxc
Adversary Model and Attack Strategy
41
http://pralab.diee.unica.it
• Max. classification error L(xc) w.r.t. xc through gradient ascent
• Gradient is not easy to compute – The training point affects the
classification function – Details of the derivation
are in the paper
Poisoning Attack Algorithm
xc(0)
xc
xc(0) xc
421. B. Biggio, B. Nelson, P. Laskov. Poisoning attacks against SVMs. ICML, 2012
http://pralab.diee.unica.it
Experiments on the MNIST digits Single-point attack
• Linear SVM; 784 features; TR: 100; VAL: 500; TS: about 2000 – ‘0’ is the malicious (attacking) class – ‘4’ is the legitimate (attacked) one
xc(0) xc
43
http://pralab.diee.unica.it
Experiments on MNIST digits Multiple-point attack
• Linear SVM; 784 features; TR: 100; VAL: 500; TS: about 2000 – ‘0’ is the malicious (attacking) class – ‘4’ is the legitimate (attacked) one
44
http://pralab.diee.unica.it
Poisoning linear models for feature selection [H. Xiao et al., ICML ’15]
• Linear models – Select features according to |w|
Data: 300 (TR) and 5,000 (TS) samples – 114 features
Similar results obtained for limited-knowledge attacks!
http://pralab.diee.unica.it
Security Measures against Poisoning
• Rationale: poisoning injects outlying training samples
• Two main strategies for countering this threat
1. Data sanitization: remove poisoning samples from training data • Bagging for fighting poisoning attacks
• Reject-On-Negative-Impact (RONI) defense
2. Robust Learning: learning algorithms that are robust in the presence of poisoning samples
48
xc(0)
xc xc(0) xc
http://pralab.diee.unica.it
Security Measures against Poisoning Data Sanitization :: Multiple Classifier Systems
• (Weighted) Bagging for fighting poisoning attacks – Underlying idea: resampling outlying samples with lower probability
• Two-step algorithm: 1. Density estimation to assign lower resampling weights to outliers 2. Bagging to train an MCS
• Promising results on spam (see plot) and web-based intrusion detection
– S: standard classifier – B: standard bagging – WB: weighted bagging
(numbers in the legend correspond to different ensemble sizes)
491. B. Biggio, I. Corona, G. Fumera, G. Giacinto, and F. Roli. Bagging classifiers for
fighting poisoning attacks in adversarial classification tasks. MCS, 2011.
http://pralab.diee.unica.it
A>ackingClustering
50
1. B. Biggio, I. Pillai, S. R. Bulò, D. Ariu, M. Pelillo, and F. Roli. Is data clustering in adversarial settings secure? AISec, 2013
2. B. Biggio, S. R. Bulò, I. Pillai, M. Mura, E. Z. Mequanint, M. Pelillo, and F. Roli. Poisoning complete-linkage hierarchical clustering. S+SSPR, 2014
http://pralab.diee.unica.it
Attacking Clustering
• So far, we have considered supervised learning – Training data consisting of samples and class labels
• In many applications, labels are not available or costly to obtain – Unsupervised learning
• Training data only include samples – no labels!
• Malware clustering – To identify variants of existing malware or new malware families
x xxx xxx
xx x
xx
x xxxx
x1 x2 ... xd
feature extraction (e.g., URL length,
num. of parameters, etc.)
data collection (honeypots)
clustering of malware families (e.g., similar HTTP
requests)
data analysis / countermeasure design (e.g., signature generation)
foreachclusterif…then…else…
51
http://pralab.diee.unica.it
Is Data Clustering Secure?
• Attackers can poison input data to subvert malware clustering
xxx
x xxx
xx
x
x
xx
xxxx
x1 x2 ... xd
feature extraction (e.g., URL length,
num. of parameters, etc.)
data collection (honeypots)
clustering of malware families (e.g., similar HTTP
requests)
data analysis / countermeasure design (e.g., signature generation)
… becomes useless (too many false alarms, low detection rate)
52
http://pralab.diee.unica.it
Our Work
• A framework to identify/design attacks against clustering algorithms – Poisoning: add samples to maximally compromise the clustering output – Obfuscation: hide samples within existing clusters
• Some clustering algorithms can be very sensitive to poisoning! – single- and complete-linkage hierarchical clustering can be easily
compromised by creating heterogeneous clusters • Details on the attack derivation and implementation are in the papers
Clustering on untainted data (80 samples) Clustering after adding 10 attack samples
531. B. Biggio et al. Is data clustering in adversarial settings secure? AISec, 2013 2. B. Biggio et al. Poisoning complete-linkage hierarchical clustering. S+SSPR, 2014
http://pralab.diee.unica.it
Conclusions and Future Work
• Learning-based systems are vulnerable to well-crafted, sophisticated attacks devised by skilled attackers
– … that exploit specific vulnerabilities of machine learning algorithms!