Personalized Privacy-Aware Image Classification

1

Personalized Privacy-aware Image Classification1Eleftherios (Lefteris) Spyromitros-Xioufis, 1Symeon Papadopoulos, 2Adrian Popescu, 1Yiannis Kompatsiaris

1Center for Research and Technology Hellas – Information Technologies Institute (CERTH-ITI)2CEA-LIST

ICMR 2016, June 6-9, 2016, New York

children drinking erotic relatives vacations wedding

2

Personalized image privacy classification Photo sharing may compromise privacy

Can we make photo sharing safer?• Yes: build “private” image detectors

• Alerts whenever a “private” image is shared• Personalization is needed because privacy is subjective!

-Would you share such an image? -It depends:

• Teenager?• Life insurance?

3

Previous work & limitations1. Focus on generic (“community”) notion of privacy• Models trained on PicAlert [1]

• Flickr images annotated according to a common privacy definition• Consequences:

• Variability in user perceptions not captured • Overoptimistic performance estimates

2. Justifications are hardly comprehensible

[1] Zerr et al., I know what you did last summer!: Privacy-aware image classification and search, CIKM, 2012.

4

Our main contributions

Study personalization in image privacy classification

• Compare personalized vs generic models

• Compare two types of personalized models

Semantic visual features

• Better justifications and privacy insights

YourAlert: more realistic than existing benchmarks

5

Personalization approaches1. Full personalization: • A different model for each user relying only his feedback• Disadvantage: requires a lot of feedback

2. Partial personalization: • Models rely on user feedback + feedback from other users• Amount of personalization controlled via instance weighting

6

Visual and Semantic Features

vlad [1]: aggregation of local image descriptors

cnn [2]: deep visual features

semfeat [3]: outputs of ~17K concept detectors

• Trained using cnn

• Top 100 concepts per image

[1] Spyromitros-Xioufis et al., A comprehensive study over vlad and product quantization in large-scale image retrieval. IEEE Transactions on Multimedia, 2014.[2] Simonyan and Zisserman, Very deep convolutional networks for large-scale image recognition, ArXiv, 2014.[3] Ginsca et al., Large-Scale Image Mining with Flickr Groups, MultiMedia Modeling, 2015.

7

Justifications via semfeat

knitwear

young-back

hand-glasscigar-smoker

smoker

drinker

Freudian

semfeat can be used to justify predictions• A tag cloud of the most discriminative visual concepts

Justifications can be noisy• concept detectors are not perfect• semfeat vocabulary is not privacy-oriented

8

semfeat-LDA: an improved semantic representation Solution: project semfeat to a latent space• Images treated as text documents (top 10 concepts)• A text corpus created from private images (Pic+YourAlert)• LDA is applied to create a topic model (30 topics)• 6 privacy-related topics are identified (manually)

A 2nd level semantic representation: semfeat-LDA

Topic Top-5 semfeat concepts assigned to each topicchildren dribbler child godson wimp niecedrinking drinker drunk tipper thinker drunkard

erotic slattern erotic cover-girl maillot backrelatives great-aunt second-cousin grandfather mother great-grandchildvacations seaside vacationer surf-casting casting sandbankwedding groom bride celebrant wedding costume

9

semfeat-LDA: more intuitive justifications

children

drinking

erotic

relatives

vacations

wedding

knitwear

young-back

hand-glasscigar-smoker

smoker

drinker

Freudian

1st level semantic representation

2nd level semantic representation

10

YourAlert: a realistic benchmark User study• Participants annotate their own photos• Loose guidance allowed adoption of personal privacy notions

• Private “would share only with close OSN friends or not at all”• Public “would share with all OSN friends or even make public”

• Automated extraction and annotation software• Reduced privacy concerns: only features and annotations shared• Users gave their informed consent to use their data

The resulting dataset: YourAlert1 • Stats: 1.5K photos, 27 users, ~16/~40 private/public per user• Main advantages:

• Facilitates realistic evaluation of privacy models• Allows development of personalized models

1Publicly available at: http://mklab.iti.gr/datasets/image-privacy/

11

Experimental evaluation Goals• Compare different visual features• Evaluate generic models in a realistic setting• Evaluate personalized and partially personalized models• Gain insights into privacy perceptions via semfeat

Experimental setup• Classifier: regularized logistic regression (LibLinear)• Evaluation measure: Area under ROC (AUC)

12

Generic models on PicAlert vs YourAlert

edch bow vlad cnn semfeat0.50

0.55

0.60

0.65

0.70

0.75

0.80

0.85

0.90

0.95

1.00PicAlert YourAlert

AUC

perfect

best visual features in [Zerr et al., 2012]

visual features based on aggregation of local

descriptors

deep visual features

semantic visual features based on

cnn

Significantly worse on YourAlert!

random

+20%

13

Key findings on generic models Almost perfect performance on PicAlert with cnn• semfeat perform similarly with cnn

Singificantly worse performance on YourAlert• Similar performance for all features

Additional findings• Using more generic training examples does not help• Large variability in performance across users

14

Personalized privacy models Evaluation carried out on YourAlert• A modified k-fold cross-validation for unbiased estimates

Personalized model types• ‘user’: only user-specific examples from YourAlert• ‘hybrid’: a mixture of user-specific examples from YourAlert

and generic examples from PicAlert• User-specific examples are weighted higher

15

Evaluation of personalized models

PicAlert YourAlertu1

3-fold cv

k=1 test set

u2 u3

Model type: ‘user’

16

Evaluation of personalized models

PicAlert YourAlertu1

3-fold cv

k=1 test set

u2 u3

𝐷𝑡𝑟𝑎𝑖𝑛𝑢 1

Model type: ‘user’ h𝑢𝑠𝑒𝑟

1

17

Evaluation of personalized models

PicAlert YourAlertu1

3-fold cv

k=1 test set

u2 u3

𝐷𝑡𝑟𝑎𝑖𝑛𝑢 2Model type:

‘user’ h𝑢𝑠𝑒𝑟2

18

Evaluation of personalized models

PicAlert YourAlertu1

3-fold cv

k=1 test set

u2 u3

Model type: ‘hybrid w=1’

19

Evaluation of personalized models

PicAlert YourAlertu1

3-fold cv

k=1 test set

u2 u3

𝐷𝑡𝑟𝑎𝑖𝑛𝑢 1

Model type: ‘hybrid w=1’ hh𝑦𝑏𝑟𝑖𝑑𝑤=1

1

20

Evaluation of personalized models

PicAlert YourAlertu1

3-fold cv

k=1 test set

u2 u3

𝐷𝑡𝑟𝑎𝑖𝑛𝑢 2Model type:

‘hybrid w=1’ hh𝑦𝑏𝑟𝑖𝑑𝑤=12

21

Evaluation of personalized models

PicAlert YourAlertu1

3-fold cv

k=1 test set

u2 u3

Model type: ‘hybrid w=2’

22

Evaluation of personalized models

PicAlert YourAlertu1

3-fold cv

k=1 test set

u2 u3

𝐷𝑡𝑟𝑎𝑖𝑛𝑢 1

Model type: ‘hybrid w=2’ hh𝑦𝑏𝑟𝑖𝑑𝑤=2

1

23

Evaluation of personalized models

PicAlert YourAlertu1

3-fold cv

k=1 test set

u2 u3

𝐷𝑡𝑟𝑎𝑖𝑛𝑢 2 hh𝑦𝑏𝑟𝑖𝑑𝑤=2

2Model type: ‘hybrid w=2’

24

Personalized privacy models

0 5 10 15 20 25 30 35 0 5 10 15 20 25 30 35 0 5 10 15 20 25 30 35vlad semfeat cnn

0.55

0.60

0.65

0.70

0.75

0.80

0.85

0.90 genericuserhybrid-g w=1hybrid-g w=10hybrid-g w=100hybrid-g w=1000

# user-specific examples / features

AUC

25

Key findings on personalized models ‘user’ catches up ‘generic’ with few examples ‘hybrid’ is better than both ‘user’ and ‘generic’• Even with very few user-specific examples• ‘user’ is expected to outperform hybrid with more examples

Weighting user-specific examples higher leads to significantly better performance!

26

Privacy insights via semfeat An exploratory analysis• Get insights into the average perception of privacy• Identify deviations from the average perception of privacy

Setup• Build 1 generic and 27 personalized (‘user’) models• Identify 50 most positive and 50 most negative coefficients

Results• Generic

• Interesting Deviations • Alcoholic is private for generic and public for • Tourist is private for and public for generic

child mate son

privateuphill

lakefront waterside

public

27

Identifying recurring privacy themes A prototype semfeat-LDA vector for each user• The centroid of the semfeat-LDA vectors of his private images

K-means (k=5) clustering on the prototype vectors

c0: {2,3,19,23,25,26,27} c1: {1,5,6,11,12,13,14,20,21

}

c2: {8,10,17,24} c3: {4,16} c4: {7,9,15,18,22}0.000.020.040.060.080.100.120.140.160.180.20 children

drinkingeroticrelativesvacationswedding

Fact

or L

oadi

ngs

28

Future work Predict fine-grained privacy classes• E.g. close-friends, all-friends, friends-of-friends, public

More sophisticated instance sharing strategies• E.g. taking inter-user similarities into account

Adaptation of the semantic vocabulary towards privacy In a larger context • Images are just one piece of the puzzle in users’ privacy

preservation…• Deal with data acquisition and sharing problems

• Collaboration with other groups to conduct larger scale study• Cross-domain collaboration (e.g. legal, social sciences)

• The USEMP1 project is a good example

1 http://www.usemp-project.eu/

29

Thank you!

Resources Datasets: http://mklab.iti.gr/datasets/image-privacy Code: https://github.com/MKLab-ITI/image-privacy Contact us

@espyromi / espyromi@iti.gr @sympap / papadop@iti.gr @kompats / ikom@iti.gr

http://www.usemp-project.eu/