Verifying Multimedia Use at MediaEval 2016 Christina Boididou 1 , Stuart E. Middleton 5 , Symeon Papadopoulos 1 , Duc-Tien Dang-Nguyen 2,3 , Giulia Boato 2 , Michael Riegler 4 & Yiannis Kompatsiaris 1 1 Information Technologies Institute (ITI), CERTH, Greece 2 University of Trento, Italy. 3 Insight Centre for Data Analytics at Dublin City University, Ireland.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Verifying Multimedia Use at MediaEval 2016
Christina Boididou1, Stuart E. Middleton5, Symeon Papadopoulos1, Duc-Tien Dang-Nguyen2,3, Giulia Boato2, Michael Riegler4 & Yiannis Kompatsiaris1
1 Information Technologies Institute (ITI), CERTH, Greece2 University of Trento, Italy.3 Insight Centre for Data Analytics at Dublin City University, Ireland.4 Simula Research Lab, Norway.5 University of Southampton IT Innovation Centre, UK.
REAL OR FAKEThe verification problem
1
Real photo Captured in Dublin’s Olympia Theatre
A photo of Eagles of Death Metal in concert
ButMislabeled on social media as showing the crowd at the Bataclan theatre just before gunmen began firing.
A TYPOLOGY OF FAKE: REPOSTING OF REAL
Photos from past events reposted as being associated to current event
‘Eiffel Tower lights up in solidarity with Pakistan’
‘Syrian refugee girl selling gum in Jordan’
A TYPOLOGY OF FAKE: PHOTOSHOPPING
Digitally manipulated photos / Tampered
‘Sharks in New York during Hurricane Sandy’
‘Sikh man is a suspect of Paris attacks’
TASK DEFINITION2
MAIN TASK
POST
IMAGE
MEDIAEVAL SYSTEM
FAKE
REAL
AUTHOR(PROFILE)
‘Given a post (image + metadata), return a decision (fake, real, unknown) on whether the information presented by the post reflects the reality’
SUB-TASK
Given an image, return a decision (tampered, non-tampered, unknown) on whether the image has been digitally modified or not.
IMAGE MEDIAEVAL SYSTEM
TAMPERED
NON TAMPERED
VERIFICATION CORPUS3
GROUND TRUTH GENERATION
Multimedia cases were labeled as fake/real after consulting online reports (articles, blogs)
Data (post) collection associated to these cases performed using Topsy (historic events) or using streaming and search API (real-time events)
Post set expansion: Near-duplicate image search + journalist debunking reports + human inspection was used to increase the number of associated posts
Crowdsourcing campaign carried out with microWorkers platform; each worker asked to provide three cases of multimedia misuse
Classic IR metricsPrecision RecallF1-score -> main evaluation metricParticipants were allowed to mark a case as “unknown” (expected to result in reduced recall)
TASK SUBMISSIONS
10 submissions for the main task
2 submissions for the sub-task (just one team)
3 teams submitted(+1 the organizers)
TRENDS IN APPROACHES
Features being used- Text features (most common)- Post and user metadata- Image forensics- Video quality metadata- Topics of post- Text similarity of posts (per image case)- Trusted sources attributed in text- Mentioned online external sources
RESULTS: MAIN TASKTeam Run Recall Precision F1-Score
Linkmedia
run1TextKnn 0.9227 0.6397 0.7556
run2CBIR1 0.3406 0.4917 0.4024
run3Sources 0.9463 0.9030 0.9241
run4Fusion 0.9121 0.7525 0.8246
MMLAB@DISI
RUN1 0.5487 0.7060 0.6175
RUN2 0.9365 0.8135 0.8707
RUN3 0.9398 0.7405 0.8283
MCGICT
hybrid 0.6097 0.7637 0.6781
image 0.5138 0.6975 0.5917
text 0.6292 0.7471 0.6831
VMU
Run1 0.8512 0.9812 0.9116
Run2 0.9056 0.7709 0.8328
Run3 0.8869 0.9882 0.9348
Run4 0.8739 0.9799 0.9239
Run5 0.9951 0.5873 0.7386
Series1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Participants and organizers F1
Series1
00.10.20.30.40.50.60.70.80.91
Participants F1
RESULTS: MAIN TASKTeam Run Recall Precision F1-Score
Linkmedia
run1TextKnn 0.9227 0.6397 0.7556
run2CBIR1 0.3406 0.4917 0.4024
run3Sources 0.9463 0.9030 0.9241
run4Fusion 0.9121 0.7525 0.8246
MMLAB@DISI
RUN1 0.5487 0.7060 0.6175
RUN2 0.9365 0.8135 0.8707
RUN3 0.9398 0.7405 0.8283
MCGICT
hybrid 0.6097 0.7637 0.6781
image 0.5138 0.6975 0.5917
text 0.6292 0.7471 0.6831
VMU
Run1 0.8512 0.9812 0.9116
Run2 0.9056 0.7709 0.8328
Run3 0.8869 0.9882 0.9348
Run4 0.8739 0.9799 0.9239
Run5 0.9951 0.5873 0.7386
Series1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Participants and organizers F1
Series1
00.10.20.30.40.50.60.70.80.91
Participants F1
RESULTS: MAIN TASKTeam Run Recall Precision F1-Score
Linkmedia
run1TextKnn 0.9227 0.6397 0.7556
run2CBIR1 0.3406 0.4917 0.4024
run3Sources 0.9463 0.9030 0.9241
run4Fusion 0.9121 0.7525 0.8246
MMLAB@DISI
RUN1 0.5487 0.7060 0.6175
RUN2 0.9365 0.8135 0.8707
RUN3 0.9398 0.7405 0.8283
MCGICT
hybrid 0.6097 0.7637 0.6781
image 0.5138 0.6975 0.5917
text 0.6292 0.7471 0.6831
VMU
Run1 0.8512 0.9812 0.9116
Run2 0.9056 0.7709 0.8328
Run3 0.8869 0.9882 0.9348
Run4 0.8739 0.9799 0.9239
Run5 0.9951 0.5873 0.7386
Series1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Participants and organizers F1
Series1
00.10.20.30.40.50.60.70.80.91
Participants F1
RESULTS: MAIN TASKTeam Run Recall Precision F1-Score
Linkmedia
run1TextKnn 0.9227 0.6397 0.7556
run2CBIR1 0.3406 0.4917 0.4024
run3Sources 0.9463 0.9030 0.9241
run4Fusion 0.9121 0.7525 0.8246
MMLAB@DISI
RUN1 0.5487 0.7060 0.6175
RUN2 0.9365 0.8135 0.8707
RUN3 0.9398 0.7405 0.8283
MCGICT
hybrid 0.6097 0.7637 0.6781
image 0.5138 0.6975 0.5917
text 0.6292 0.7471 0.6831
VMU
Run1 0.8512 0.9812 0.9116
Run2 0.9056 0.7709 0.8328
Run3 0.8869 0.9882 0.9348
Run4 0.8739 0.9799 0.9239
Run5 0.9951 0.5873 0.7386
Series1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Participants and organizers F1
Series1
00.10.20.30.40.50.60.70.80.91
Participants F1
RESULTS: MAIN TASKTeam Run Recall Precision F1-Score