Phillip Isola, Jianxiong Xiao, Devi Parikh, Antonio Torralba, Aude Oliva High-level attributes of images: How memorable is an image? Motivations How to measure subjective attributes? What content makes an image memorable? Prediction algorithms Applications and future directions Prediction algorithm: SVM Regression with non-linear kernels on following features: “Aquarium, indoors” 1) Scene annotations scene categories of entire image l = 0.42 Predicting image memorability Database Memory Game ... Vigilance repeat Memory repeat 100 1-7 back 91-109 back time + + + + + 665 participants on Amazon’s Mechanical Turk. 200 1000 1800 40% 50% 60% 70% 80% 90% 100% Image rank N, according to specified group Average % memorability, according to Group 1, of 25 images centered about rank N Group 1 Group 2 Chance l = 0.75 Memorable Average Forgettable What content makes an image memorable? Object score = (prediction when object included in image’s feature vector) - (prediction when object removed) - 0.15 + 0.09 0 ... person sitting building mountain person floor sky tree seats natural lake (52%) broadleaf forest (52%) art studio (81%) campus (53%) bedroom (76%) bakery shop (81%) botanical garden (52%) bathroom (84%) ... Objects shaded according to object score (computed per image) Objects ranked according to object score (averaged across images) Scenes ranked according to their average memorability Database: 2222 photographs from SUN database (Xiao et al. 2010). Memorability = probability of correctly detecting a repeat after a single view of an image in a long stream. Wide range of memorabilities and high inter-subject consistency enclosed_space < 0.47 face_visible < 0.49 single_focus < 0.20 peaceful < 0.75 peaceful < 0.75 face_visible < 0.21 recognize < 0.55 recognize < 0.45 0.78 0.56 0.64 0.62 0.69 0.73 0.65 0.61 0.83 U M corr: 0.11 A M corr: 0.28 Aesthetics versus memorability Unusualness versus memorability Understanding memorability 2 4 6 8 0.36 0.38 0.4 0.42 0.44 0.46 0.48 # Features Rank corr Testing Training 2) Object annotations number, size, and rough position of each object class l = 0.49 “Funny, peaceful, eye contact” 3) Attribute annotations l = 0.53 Understandable attributes describ- ing layout, aesthetics, emotions, ac- tions, and appearche of people 5) All features l = 0.59 All annotations and global image features GIST SIFT HOG SSIM Pixels 4) Global image features pixel histograms, GIST, spatial pyramids of SIFT, HOG, SSIM l = 0.47 Understanding memorability Applications Predicted memorable Predicted average Predicted forgettable Automatic predictions from global image features Information theoretic feature selection Retrieve better images from search Make an image more memorable Understand human memory Diagnose memory problems Summarize photo album or video Design mnemonic aids + + “lourds” “heavy”