Visual Texture, Music, and Emotion: Cross-modal associations between visual textures and music Thomas Langlois, Joshua Peterson & Stephen E. Palmer, Department of Psychology, University of California, Berkeley METHODS STIMULI BACKGROUND RESULTS EXPERIMENT 1 EXPERIMENT 1 CONCLUSION RESULTS EXPERIMENT 2 EXPERIMENT 2 REFERENCES Emotion Dimensions Not Angry-Angry Active-Passive Smooth Jazz Piano Folk Mozart Country Bach Big Band Indie Soundtrack Balkin Folk Reggae Hip Hop Progressive Rock 80s Pop Electronic Arabic Hindustani Sitar Blues Dixieland Jazz Alternative Funk Gamelan Salsa Bluegrass Irish Trance Classic Rock Progressive House Psychobilly Stravinsky Ska Dubstep Heavy Metal Piano Bach Country Smooth Jazz Mozart Balkin Folk Folk Indie Reggae Big Band Hip Hop Soundtrack 80s Pop Alternative Blues Gamelan Progressive Rock Arabic Irish Jazz Dixieland Hindustani Sitar Funk Salsa Bluegrass Electronic Trance Progressive House Psychobilly Stravinsky Classic Rock Ska Dubstep Heavy Metal Harmonious-Disharmonious Orderly or Simple-Chaotic or Complex Sharp-Smooth Happy-Sad Texture-Music Association (TMA D ): The weighted average of the three textures picked as being most consistent with the music (C d,m ) minus the weighted average of the three textures picked as being most inconsistent with the music ( I d,m ) along a given dimension D: C d,m = (3c 1,d,m + 2c 2,d,m + c 3,d,m )/6, I d,m = (3i 1,d,m + 2i 2,d,m + 1 3,d,m )/6, TMA d,m = C d,m - I d,m 1 st 3 rd Choices 2 nd 2AFC Experiment Country Balkin Folk Bach Gamelan Folk Big Band Smooth Jazz Reggae Dubstep Mozart Psychobilly 80s Pop Irish Piano Electronic Ska Progressive Rock Salsa Heavy Metal Soundtrack Calm-Agitated Piano Bach Folk Mozart Country Smooth Jazz Balkin Folk Indie Big Band Soundtrack Reggae Hip Hop Blues Hindustani Sitar Jazz Dixieland Gamelan Progressive Rock Alternative 80s Pop Arabic Irish Funk Electronic Bluegrass Salsa Psychobilly Classic Rock Progressive House Stravinsky Trance Ska Dubstep Heavy Metal Heavy Metal Dubstep Ska Stravinsky Classic Rock Trance Psychobilly Progressive House Irish Funk Bluegrass Salsa Gamelan Electronic Dixieland Arabic Jazz Hindustani Sitar Blues Progressive Rock 80s Pop Alternatives Hip Hop Reggae Soundtrack Indie Big Band Balkin Folk Folk Country Smooth Jazz Mozart Bach Piano Ska Heavy Metal Classic Rock Trance Progressive House Psychobilly Stravinsky Gamelan Soundtrack Funk Hip Hop 80s Pop Reggae Big Band Alternative Balkin Folk Country Piano Indie Mozart Hindustani Sitar Smooth Jazz Electronic Jazz Blues Irish Bach Folk Salsa Progressive Rock Arabic Bluegrass Direct Music Ratings Texture Music Associations (TMA) Mozart Piano Folk Bach Country Smooth Jazz Balkin Folk Indie Big Band Reggae Hip Hop Soundtrack Alternative 80s Pop Gamelan Blues Irish Progressive Rock Jazz Dixieland Bluegrass Arabic Hindustani Sitar Electronic Funk Salsa Classic Rock Psychobilly Progressive House Stravinsky Trance Ska Dubstep Heavy Metal Heavy Metal Dubstep Ska Progressive House Stravinsky Trance Psychobilly Classic Rock Alternative Irish Bluegrass Gamelan Blues Jazz Salsa Funk Hindustani Sitar Big Band 80s Pop Progressive Rock Dixieland Hip Hop Reggae Electronic Bach Indie Mozart Arabic Folk Soundtrack Country Piano Balkin Folk Smooth Jazz 80s Pop Reggae Folk Soundtrack Dixieland Arabic Smooth Jazz Country Mozart Progressive Rock Electronic Funk Balkin Folk Piano Blues Hip Hop Big Band Bluegrass Alternative Progressive House Trance Gamelan Hindustani Sitar Bach Irish Classic Rock Indie Salsa Psychobilly Jazz Heavy Metal Stravinsky Ska Dubstep High correlations indicate support for the emotion mediation hypothesis (Palmer, S. E., Schloss, K., et al (2013) MDS of Emotional Space for thirty-four genres of music based on pairwise correlations of average emotional ratings Dimension 1 Dimension 2 Happy Sad Agitated Calm -1.5 -1 -0.5 0 0.5 1 1.5 -1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 Salsa Progressive Rock Arabic Bluegrass Alternative Blues Gamelan Bach Balkan Folk Country Classic Rock Dixieland Trance Progressive House Ska Electronic Piano Irish Soundtrack Indie Funk Hip Hop 80s Pop Psychobilly Mozart Dubstep Heavy Metal Hindustani Sitar Stravinsky Jazz Reggae Smooth Jazz Big Band Folk Correlations between the emotional content of the music and the corresponding emotional content of the colors chosen to go with it p < .05 0.2 0.4 0.6 0.8 1.0 Correlation (r) Calm/ Agitated Happy/ Sad Whim./ Serious Appeal./ Disgust. Pref. Thirty-four musical genres clustered by similar texture profiles*: Associations among music, texture, emotion, and non-emotional audio-visual features: Ska Heavy Metal Gamelan Psychobilly Progressive House Dixieland Salsa Dubstep Trance Big Band Funk Reggae 80s Pop Soundtrack Classic Rock Bluegrass Stravinsky Electronic Smooth Jazz Hip Hop Arabic Jazz Progressive Rock Irish Balkin Folk Blues Country Hindustani Sitar Piano Folk Bach Indie Alternative Mozart Granular-Fibrous Arabic Salsa Dixieland Electronic Funk Progressive Rock Hindustani Sitar Smooth Jazz Soundtrack 80s Pop Hip Hop Piano Reggae Bluegrass Balkin Folk Big Band Trance Indie Jazz Ska Blues Gamelan Stravinsky Country Classic Rock Mozart Folk Psychobilly Heavy Metal Bach Dubstep Irish Progressive House Alternative Curved-Straight Non-synesthetes make systematic associations between 34 musical genres and color. The associations are mediated by emotion (Whiteford, Schloss, Palmer, VSS-13). Twenty-eight different visual textures synthesized from spline curves: Do cross-modal associations exist between music and visual texture? Hindustani Sitar Indie Dixieland Bluegrass Thirty-four excerpts of songs, each representing a unique musical genre: Stravinsky Funk Trance Alternative 1 Sound-Texture Choices 1) Pick 3 textures that are most consistent with the music. Music-Feature Ratings Happy Sad Happy Sad Texture-Feature Ratings 2) Pick 3 textures that are least consistent with the music. 1 2 3 Rate features of each stimulus: Simple-Complex Granular-Fibrous Slanted-Not Slanted Curved-Straight Separate-Connected Sharp-Smooth Natural-Artificial Happy-Sad Calm-Agitated Weak-Strong Active-Passive Not Angry-Angry Harmonious-Disharmonious r = .92**** MUSIC RATING TMA r = .95**** MUSIC RATING TMA r = .89**** MUSIC RATING TMA r = .89**** MUSIC RATING TMA r = .91**** MUSIC RATING TMA r = .84**** MUSIC RATING TMA r = .55*** MUSIC RATING TMA r = .44** MUSIC RATING TMA Jazz Hip Hop House Blues Classic Rock Arabic -150 -100 -50 0 50 100 150 Dixieland Country Smooth Jazz Hip Hop Piano Balkin Folk Bach Soundtrack Folk Hindustani Sitar Gamelan Funk Mozart Jazz Big Band Indie Reggae 80s Pop Progressive Rock Salsa Electronic Blues Ska Irish Arabic Stravinsky Psychobilly Bluegrass Alternative Heavy Metal Classic Rock Progressive House Trance Dubstep Strong Weak r = .75**** MUSIC RATING TMA 0 0.5 1 0 0.5 1 Visual Dimensions Angry-Not Angry Calm-Agitated Weak-Strong Natural-Artificial Natural-Artificial Sharp-Smooth Sharp-Smooth Curved-Straight Curved-Straight Granular-Fibrous Granular-Fibrous Simple-Complex Simple-Complex Harmonious-Disharmonious Harmonious-Disharmonious Active-Passive Active-Passive Right Better Left Better Thick Thin Dark Background Medium Brightest Darkest Medium Bluegrass Blues Gamelan Bach Dixieland House Funk Piano Heavy Metal Folk Variance Explained 20 40 60 80 100 0 Thickness Contrast Lightness As with music-color cross-modal associations, non-synesthetes appear to make systematic cross-modal associations from music to texture. Music-texture cross-modal associations were mediated by emotion, but unlike music-color associations, happy-sad was not relevant to texture. Music-texture associations were also mediated by other visual features. Variance explained by perceived lightness, contrast, and thickness of visual textures Shared features: Angry-Not Angry Harmonious-Disharmonious Active-Passive Weak-Strong Texture-Music features: Sharp-Smooth Curved-Straight Granular-Fibrous Color-Music features: Happy-Sad Listen to Music Evaluate Emotional content and other features of the music Texture-Music Association Task: People chose textures with match- ing emotional content, and match- ing non-emotional audio-visual content Color-Music Association Task: People chose colors with matching emotional content Happy-Sad does not mediate texture-music associations r = -.03 MUSIC RATING TMA Visual Dimensions Emotional Dimensions *Contrast of texture corresponds to frequency of choice across all subjects Direct Music Ratings Texture Music Associations (TMA) Direct Music Ratings Texture Music Associations (TMA) Direct Music Ratings Texture Music Associations (TMA) Direct Music Ratings Texture Music Associations (TMA) Direct Music Ratings Texture Music Associations (TMA) Direct Music Ratings Texture Music Associations (TMA) Direct Music Ratings Texture Music Associations (TMA) Direct Music Ratings Texture Music Associations (TMA) Direct Music Ratings (TMA) First color picks for first twenty participants Bright Background Acknowledgements: We thank Will Griscom, Jan Flatley-Feldman, Vivian Wung, Candita Wager, Melissa Gertler, Sheila Rajagopalan for their help with this project. This material is based on work supported by the National Science Foundation under Grants Nos. 1059088 and 0745820, and by a Google Gift to the senior author, Stephen E. Palmer. -Whiteford, K., Schloss, K., Palmer, S., (VSS-2013). “Bach to the Blues” Presented at the 14th Annual meeting of the Vision Sciences Society, Naples, FL, May 2013. (r) -150 -100 -50 0 50 100 150 -150 -100 -50 0 50 100 150 -150 -100 -50 0 50 100 150 -150 -100 -50 0 50 100 150 -150 -100 -50 0 50 100 150 -150 -100 -50 0 50 100 150 -150 -100 -50 0 50 100 150 -150 -100 -50 0 50 100 150 -150 -100 -50 0 50 100 150 **** p < 0.00001 *** p < 0.0001 ** p < 0.001 * p < 0.01 -Palmer, S. E., Schloss, K., B., Xu, Z. & Prado-Leon, L., (2013) Music-color associations are mediated by emotion, Proceedings of the National Academy of Sciences. 110 (22), 8836-8841.