Classifying Motion Picture Audio
Eirik Gustavsen07.06.07
Outline
• Motivation • Thesis• State of the Art• Proposed system• Experimental setup• Results• Future work• Conclusion
Motivation
• Most projects classify clear classes or classes with noise.
• Few clear boundaries in motion picture audio• Subjective descriptions of movies• Dificult to compare movie content
Thesis
It is possible to automatically create a table of contents of a motion picture, based on its audio track only.
Research questions
• Find best LLDs to classify motion picture audio
• Detect boundaries between audio classes within complex audio segments
• Automatically create a TOC based on the audio track only
Pre-Processing44100 Hz sample rateMono16 bits
30 ms windows (LW)
Low Level Descriptors
Time domain Frequency domain
Low Level Descriptors
• Total of 23 low level descriptors
TIME DOMAIN
• Audio Power• Audio Wave Form• Root-Mean Square• Short Time Energy• Low Short Time Energy Ratio• Zero-Crossing Rate• High Zero-Crossing Rate Ratio
FREQUENCY DOMAIN
• Audio Spectrum Centroid• Fundamental Frequency• 10 Mel-Frequency Cepstral Coefficients• Spectrum Flux
Dimensionally reduction
Principal components analysis (PCA) is a technique used to reduce multidimensional data sets to lower dimensions for analysis.
f(1)f(2)f(3)f(4)f(5)...f(23)
PCAd(1)d(2)d(3)
K Nearest Neighbors
Proposed system
Pre- Prosessing LLD Norm
PCAKNNPost- Prosessing
TOC Generation
Classifying Audio
Speech
Noise (white)
Music
”Silence”
Mixed audio classes
Class Boundary Detection
Class Boundary Detection
Class Boundary Detection
Finding most suitable LLDs
Most Suitable:
ASCAWFRMSHZCRR
Sample Results
Music with low volume
Clear speech
Speech with background environmental sounds
Fading between music and speech
Speech with Background music
Jingle
” Some mistakes”
Future Work
• To be done in this thesis– Post processing– TOC
• Open research questions for future works– New motion picture audio classes– Detecting sound objects– Speech recognition
Conclusion
• Pre-processing makes it possible to classify motion picture audio correctly
• Using right combination of LLDs enhances the result of the classification
Questions
?
