Music Coloring Music Coloring for Biofeedback for Biofeedback Presented By Presented By Eyal Balla Eyal Balla [email protected] [email protected]
Jan 24, 2016
Music ColoringMusic Coloringfor Biofeedbackfor Biofeedback
Presented ByPresented By
Eyal BallaEyal Balla
[email protected]@tau.ac.il
The Demo System
EKG Monitor
PC with A/Dand music coloring software
Patient
Original Music
Colored Music
Music Coloring What is it?
Music coloring is changing of music in different aspects (such as pitch, rhythm, timber, volume, reverberation and other aspects of the music) in order to embed data into the music.
Music Perception in Humans
Recent research shows that several distinct neural functions exist in music perception: Volume Pitch Rhythm Timber Direction of source of music
This introduces the possibility of multidimensional display with audio.
Moreover music has been shown to have effect on human breathing and heart rate
Music Perception in Humans
Visual DisplayAuditory Display
Number of dimensions is limited to 3D
Able to percept many dimensions such as pitch, rhythm, timber etc.
Uses full attention of human using it.
Uses partial attention. Other tasks can be done simultaneously.
Previous Works in Auditory Display
Sonification Mappings: sMax: A Multimodel Toolkit for Stock Market Data
Sonification. Maps stock market data into multi-streams of audio. User has control of mapping. Requires training. Mapping is through pitch, pulse speed, brightness and attack time of different instruments
MarketBuzz. A sonification of real-time financial data. 4 behaviors were identified and sonified in beeps and pitch sonifications. Requires training.
Previous Works in Auditory Display
Polyrhythm in the Human Brain. Sonification of EEG brain waves in four different streams according to EEG wave types.
Listening To the Mind listening. Sonification of ECG heart rate and two streams of EEG with difference between the two temporal sections (above ears) were used as stereo positioning.
Previous Works in Auditory Display
Feedback Systems: Auditory Feedback of Human EEG for Direct
Brain Computer Communication. A spelling machine for the handicapped was built. User would use SCP EEG brain waves to control it. Learning was through auditory instruction. They also found other EEG waves controlable using multi-stream audio teaching.
Previous Works in Auditory Display
Previous Works in Auditory Display
Data Analysis through Auditory Display: Applications in Heart Rate Variability. The work deals with creating a auditory system for heart rate variability analysis. The auditory display created is made of several streams each representing a different aspect of heart condition. The work mentions the importance of using pitch and volume.
Conclusion of Current Works
Several orthogonal features of music have been proved as distinguishable by humans in experiments
Sonification is a good way of displaying multidimensional data.
Works usually create new music according to data streams extracted from the raw data.
Current Directions
Last year’s workshop created an interface which inserts samples of EKG into MAX/MSP environment using a A/D sampler.
The workshop showed that online real-time analysis of this data is possible using the MAX/MSP development environment.
Current Directions
We are looking for auditory “Axis” which can be precept orthogonally.
We are using already build components which color music to learn the reaction graph of humans to change in different music features.
References
The structural Components of music perception. Herve´ Platel, Cathy Price, Jean-Claude Baron, Richard Wise, Jany Lambert, Richard S. J. Frackowiak, Bernard Lechevalier and Francis Eustache
Max/Msp 4.5 Documentation. http://www.cycling74.com/products/dldoc.html
sMax: A Multimodel Toolkit for Stock Market Data Sonification. Fabio Cifariello Ciardi
MARKETBUZZ: Sonification of real-time financial data. Petr Janata and Edward Childs
References
Polyrhythm in the Human Brain. Thomas Hermann, Gerold Baier, Markus Muller.
Listening To the Mind listening. Hans van Raaij
Listening to the mind listening concert. http://www.icad.org/websiteV2.0/Conferences/ICAD2004/concert.htm
Auditory Feedback of Human EEG for Direct Brain Computer Communication. Thilo Hinterberger,
Gerold Baier, Jürgen Mellinger, Niels Birbaumer Data Analysis through Auditory Display:
Applications in Heart Rate Variability. Mark Ballora
MAX/MSP Introduction
MAX/MSP is an object oriented development environment for music synthesis and creation.
The MAX part of the environment handles MIDI format manipulation.
The MSP part of the environment handles signal processing and manipulation.
MAX/MSP Example
The following example outputs via audio several 2 dimension data items using 2 streams of data for pitch and volume.