Web Audio API Let your WebGL program dance
Jan 19, 2018
Web Audio APILet your WebGL program dance
Announcement• Final project proposal due in one week.• Choice of demo date?• Plan for the 12/23 class.
2
Final Projects• Individual or Team of 2.• Voting during the demo.• Do NOT use THREE.js• Themes for this semester: Interactive Art in
WebGL– Animating objects with changing colors,
positions, shapes, …etc.– Interacts with mouse input or audio input (music).
3
Possible Interactions• Input:
– Mouse, keyboard– Audio– Video or camera?
• Output – changing the following– Size, number, position, color, …etc.– Viewpoint– Lighting
4
For Further Reading• Web Audio API:
– Good introduction at: http://www.html5rocks.com/en/tutorials/webaudio/intro/
– More information on the frequency analyzer: http://srchea.com/experimenting-with-web-audio-api-three-js-webgl
• Watch out for deprecated functions:– e.g., createJavaScriptNode() replaced by
createScriptProcessor()
5
WEB AUDIO6
Loading the Music: Option #1
• Option #1: <audio> element in HTML• The following code produces an audio
element with playback controls.
7
<button id="xButton">Rotate X</button><button id="yButton">Rotate Y</button><button id="zButton">Rotate Z</button><br/><button id="pButton" ...>Pause</button><button id="dButton" ...>Depth Test</button><br/><audio id="myAudio" src="Sleep Away.mp3" controls></audio>
A Simple Player
// Experimenting with HTML5 audio var context = new AudioContext(); var audio = document.getElementById('myAudio'); var audioSrc = context.createMediaElementSource(audio);
audioSrc.connect(context.destination);
audio.play();
Adding Frequency Analyzer
var context = new AudioContext(); var audio = document.getElementById('myAudio'); var audioSrc = context.createMediaElementSource(audio); var sourceJs = context.createScriptProcessor(2048);
analyser = context.createAnalyser(); analyser.smoothingTimeConstant = 0.6; analyser.fftSize = 512;
// Connect the MediaElementSource with the analyser audioSrc.connect(analyser); analyser.connect(sourceJs); sourceJs.buffer = audioSrc.buffer; sourceJs.connect(context.destination); audioSrc.connect(context.destination);
sourceJs.onaudioprocess = function(e) { // frequencyBinCount: how many values from the analyser frequencyData = new Uint8Array( analyser.frequencyBinCount); analyser.getByteFrequencyData( frequencyData );};
Visualizationfunction render() {
...
// update data in frequencyData analyser.getByteFrequencyData(frequencyData);
// render frame based on values in frequencyData gl.uniform1f( volumeLoc, frequencyData[160] / 255 ); gl.drawArrays( gl.TRIANGLES, 0, numVertices );
requestAnimFrame( render );}
Loading the Music: Option #2
• Option #2: XMLHttpRequest() in JS
12
loadSound("your_song.mp3");function loadSound(url) { var request = new XMLHttpRequest(); request.open('GET', url, true); request.responseType = 'arraybuffer'; // When loaded decode the data request.onload = function() {
context.decodeAudioData( request.response, ... ...}
request.send();}
Lab Time!• Download cube3_music.zip• Change frequencyData[160] to other entry.
– How does it respond to the music?• Uncomment console.log(frequencyData)
– What do you see in the console?• How about visualizing the whole array of
frequencyData[0..255]?
13