Google Docs for Music

Building a Collaborative Music Building a Collaborative Music Notation Software with HTML 5Notation Software with HTML 5

ByBy

Joshua Koo

Building a CollaborativeBuilding a CollaborativeMusic Notation SoftwareMusic Notation Software

with HTML 5with HTML 5

Text?

Word ProcessorsWord Processors

Music?

Music Notation SoftwareMusic Notation Software

What is Music Notation Software?

Software for writing musicg

Produces sheet music or enable music playback



Col·lab·o·ra·tion

nounnoun

The act of working togetherWebster's Revised Unabridged Dictionary

Collaborative

dj tiadjective

Working together toward a common endg g



(R l Ti ) (Real-Time) bCollaborative Software

Mother of All DemosMother of All Demos



HTML 5HTML 5

the next major revision of the HTML standard, currently under development.y p

(wikipedia, Nov10 2010)

“A plateform”phttp://blogs.msdn.com/b/tims/archive/2010/10/03/html5-a-specification-or-a-

platform.aspx



Existing PackagesExisting Packages

Finale

SibeliusSibelius

Noteworthy Composer

Anvil Studio

LilypondLilypond

Musescore

Noteflight

MotivationsMotivations

Music is a collaborative activity,

So why not?y

W h ll b i / i diWe have collaborative text/programming editors,

So why not?y

Nobody else seems to be doing itNobody else seems to be doing it,

So why not?

Building ChallengesBuilding Challenges

Music software alone is very difficultMusic software alone is very difficult

Decades of research and development

What Design Patterns to use?

Building a (Real-Time) CollaborativeBuilding a (Real Time) CollaborativeMusic Notation SoftwareMusic Notation Software


So, can we have a Google Docs for music?

Let’s Think About ItLet s Think About It

Ah!Ah!

Targeted PlatformsTargeted Platforms

ServerServer

(Portable Colinux) Ubuntu Linux 9

ClientClient

Modern Web Browsers

Modern BrowsersModern Browsers

• Google Chrome Browser 4

• Apple Safari 5

• Mozilla Firefox Browser 3 5• Mozilla Firefox Browser 3.5

• Opera Browser 10.50Opera Browser 10.50

• Microsoft IE9 Developer Platform Preview 3

• Others? – Webkit based, Mobile

Development EnvironmentDevelopment Environment

• Intel Quad Core Q8400, Windows 7, 3GB ram, Desktopp

D ll I i Wi d Vi GB• Dell Inspiron 1440, Windows Vista, 2GB ram, Laptop

A E PC 1002HE Wi d XP At• Asus Eee PC 1002HE, Windows XP, Atom Processor, 1GB ram, Netbook

Lean Software DevelopmentLean Software Development

IterativeIterative

Continuous Integration (CI)

Reduce Waste

Architectural OverviewArchitectural Overview

Brrowser/Clie

1. UI component3. Canvas Framework

2. Notation 4 Audio ent Side

Core 4. Audio Framework

5. Network Layer

CouchDB Node.js

Socket.io

6. Server Components

Server Side ModuleServer Side Module

• Hosts the web application

• Handles connections from clientHandles connections from client

• Connects to the persistence layer

• Node jsNode.js

• CouchDB

Node jsNode.js

• Non blocking I/O

• Event DrivenEvent Driven

• V8 Javascript Engine

• Can Handle 100,000 connections (solves C10K problem http://www.kegel.com/c10k.html ) p p // g / )http://amix.dk/blog/post/19490

Sh d ith li t id• Share code with client side

CouchDbCouchDb

• Document Orientated – Not relational, not OO database

• MVCC

R f l• Restful

• Schema-Free

• Distributed

• Javascript Query Language

Concurrencies IssuesConcurrencies Issues

• State always reflected in Database

• Music score first persistedMusic score first persisted

• Deltas also persisted

• Disconnected Dataset

• Updates never persisted by unsynchronizedUpdates never persisted by unsynchronized clients

• Optimistic Locking

Database DesignDatabase Design

Music Action LogsMusic Action Logs

CouchDB-Node js CommunicationCouchDB-Node.js Communication

• Save Music/ Action Pseudo– New? Create and save new Document

– Old? Are you using the lastest revision?• Yes? Save• Yes? Save

• No? Send Conflict Feedback

Tricky Scenarios ITricky Scenarios I

User Alice• Connected

User Bob• Connected

• Synchronized • Synchronized

• Disconnected

• Make Changesg

• Reconnected

Tricky Scenarios IITricky Scenarios II




• Disconnected

• Make Changes

• Reconnected

g

Tricky Scenarios IIITricky Scenarios III




• Disconnected

• Make Changesg

• Reconnected

• Make Changes


Brrowser/Clie




5. Network Layer

CouchDB Node.js

Socket.io


Network ModuleNetwork Module

• Communication with the server for real time collaboration

• In web browser, “real-time” solutionsAJAX/XHR P lli / L P lli / St i– AJAX/XHR Polling / Long-Polling / Streaming

– Html5 Websockets

• Handles Network Protocol

• Socket io and Socket io node• Socket.io and Socket.io-node (http://socket.io/)

Network ProtocolNetwork ProtocolC ti t t d lt• Can pass entire state or deltas

• JSON messages

• Types – Actions– Status

M– Message– Etc…

• Node to take care of client+ document sessions


Brrowser/Clie




5. Network Layer

CouchDB Node.js

Socket.io


Audio FrameworkAudio Framework

• Handles Music playback in the browser

• Possible SolutionsPossible Solutions– Server side solution

B di– Base64 encoding

– Utilize Flash Sound Object

– Bridge Java access to the sound system

• Implemented a loadable sound engine module• Implemented a loadable sound engine module

Bridged Applet InterfaceBridged Applet Interface//Obj i h l d bl l//Object is the loadable appletObject.loadMusic(musicJson);Object.play(musicJson);

//register javascript interfaces for JS to call ASExternalInterface.addCallback("loadMusic", loadMusic);ExternalInterface.addCallback("play", play);ExternalInterface.call('jabtunes.sound.loaded', true);

For java, a JSObject package needs to be imported and called.JSObject address = (JSObject) jso.eval("'jabtunes.sound.loaded();");

Sound EnginesSound Engines

• Standingwave

• Pitch Shifting

• FluidSynth Alchemy Port

• Sion Project

• Java Midi Applet


Brrowser/Clie




5. Network Layer

CouchDB Node.js

Socket.io


Notation CoreNotation Core

• Handles Complex Music Notation Logic

• The glue for different componentsThe glue for different components

• Consist of following components– Undo / Redo Stack

– Music Data Structures

– Music Interpretation / Conversion

Note Spacing Algorithms– Note Spacing Algorithms

Design PatternsDesign PatternsMVC?MVC?

Modified MVC

Rule based?Used for various sub-componentsp

Event driven?Observer pattern used when possible

D t t ti ?Data representation?JSON / jObj

The JSONThe JSON

• JavaScript Object Notation

• lightweight data-interchange formatlightweight data interchange format

• Text format

• Language independent

• Subset of JavascriptSubset of Javascript

• Name value pairs and ordered list

• http://www.json.org/

A Json DatastructureA Json Datastructure{{

“title": “This is my music”,“composer": {

“ “ h ”“name": “Me, Joshua”,“year": “4 Nov 1985”

},“staff": {

“notes” :[ {"dur": 32,,"line": 2 }, {"dur": 8,"line": 5line : 5

}]}}

Name SpacesName Spacesb• jabtunes

• jabtunes.editj• jabtunes.font• jabtunes layout• jabtunes.layout• jabtunes.music• jabtunes.network• jabtunes soundjabtunes.sound• jabtunes.util

Usage of JSONUsage of JSONP i D b S f J i• Persistence – Database, Storage of JavascriptFonts (CUFON format)

• Music Data• Configurationg• Namespaces• Network API - Data transmission between client• Network API - Data transmission between client

and server• Audio API Data exchange between app and• Audio API – Data exchange between app and

sound engine

Undo / RedoUndo / Redo• Memento?Memento?

• Command Patternvar action = {

toDo: [this.makeAction('set', 'score.staff.'+uid, o),this.makeAction('add', 'score.staffIndex', uid)

],toUndo: [

this.makeAction('remove', 'score.staff.'+uid),this.makeAction('remove', 'score.staffIndex', uid)

],desc: "Add Staff"desc: dd Sta

};

• Command Pattern used• Command Pattern used• However need special considerations for

ltimulti-users

Local StorageLocal Storage• Caching Big Files – FontsCaching Big Files Fonts

• Saving Music Score / Offline

N S iNote SpacingD i l i l f• Determine actual time value of notes

• Use logarithmic spacing by time valueg p g y

• Flatten staffs, get absolute time of note, t ff d t th id tcompare across staff, expand to the widest

The jObjThe jObj

• jobj extends json

• Hashmap like features with setter getterHashmap like features with setter getter

• Xpath like features

• Used forUsed for – main score object

– Used for custom canvas handler

Sample jObj usageSample jObj usagej j b j bj()var j = new jabtunes.jobj();

j.set(‘score.title’, ‘New Title’);j.set(‘score.composer, ‘Joshua Koo’);j.set( score.composer, Joshua Koo );j.set(‘score.staff.instrument, ‘violin’);j.set(‘score.staff.notes, {});j.get(‘score.owner’); // returns ownerj.add(‘score.collabrators’,’joshuakoo’); // Add t th li t f ll b t// Adds a name to the list of collabrators

Main Music DatastructureMain Music Datastructure

• Staffs

Notes– Notes

– Symbols

• Graphical Objects

• Title of the Piece

• Most items are hashed and UUIDed!• Most items are hashed and UUIDed!

• Etc etc.

Architectural pattern?Architectural pattern?

ViewView

Modelb

graphicalDoc

Controller musicDoc

abstractDoc

MediatorsMediators


Brrowser/Clie




5. Network Layer

CouchDB Node.js

Socket.io


Canvas Rendering FrameworkCanvas Rendering Framework

• Layer dealing with lower level drawing api

• arrangement and display of music symbolsarrangement and display of music symbols and objects in the browser

H dl d i i i lli• Handles rendering, repainting, scrolling,..

• Also handles Event handling of graphical g g pobjects

Canvas DrawingCanvas Drawing

Design of Virtual Canvas SpaceDesign of Virtual Canvas Space

• Handles 10K objects

• 120K wide virtual pixel canvas space120K wide virtual pixel canvas space

Custom HandlersCustom Handlers

_handlers = new jabtunes.jobj()_handlers.set('mousedown', {});handlers set('mouseup' {});_handlers.set( mouseup , {});

_handlers.set('mouseclick', {});_handlers.set('mousedrag', {});_handlers.set('mousemove', {});handlers set('mouseout' {});_handlers.set( mouseout , {});

_handlers.set('mouseenter', {});

addHandler('mousemove', 'dragnote', function(x,y) {}


Brrowser/Clie




5. Network Layer

CouchDB Node.js

Socket.io


UI ModuleUI ModuleG l k d l• General Look and Feel

• Deals with High Level Interactionsg• Uses DOM, CSS.

• jQuery• Menus• Window LayoutWindow Layout• Dialogs

UI ComponentUI Component

DemonstrationDemonstration

ConclusionConclusion• Working prototypeWorking prototype

– Simple Collaborative featuresSafe persistence– Safe persistence

• Custom components– UI Widgetsg– Canvas– Audio frameworkAudio framework– Network Stack

Areas for Future ImprovementAreas for Future Improvement

• UX and User studies

• Collaborative Undo/Redo

• Merge Algorithms

• Features for making a ready-to-market

dproduct

Lessons LearntLessons Learnt

• Appreciate of academic papers

• Experience old and cutting edge technology

• Understanding challenges

• Satisfaction in solving one problem at a time

• And pushing the limits

References / PapersReferences / PapersC ll b ti S l ti f M i Ed tiCollaborative Solution for Music Education

Automatic formatting of Music Sheets

Justification of Printed Music

Music Notation as Objects - An Object-Oriented Analysis of the Common Western Music Notation System by Kai Lassfolk 2004

Algorithms and Data Structures for a Music Notation System based on GUIDO Music Notation

KRenz, 2002

CouchDB http://couchdb.apache.org/docs/intro.html

AcknowledgmentsAcknowledgments

My supervisor, Professor Tan Kian Lee

Associate Professor Abhik

Assistant Professor Wang Ye

Professor Bernard Tan Tiong Gie

Final Questions?

[email protected]

http://www.facebook.com/joshuakoop j

Thank you! :)Thank you! :)

Google Docs for Music

Technology