The Music Sphere Name: Andreas Papadopoulos Degree: MA in Music Technology Supervisors: Oliver Larkin and Dave Malham Date: 31-08-2010
Mar 09, 2016
The Music Sphere
Name: Andreas Papadopoulos
Degree: MA in Music Technology
Supervisors: Oliver Larkin and Dave Malham
Date: 31-08-2010
Abstract
This project report describes the concept and the development process of a multi-
touch musical interface, the Music Sphere, an application made for the iPad. The
main means of interaction between the user and the produced sounds is a Sphere.
As the user performs gestures on the screen, various visual and audio effects are
applied to the sphere and the sounds respectively, resulting in an interactive
experience. The project researches into existing multi-touch technology and musical
interfaces that rely on that technology. It also explores gestural interactivity as a way
of composing and manipulating electronic music in real time. Such multi-touch
interfaces present a more natural way of control and can consequently help in crucial
elements of a live performance such as expressiveness and spontaneity.
Y4796421
2
Table of Contents
Glossary 5
1. Introduction 6
1.1 History of Multi-touch Technology in HCI 6
1.2 Overview of this work 8
2. Multi-touch Technologies 9
2.1 Optical-based 9
2.2 Resistive 11
2.3 Capacitive 12
3. Multi-touch Technology in Music 13
3.1 The reacTable* 13
3.2 Jazzmutant Lemur 17
3.3 Apple iPhone/iPad 20
3.4 Subcycle 26
4. The Music Sphere 27
4.1 The Concept 27
4.2 Scene One - Pre-Recorded Material 29
4.3 Scene Two - FM Synthesiser 31
Y4796421
3
4.4 Other Synthesis Methods 32
5. Development Process 33
5.1 Program Structure 33
5.2 The C++ Sphere Method 35
5.3 Gestures 38
5.4 Communication Protocol 40
6. Conclusions 44
6.1 Evaluation 44
6.2 Further Development 45
7 References 47
8 Appendices 52
8.1 Sphere C++ Function 52
8.2 Apple SDK Gesture Recognisers 53
8.3 Custom Gesture Recogniser 55
8.4 Custom Gesture Recogniser Responders 57
8.5 OpenGL ES 58
8.6 OSC 58
Y4796421
4
Glossary
AM - Amplitude Modulation
CPU - Central Processing Unit
DAW - Digital Audio Workstation
DI - Diffused Illumination
DIY - Do It Yourself
DSI - Diffused Surface Illumination
FFT - Fast Fourier Transformation
FM - Frequency Modulation
FPU - Floating Point Unit
FTIR - Frustrated Total Internal Reflection
GPU - Graphics Processing Unit
GUI - Graphical User Interface
HCI - Human Cmputer Interaction
LFO - Low Frequency Oscilator
LLP - Laser Light Plane
MaxMSP - Object oriented audio programming environment
MCU - Mackie Control Universal
MIDI - Music Instrument Digital Interface
NI - Native Instruments
OpenGL ES - OpenGL Embedded Systems
OSC - Open Sound Control
PD - Pure Data
PDA - Personal Digital Assistant
RGBA - Red Green Blue Alpha
SDK - Software Development Kit
Y4796421
5
1. Introduction
1.1 History of Multi-touch Technology in HCI
In 1997 Matsushita and Rekimoto unveiled the HoloWall, a multi-touch sensitive
interactive wall. The “wall” consisted of a sheet of glass coupled with a sheet of rear-
projection material. Behind it, a projector was used to display material while an
infrared light source and a camera were used for the touch detection. Anything that
would touch the glass wall would reflect the infrared light back to the camera
(Matsushita, 1997:209).
In 2004, Wilson presented TouchLight, a technology suited for installations and
wall interaction. It uses two cameras placed behind a semi-transparent sheet of
acrylic, which acts as the interactive surface. The outputs of these cameras are
combined and further processed with various image processing techniques to result
in touch detection. This technique also uses a projector to display material on the
acrylic surface (Wilson, 2004:69).
Recently, multi-touch systems have become more affordable. Han (2005:115)
presented such a low cost multi-touch technique in 2005 called Frustrated Total
Internal Reflection. Infrared light is emitted within a sheet of acrylic which forms the
touch screen surface. Contact with objects or fingers on the screen will cause the
Infrared light to scatter towards a camera. After analysis on the captured data is
possible to track the spots that are being touched.
In June 2007 Apple Computer releases the iPhone, a cellular phone device
that features a 480x320 multi-touch screen. It achieves multi-touch sensitivity by
using a technology where tiny electrical fields are spread across the multi-touch
surface. These fields will change their value when they are in contact with a finger.
In 2007 Microsoft Research Cambridge presented ThinSight, an optical
sensing system that can be used in conjunction with standard LCD displays to form
Y4796421
6
multi-touch panels. ThinSight’s operational principle relies on the properties of a
optosensor, a device that can both emit and detect amounts of infrared light. When
an object or finger touches the surface, some of the emitted light is reflected back to
the optosensor and that is how detection is achieved. Multiple optosensors are used
in a grid configuration to cover the display surface and transform it into multi-touch
system (Hodges, 2007:259).
There is also a number of interfaces that demonstrated multi-touch
functionality and were oriented towards musical expression and composition. The
AudioPad works by placing electromagnetically tagged objects on a surface. Each
object includes a simple circuit with a capacitor that resonance at different frequency.
A specifically designed antenna can measure the amplitudes of the resonant
frequencies and, as a result, track the object. Feedback is provided by a projector
which is placed above the user (Patten, 2002:1).
The SoundScape Renderer is a software system aimed to reproduce spatial
audio. The current version can render acoustic scenes by either Wave Field
Synthesis (WFS) or Vector Based Amplitude Panning (VBAP). The multi-touch
SoundScape renderer is a multi-touch controller for the SoundScape Renderer
software. It is basically a FTIR multi-touch table (Han, 2005, 115) that controls
various parameters of the software in real-time, allowing multiple musicians to
collaborate in creating acoustic music (Brown, 2009:446).
DrawSound, introduced in 2008, combines multi-touch input technology with
the expressive characters of drawing on the ordinary paper in a sound performance.
It uses a DiamondTouch multi-touch interface for touch sensitivity and data extraction
while MaxMSP is used for mapping the data from DiamondTouch to different sound
synthesis techniques. DrawSound has been used in two projects, the Sine Wave
Quartet and aeo (Jo, 2008:59).
Y4796421
7
1.2 Overview of this work
Multi-touch interaction both in Human Computer Interaction (HCI) and in music is a
area with particular technological interest. The initial goal for this project was to
develop a multi-touch interface targeted for musical interactivity. The first and most
important principle in this work was to avoid conventional and widely used control
concepts such as sliders, knobs and buttons. Instead, the interface explores and
evaluates the scenario where a 3D shape like a sphere could serve as the main
means of interaction between the performer and the music.
Despite the fact that a custom build multi-touch interface could be possible,
the development of this project is targeted towards the Apple iPad multi-touch device
for a number of different reasons. First of all, the iPad’s multi-touch technology is
proven to work effectively without issues, eliminating any errors and instability that a
custom interface might have. In addition, there is a growing community of developers
who provide support and help to new and inexperienced developers that wish to take
the first steps into programming. There are also numerous examples bundled in the
iPhone SDK which make the development process easier. Finally, developing
applications for such multi-touch mobile devices is an area of particular interest for
me, as it open the possibility of developing commercial applications in the future.
The objectives that were set for this project where very straightforward: to
create a multi-touch musical prototype interface where the user could interact with
audio. The project also evaluates the level of success of using the 3D concept for
controlling music through gestural interactivity. In addition, the development process
would enable me to investigate how multi-touch interfaces work as well as make me
familiar with developing apps for the iPhone, which is something I would like to do in
the future.
Y4796421
8
2. Multi-touch Technologies
Different inventors have taken different technological approaches to achieve multi-
touch functionality, depending on factors such as cost, ease of construction, size of
the interface and more. There are three main categories of technology that evolved
over the years which are: optical-based, resistive and capacitive. Each one has its
own weaknesses and strengths while which one is the best depends on the desired
functionality of the final interface. As a general rule, optical-based multi-touch
technology is preferable for installations and large multi-touch surfaces while the
remaining technologies are suited for smaller surfaces and commercial devices.
2.1 Optical-based
One very commonly used method is achieving touch sensing via optical techniques
and IR light sensing. These approach to multi-touch sensing requires an optical
device (camera), IR light generator (IR LEDs or other) and an visual feedback device
(LCD display or digital projector). There is a number of variations in the construction
when utilising optical multi-touch sensing methods each having its positive and
negative aspects.
Generally, all the optical techniques share the same operational principle. The
camera scans the touch surface at a fixed amount of frames per second and the
resulting images are processed with various image rectification algorithms in order to
eliminate any errors that might occur. The resulting clearer view of the interaction
points, or else “blobs”, is fed to the tracking algorithm. The tracking algorithm starts
here and exports data as lists of XY coordinates which are used by the software that
runs on the computer to perform tasks (e.g. make sounds). These XY coordinates
are also used to crate visuals that the projector displays on the touch surface to
provide feedback to the user.
Y4796421
9
Frustrated Total Internal Reflection (FTIR)
This multi-touch sensing technique relies on Total Internal Reflection, a well-
known optical phenomenon. In FTIR, an acrylic sheet (about 1 cm thick) is run
through IR light which is emitted by IR LEDs that are placed along the perimeter of
the sheet. The IR light is trapped inside the acrylic surface and whenever the user
interacts (a finger comes into contact with the surface), the IR light finds a way to
escape the acrylic surface as it is scattered by the skin (the light is frustrated). This
light is then scanned by the camera (Han, 2005:115).
Diffused Illumination (DI)
The Diffused Illumination technique comes in two variations, Front Diffused
Illumination and Rear Diffused Illumination. Both methods rely on the shadow that is
created when a finger touches a surface that is projected with light. The Front
Diffused Illumination method uses visible light which comes from above while the
Rear Diffused Illumination utilises IR light that is place underneath the touch surface
(NUI Group Authors, 2009:13).
Laser Light Plane (LLP)
The LLP technique differs from the others in terms of the light source, as it
uses laser technology instead of IR light. The light is shed just above the surface so
when something touches the surface the camera will pick up a blob.
Diffused Surface Illumination (DSI)
DSI combines characteristics from the FTIR and DI methods. Physically, it is
very similar to FTIR technique (a frame of IR LEDs is needed) apart from the fact that
a special type of acrylic is needed, the EnLighten Acrylic. Due to the nature of this
special acrylic, the light get more even distribution than the DI method (NUI Group
Authors, 2009:15).
Y4796421
10
All of the aforementioned techniques have their strengths and weaknesses
and choosing which one is the most suitable depends largely on the applications and
features that are needed. Also, some of these techniques might be affected by
external factors such as the ambient light and they might only be suitable for
particular applications (installations etc).
Since the camera is a very important aspect of this touch sensing technology,
the reliability and usability of such a system will depend largely on the quality of the
camera. First of all, the camera needs to be sensitive to IR light, since this is used for
the finger tracking. Many consumer USB cameras utilise an IR filter to filter out the IR
information as this band is not visible and not needed in an average consumer
device. The camera has to be modified in order to be sensitive to the IR light by
taking the IR filter out. Also, other technical characteristics (specifications) of the
camera device are important such as the resolution and the frame rate. Typically, a
standard VGA 640x480 pixels frame size is adequate but more resolution will
improve precision. Also, the frame rate plays an important role in the responsiveness
of the system and largely determines the latency of the system. Finally, it would be
sensible to go through the technical specifications of the camera to see the exact
response of the lens to IR light and find IR LEDs or other IR light source that emit IR
light in the IR band that the camera is most sensitive at (Muller, 2008:11).
2.2 Resistive
Resistive multi-touch sensing is achieved by implementing a structure of two
electrically conductive layers coated with indium tin oxide (ITO) separated by a thin
insulating layer. A specified amount of electric current is sent to one layer while the
current of the other layer is being monitored. When the user touches the screen, the
two layers are connected and that makes some electrical current to pass through the
Y4796421
11
two layers. By measuring the amount of that current it is possible to determine the
exact position of the touch (Schöning, 2008:2).
The production of resistive touch screens is cheap and as a result such touch
screens have been very successful commercially. Numerous mobile devices, PDAs
and portable gaming devices like the Nintendo DS feature such a touch screen not
only because its affordable, but also because it is durable and robust under normal
use conditions.
However, providing additional protection to the screen by adding a protective
layer is not an option because the touch sensing responsiveness decreases. As a
result, the touch sensor is vulnerable to damage by sharp objects. Moreover, adding
a resistive touch screen layer on top of an existing LCD or TFT screen reduces the
clarity of the screen to approximately 75-85%, resulting in a blurry image. (Schöning,
2008:2) Finally, resistive touch sensing is considered as a low-resolution technology
because of the fact that the screen responds only when a touch is made at the exact
point of the resistors.
2.3 Capacitive
Electrodes are placed around the edges of the screen surface, establishing an
electrical field of a specified amount. Whenever a finger comes in touch with the
screen, an amount of the electrical charge is passed through, resulting in an
alteration of the original electrical field. By measuring the amount of current that is
passed through the adjacent object the position of the touch is determined
(Schöning, 2008:3).
Capacitance based multi touch technology does not reduce the clarity of the
screen like the resistive technology and that is an important advantage. This
advantage in combination with features like robustness and durability makes this
Y4796421
12
technology more suitable for a much wider application area. Capacitive multi-touch
surfaces can be used equally well in installations and in commercial products where
reliability is crucial.
This technology has its own disadvantages, such as the high construction
cost. In addition, capacitive sensing can only respond touches made with fingers or
other objects with similar electrical properties, making use of regular stylus pens not
possible. However, special stylus pens are becoming available.
3. Multi-touch Technology in Music
As seen in the introduction, multi-touch technology has not left the music community
unaffected. Various projects have been undertaken either by researchers or
commercial companies in quest of new and innovative ways to incorporate this
promising new technology in musical interfaces. Several projects have been
successfully completed and presented to the public.
This section covers examples of multi-touch musical interfaces that use the
three different types of multi-touch technology presented in the previous chapter. The
fact that each of these projects achieve multi-touch functionality by utilising different
type of multi-touch technology, proves that musical expression is not affected by the
technology itself but rather, by the principle of multi-touch interaction.
3.1 The reacTable*
The reacTable is an modern musical instrument that features a circular multi-touch
surface. In addition, it is a collaborative instrument, meaning that more than one
users can operate the instrument at the same time. The users provide input to the
device by placing, moving and rotating pre-determined objects on the illuminated
multi-touch surface. Each object represents a link in the audio signal chain which
Y4796421
13
affects the sonic outcome in different ways. Various parameters of this signal path
are controlled by rotating the objects or moving them to be either closer or further
from each other.
The reacTable is the perfect example of FTIR multi-touch technology,
presented by Han in 2005. Physically, the reacTable is a round table of one meter
diameter, space that in most cases will be enough for two to four performers
simultaneously. Infrared emitters provide the necessary infrared illumination which
runs through the acrylic surface. Whenever an object of a finger comes in contact
with the surface, the infrared light is scattered towards in the inside of the reacTable’s
body and picked up by the infrared-sensitive camera. The images captured by the
camera are processed and analysed by the reacTIVision software, which produces
data based on that analysis (Kaltenbrunner, 2009:9). Examples of such data are the
the exact position of a finger touch or shape recognition. The visuals are
accomplished with a digital projector which projects the generated images on the
acrylic surface.
As described earlier, computer vision systems tend to have several
disadvantages like occlusion issues, latency and instability over other more
sophisticated technologies. However, the reacTable utilises a camera based
computer vision system to get input. By using higher quality components such as
cameras sampling 60 frames per second or more, the inherited technical limitations
of the system are not noticeable to the end user, making FTIR preferable in terms of
cost and ease of construction. Furthermore, choosing FTIR technology presented
some advantages like support for virtually unlimited touches, support for recognition
of several hundreds markers and the ability to detect the orientation of an object.
Furthermore, this approach improves the aesthetics of the device, by hiding the
components (camera, projector, IR source) within the reacTable’s body.
Y4796421
14
The audio generation engine is made in Pure Data (PD) and is based on the
principle of modularity. Basic sound synthesis and processing concepts like wave-
table oscillators and granular synthesis were implemented in PD patches which were
encapsulated in order to fit in the reacTable’s modular structure. Each of these
abstractions is represented, instantiated and controlled by a particular object that the
user might choose to use in the composition (Jordà, 2004:20).
The visual aspect of the reacTable is a crucial part of the instrument as it
serves as the only means of user feedback. As the users interact with the reacTable,
various lines, shapes and abstract graphics are drawn on the screen which reflect the
way audio is synthesised and processed. These graphics derive from the audio part
of the reacTable, giving instant and comprehensive visualisation of the whole signal
path. (Jordà, 2004:20) In addition, part of the visual generation provides information
about the state of the objects, how these objects are connected and various other
essential information about the composition. For example, the circles around It is
worth noting that numeric values and characters are intentionally avoided in order to
let the users interact in a more natural way.
The idea of collaboration has been a serious objective in the reacTable’s
development, and can take the form of either Local or Remote Collaboration (Jordà,
2006:408). The reacTable is designed to host up to four users at the same time.
These users can choose to have separate workspaces or to share the the whole
reacTable surface. In the first case, each user produces sounds individually which
are mixed together to produce the final musical outcome. In the second case all
users contribute to the same patch, each affecting various parameters of the same
signal path. Musical instrument players can also contribute to the session by
providing the output of their instruments as a sound source for the reacTable
performers.
Y4796421
15
The reacTable development team has taken it a step further, incorporating the
Remote Collaboration feature. In Remote Collaboration, two or more reacTables can
be linked for a performance. The two tables can be placed in different rooms, cities or
countries, as they communicate via a the internet. The actions, and consequently the
sounds, that a user performs in one location are reflected and reproduced to other
location, giving the performers of both locations the ability to interact musically and
creatively.
Finally, there is also the Remote Participation scenario, where people from the
audience can contribute to the composition. This form of collaboration does not
require a physical reacTable but rather, a reacTable simulator application that can run
on mobile devices like PDAs, net-books and mobile phones. Even though the
simulator is fully integrated into the reacTable setup and theoretically provides the
same flexibility and features, it is obviously not capable of delivering the same feeling
and level of creative freedom as the real reacTable.
The reacTable development team has also been working on the scoreTable,
an interface very similar to the reacTable that will be primarily used for sequencing
purposes. The two tables can interact with each other by sending and receiving pitch
and tempo values, expanding the creative potential of the interface.
The objects that a user can use are divided into seven groups. Each group
has a different functionality and affects the musical result in a different way. The first
group of objects includes all the Generator objects. As their name suggests, these
objects are used to instantiate a sound synthesis engine or playback of pre-recorded
material. They also have the ability to interconnect with control objects (explained
later) in order to let the user access various parameters of the generator. The second
category is called Audio Filters and consists of all the objects that are capable of
changing and manipulating audio streams in real time. The Audio Filters objects do
Y4796421
16
not produce sound on their own, but they require the output of a Generator object (or
the output of another Audio Filter) in order to function. Each object from this group is
associated with a particular audio processing algorithm like low-pass filter and delay.
Controlling the algorithm’s parameters in greater detail is achieved by using control
objects.
The next two groups are Controllers and Control Filters and are auxiliary
objects that work with data rather than audio streams. The output of these objects
can feed the control inputs of the aforementioned Generator and Audio Filter object
groups. A Controller object generates data based on an algorithm. Example.
The next group consists of a single object, the Mixer object. Unsurprisingly, this
object sums many audio streams into one single audio stream. There is also the
ability to use the Mixer object inverted in order to create a Splitter object which
outputs duplicates of the incoming audio stream. The Clock Synchronisers form the
sixth group of objects and have a higher level role in the reacTable’s workflow. Clock
Synchronisers control multiple objects simultaneously in order to fine tune the
parameters of these objects in terms of time (tempo) and pitch. The last group
includes all the Container objects, objects that encapsulate the functionality of two or
more combined objects. These objects are particularly useful for recall of complex
effect chains and synthesis engines. The Container objects need to be pre-
configured, meaning that a user cannot make one in real time (Jordà, 2004:21).
3.2 Jazzmutant Lemur
The Jazzmutant Lemur is an interface that features a 12” multi-touch TFT screen.
The screen has a resolution of 800x600 pixels and supports up to 10 simultaneous
finger touches. The device achieves multi-touch functionality by utilising the resistive
technology. The purpose of the Lemur is to provide real-time control over different
Y4796421
17
types of musical hardware and software in an intuitive way. The user can directly
control and even modulate various parameters of the available sound processors by
touching and changing UI controls on the screen. Due to its ease of use, robustness
and mobility, the device is very popular among live performers such as DJs.
The Lemur operates as a stand-alone unit, meaning that can run
independently from the computer, without relying on the computer’s CPU. The device
draws the required computational power from its own CPU, GPU and FPU
processors. The GPU is responsible for rendering the graphics on the multi-touch
screen while the CPU and FPU undertake other operations like touch detection. The
communication between the Lemur and the computer is accomplished via an
Ethernet cable. The Ethernet 100-baseT protocol has more than sufficient Data
Transfer Rate (DTR) for transmitting and receiving OSC messages and,
consequently, guarantees that even large numbers of OSC data will successfully
stream through the cable without any noticeable artifacts like data loss or latency
(Jazzmutant, 2010).
Apart from musicians, the Lemur has also gained popularity among other
types of performers who want to control their equipment in real-time such as VJs and
lights engineers in concerts and night clubs. These creative professionals rely more
and more on computer software for their performances, and the Lemur has proved to
be the perfect companion to accomplish great performances. In fact, the Lemur will
communicate with any OSC and/or MIDI compliant piece of software, giving the
potential for new and interesting control concepts not only for music, but also for
human computer interaction. In addition, the device is capable of communicating with
many computers as well as many different software applications at the same time,
meaning that performers can easily control their entire setup from a single device.
Y4796421
18
One of the Lemur’s main strengths is the ability to create customised User
Interfaces (UIs) that suit a particular application. Users can create a custom UIs
(patches) on the computer using the JazzEditor application, and then upload these
patches to the hardware. It is worth noting that JazzEditor also works as an OSC to
MIDI converter and forms the way the Lemur interfaces with MIDI applications. There
is a number of pre-determined modular UI elements (objects) that a user can choose
from, ranging from simple sliders to more sophisticated and interactive components.
Starting with conventional objects, the Lemur offers the following types of objects: a
fader (horizontal and vertical), knob (endless and normal), switches (arrays of on/off
toggles), multi-slider and pad. In most cases these objects act like their physical
counterparts, occasionally offering enhanced functionality. For example, the multi-
slider object reacts to finger swipes (physically difficult to do with real faders) and
also offers a gravity mode, which is particularly useful in cases where the performer
wants not only to control but also to modulate a target parameter.
There are also objects that are used only to provide complementary user
feedback and do not react when they are touched. These objects are Signal Scope,
Monitor and SurfaceLCD. These objects can be set to display either the incoming
OSC/MIDI data or various parameters from Lemur’s internal operations. Some
objects are made with a specific use in mind and have special features like displaying
the history of a parameter’s state or number formatting for timecode display.
The Lemur also offers two unconventional UI elements, the Ring Area and the
MultiBall. The Multiball object is a 2D controller and is basically an enhanced version
of an XY pad. The object can host up to ten balls, one for each finger, and apart from
XY functionality each ball also has a Brightness property which can be used to
expand the object’s functionality even further. The Ring Area object is visually similar
to the multiball object where a ball is confined within a circular space. There is the
Y4796421
19
option to add attractors to the Ring Area so the ball returns to a pre-determined spot
after the object is released.
Jazzmutant also offer Dexter, which is a software add-on to the Lemur
hardware. Dexter is oriented towards Digital Audio Workstation (DAW) control,
bringing the benefits of multi-touch technology to studio engineers and producers
(Jazzmutant, 2010). It is worth noting that such technology was not previously
available for dedicated studio equipment. Similarly with the Lemur, Dexter’s UI is
completely customisable by using the JazzEditor application, meaning that a user
can map any UI control to any parameter of a DAW project. The creative potential of
this approach is obvious, as it allows DAW based audio professionals to explore and
experiment with new control concepts and possibilities in typical DAW workflows.
Finally, sound engineers can also reduce the required time for tedious tasks like
editing automation by incorporating Dexter in their workflow. Apart from the common
mixer view (volume faders, pan pots etc), Dexter also offers a wide range of
innovative controls. The user can change the panorama position of many tracks at
the same time by using a special multi-XY pad.
The Lemur has had a huge success around the world and that is reflected by
the fact that well known and established artists like Bjork have been using the Lemur
in live performances as well as in the studio. Many artists also take advantage of the
visuals on the Lemur’s screen to supplement the performance and give the audience
an idea of how the sounds are created (Jazzmutant, 2010).
3.3 Apple iPhone/iPad
In June 2007 Apple Computer releases the iPhone, a cellular phone device that
features a 480x320 multi-touch screen based on the capacitive technology. The
touch screen acts as the main means of interaction between the user and the device,
Y4796421
20
but the user can also take advantage of the accelerometer and the built-in
microphone to provide input, depending on the application. In addition, the bluetooth
and wifi capabilities of the device provide an effective way to networking with
computers or other mobile devices (Apple, 2010). The latest models of the iPhone
also have a built-in camera which can be used as input for applications. One of the
key features that contributed to the iPhone’s commercial success is its software and
hardware expandability. Regarding software, the iPhone SDK, which is freely
available from the Apple web site, enables developers to code iPhone specific
applications, also known as apps. These apps are available to download from a
special internet location called the App Store either for free or for a relatively small
amount of money. There is also a vast variety of hardware accessories that plug into
the iPhone to expand its functionality. Due to the iPhone’s popularity, Apple released
a very similar multi-touch device called the iPod Touch, which is essentially an
iPhone lacking the cellular phone capabilities.
In April 2010 Apple released iPhone’s successor, the iPad, a device that aims
to bridge the gap between a cellular phone and a laptop computer. Externally, the
iPad looks like a scaled up iPhone with a 9.7” multi-touch screen. In fact, the two
devices have more similarities than differences in terms of both hardware and
software. The iPad includes most of the iPhone’s hardware components such as the
multi-touch screen, accelerometer, built-in microphone and wifi connectivity. It also
runs the same operating system as the iPhone, called iOS, which is capable of
running the same apps and uses the same SDK for software development.
Apart from the screen size, the only difference between the two Apple devices
is the available processing power. While Apple has refused to disclose any
information about the iPhone’s CPU, it has been reported that it uses a 600 MHz
ARM processor, which, depending on the device generation and firmware, might run
Y4796421
21
on a speed range from 400 MHz to 667 MHz (Ryan, 2008). The iPad features a
custom Apple A4 1GHz CPU, which significantly boosts the performance and
capabilities of the device when compared to the iPhone (Apple, 2010). This
improvement has enabled developers to deliver more sophisticated apps that can
undertake more tasks than was previously possible. The latest incarnation of the
popular phone, iPhone 4 released in June 2010, incorporates the same A4 CPU as
the iPad (Apple, 2010), fact that is expected to eliminate the performance difference
between the two devices.
Since the launch of the first generation iPhone in 2007, many developers
embraced the idea of using a mobile device for artistic purposes. It is worth noting
that the iPhone has become a rapidly expanding platform where developers around
the world make new musical applications very often, adding new possibilities for
experimentation and musical expression. Apps developed for musicians as well as
audio professionals range vastly in functionality and complexity. A metronome and an
instrument tuner are invaluable tools for most music performers, and there are
numerous apps that bring these tools to the iPhone user. At the time of writing there
are hundreds of music oriented apps in the App Store, and their number keeps
growing. While it would be almost impossible to cover all of them, a basic
categorisation of these apps gives a complete understanding of the capabilities of
this device as well as what functionality is available.
The first category of apps are the ones that can record incoming audio. As
mentioned earlier, the Apple devices have the advantage of being expandable with
software and hardware. Particularly useful hardware accessories to musicians for the
iPhone are external microphones which, paired with an app that has audio recording
capabilities, effectively transforms the iPhone into a portable field recorder. At the
time of writing there is a number of such apps that can take advantage of this
Y4796421
22
external hardware while some of them even provide multi track playback with basic
audio editing features and effects, resembling a DAW.
Along with the iPad, Apple presented a series of accessories for this new
interface. One of the most interesting and eagerly anticipated accessories is the
Camera Connection Kit, which adds USB and SD card expandability for the iPad
(Apple, 2010). As its name suggests, the Camera Connection Kit’s main objective is
to connect a digital camera via USB and download photos on the device. Despite the
fact that Apple does not officially support other types of USB devices with Camera
Connection Kit and the iPad, it has been reported that some USB microphones and
USB audio interfaces work (Rogerson, 2010). This non-documented functionality
paired with an app capable of recording audio can make the iPad a very powerful
and robust solution for recording audio either on-location or in the studio.
Unfortunately, audio recording on the iOS platform is restricted to standard CD
quality (44.1 kHz sample rate, 16 bit resolution), fact that may discourage many
audio professionals who seek better audio quality to use the iPad in a professional
project.
Another category of music-oriented apps available for iPad users are apps that
send data messages to synthesis engines. Such apps do not produce sound on their
own, they just control and communicate with other musical equipment in a very
similar way to how a MIDI controller communicates with hardware synths, computers
or any other devices equipped with MIDI. These apps take advantage of the iPhone’s
networking capabilities to send their messages over a wireless local network. The
connection is established via a server/client configuration where the server is a
computer (could be any other device with wifi connectivity and capable of running the
server app) and the client is the iPad. In most cases communication is bi-directional,
Y4796421
23
meaning that both the server and the client update their values when a parameter
changes on either device.
Recently, there have been made a few additional hardware accessories
available that are attached to the iPad and provide wired MIDI functionality. This
configuration allows the iPhone to be plugged directly into a MIDI setup and instantly
become an active controller.
There are currently two major protocols that musicians use for control
purposes, MIDI and OSC. Both are very popular in the music making community and
have been used in professional as well as amateur projects. The fact that both
protocols can be carried through a network by sending UDP messages makes it
possible to implement MIDI and/or OSC functionality into an iPad app.
There are numerous advantages of using MIDI and OSC apps on a device like
the iPad. First of all, the iPad offers wireless connectivity with the computer, making it
a truly mobile controller for an existing rig. It also brings multi-touch functionality that
was not previously available at this price tag. The potential of future software
expandability is also a strength of the iPhone platform. However, the device has its
own weaknesses compared to similar devices. The wireless connection might slow
down the responsiveness of the system. Also, the integration in a existing setup
might be time-consuming, especially when the OSC protocol is used.
At the time of writing, there are numerous apps available for download that
provide MIDI and OSC functionality. Most available MIDI apps so far tend to simulate
various elements of a hardware MIDI controller, such as the keyboard, knobs and
sliders. Other MIDI apps are dedicated to controlling DAWs by using the Mackie
Control Universal (MCU) MIDI mapping.
There are also a few OSC apps. Most of them use the same control concepts
as the MIDI apps, providing various UI elements that resemble hardware
Y4796421
24
components like knobs and sliders. However, there are a few apps that do not follow
the same functionality principles, enabling the user to modify various parameters in a
non-conventional manner. These apps communicate with the computer through a
wireless network, making the iPad one of the first wireless devices widely available to
musicians.
One of the most popular apps for the iPhone platform is hexler’s TouchOSC
app. Instead of utilising an UI with a fixed number of controls, TouchOSC enables the
user to choose from a list of available UI elements the ones that are needed and
assemble a customised user interface. The range of available UI elements consists
of pushbutton, toggle, slider, text, XY pad and accelerometer as well as user
feedback objects such as title and textview. (hexler, 2010) As with the Lemur, the
user creates the customised UI on an editor application on the computer and then
uploads it to the device.
Features like robustness, stability and active community have made
TouchOSC very popular among iPhone users. People around the world create new
UIs that are customised to work with a particular software application like NI Traktor
Pro and Ableton Live and share them so other users get the functionality instantly
(Osculator Forum, 2010).
One last category of music apps consists of the apps that produce offer multi-
touch control while at the same time produce sound on the iPad, making the iPad an
individual and autonomous instrument. Unlike control apps like TouchOSC, such
apps rely exclusively on the iPad’s CPU to produce the sound. Some of the available
apps tend to emulate hardware electronic instruments like synthesisers, drum
machines, samplers and sequencers. Other apps provide more sophisticated UIs that
do not resemble any hardware instrument. There is a plethora of such apps available
while some of them are made by established software and hardware manufacturers,
Y4796421
25
reflecting that the iPhone/iPad platform despite its comparatively low cost is capable
of delivering very goo results.
3.4 Subcycle
Subcycle is a combination of pre-existing hardware and software technologies rather
than an individual system and is a perfect example of DIY multi-touch music
interface. It also proves that multi-touch technology is not limited to laboratories and
sophisticated products, but it can be successfully implemented by individuals for use
in a particular project and for a particular purpose.
Because of the fact that the project is still under development, there is lack of
detailed documentation while the only available information come from the
developer’s blog. In addition, as the project progresses to higher build versions the
developer presents different and often contradicting interaction concepts, which
makes an accurate description of the system a difficult task.
Even though the project uses an optical based technology, the details of the
multi-touch sensing setup are not yet available. The core of the project is a computer
than runs a number of different programs, each one having its own role in the whole
interactive outcome. First of all, the computer provides its DSP power to run the
audio synthesis engine which in this project is Logic Pro hosting Native Instruments’
Massive virtual instrument plugin. The instrument aims to provide control over
various parameters of this Wave Scanning synthesis engine. (Native Instruments,
2010) Equally important, the computer hosts the visual generation software which in
this case is the open source Processing. The visuals are fed to the screen through a
digital projector. The last required piece of software is a utility made in MaxMSP that
converts and maps MIDI messages to OSC messages. This last bit of software is
essential in order to interconnect Logic Pro with Processing (Bannister, 2010).
Y4796421
26
It is important to note that no audio analysis takes place. The visuals that are
presented on the screen are composed in Processing and modulated by various
parameters that derive from Logic Pro. For example, the LFO that is used in a
synthesiser for sound generation is also used to modulate various shapes and
graphics in Processing. Similarly, the grain time of a granulation effect will also
control the respective graphic representation.
4. The Music Sphere
4.1 The Concept
The Music Sphere project is a multi-touch interface for creating and processing
music. Unlike many similar multi-touch interfaces that are developed for musical
purposes, the Music Sphere’s user interface does not utilise the conventional
elements like sliders, buttons and knobs. Instead, it uses a new concept for creating
and processing sounds and music, a 3D object.
The Music Sphere is made with a very specific idea in mind, to serve a
particular musical purpose. As Bown suggests (2009:190), these instruments are
described as composed instruments:
In electronic and digital instruments, controllers, generators and the sets of
mappings that link them are independent units that can be combined with great
structural and functional flexibility. There is no longer a fixed and direct
correspondence between the interface and sound production mechanism. This
leads to several distinct (but not mutually exclusive) senses in which an instrument
can be composed: a software instrument can combine a control interface with a
generative algorithm, such that explicit temporal structure can be predefined; the
instrument itself can be composed, in the sense that modules (controllers, mappings
and generators and their subcomponents) are arranged with specific musical intent;
in both these cases, the software system embodies some aspect of the maker’s
Y4796421
27
musical intent, and acts (like a score) as a vehicle for sharing musical ideas across
culture. (Bown, 2009:190)
The interface operates by performing gestures on the sphere that alter both
the sonic outcome and the sphere’s geometrical properties, resulting in a
combination of audio/visual real time manipulation. These alterations in both the
audio and the visual aspect have to be perceptually similar and complementary in
order to achieve a great level of interactivity and also help in crucial aspects of a live
performance such as expressiveness and precision. In addition, matching audio and
visual effects can enable non-musicians or people that have no background
knowledge on digital audio synthesis and processing to get an idea about how sound
is made on a digital system.
The audio aspect in this particular project concentrates in manipulating
parameters in two different sound generators, or scenes. The user can select the
desired scene by tapping the corresponding coloured spheres at the bottom right
corner of the screen. As soon as a scene is selected, the background colour changes
to signify the event of scene change and provide feedback to the user. The first
scene deals exclusively with pre-recorded material while the second scene controls
an FM synthesiser. This selection of sound sources might seem somehow limiting in
terms of possible sounds that can be created, however, the same operational
principles can be easily applied to different kinds of synthesis techniques, expanding
the sonic outcome of the interface.
Each scene consists of a sphere placed in the middle of the screen, the
sphere in focus, and four more spheres revolving around the central sphere, each
having its own different colour. The sound generator and processing algorithms are
divided into four smaller individual parts, for example LFO, distortion, filter and
Y4796421
28
modulation effect while each of these parts is represented by a revolving sphere.
Tapping on a revolving sphere makes that sphere the sphere in focus and the user
can then access various parameters associated with that sphere/sound generator
part by performing gestures on the screen.
4.2 Scene One - Pre-Recorded Material
The first sphere controls various aspects of the audio file’s playback. The playback of
the audio files is accomplished by using the elasticx~ MaxMSP external object
developed by Joe Jarlett and Simon Adcock, mainly because of its high quality and
low latency pitch shifting algorithm. (Jarlett, 2008) The pitch parameter is easily
accessible by the simple two-finger pinch gesture. The sphere deformation effect that
is used as visual feedback is egg for pitch shifting to a higher note and disk for pitch-
shifting to lower note.
Gestures with more than two fingers will control more complex audio
processes like the slicer processor. The slicer processor is a MaxMSP patch that was
developed for use in this particular project. The processor randomly choses a small
part of an audio file and plays it repeatedly until the user ends the gesture. As soon
as the gesture starts, an algorithm detects the areas with the . These values are set
as the looping points for the elasticx~ object. The corresponding visual effect for this
audio process is a combination of the extrude and sculpt effects.
The second sphere responds only to more than two finger gestures and is
dedicated to an audio process that utilises the resonators~ MaxMSP external object
made and maintained by CNMAT. This objects is essentially a resonant filter. (Freed,
1999) The process includes four instances of the resonators~ object and a simple
reverb algorithm. The numeric values that derive from the gesture analysis are
scaled and mapped to the resonator~ object inputs as well as the reverb Size, Decay
Y4796421
29
Time, Damping and Diffusion parameters. By performing a multi-finger rotation
gesture the user can change the pitch of the resonator~ object and produce a range
of interesting sounds.
The visual feedback for this audio process is a combination of many sphere
alterations. As the resonance of the resonator~ object increases, the density and
sculpt (set to form a star with seven points) effects increase as well.
The third sphere enables the user to control a flanger effect and it works in a slightly
different way compared to the other ones, is as it provides visual feedback even
without the user’s interaction. The sphere distorts and gradually changes shape
reflecting the LFO modulation. The deformation that is used for this purpose is
reshape because of it looked very appropriate for this purpose. When the LFO
reaches its minimum value the sphere remains unaltered while at the maximum value
the sphere is fully deformed. The sphere responds to multi-touch gestures by
becoming bigger in size as the wet/dry ratio increases. By performing the five finger
rotation gesture the user can change the modulation rate to pre-determined values.
These pre-determined values are calculated based on the tempo of the composition,
so the modulation rate will always be on time. Spreading the fingers away from each
other will increase the feedback of the flanger along with the depth and the wet/dry
output ratio.
The fourth sphere controls a combination of distortion effects. The combination
of distortion effects consists of an audio quality degrader and an FFT noise gate.
Touching the sphere with five fingers will cause reduction in the playback sampling
rate and bit depth depending on the spread property of the gesture. Moreover,
spread will also control the threshold of the FFT noise gate. The visual feedback for
this effect is a combination of density, completeness and extrusion effects that it
reveals certain parts of the sphere.
Y4796421
30
4.3 Scene Two - FM Synthesiser
The second scene controls an FM synthesiser. Similarly with the first scene, the user
taps on the coloured spheres to select which part of the available synthesis and
audio processing parameters will be modified by the gestures. The FM synthesis
implementation is rather basic, but adequate to demonstrate the operational
principle.
The first sphere provides the synthesiser with pitch information to produce sounds.
The pitch is provided by a multi-touch pinch gesture. When the gesture spreads the
synthesiser produces a higher pitched note while when the gesture shrinks the
synthesiser produces a lower pitched note. This is accomplished by mapping the
spread property of the gesture to MIDI values. The interface also supports recording
and playback of the MIDI sequences by performing simple long press gestures
anywhere on the screen. Finding appropriate visual representation for this
functionality was a very difficult task and this detail of the project could be
significantly improved.
The second sphere controls four distortion units and a XY pad which
determines the volume of each distortion unit. Each corner of the XY pad represents
a different distortion technique (overdrive and phase distortion variations). The XY
pad is controlled by calculating the average coordinates of all touches and extracting
a single touch equivalent. The visuals are a combination of completeness, reshape,
sculpt and density effects. These effects result in a deformation that is particularly
suited for distortion effects, as they make the sphere look grainy and intermittent.
The third sphere represents a low pass filter. The user controls the resonance
of the filter with multi-touch pinch gestures while multi-touch rotation gestures change
the cutoff frequency. The visual feedback for this sphere is a combination of egg,
Y4796421
31
density and extrude effects. Unfortunately, the resulting deformation is not very suited
for the low pass filter effect but the limited amount of available choices made this
choice unavoidable.
The last sphere controls the synthesiser's LFO in exactly the same fashion as
the flanger LFO found in the first scene. The user controls the rate by multi-touch
rotation while other parameters are controlled with the multi-touch pinch.
4.4 Other Synthesis Methods
The development process also included a Waveshaping synthesis engine.
Waveshaping synthesis uses a sine wave as the primitive sound and distorts it in
order to create a new shape and, consequently, a new sound. The waveshaper is a
fundamental part of a waveshaping synthesis engine which distorts the signal
depending on the signal’s amplitude. The amount of distortion depends on the
waveshaper’s transfer function which usually is a small audio file that serves as the
distortion index. In the case where the distortion index is linear, no distortion is
introduced and the sound passes through the waveshaper unaltered. (Dodge,
1997:140)
The development of the Music Sphere interface explored the idea of
manipulating the distortion index in real-time instead of using an audio file. This
functionality would involve drawing the distortion index using touches on the screen.
By moving the fingers the user would also be able to change the distortion index and
create different sonic timbres instantly. A MaxMSP implementation of this idea was
achieved by extracting the coordinates of each touch on the screen, scaling the
numeric values to the appropriate value range and mapping them into a function
object.
Y4796421
32
Even though this approach did work and the contents of the distortion index
were successfully updated in real time according to the user’s input, the calculation of
the distortion index proved to be very CPU intensive, resulting in serious issues and
instability of the system. More specifically, the MaxMSP DSP monitor reported 100%
CPU usage as soon as the distortion index was modified, while audible artefacts
such as clicks, drop outs and latency appeared.
5. Development Process
The project is a combination of a computer running the MaxMSP programming
environment and an Apple iPad serving as the multi-touch controller for the MaxMSP
patch. These two parts communicate via a USB cable, exchanging Open Sound
Control (OSC) messages. The audio synthesis and processing is done in MaxMSP
mainly because of the ease of use and the availability of a wide range of objects. The
original idea was to code the audio synthesis engine and the effect algorithms to run
natively on the iPad but, even though it is possible, it would have taken considerably
more time to develop. However, using an object oriented programming environment
like MaxMSP or Pure Data is a very good alternative and serves well for prototyping
and demonstrating the concept.
5.1 Program Structure
The user provides input on the iPad by touching the screen. Essential algorithms for
touch recognition, sphere selection and basic gesture recognition are performed on
the iPad’s CPU. These algorithms can extract the exact location (XY coordinates) of
each touch on the screen as well as various other parameters that correspond to
multi-touch gestures such as velocity (the speed at which a multi-touch gesture is
performed) and spread (how close/far away are the touches). The calculated data
Y4796421
33
that derive from the touches are scaled to an appropriate the range, formatted into
the OSC messaging style and then sent to MaxMSP through the USB cable.
The data received by MaxMSP is rescaled and mapped to many synthesis and effect
controls simultaneously. For example, the spread property of a multi-touch gesture
could be mapped to a filter’s resonance parameter as well as another effect’s wet/dry
control. Of course, some gestures are exclusive to some processes. In some cases,
additional and more complex gesture recognition algorithms are performed in
MaxMSP in order to provide more control and flexibility over the desired parameters.
For instance, the user could control the filter’s resonance with the spread gesture
property and, while still touching the screen, start a multi-touch rotation gesture which
will affect the filter’s cutoff frequency, leaving the resonance set to its last state.
The MaxMSP patch is made in a modular manner in order to simplify the
development process and allow future expandability. The signal path consists of
individual processing blocks of effects that are placed one after the other.
As the audio processing is performed, certain parameters that were affected
by the user input are re-scaled to an appropriate value range, formatted into the OSC
style and sent back to the iPad through the USB cable. The incoming OSC
messages trigger certain methods in the code which alter and deform the spheres in
ways that could describe the audio precesses. For example, pitch-shifting an audio
file could be represented on the screen with stretching the sphere.
The user interface is made by using the OpenGL ES framework. OpenGL ES is
OpenGL’s equivalent for mobile devices such as the iPhone and Android. Even
though OpenGL ES lacks of many features that are present in the full OpenGL
framework, the available capabilities are sufficient for basic shape drawings. Open
source developers have also been contributing to expand the standard OpenGL ES
Y4796421
34
library by either porting existing OpenGL code to mobile platforms or writing new
code to suit the specific needs of a project.
Figure 1: Data flow of the Music Sphere project
5.2 The C++ Sphere Method
Typically, drawing a sphere in an OpenGL project is done by calling the gluSphere
function. However, OpenGL ES does not include this functionality, meaning that a
separate function which will be responsible for calculating the sphere geometry will
have to be used. In this particular project, an external to OpenGL ES function was
used in order to draw the spheres. The source code for the sphere is written by Jeff
LaMarche and published on his personal blog while a large part of it was modified in
order to be successfully used for this project (Appendix 8.1). This method calculates
a approximation of a sphere based on a mathematical procedure. The level of
approximation depends on the amount of stacks and slices of the sphere. As the
developer explains:
Y4796421
35
These basically just define how many vertices there are latitudinally and
longitudinally. Think of an orange. If you slice it with a knife vertically, those are
"slices". If we cut it horizontally, those are "stacks". (LaMarche, 2009)
A high stacks and slices number will result in a smoother and rounder circle but will
also require more CPU power. A value of 50 stacks and slices results in a smooth
circle with no obvious dents while the required calculations are kept within the iPad’s
CPU threshold.
In order to create the effects on the sphere, various variables were inserted
into the code. By trial and error, all the spots in the code where additional variables
could be inserted were identified. As these variables change values, they affect the
procedural drawing of the sphere resulting in various deformations of the sphere.
These basic deformations can also be combined in order to make more complex
effects that seem more suitable for a particular audio process.
There are eight basic effects that derive from arithmetic manipulation in the
sphere method. These primitive effects were named after the result they have on the
sphere and are: scale, density, completeness, egg, disk, extrude, reshape and sculpt.
Some of them are self-explanatory while others affect the sphere in a more
unpredictable way. Figure 2 depicts the these effects. Some of these effects were
used for direct manipulation for the sounds while others were used as user feedback
only. All of the effects were equally important to achieve the interactivity and
responsiveness required for the project.
There is a number of very simple effects like the scale effect which even though are
not particularly impressive on their own, they are heavily used in combination with
other effects. The scale effect just scales the sphere, but when used in combination
Y4796421
36
with another effect it gives the impression of fading in the second effect. The egg and
disk effects affect how symmetrical is the sphere towards an axis, stretching the
sphere to look like an egg, or shrinking it to resemble a disk. Similarly to the egg
effect, the extrude effect stretches the sphere but, unlike the egg effect, it also
creates ripples on the surface of the stretched sphere.
The density effect affects the sphere in a very interesting way as it only draws
horizontal slices of the sphere rather than the whole sphere surface. The
completeness effect, as its name suggests, allows only a portion of the sphere to be
drawn. This effect was used to provide feedback over the pre-recorded loop playback
percentage, allowing the user to instantly figure out how much of the loop has been
played.
The reshape and sculpt effects are more sophisticated effects compared to the
other effects, as their results can be varying and unpredictable. These effects depend
largely on the value range of the input and exceeding these value ranges might
cause instability or even crash. The reshape effects stretches or shrinks certain parts
of the sphere surface and it results is weird and unusual shapes. The sculpt effect
gradually transforms the sphere into other shapes like a star or a hexagon. Similarly
to the reshape effect, the resulting shape depends on the range of the input. There is
a wide range of resulting shapes which in combination with other effects can produce
very interesting deformations on the sphere.
Y4796421
37
Figure 2: the eight basic effects (a) scale, (b) completeness, (c) density, (d) egg, (e) disk, (f) extrude, (g) reshape, (h) sculpt to hexagon, (i) sculpt to a stat with seven points, (j) the original unaltered sphere
5.3 Gestures
Apple has provided support for a few common multi-touch gestures in the iPhone
SDK, making gesture recognition a fairly easy task (Appendix 8.2). The standard
gesture recognition system found in the iPhone SDK supports six different gestures:
tap, swipe, pinch, rotate, pan and long press. (Apple, 2010:49) While all these
gestures are heavily used throughout iOS and are guaranteed to work efficiently for
the most tasks, there is also the ability to implement additional user-definable
gestures.
Y4796421
38
The tap gesture similar to a “mouse click” in the iOS. The users tap and
instantly release the screen to push buttons on the screen and more. The tap gesture
also has properties where the it can be specified how many sequential taps are
required (for example double-tap), or how many touches are required (for example
two-finger tap). A special kind of tap is the long press gesture that is performed by
tapping and holding the finger for a certain amount of time. Swipe is a gesture that is
usually mapped to the “scroll” function of a computer mouse. The user touches the
screen and quickly moves the finger towards one direction to scroll there. Similar to
the tap gesture, swipe can be specified to respond only for swipes on an axis (up-
down or left-right) or towards a pre-determined direction (only for right swipes).
The pinch gesture is a multi-touch gesture which requires two touches on the screen
and is performed by moving the two fingers away from each other, both following a
straight line. Many applications use the pinch gesture for zooming and scaling
purposes. The rotation gesture is used, as expected, for rotating images and other
elements in iPad apps. Is performed by touching the screen with two fingers and
moving them in a circular manner. Finally, the pan gesture is performed by touching
the screen and simply moving the finger around. The pan gesture can be also
constructed to respond for more touches. For the last two described gestures,
rotation and pan, there is also the option to access the gesture’s velocity, the speed
in which the gesture is performed.
Figure 2: Common gestures in the iPhone SDK: (a) pinch gesture outwards, (b) pinch gesture inwards and (c) rotation gesture
Y4796421
39
(a) (b) (c)
Apart from the gesture recognition bundled with the Apple SDK, there was the
need to implement custom gesture recognisers specifically for this project. These
additional gesture recognisers work in a very similar fashion with the Apple ones,
providing a consistent way of controlling the interface (Appendix 8.3 and Appendix
8.4). The need for extra gesture recognition derives from the fact that Apple only
supports gesture recognition for gestures that use two fingers at most. Even though
such simple gestures are adequate for controlling the iOS and the vast majority of the
iPad applications available to date, the need for a more sophisticated gesture
recognition system was apparent in this project.
The custom gestures that can be performed are multi-finger pinch and multi-
finger rotation. These gesture recognition algorithms present a significant advantage
compared to the Apple system: the user can make a gesture, for instance a multi-
finger pinch, and without releasing the fingers from the screen start another gesture,
for example a multi-finger rotation. This feature plays a very important role in the
spontaneity and the expressiveness of a performance.
Figure 3: Custom Gestures (a) multi-touch rotation and (b) multi-touch pinch
5.4 Communication Protocol
As described earlier, the networking features of the iPad allow both the MIDI and
OSC protocols to be integrated into iPad apps. Dealing with music oriented tasks has
Y4796421
40
(a) (b)
been the primary objective in the creation of both protocols, and their success is
proven by the fact that both have been embraced by the music making community,
allowing musicians to interconnect many pieces of hardware and software. In
addition, features like simplicity, robustness and unnoticeable latency were
appreciated by non-musicians, bringing MIDI and OSC support to many different
types of creative applications like robotics, VJing and more.
MIDI, defined in 1982, is a communication protocol that allows MIDI
compatible devices to control and synchronise with each other by sending data.
Since 1982, countless hardware synthesisers, effect units, drum machines and audio
interfaces are equipped with MIDI DIN connectivity, achieving instant interoperability.
The most common MIDI message is a 7-bit message which has a range of 128 steps
(integer values ranging from 0 to 127). These messages can represent Velocity,
Volume, Pan and numerous other parameters. Other supported MIDI messages are
14-bit messages which have a range of 16383 discrete steps (often used as integer
values ranging from -8192 to 8191 to represent pitch bend information), Note On,
Note Off messages and more.
Partly because of its age, the MIDI protocol is a popular and well established
method of interconnecting music equipment. The MIDI protocol is maintained by the
International Standards Association [ref], fact that has numerous advantages to MIDI
users like standardisation of hardware-specific MIDI mappings and plug-and-play
functionality of any MIDI equipped device.
OSC is a protocol developed and maintained by CNMAT. Its purpose is to
allow communication between synthesisers, computers and other multimedia
devices by taking advantage of the today’s advanced networking technology. One of
the most prominent advantages of OSC over MIDI is that OSC can deal with a wide
range of data types such as integers, floating point numbers and strings instead of
Y4796421
41
just integer values. (Freed, 2008) Recent OSC implementations also include support
for boolean values, RGBA colour values and more. While MIDI still offers the
possibility to carry other data types (float numbers etc) by using System Exclusive
messages, this approach has several disadvantages like difficulty to set up. In
addition, MIDI has pre-determined binary message fields such as note, velocity, pitch
etc while OSC allows the user to easily create customised message structures that
suit the needs of a particular application. Also, this message structure is in a human
readable URL-style, which eliminates the complexity of binary data and enables
users to easily understand and re-configure an OSC system. Finally, OSC supports a
high timing resolution timestamp (picosecond) that allows OSC messages and
performances to be scheduled, recorded and re-produced. While the same
functionality is achievable with MIDI, the resolution is several times lower
(millisecond) and that can result in inaccuracies in capturing and re-producing MIDI
performances. There is also the possibility to implement OSC communication over
the internet, fact that can help in collaborative musical projects. However, such a
scenario would most probably be vulnerable to latency and instability.
Despite its many strengths, the OSC protocol has its own weaknesses that
have mostly to do with the fact that it is a relatively new technology. First of all, major
music hardware manufacturers have been slow to implemented support for OSC,
narrowing down the practical capabilities of this protocol. Moreover, and due to the
fact that OSC is completely customisable, there is no standardisation of control
messages as opposed to MIDI, leading in interoperability issues.
Both MIDI and OSC are Hardware Transport Independent, meaning that no specific
hardware is required in order to achieve transmission and receipt of messages. Even
though the MIDI 1.0 Specification includes the MIDI DIN transport specification,
realising MIDI through this transport is optional as it is possible to implement the MIDI
Y4796421
42
protocol through other transports like USB, Firewire and Ethernet. The transport
determines the data rate of the system and, consequently, plays a very important role
in the system’s stability and responsiveness. Since OSC was introduced there has
been the false impression that OSC is a faster protocol than MIDI. This is only true
when MIDI is implemented through a low data rate transfer such as the original MIDI
DIN. In the case where another transfer like Ethernet is used, the MIDI protocol does
not show any difference to the OSC protocol in terms of speed (MIDI Manufacturers
Association, 2008).
The Music Sphere project implements the OSC protocol instead of MIDI for a
number of different reasons. First of all, there is existing OSC open source code for
the iPhone platform (rayculter, 2009) which is reliable and works efficiently, fact that
made the development process fast and easy. Moreover, the typical 0-127 value
range provided by MIDI could be proved insufficient in terms of resolution and ease
of dealing with the values. The customisability of the OSC protocol was also a very
significant reason for choosing OSC, as the code is human readable instead of
binary numbers, fact that helped for a better understanding of the programming
structure that had to be followed. Finally, future development will benefit from the
OSC support, allowing expandability with RGBA values and character strings.
During initial development process, many tests were performed in order to
make sure that the OSC implementation works and its speed is adequate for the
project’s needs. Initially, OSC communication was implemented over a wireless
network, taking advantage of iPad’s wifi connectivity. Unfortunately, this method
resulted into very poor results in terms of responsiveness and accuracy, especially
when values changed rapidly. This drawback is caused by the iPad’s poor wifi
hardware, which presents flaws and malfunctions even in simple tasks like browsing
Y4796421
43
the internet. The solution to this problem was to use the wired USB transfer for
exchanging OSC messages instead. The responsiveness of the system was radically
improved up to an acceptable level, even though in some cases issues like latency
and sluggishness still are apparent.
6. Conclusions
There are numerous advantages that the rise of multi-touch technology has brought
to HCI, and that is demonstrated by the fact that more and more companies and
organisations research and incorporate multi-touch devices. Compared to traditional
input methods like mouse and keyboard, multi-touch interaction enables the users to
control the computer in a more comfortable and natural way, which increases
productivity. This technological development has not left the music community
unaffected, as many research projects (reacTable*) as well as commercial products
(Lemur) have incorporated multi-touch technology and successfully reached their
goal which is to create and manipulate sounds and music in real time.
6.1 Evaluation
The project achieves its primary objective which was to create an interactive multi-
touch musical interface. It also accomplishes, in different levels of success, many of
its secondary objectives such as gesture recognition and mapping gesture values to
audio processes. Even though the customised gesture recognition system presents
flaws, it is a very good first step into understanding the underlaying principles and
also developing gesture recognisers for future projects. Due to the lack of more multi-
touch gesture recognisers the mapping is very basic, as in some cases a single
gesture maps to many parameters at the same time, which results in limited control.
Y4796421
44
! In many cases the deformations made on the sphere are aesthetically similar
to the alterations made to the sound. However, not all sphere deformations are
suitable for the corresponding audio processes. This is because the range of
available sphere graphics was limiting, while effect combinations in many cases
would result into very similar visuals, which made it difficult to achieve individual and
instantly distinguishable deformations for each sphere/audio process.
Finally, the steep learning curve of programming affected this project greatly,
due to lack of previous programming experience. Unfortunately, a lot of time and
effort were spent on matters that were not directly related to the musical or artistic
aspect of the project. For instance, managing to stream OSC data through USB,
drawing 3D objects in OpenGL and fine-tuning the settings in XCode were crucial
parts of the development process and took a considerable amount of time to
implement, without contributing directly to the aesthetic and musical outcome.
However, this project was an excellent opportunity to learn more about programming
and develop programming skills, which will be invaluable in the future.
6.2 Further Development
The Music Sphere project concentrates on controlling sounds and music by means of
a 3D shape instead of other common elements like virtual keyboards, sliders and
buttons. It successfully achieves its goal in terms of compositional freedom and
expression. However, there is the potential of future improvement and development.
Many different aspects of the project could undertake improvement, while the
modular structure of the program allows the addition of new features.
One of the first things that could improve the project would be to translate the
MaxMSP patch into code that runs natively on the iPad’s CPU in order to make Music
Sphere a complete and independent instrument. Not only that would eliminate the
Y4796421
45
need for a computer running MaxMSP, but it also would mean that the instrument
would not require OSC support and the USB cable, making the interface truly
autonomous. Another area of improvement would be the graphics rendering. An anti-
aliasing filter could considerably improve the edgy graphics and present smoother
image. At the time of development, anti-aliasing for the iOS 3 was not supported by
Apple and, consequently, was not implemented. Even though there are ways to apply
anti-aliasing filters, the development process would take much more time without
contributing enough to the final result.
Probably the most limiting area of this project is the lack of many different
sound generators. Even though the purpose of this project is to research into modern
music multi-touch interface and demonstrate a new operational concept, the
available sound generators are not enough to make the project a complete
instrument. In the future, more synthesis engines could be incorporated in the project
to produce a wider palette of sounds that can be created with the interface. The most
prominent feature would be an AM synthesiser which would also be very easy to
incorporate, due to the operational similarities with FM synthesis. Additional sound
processors like modulation effects (tremolo, chorus, auto-pan) could be a nice
complementary addition to the current flanger effect while spatial processors like
reverb and delay could provide more options to the user. More sophisticated audio
processes like granulation could also be added in order to expand the sonic
possibilities of the interface.
Apart from sound generation using common synthesis techniques, the
interface could also utilise the built-in microphone to either record audio files and load
them in the pre-recorded material scene or process the incoming audio in real time.
This feature would greatly improve the mobility of the device, making it a portable
instrument capable of creating and manipulating sounds instantly.
Y4796421
46
There is also a lot of room for improvement in the gestural interactivity of the project.
The Music Sphere app currently supports up to five fingers even though the iPad
platform supports much more. The reason for this limitation is that the 9.7” screen is
just big enough to accommodate a hand with the fingers fully spread. However,
support for more touches could be possible to implement in the case that gesture
recognisers are made that can respond to six or more touches. Alternatively, a
system where multiple gestures are recognised simultaneously could also expand
the controllability of the system. For example, the user could perform a five finger
rotation gesture with one hand and a pinch gesture with the other in order to control
multiple parameters at the same time.
7 References
Benko, H., Wilson, A.D. & Balakrishnan, R. (2008) ‘Sphere: multi-touch interactions on a spherical display’ (Proceedings of the 21st annual ACM symposium on User interface software and technology). Monterey, CA, USA: ACM
Bown, O., Eldridge, A. & Mccormack, J. (2009) ‘Understanding interaction in contemporary digital music: From instruments to behavioural objects’ (Organised Sound, vol.14 no.2) Bredies, K. et al. (2008) ‘The multi-touch SoundScape renderer’ (Proceedings of the working conference on Advanced visual interfaces). Napoli, Italy: ACM
Couturier, J. & Arfib, D. (2003) ‘Pointing fingers: using multiple direct interactions with visual objects to perform music’ (Proceedings of the 2003 conference on New interfaces for musical expression). Montreal, Quebec, Canada: National University of Singapore[, pp. 184-187.
Dodge, C. & Jerse, T. (1985/R1997) Computer Music: Synthesis, Composition and Performance. Boston, Schirmer
Essl, G. & Rohs, M. (2009) ‘Interactivity for mobile music-making’ (Organised Sound, vol.2 no14), 197-207.
Freed, A., Dudas, R. & Jehan, T. (1999) Musical Applications of New Filter Extensions to Max/MSP.
Y4796421
47
Geiger, G. (2006) ‘Using the touch screen as a controller for portable computer music instruments’ (Proceedings of the 2006 conference on New interfaces for musical expression). Paris, France: IRCAM — Centre Pompidou, pp. 61-64.
Han, J.Y. (2005) ‘Low-cost multi-touch sensing through frustrated total internal reflection’ (Proceedings of the 18th annual ACM symposium on User interface software and technology). Seattle, WA, USA: ACM, pp. 115-118.
Hodges, S., Izadi, S., Butler, A., Rrustemi, A., and Buxton, B (2007) ‘ThinSight: versatile multi-touch sensing for thin form-factor displays’. (Proceedings of the 20th annual ACM symposium on User interface software and technology). Newport, Rhode Island, USA: ACM, pp. 259-268.
Izadi, S., Hodges, S., Butler, A., Rrustemi, A., and Buxton, B. (2007) ‘ThinSight: integrated optical multi-touch sensing through thin form-factor displays’ (Proceedings of the 2007 workshop on Emerging displays technologies: images and beyond: the future of displays and interacton). San Diego, California: ACM, p. 6.
Jo, K. (2008) ‘DrawSound: a drawing instrument for sound performance’. (Proceedings of the 2nd international conference on Tangible and embedded interaction). Bonn, Germany: ACM, pp. 59-62.
Jordà, S. (2010) ‘The reactable: tangible and tabletop music performance’. (Proceedings of the 28th of the international conference extended abstracts on Human factors in computing systems). Atlanta, Georgia, USA: ACM, pp. 2989-2994.
Jordà, S. & Alonso, M. (2006) ‘Mary had a little scoreTable* or the reacTable* goes melodic’. (In Proceedings of the 2006 conference on New interfaces for musical expression). Paris, France: IRCAM — Centre Pompidou, pp. 208-211.
Jordà, S., Geiger, G., Alonso, M., and Kaltenbrunner, M. (2007) ‘The reacTable: exploring the synergy between live music performance and tabletop tangible interface’s’. (Proceedings of the 1st international conference on Tangible and embedded interaction). Baton Rouge, Louisiana: ACM, pp. 139-146.
Jordà, S., Kaltenbrunner, M., Geiger, G., and Alonso, M. (2006) ‘The reacTable: a tangible tabletop musical instrument and collaborative workbench’. (ACM SIGGRAPH 2006 Sketches). Boston, Massachusetts: ACM, p. 91.
Kaltenbrunner, M., Geiger, G. & Jordà, S. (2004) ‘Dynamic patches for live musical performance’. (Proceedings of the 2004 conference on New interfaces for musical expression). Hamamatsu, Shizuoka, Japan: National University of Singapore, pp. 19-22.
Kaltenbrunner, M., Jordà, S., Geiger, G. & Alonso, M. (2006) ‘The reacTable*: A Collaborative Musical Instrument’. (Enabling Technologies, IEEE International Workshops on). Los Alamitos, CA, USA: IEEE Computer Society, pp. 406-411.
Kaltenbrunner, M. (2009) ‘reacTIVision and TUIO: a tangible tabletop toolkit’. (Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces). Banff, Alberta, Canada: ACM, pp. 9-16.
Y4796421
48
Matsushita, N. & Rekimoto, J. (1997)’ HoloWall: designing a finger, hand, body, and object sensitive wall’. (Proceedings of the 10th annual ACM symposium on User interface software and technology). Banff, Alberta, Canada: ACM, pp. 209-210.
Maximo, A., Saba, M.P. & Velho, L. (2009) ‘collecTable: a natural interface for music collections’. (SIGGRAPH '09: Posters). New Orleans, Louisiana: ACM, pp. 1-1.
NUI Group Authors (2009) Multi-Touch Technologies. New York: NUI Group
Patten, J., Recht, B. & Ishii, H. (2002) ‘Audiopad: a tag-based interface for musical performance’. (Proceedings of the 2002 conference on New interfaces for musical expression). Dublin, Ireland: National University of Singapore, pp. 1-6.
Schöning J., Brandl P., Daiber F., Echtler F., Hilliges O., Hook J, Löchtefeld M, Motamedi N., Muller L., Olivier P, Roth T, & on Zadow U. (2008) ‘Multi-Touch Surfaces: A Technical Guide’ (Technical Report TUM- I0833). Munich, Technical Reports of the Technical University.
Shaer, O. & Hornecker, E. (2010) ‘Tangible User Interfaces: Past, Present, and Future Directions. Found’. Trends Hum.-Comput. Interact., 3(1–2), 1-137.
Wilson, A.D. (2004) ‘TouchLight: an imaging touch screen and display for gesture-based interaction’. (Proceedings of the 6th international conference on Multimodal interfaces). State College, PA, USA: ACM, pp. 69-76.
Wright, M., Freed, A. & Momeni, A. (2003) ‘OpenSound Control: state of the art 2003’. (Proceedings of the 2003 conference on New interfaces for musical expression). Montreal, Quebec, Canada: National University of Singapore, pp. 153-160.
Internet Sources
Apple. iPad Programming Guide. Internet WWW page, at URL: <http://
developer.apple.com/iphone/library/documentation/General/Conceptual/
iPadProgrammingGuide/GestureSupport/GestureSupport.html> (version current at
September 2010).
Apple. iPhone 4 Technical Specifications. Internet WWW page, at URL: <http://
www.apple.com/iphone/specs.html> (Accessed 10 August 2010).
Bannister, Christian. Subcycle Labs. Internet WWW page, at URL: <http://
www.subcycle.org/> (Accessed 18 July 2010).
Y4796421
49
Freed, Adrian. The Open Sound Control 1.1 Encoding Specification. Internet WWW
page, at URL: <http://cnmat.berkeley.edu/user/adrian_freed/blog/2008/10/06/
open_sound_control_1_1_specification> (last updated 06 October 2008).
hexler. TouchOSC. Internet WWW page, at URL: <http://hexler.net/software/
touchosc> (26 July 2010).
Jarlett, Joe & Adcock, SImon. elastic~ in more detail Internet WWW page, at URL:
<http://www.elasticmax.co.uk/elastic~details.php> (Accessed January 2010).
Jazzmutant. Lemur Hardware Features. Internet WWW page, at URL: <http://
www.jazzmutant.com/lemur_features_hard.php> (Accessed August 2010).
Jazzmutant. Hands on Control. Internet WWW page, at URL: <http://
www.jazzmutant.com/dexter_intuition.php> (Accessed August 2010).
Jazzmutant. Damian Taylor (Björk Volta Tour). Internet WWW page, at URL: <http://
www.jazzmutant.com/artists_bjork.php> (Accessed August 2010).
Khronos Group. OpenGL ES - The Standard for Embedded Accelerated 3D
Graphics. Internet WWW page, at URL: <http://www.khronos.org/opengles/>
(Accessed June 2010).
LaMarche, Jeff. Procedural Spheres in OpenGL ES. Internet WWW page, at URL:
<http://iphonedevelopment.blogspot.com/2009/05/procedural-spheres-in-opengl-
es.html> (version current at 5 May 2009).
MIDI Manufacturers Association. White Paper: Comparison of MIDI and OSC.
Internet WWW page, at URL: <http://www.midi.org/aboutmidi/midi-osc.php>
(Accessed November 2008).
Native Instruments. Massive Feature Details. Internet WWW page, at URL: <http://
www.native-instruments.com/#/en/products/producer/massive/?page=675>
(Accessed 10 August 2010).
Y4796421
50
Osculator Forum. Mobile Apps (iOS, Android). Internet WWW page, at URL: <http://
www.osculator.net/forum/forum/7> (Accessed August 2008).
rayculter. vvopensource. Internet WWW page, at URL: <http://code.google.com/p/
vvopensource/> (version current at 19 July 2009).
Rogerson, Ben. iPad USB audio interface compatibility confirmed. Internet WWW
page, at URL: <http://www.musicradar.com/news/tech/ipad-usb-audio-interface-
compatibility-confirmed-250703> (12 May 2010).
Ryan. iPhone CPU Speed. Internet WWW page, at URL: <http://macosrumors.com/
articles/iphone-cpu-speed/> (Accessed 2 January 2008).
Y4796421
51
8 Appendices
8.1 Sphere C++ Function
- (void) drawSphere1: (GLfloat) radius ! ! ! slcs:(GLfloat) slices! // slices number! ! ! stcks:(GLfloat) stacks! // stacks number! ! ! tSVH:(Vertex3D **)triangleStripVertexHandle! ! ! tSNH:(Vertex3D **)triangleStripNormalHandle! ! ! tSVC:(GLuint *)triangleStripVertexCount
! ! ! tFVH:(Vertex3D **)triangleFanVertexHandle! ! ! tFNH:(Vertex3D **)triangleFanNormalHandle! ! ! tFVC:(GLuint *)triangleFanVertexCount
! ! ! nsgn:(GLfloat) _nsign! // disk variable! ! ! scale:(GLfloat) _scl! // scale variable!! ! ! density:(GLfloat) _dnst! // density variable! ! ! extrude:(GLfloat) _extrd! // extrude variable! ! ! makeegg:(GLfloat) _egg! // egg variable! completeness:(GLfloat) _cmplnss!! // completeness variable! ! ! reshape:(GLfloat) _rshp! // reshape variable! ! ! sculpt:(GLfloat) _sclpt! // sculpt variable! ! auxiliary:(GLfloat) _sth;{! GLfloat rho, drho, theta, dtheta; GLfloat x, y, z; GLfloat s, ds; drho = M_PI / ((GLfloat) stacks*_rshp);! // RESHAPE dtheta = 2.0 * M_PI / (GLfloat) slices; // Calculate the Triangle Fan for the endcaps *triangleFanVertexCount = slices+2; triangleFanVertices = calloc(*triangleFanVertexCount, sizeof(Vertex3D)); triangleFanVertices[0].x = 0.0; triangleFanVertices[0].y = 0.0; triangleFanVertices[0].z = _nsign * radius; int counter = 1; for (int j = 0; j <= slices; j++) { theta = (j == slices) ? 0.0 : j * dtheta; x = -sin(theta) * sin(drho); y = cos(theta) * sin(drho); z = _nsign * cos(drho); triangleFanVertices[counter].x = x * radius; triangleFanVertices[counter].y = y * radius; triangleFanVertices[counter++].z = z * radius; } // Normals for a sphere around the origin are darn easy - just treat the vertex as a vector and normalize it. triangleFanNormals = malloc(*triangleFanVertexCount * sizeof(Vertex3D)); memcpy(triangleFanNormals, triangleFanVertices, *triangleFanVertexCount * sizeof(Vertex3D)); for (int i = 0; i < *triangleFanVertexCount; i++) Vector3DNormalize(&triangleFanNormals[i]); // Calculate the triangle strip for the sphere body *triangleStripVertexCount = (slices + 1) * 2 * stacks; triangleStripVertices = calloc(*triangleStripVertexCount, sizeof(Vertex3D)); counter = 0; for (int i = 0; i < stacks*_scl; i++) {! ! // SCALE rho = i * drho*_dnst;! ! ! ! // DENSITY! ! s = 0.0; for (int j = 0; j <= slices*_cmplnss; j++)! // COMPLETENESS {
Y4796421
52
theta = (j == slices) ? 0.0 : j * dtheta*_sclpt;// SCULPT x = -sin(theta) * sin(rho); y = cos(theta) * sin(rho)*_egg;! ! ! // EGG z = _nsign * cos(rho)*_extrd;! ! ! // EXTRUDE
triangleStripVertices[counter].x = x * radius; triangleStripVertices[counter].y = y * radius; triangleStripVertices[counter++].z = z * radius; x = -sin(theta) * sin(rho + drho); y = cos(theta) * sin(rho + drho); z = _nsign * cos(rho + drho); // TXTR_COORD(s, t - dt); s += ds; triangleStripVertices[counter].x = x * radius; triangleStripVertices[counter].y = y * radius; triangleStripVertices[counter++].z = z * radius; } } triangleStripNormals = malloc(*triangleStripVertexCount * sizeof(Vertex3D)); memcpy(triangleStripNormals, triangleStripVertices, *triangleStripVertexCount * sizeof(Vertex3D)); for (int i = 0; i < *triangleStripVertexCount; i++)! ! Vector3DNormalize(&triangleStripNormals[i]); *triangleStripVertexHandle = triangleStripVertices; *triangleStripNormalHandle = triangleStripNormals; *triangleFanVertexHandle = triangleFanVertices; *triangleFanNormalHandle = triangleFanNormals;}
8.2 Apple SDK Gesture Recognisers
This the source code that handles the gesture recognisers provided by Apple in the iPhone SDK.
// ----------- S T A N D A R D S D K G E S T U R E S ---------- //// ---------- tap, swipe, pinch, pan, rotation, long press -------- //!! // * TAPS * TAPS * TAPS * TAPS * TAPS * TAPS * TAPS !! // SINGLE TAP singleTap = [[UITapGestureRecognizer alloc]! ! ! ! ! ! ! ! ! ! initWithTarget:self action:@selector(handleSingleTap)]; singleTap.numberOfTapsRequired = 1; [self addGestureRecognizer:singleTap]; [singleTap release];
! // DOUBLE TAP doubleTap = [[UITapGestureRecognizer alloc]! ! ! ! ! ! ! ! ! ! initWithTarget:self action:@selector(handleDoubleTap)]; doubleTap.numberOfTapsRequired = 2; [self addGestureRecognizer:doubleTap]; [doubleTap release];!! // TRIPLE TAP tripleTap = [[UITapGestureRecognizer alloc]! ! ! ! initWithTarget:self action:@selector(handleTripleTap)]; tripleTap.numberOfTapsRequired = 3; [self addGestureRecognizer:tripleTap]; [tripleTap release];!! // QUADRUPLE TAP quadrupleTap = [[UITapGestureRecognizer alloc]! ! ! ! initWithTarget:self action:@selector(handleQuadrupleTap)]; quadrupleTap.numberOfTapsRequired = 4;
Y4796421
53
[self addGestureRecognizer:quadrupleTap]; [quadrupleTap release];! // TWO FINGER SINGLE TAP twoFingerSingleTap = [[UITapGestureRecognizer alloc]! ! ! ! ! ! ! ! ! ! ! ! initWithTarget:self action:@selector(handleTwoFingerSingleTap)]; twoFingerSingleTap.numberOfTapsRequired = 1; twoFingerSingleTap.numberOfTouchesRequired = 2; [self addGestureRecognizer:twoFingerSingleTap]; [twoFingerSingleTap release];!! // distinguish between single and multiple taps so it doesnt trigger single taps in case of double tap and so on//! [singleTap requireGestureRecognizerToFail:doubleTap];//! [doubleTap requireGestureRecognizerToFail:tripleTap];//! [tripleTap requireGestureRecognizerToFail:quadrupleTap];//! [twoFingerSingleTap requireGestureRecognizerToFail:doubleTap];!! // * SWIPES * SWIPES * SWIPES * SWIPES * SWIPES * SWIPES * SWIPES!! // SWIPE FROM RIGHT TO LEFT swipeLeft = [[UISwipeGestureRecognizer alloc]! ! ! ! ! ! ! ! ! ! initWithTarget:self action:@selector(handleSwipeLeft:)]; swipeLeft.direction = UISwipeGestureRecognizerDirectionLeft;! swipeLeft.numberOfTouchesRequired = 1; [self addGestureRecognizer:swipeLeft]; [swipeLeft release];!! // SWIPE FROM LEFT TO RIGHT swipeRight = [[UISwipeGestureRecognizer alloc]! ! ! ! ! ! ! ! ! !! initWithTarget:self action:@selector(handleSwipeRight:)]; swipeRight.direction = UISwipeGestureRecognizerDirectionRight;! swipeRight.numberOfTouchesRequired = 1; [self addGestureRecognizer:swipeRight]; [swipeRight release];!! // SWIPE FROM DOWN TO UP swipeUp = [[UISwipeGestureRecognizer alloc]! ! ! ! initWithTarget:self action:@selector(handleSwipeUp:)]; swipeUp.direction = UISwipeGestureRecognizerDirectionUp;! swipeUp.numberOfTouchesRequired = 1; [self addGestureRecognizer:swipeUp]; [swipeUp release];!! // SWIPE FROM UP TO DOWN swipeDown = [[UISwipeGestureRecognizer alloc]! ! ! initWithTarget:self action:@selector(handleSwipeDown:)]; swipeDown.direction = UISwipeGestureRecognizerDirectionDown;! swipeDown.numberOfTouchesRequired = 1; [self addGestureRecognizer:swipeDown]; [swipeDown release];!! // * MULTI-TOUCH GESTURES * PINCH * PAN * ROTATION!! // PINCH! pinchGesture = [[UIPinchGestureRecognizer alloc]! ! ! ! ! ! ! ! ! ! ! initWithTarget:self action:@selector(handlePinchGesture)];! [self addGestureRecognizer:pinchGesture];! [pinchGesture release];!! // ROTATION! rotationGesture = [[UIRotationGestureRecognizer alloc]! ! ! ! ! ! ! ! ! ! ! ! !initWithTarget:self action:@selector(handleRotationGesture)];! [self addGestureRecognizer:rotationGesture];! [rotationGesture release];!! // PAN
Y4796421
54
! panGesture = [[UIPanGestureRecognizer alloc]! ! ! ! initWithTarget:self action:@selector(handlePanGesture)];! panGesture.minimumNumberOfTouches = 1;! panGesture.maximumNumberOfTouches = 1;!! [self addGestureRecognizer:panGesture];! panGesture.delegate = self;! [panGesture release];
! [panGesture requireGestureRecognizerToFail:pinchGesture];! [panGesture requireGestureRecognizerToFail:rotationGesture];!! /*! // PAN TWO FINGERS! panGestureTwoFingers = [[UIPanGestureRecognizer alloc]! ! ! ! initWithTarget:self action:@selector(handlePanGestureTwoFingers)];! panGestureTwoFingers.minimumNumberOfTouches = 2;! panGestureTwoFingers.maximumNumberOfTouches = 2;!! [self addGestureRecognizer:panGestureTwoFingers];! panGestureTwoFingers.delegate = self;! [panGestureTwoFingers release];!! [panGesture requireGestureRecognizerToFail:pinchGesture];! [panGesture requireGestureRecognizerToFail:rotationGesture];! */!! // LONG PRESS! longPressGesture = [[UILongPressGestureRecognizer alloc]! ! ! ! ! ! initWithTarget:self action:@selector(handleLongPressGesture)];! //! longPressGesture.numberOfTapsRequired = 1;! [self addGestureRecognizer:longPressGesture];! [longPressGesture release];! // ***************** Standard SDK Gestures END ******************* //
8.3 Custom Gesture Recogniser
In order to maek custom gesture recognisers, the developer has to implement the following methods from the iPhone SDK:
- (void)touchesBegan:(NSSet *)touches withEvent:(UIEvent *)event {!
}- (void)touchesMoved:(NSSet *)touches withEvent:(UIEvent *)event{! UITouch *touchA, *touchB, *touchC, *touchD, *touchE;! CGPoint pointA, pointB, pointC, pointD, pointE;!! NSSet *touchEnum = [event allTouches];! // get all touches on the screen!! // No point in making 2 finger recognisers cause they are covered in the SDK Gesture Recognisers!! // ------- THREE FINGERS ------- //! if ([touchEnum count] == 3)! {! ! // Disable pinch, pan and rotation recognisers so data is not passed to the pinch and rotation responders! ! pinchGesture.enabled! = NO;! ! rotationGesture.enabled = NO;! ! panGesture.enabled! ! = NO;! !! ! touchA = [[touchEnum allObjects] objectAtIndex:0];! ! touchB = [[touchEnum allObjects] objectAtIndex:1];! ! touchC = [[touchEnum allObjects] objectAtIndex:2];! !! ! pointA = [touchA locationInView:self];
Y4796421
55
! ! pointB = [touchB locationInView:self];! ! pointC = [touchC locationInView:self];! !! ! float currDistanceAB = [self distanceFromPoint:pointA toPoint:pointB];! ! float currDistanceAC = [self distanceFromPoint:pointA toPoint:pointC];! ! float currDistanceBC = [self distanceFromPoint:pointB toPoint:pointC];! !! ! float abcSpread = [self threePointsSpread:pointA andPoint:pointB andPoint:pointC];! ! float abcAverage = [self threePointsAverage:pointA andPoint:pointB andPoint:pointC];! ! float averagex = (pointA.x + pointB.x + pointC.x)/3.f;! ! float averagey = (pointA.y + pointB.y + pointC.y)/3.f;! !! ! [self drawView];! !! ! NSLog(@"%@", [NSString stringWithFormat:@"%0.2f", abcSpread]);! }!! // ------- FOUR FINGERS ------- //! if ([touchEnum count] == 4)! {! ! // Disable pinch, pan and rotation recognisers so data is not passed to the pinch and rotation responders! ! pinchGesture.enabled! = NO;! ! rotationGesture.enabled = NO;! ! panGesture.enabled! ! = NO;! !! ! touchA = [[touchEnum allObjects] objectAtIndex:0];! ! touchB = [[touchEnum allObjects] objectAtIndex:1];! ! touchC = [[touchEnum allObjects] objectAtIndex:2];! ! touchD = [[touchEnum allObjects] objectAtIndex:3];! !! ! pointA = [touchA locationInView:self];! ! pointB = [touchB locationInView:self];! ! pointC = [touchC locationInView:self];! ! pointD = [touchD locationInView:self];! !! ! float abcdSpread = [self fourPointsSpread:pointA andPoint:pointB andPoint:pointC andPoint:pointD];! ! float abcdAverage = [self fourPointsAverage:pointA andPoint:pointB andPoint:pointC andPoint:pointD];! ! float averagex = (pointA.x + pointB.x + pointC.x + pointD.x)/4.f;! ! float averagey = (pointA.y + pointB.y + pointC.y + pointD.y)/4.f;! !! ! [self drawView];! !! ! NSLog(@"%@", [NSString stringWithFormat:@"%0.2f", abcdSpread]);! }!! // ------- FIVE FINGERS ------- //! if ([touchEnum count] == 5)! {! ! // Disable pinch, pan and rotation recognisers so data is not passed to the pinch and rotation responders! ! pinchGesture.enabled! = NO;! ! rotationGesture.enabled = NO;! ! panGesture.enabled! ! = NO;! !! ! touchA = [[touchEnum allObjects] objectAtIndex:0];! ! touchB = [[touchEnum allObjects] objectAtIndex:1];! ! touchC = [[touchEnum allObjects] objectAtIndex:2];! ! touchD = [[touchEnum allObjects] objectAtIndex:3];! ! touchE = [[touchEnum allObjects] objectAtIndex:4];! !! ! pointA = [touchA locationInView:self];! ! pointB = [touchB locationInView:self];! ! pointC = [touchC locationInView:self];! ! pointD = [touchD locationInView:self];! ! pointE = [touchE locationInView:self];! !
Y4796421
56
! ! float abcdeSpread = [self fivePointsSpread:pointA andPoint:pointB andPoint:pointC andPoint:pointD andPoint:pointE];! ! float abcdeAverage = [self fivePointsAverage:pointA andPoint:pointB andPoint:pointC andPoint:pointD andPoint:pointE];! ! float averagex = (pointA.x + pointB.x + pointC.x + pointD.x + pointE.x)/5.f;! ! float averagey = (pointA.y + pointB.y + pointC.y + pointD.y + pointE.y)/5.f;! !! ! [self drawView];! !NSLog(@"%@", [NSString stringWithFormat:@"%0.2f", abcdeSpread]); // debug line! }}
- (void)touchesEnded:(NSSet *)touches withEvent:(UIEvent *)event{! // Re-enable pinch, pan and rotation gesture recognisers so they are ready for use! pinchGesture.enabled! ! = YES;! rotationGesture.enabled ! = YES;! panGesture.enabled! ! = YES;}
8.4 Custom Gesture Recogniser Responders
These methods are called in order to calculate the spread of the multi-touch pinch and the equivalent point of the gesture.
- (float)distanceFromPoint:(CGPoint)pointA toPoint:(CGPoint)pointB {! float xD = fabs(pointA.x - pointB.x);! float yD = fabs(pointA.y - pointB.y);!! return sqrt(xD*xD + yD*yD);}
- (float)fivePointsAverage:(CGPoint)pointA andPoint:(CGPoint)pointB andPoint:(CGPoint)pointC andPoint:(CGPoint)pointD andPoint:(CGPoint)pointE {! float xD = fabs(pointA.x + pointB.x + pointC.x + pointD.x + pointE.x)/5.f;! float yD = fabs(pointA.y + pointB.y + pointC.y + pointD.y + pointE.y)/5.f;!! return sqrt(xD*xD + yD*yD);}
- (float)fivePointsSpread:(CGPoint)pointA andPoint:(CGPoint)pointB andPoint:(CGPoint)pointC andPoint:(CGPoint)pointD andPoint:(CGPoint)pointE {
! float ab = [self distanceFromPoint:pointA toPoint:pointB];! float ac = [self distanceFromPoint:pointA toPoint:pointC];! float ad = [self distanceFromPoint:pointA toPoint:pointD];! float ae = [self distanceFromPoint:pointA toPoint:pointE];! float bc = [self distanceFromPoint:pointB toPoint:pointC];! float bd = [self distanceFromPoint:pointB toPoint:pointD];! float be = [self distanceFromPoint:pointB toPoint:pointE];! float cd = [self distanceFromPoint:pointC toPoint:pointD];! float ce = [self distanceFromPoint:pointC toPoint:pointE];! float de = [self distanceFromPoint:pointD toPoint:pointE];!! float spread = (ab+ac+ad+ae+bc+bd+be+cd+ce+de)/10.f;!! return spread;}
Y4796421
57
8.5 OpenGL ES
The following code indicates OpenGL which vertices to use in order to draw the spheres. This code is called for every frame of the animation which is set to 60 frames per second. However, the actual frame per second rate is not as high. The real frame rate is about 30 frames per second as measured with the performance tool Instruments.
! // OpenGL code to draw the sphere on the screen! // This code is called for every frame of the animation!! glPushMatrix();!! /* color the sphere */! GLfloat s2Ambient[] = {RGBToFloat(82), RGBToFloat(46), RGBToFloat(4), 1.0}; glMaterialfv(GL_FRONT_AND_BACK, GL_AMBIENT, s2Ambient); GLfloat s2Diffuse[] = {RGBToFloat(29), RGBToFloat(50), RGBToFloat(71), 1.0}; glMaterialfv(GL_FRONT_AND_BACK, GL_DIFFUSE, s2Diffuse);!! /* rotate the sphere */! _x3 = 3.000f*sin(_rotationTemp+SphreSpacing(numberOfSpheres))*_diaspora;! _y3 = 3.000f*cos(_rotationTemp+SphreSpacing(numberOfSpheres))*_diaspora;
! /* place the sphere into the drawing */! glTranslatef((_x3*_vathos), (_y3*_vathos), (-11*_vathos));! glRotatef(_rotMid,0.1f,0.1f,0.1f);!! /* OpenGL commands */ glVertexPointer(3, GL_FLOAT, 0, sphereTriangleFanVertices3); glNormalPointer(GL_FLOAT, 0, sphereTriangleFanNormals3); glDrawArrays(GL_TRIANGLE_FAN, 0, sphereTriangleFanVertexCount3); glVertexPointer(3, GL_FLOAT, 0, sphereTriangleStripVertices3); glNormalPointer(GL_FLOAT, 0, sphereTriangleStripNormals3); glDrawArrays(GL_TRIANGLE_STRIP, 0, sphereTriangleStripVertexCount3);! glPopMatrix();
8.6 OSC
This is the code that sends and receives OSC messages. The receiver method also works as a parser.// Send OSC message Method- (void)sendOSCMessage:(float)floatMessage label:(NSString*)label {! OSCMessage *msg = [OSCMessage createWithAddress:label];! [msg addFloat:floatMessage];! [outPort sendThisPacket:[OSCPacket createWithContent:msg]];}
- (void) receivedOSCMessage:(OSCMessage *)m! {!!! NSString *address = [m address];! OSCValue *value = [m value];! float! incomingOSC = [value floatValue];! int! incomingIntegerOSC = [value intValue];!! if ([address isEqualToString:@"/something"]) {! !! ! // do stuff here...! !! } else if ([address isEqualToString:@"/something_else"]) {! !! ! // do some other stuff here...! !! }NSLog(@"%@ %@", address, value);!// debug line!}
Y4796421
58
Y4796421
59