© Copyright Khronos Group, 2009 - Page 1 Erik Noreke ST Ericsson | OpenSL ES Chair
© Copyright Khronos Group, 2009 - Page 1
Erik Noreke ST Ericsson | OpenSL ES Chair
© Copyright Khronos Group, 2005-2009 - Page 2
Over 3.5 Billion phones built based on our technologies
Offers the full range of access technologies from 2G to 3G to LTE
LET’S CREATE IT
Leading global supplier of platforms and semiconductors for wireless
Joint Venture owned equally by STMicroelectronics and Ericsson
A workforce of about 8000
© Copyright Khronos Group, 2005-2009 - Page 3
• Lots of fragmentation!
• Many proprietary audio APIs of varying functionality - Even playing a simple sound on different platform requires different code.
• No standard way to access audio hardware acceleration - Lots of work for developers to re-write code for every platform – no application/source code
level portability
• Newer multimedia devices incorporating more advanced audio functionality - Increases in audio quality & functionality further complicate the efforts of content developers
aggravating the portability problem.
Solution - Open standard application level API for embedded audio !
© Copyright Khronos Group, 2005-2009 - Page 4
High performance & low latency access to audio
features
Support a diversified market space of
embedded devices
• Diversified collection of embedded devices - Target devices include “Mobile phones, personal media players & handheld gaming consoles”
• High performance, low latency access to audio • Application developer friendly - Source-level portability of native code from platform to platform - Same interface for both hardware and software solutions
• Royalty-free open standard
Open standard for embedded audio
Open Standard API for application developers enabling application
portability
© Copyright Khronos Group, 2005-2009 - Page 5
• What about ALSA, OSS, GStreamer, OpenAL? - OpenAL is targeted for desktop PCs - OSS is obsolete, replaced by ALSA - ALSA is Linux specific - GStreamer is not designed to be hardware accelerated - Are all released under variations of GNU Public License
- They can provide the foundation for an OpenSL ES implementation
• OpenSL ES - Designed specifically with embedded software and hardware in mind - Designed to be hardware accelerated - Pure API, independent of underlying implementation and architecture - Royalty-free open standard – No copy-left implications
© Copyright Khronos Group, 2005-2009 - Page 6
4Q07 1H08 2H08 1Q09
OpenSL ES 1.0 Provisional Release
Conformance Tests and OpenSL ES 1.0 Final
2Q09 3Q09
• OpenSL ES 1.0 released in provisional form in September 2007 - Encouraged developer feedback before specification finalization
• Final specification released March 2009 - Full set of conformance tests available
© Copyright Khronos Group, 2005-2009 - Page 7
• Target devices include: - Basic, Feature & Gaming mobile phones - Personal Media Players & PDA’s - Portable gaming consoles - Just about any portable device
• Target applications include - User Interface sounds - Music and Ring-tone playback - Simple 2D and advanced 3D games - Media recording and sequencers
© Copyright Khronos Group, 2005-2009 - Page 8
• Playback of audio files - Playback PCM and encoded content - Good for sound effects; device UI sounds
• SP-MIDI, Mobile DLS, Mobile XMF - For interactive music and ring-tones
• Effects & Controls - Music and media player effects - Advanced environmental effects for gaming
• 3D Audio - Provided for gaming as companion to OpenGL ES
Doppler
MIDI messages
Preset Reverb
EQ Pitch
Volume
Buffer Queues Rate Metadata extraction
Environmental Reverb
Virtualization
Stereo widening LED & Vibra
3D position
© Copyright Khronos Group, 2005-2009 - Page 9
• OpenSL ES 1.0 consists of: • The specification - 58-page user guide providing a comprehensive overview of the API - 497-page API reference that goes into details of the objects and interfaces
• Header files - Available to the public for download
• Conformance Tests - Ensures application portability across implementations - Passing required for implementations before adoption is accepted - Adopters that pass the conformance tests are able to use the OpenSL ES logo
• Adopters Package - Defines the process and requirements for a conformant implementation
© Copyright Khronos Group, 2005-2009 - Page 10
Basic mobile phones Ring tone and alert tone playback (basic MIDI functionality), basic audio playback and record
functionality, simple 2D audio games
Game-centric mobile devices Advanced MIDI functionality, sophisticated audio
capabilities such as 3D audio, audio effects, ability to handle buffers of audio, etc.
Music-centric mobile devices High quality audio, ability to support
multiple music audio codecs, audio streaming support
© Copyright Khronos Group, 2005-2009 - Page 11
• Phone + Music = Phone combined with portable music player - Advanced stereo ring tones - Digital music playback
• Game + Music = Advanced music player / Musical games - High quality music playback - 3D audio games with advanced audio effects
• Phone + Game = Gaming phone - 3D ring signals - Games utilizing advanced audio techniques
• Phone + Game + Music = Ultimate Audio Rendering Device - Multiple audio sources perceived - Encapsulates the entire functionality of OpenSL ES
© Copyright Khronos Group, 2005-2009 - Page 12
• Same API for H/W and S/W solutions
Middleware, JSR-135, JSR-234, High-level Audio Libraries
Applications
Native Audio Solution
Media silicon (audio HW, CPUs, DSPs)
OpenMAX IL
© Copyright Khronos Group, 2009 - Page 13
© Copyright Khronos Group, 2005-2009 - Page 14
• OpenSL ES has an object-oriented programming model - Simplifies common use cases – but also extensible
• Engine Objects are central to any OpenSL ES session - Objects created using methods on the Engine Object interfaces
• OpenSL ES Objects enable PLAY and RECORD of audio - Perform some operation on an input and emit the result as output - Can handle almost any audio use case
• Objects have control interfaces - Play, Seek, Rate, Audio, Metadata Extraction - Record, Audio Encoder, MIDI
• Audio Objects input and output to Devices
OpenSL ES Audio Object URI
Memory
Audio Input
URI
Memory
Audio Mix
DSrc DSnk
© Copyright Khronos Group, 2005-2009 - Page 15
• Create Engine object - To drive this session
• Create Audio Output Mix object - Method on Engine interface - Mix object drives audio output devices
• Create Media Player object - Method on Engine interface - Input is URI pointing to a local mp3 file - Output drives audio output mix
• Register event callback - Method on Media Player interface
• Set PlayState to Playing - Method on Media Player interface
• Wait for end of file event - Via registered callback
Engine Object
Media Player Object
Output Mix Object
EngineItf
PlayItf
Play Event Callback
Application
© Copyright Khronos Group, 2005-2009 - Page 16
Audio Player
SLPlayItf
SLPlayItf
SL3DLocationItf
URI
Memory
Buffer Queue
URI
Memory
Buffer Queue
Output Mix
Audio Player SLVolumeItf
SLVolumeItf
© Copyright Khronos Group, 2005-2009 - Page 17
Overlap streaming media functionality uses consistent APIs
3D Audio
Audio Effects
Advanced MIDI
Audio Playback
Audio Recording
MIDI
Video playback
Camera
Video recording
Image capture & display Buffer
queues
Analog Radio & RDS
(Enhanced audio API) (Streaming Media APIs)
© Copyright Khronos Group, 2005-2009 - Page 18
• Independent - No dependency between the APIs – either one can exist by itself - A device may support any combination of the APIs that most suits the device: - OpenSL ES only (Phone, Game, Music) - OpenMAX AL only (Multimedia Playback/Recording) - OpenMAX AL + OpenSL ES (Music + Multimedia Playback/Recording) - OpenMAX AL + OpenSL ES (Game + Multimedia Playback/Recording) - ….
• Compatible - Working groups collaborated to make sure the APIs work together well
• Consistent - Identical API architecture - Identical APIs for same functionality
• Distinct - OpenSL ES provides advanced audio functionality - 3D, audio effects and advanced MIDI
- OpenMAX AL provides wide-range, basic multimedia functionality - audio, video and image
© Copyright Khronos Group, 2005-2009 - Page 19
Audio Player URI Output Mix
File Reader
Parser
Codec Audio Effect
Audio Effect Mixer
Sink
© Copyright Khronos Group, 2005-2009 - Page 20
• Same API regardless of underlying solution - hardware accelerated - software implementations
• Supports JSR-135 and JSR-234 audio functionality - Simplifies integration of third party JSR implementations - Provides a stable foundation for Java support
• Simplifies porting between platforms - Spend application development time on application development, not porting
• Does for audio what OpenGL ES does for graphics - Brings your applications and devices to life!
© Copyright Khronos Group, 2009 - Page 21
The OpenSL ES Workgroup