This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Dynamic Media ProgrammingDynamic Media ProgrammingSynergistic integration of OpenML and OpenGL Synergistic integration of OpenML and OpenGL • What OpenGL is to GRAPHICS - 2D, 3D and image processing…
…OpenML is to MEDIA - video and audio processing• OpenML and OpenGL together create a unified programming environment
- With seamless integration of graphics and media
• Tightly integrated graphics, video and audio = DYNAMIC MEDIA
Workstation3D graphics
Professional video and audio authoring
OpenML enables interoperability of media authoring applications across: Platforms, Operating Systems, Hardware devices
Technical Strategy for OpenML 1.0Technical Strategy for OpenML 1.0Move PROVEN technologies into Open StandardsMove PROVEN technologies into Open Standards
UST/MSCSynchronization
#4: Synchronization of diverse media streams (graphics, audio, video)
SGI’s dmSDKDigital Media Handling
#1: Configure and control the Input/Output and processing of video and digital audio ML
OpenGL ExtensionsIntegrating graphics and video
#3: Enable OpenGL hardware to add more value to the media authoring workflow
OpenML 1.0 SDK ReleaseOpenML 1.0 SDK ReleaseEnabling and encouraging OpenML AdoptionEnabling and encouraging OpenML Adoption
Executable ML and MLdc Executable ML and MLdc libraries and utilities to run libraries and utilities to run OpenML applications for OpenML applications for IRIX, Linux and WindowsIRIX, Linux and Windows
ML and MLdc library ML and MLdc library source and header files source and header files to enable new OpenML to enable new OpenML applications to be builtapplications to be built
Source code for example Source code for example audio and video device audio and video device
modules with full modules with full documentation documentation
Source code for example Source code for example OpenML applications OpenML applications
with full documentation with full documentation
SDK
Available today at Available today at www.khronos.orgwww.khronos.org
Free of charge for Free of charge for commercial use commercial use under Khronos under Khronos
licensing agreementlicensing agreement
Can be used as a Can be used as a starting point for starting point for
building commercial building commercial applicationsapplications
OpenML Outreach and EngagementOpenML Outreach and EngagementIndustry Involvement will drive momentumIndustry Involvement will drive momentum• SDK Released - OpenML Forums at www.khronos.org
- Adopters can request, discuss programming and the evolution of OpenML and the SDK
• OpenML - for multimedia application vendors- Complete software environment for advanced media on wide selection of systems and devices
• OpenML – for hardware vendors- Gives more ISVs better access to hardware device capabilities
• OpenML - for end users- Better functionality across a wider range of platforms at cheaper prices
The Khronos Group encourages your participation to help grow the dynamic media market!
OpenGL is a registered trademarks of Silicon Graphics, Inc., and OpenML is a trademark of Silicon Graphics, Inc., used with permission by the Khronos Group. All other product names, trademarks,
and/or company names are used solely for identification and belong to their respective owners.
OpenML’s “ML” API OpenML’s “ML” API Media device control interfaceMedia device control interface• API to configure/control video/audio device input, output and processing
- Low level - provides device independence without imposing policy
• Perfect environment for implementing higher-level software - Such as QuickTime
OpenML’s “ML” API OpenML’s “ML” API Market-proven audio/video processingMarket-proven audio/video processing• ML enables abstracted query and control of media processing devices
- E.g. IEEE1394 cameras, video transcoders, OSS audio processors
• ML is used to setup processing pipelines between I/O and devices- With optimized buffering and synchronization between devices and the application- Asynchronous communication between application and media devices
ApplicationApplicationMedia Input Jacks Media Output Jacks
TranscoderTranscoder
Control over input/output parameters such as data format and quality levels
Control over transcoder operation
Buffered, synchronized message-based data communication
OpenML’s “MLdc” APIOpenML’s “MLdc” APIAbstracted display control Abstracted display control • Allows an application to control what is displayed on monitors
- Sophisticated level of control beyond most windowing systems
• MLdc sets up channels to drive each display screen- Channels that are driven from sections of physical frame buffers - Number and type of supported channels depends on the display device
• Properties that can be controlled include:- The number of channels and where their display data is derived- Interrogating monitor information- Precise video positioning- Video format query and settings- Vertical retrace rate - Interlaced vs. progressive- Gamma ramp control- Video sync sources- Genlock state notification
Media SynchronizationMedia SynchronizationA critical part of any authoring environmentA critical part of any authoring environment• Need to ensure that all media streams start at the same time
- And stay in synch
• Non-trivial problem to solve- Devices from various manufacturers - Device latencies are unpredictable- OS scheduling latencies
Audio Device AAudio Device A
Video Device AVideo Device A
Audio Device BAudio Device B
Gfx Device BGfx Device B
Gfx Device AGfx Device A
ApplicationApplication Do all the streams come out in sync?
The UST/MSC/SBC SolutionThe UST/MSC/SBC SolutionAccurate media synchronizationAccurate media synchronization• UST - Unadjusted System Time – time reference for software and devices
- 64-bit monotonic counter with nanosecond resolution, microsecond system-wide accuracy
• MSC - Media Stream Counter - incremented for each channel sample- i.e. Video - per frame, Graphics - per retrace, audio - per audio sample, e.g. 44.1KHz
• SBC – Swap Buffer Count - incremented on graphics buffer swap- Not necessarily in synch with retrace
Using UST/MSC/SBCUsing UST/MSC/SBCRobust Synchronization – even on non real-time OSRobust Synchronization – even on non real-time OS• Every media sample is time-stamped
- Timestamp can be accessed through OpenGL and ML APIs for temporal analysis and control
• Application can synchronize output events to UST- Enabling precise timing and synchronization in non-real-time operating systems
• Applications can detect dropped samples- And take corrective action
OpenML’s OpenML’s OpenGL ExtensionsOpenGL ExtensionsVideo imaging and 3D graphics integrationVideo imaging and 3D graphics integration• OpenML is built around OpenGL• OpenML and OpenGL: A tight and synergistic integration
OpenML and OpenGLOpenML and OpenGLTight and synergistic integrationTight and synergistic integration• Compliant OpenML assumes a robust OpenGL implementation
- OpenGL 1.2.1 with GLX 1.3 (Linux/UNIX) or WGL (Windows)
• OpenML extends OpenGL with video primitives and DCC rendering features- Defines a set of OpenGL extensions that OpenML compliant hardware must implement
• Pbuffers are the fundamental method of integrating graphics and video- Enables buffering and off-screen processing of video data
• OpenGL imaging extensions are used to process video streams- Filtering, color correction, blending, image enhancement, color space conversion
• Significant opportunity for OpenGL hardware vendors to add more value- Use OpenGL graphics hardware for 3D and video acceleration
OpenML’s OpenGL ExtensionsOpenML’s OpenGL ExtensionsBringing together the worlds of 3D and videoBringing together the worlds of 3D and video• Synchronization control
- Primitives to control UST/MSC synchronization from OpenGL- Functions added: WaitForMSC, SwapBuffersMSC, GetSyncValues
• Interlace support- Based on two previous extensions: SGIX_interlace, INGR_interlace_read
Enables reading and writing of the frame buffer while skipping every other line- Enables processing (e.g., convolution) on interlaced video streams
• Texture color mask management to enable advanced compositing- Enables independent loading of RGB images and Alpha masks in texture memory- Recently promoted from an SGI extension to an ARB extension- Works for textures like the OpenGL color mask works for the color buffer
• Chroma resampling to support pixel formats such as YCrCb- Upsampling and downsampling when converting between YCrCb and RGB- Sampling options are: Replicate, Zero fill, Average