Introduction to 2D/3D Graphics

TIITC2009 : Customer Centric Strategies

1

Engaging the consumer with visual effects –

Architectures and implementations for graphics

Prabindh Sundareson ([email protected])

Texas Instruments India

DSPS


2

Outline

• 2D Graphics

• 3D Graphics

• The OpenGL API

• Internals of SGX, benchmarking

• Future roadmap

• Conclusion

• [Graphics Cheatsheets]


3

What can Graphics do for me ?

http://cg.cs.tsinghua.edu.cn:8080/cmm/?page_id=155


4

Graphics Frameworks Cheatsheet

FB, V4l2FB, V4l2

CairoCairo

Clutter, QT, OperaClutter, QT, Opera

GDI

DirectFBDirectFB DDraw

Surfaces

XX Win32/ Windowing SystemWindow System/Mgrs

Application Frameworks

Buffers / blits

Win32, .NET, Silverlight

Note: Cheatsheet table in appendix


5

2D Graphics

• Almost all existing Consumer Graphical User Interfaces are still built around 2D, and not around OpenGL– Exception of course, iPhone, other Mobile phones, Games…

– Specific operations can still be accelerated with 3D accelerators. Like blits of large sizes

• What are the typical 2D applications ?– All Windowing Systems (X, Win32, …)

– Application Frameworks like QT, Flash

– Browsers

– All 3D operations ultimately end in 2D blitting (ex, EGL uses PVR2D)

• An example, a Browser application…


6

Case study: Webkit’s Call Sequence

FILL RECTANGLE

SHOW GLYPHS

SET CLIP REGION

ACQUIRE FONT CACHE

ALLOCATE FONT CACHE

SURFACE COMPOSITE

PREPARE/FINISHCOMPOSITE

FillRectangles()

BatchBlit()

Blit(), mem_copy()

Clear()Blit()TileBlit()


7

Summary of 2D – Browser Rendering Processes

• To render one page of Youtube, for example

– A large blit of size = page width x block size ~ typical (500 x 100 pix)

– Followed by multiple blits of a glyph cache line of one line of text (page width x text height ) ~ ex (50 x 8), this renders blocks line by line

– Followed by multiple compositing operations for each block to render the whole page ~ ex (16 x 16 to page size)


8

2D Features - Anti-aliasing

Coverage proportional to (1/x)

xx

Many advances – Supersampling is common now in all GPUs


9

2D Features - Compositing

SRC

1 -asrc

DST

1-adst adst

SRC & DST

SRC & NO-DST

NO-DST & NO-SRC

Compositing


10

Porter-Duff OperationsComposition modes were originally proposed by Thomas Porter and Tom Duff in

the article Compositing Digital Images in 1984Describes a model for combining the pixels in a source image with the pixels in a

destination, there are 12 basic composition modes

Formula for combining source and destination pixels

cdst and csrc represent color components of destination and source

adst and asrc represent alpha components of destination and source

Cdst and Csrc are the color components pre-multiplied by the alpha components

The parameters X, Y and Z are either 0 or 1.

Permutations of X, Y, Z, and f produce various composition modes Most Common = SRC, SRC_OVER


11

A typical 2D Graphics HW Feature Set

• Bit blit feature supporting all P-D combinations and ROPs and blending

• Data expansion (Gray scale to full colour)

• Line/curve drawing based on paths

• Area/pattern filling

• Anti-aliased drawing with enable/disable

• Alpha

• Color-keys

• Hardware cursor support

• SW implementations also have to take care of Glyphs/cache


12

3D vs 2D

• Windowing systems already comprehend z-order

• 3D != 2D with depth

• So what is 3D ?


13

Pixology – – Video & Graphics

• Video Compression– Analytic approach to describe a scene

• Describes a scene using its pixel values. Quality is dependent on the source content/ encoder.

• Encoder can be highly complex, decoder is constrained by spec• Residual compressor

• OpenGL– Object based

• Describes a scene using its components and properties. Output quality is dependent on the renderer, not source

• GL authoring tools are complex, but quality depends on renderer– GL Spec is intentionally non-pixel compliant !

• Each frame rendering is independent of any other– Complexity now dependent on shaders !!

Basics


14

3D Concepts

This is hidden by another object !

Light Source

Shadows,Realism

Not visible in the currentViewport !

1

2

3

4


15

3D Graphics

• 3D Graphics APIs allow recreating a natural scene, by using– One or more light sources

– Replicating natural phenomena like reflections, shadows, surface textures, and cloth surface properties like bending or water properties like waves

– A specified viewport from where viewer is looking

• This emphasis on recreating realistic “Physics”, has resulted in driving computational complexity higher and higher, in GPUs

• In order to accelerate this in HW, several standard blocks are defined• What is OpenGL anyway ?


16

Key OpenGL Terminologies

• OpenGL– An API from Khronos (from SGI), for constructing a 3D object, doing operations on

the object, and displaying it

• Primitives– Triangles, Lines, Points, that can be specified through vertices to define an

arbitrary shape

• Texture– Small (!) bitmap to make objects more realistic

• EGL– The EGL API defines a portable mechanism for creating GL contexts and windows

for rendering into, which may be used in conjunction with different native platform window systems using the WSEGL layer

• Shader Language– A C like language, for writing vertex and fragment shaders. For GL ES2.0.


17

The OpenGL BazaarThe OpenGL Bazaar

OPENGL Full versionOPENGL Full version

ES versionES version

CommonCommon

Common-LiteCommon-Lite

GLSL companionGLSL companion

GLSL-ES companionGLSL-ES companion

What we miss in ES compared to desktop version:Polygons, Display lists, Accumulation buffers,…

Currently in 3.0

Currently in 2.0Currently in 1.0.16

Currently in 1.20

EGLEGLCurrently in 1.3

Core GL SpecCore GL Spec Vertex/FragshadersVertex/Fragshaders

Platform interface and Configuration setupPlatform interface and Configuration setup

EGLEGLCurrently in 1.3

2.0 vs 1.1


18

ES2.0,ES1.1 and Full FunctionES2.0,ES1.1 and Full Function

• ES2.0 offers a Programmable pipeline vs Fixed function in ES1.1– Ex, Lighting, Fog functions in ES1.1 not present in ES2.0, so need to write

code to generate these effects.– More flexibility, little more work, but lot of available code– Android uses ES 1.1

• ES2.0 NOT backward compatible with ES1.1

• See “Difference Specification” documents in Khronos OpenGL website, for summary of changes compared to openGL “full” function specification

• Ok, so what exactly is the OpenGL pipeline ?


19

ES2.0 Pipeline


20

Sample OpenGL ES 2.0 program

Handle = get_platform_specific_window_handle();eglGetDisplay(handle);eglInitialize();

eglBindAPI(EGL_OPENGL_ES_API);eglChooseConfig();eglCreateWindowSurface();eglCreateContext();eglMakeCurrent();

//Compile and Load the shaders before this step …float afVertices[] = {G,H,F, F,H,E, E,A,F, F,A,B, …};glEnableVertexAttribArray(0);glVertexAttribPointer(VERTEX_ARRAY, 3, GL_FLOAT, GL_FALSE, 0, (const void*)afVertices);glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);

eglSwapBuffers(eglDisplay, eglSurface);

Setup

Actors

Show

Keep showing the actors, in a loop, change view/position/texture .. How does the shader code look like ?


21

Simple ShaderSimple Shader

• char* pszFragShader = "\

• uniform sampler2D sTexture; \

• varying mediump vec2 TexCoord; \

• void main (void)\

• {\

• gl_FragColor = texture2D(sTexture, TexCoord);\

• }";

• char* pszVertShader = "\

• attribute highp vec4 inVertex;\

• attribute mediump vec2 inTexCoord;\

• uniform mediump mat4 MVPMatrix;\

• varying mediump vec2 TexCoord;\

• void main()\

• {\

• gl_Position = MVPMatrix * inVertex;\

• TexCoord = inTexCoord;\

• }";

GL ES 2.0 permits shader code to be precompiled and loaded, or dynamically compiled and loaded


22

Alpha Blending

• Many options for blending –

– Multi-textures, Blending with Framebuffer, Can be looped…

• Blending with framebuffer

– glEnable(GL_BLEND); //disabled by default– glBlendEquation(GL_FUNC_ADD); //many other equations

available– glBlendFunc(GL_SRC_ALPHA, GL_DST_ALPHA); //many other

modes available


23

Texturing using OpenGL

• Pass Texture coordinates– glEnableVertexAttribArray();

– glVertexAttribPointer();

• Use glTexImage2D() to load image buffer for texture

• Apply texture in the fragment shader program– gl_FragColor = texture2D();

• OMAP3 supports ARGB 8888 textures– Also YUV textures using extensions (IMG_STREAM)

When we do texturing, scaling is automatic


24

Rotation with OpenGL

• Rotation is accomplished by modifying the viewing angle

• Update the Model View Projection Matrix before drawing

• Update the vertex position in the vertex shader

– gl_Position = MVPMatrix * inVertex; //

• Along with texturing, can be used to create “wet floor” effects

– Apply texture in reverse way


25

Texturing with SGX - Example

An exampleWindow managerusing texturing and SGX only

Efficient Rendering


26

Efficient Rendering Paths

Discussion:How many ways can we implement this ?

..SGX Core

Dis

play

Draw


27

Overview of the SGX core

SGX uses “deferred” rendering

REF: POWERVR SGX.OpenGL ES 2.0 Application Development Recommendations.1.8f.External.pdf (from AnandTech)


28

Measuring Performance

• Performance of Graphics is not a single number, but a collection of parameters

• Almost all operations other than texturing, take about 10-15% of 500 MHz of Cortex-A8. 2D Texturing can take 100% CPU depending on size of texture

• Optimised 2D Texturing methods using SGXSink bring CPU utilisation to minimum

Raw Throughput Qua

lity

Raw Throughput Qua

lity

CPU Utilisation

DDR Bandwidth


29

Benchmarking Tools

• Throughput measured under 2 cases,– Vertex Throughput– Pixel Throughput

• Vertex throughput is very important for gaming scenerios (Millions of vertices)

• UI/ Blit use cases have more dependency on Pixel throughput

• SGXPERF – tool allows benchmarking of various texturing methods (2D Pixel throughput)

– https://gforge.ti.com/gf/project/gleslayer/

• How do we compare against ARM-NEON, and all APIs available on SGX ?


30

SGX Debugging Tools

• For debugging, PDUMP tools are provided that is similar to a stack trace – useful for debugging crash scenarios during power management transitions

• In addition, Imagination Tech provides PVRTUNE for tuning the performance– Shows loading of each shader, number of tasks

GPU futures…


31

Texture as data

• Fragment Shader is primarily SIMD, processing all pixels as a parallel stream

• Each computation is independent of the other

• Vertex Shader can do MIMD processing, but General Purpose processing is done with the more efficient Fragment Shader

• Right way to represent data arrays is as a Texture Stream


32

The OpenCL standard

• OpenCL/DirectCompute take the concept of multi-core programming to a common model across the industry !

• Each core is considered a “Compute Unit”

• The host can “submit” “kernels” (like FFT or DCT or any C function) to any Compute Unit

• Event interactions are defined, so that results of computation can be retrieved

• Textures as Arrays– N elements of 1D array, as AxB 2D

Texture, where AxB = N– 1:1 mapping of buffers to be used

• GPU Computing != GP.GPU which is shader based

• Refer OpenCL Cheatsheet Possibilities

Possibilities of GPU moving forward

• General Purpose programming - C/C++ (+) OpenGL (OpenCL)– More complicated because, Games just have to look good, Computations have to be accurate !

– Will make more processing power available to general applications !

• Integrated Web with OpenGL (WebGL)

• Medical Imaging applications

• Usage of higher computing power - Raytracing, Physics processing

Conclusion


34

Conclusion

• Core clocks are not going to get much faster, but multi-core is here to stay

• Requirements for 2D Graphics vastly different from 3D Graphics

• Continued focus needed on enabling UI frameworks like QT, Browsers using 2D accelerators

• Full usage of 3D in commercial UI is soon approaching - Video texturing, Coverflow are examples that show that it is possible

• Video+Graphics+display integration is key to having a high texturing performance


35

References

• Porter & Duff Operations

• SGXPERF

• The SGX architecture used in IPhone– http://www.anandtech.com/printarticle.aspx?i=3579

• The OpenGL Redbook– http://www.glprogramming.com/red/chapter01.html

• OpenGL ES2.0 programming handbook– OpenGL® ES 2.0 Programming Guide, Aaftab Munshi; Dan Ginsburg; Dave Shreiner

• Khronos organisation (OpenGL, OpenMAX, OpenVG ..)– http://Khronos.org


36

Which Graphics ?


37

Graphics CheatsheetName Author Description

Cairo Cairo-graphics API that can take care of interfacing to multiple backends like X, DirectFB etc

Flash Adobe Application framework for games, animation, video and others

Clutter Intel UI development framework, based completely on OpenGL, with different backends

Gst-Gl Gstreamer Plugin for allowing video to be played as OpenGL ES textures

XGL,WGL,EGL Multiple Interfaces for native platform support for interfacing to the windowing system

OpenGL/ES Khronos An API for 3D Graphics programming

OpenCL Khronos An API for multi core programming

QT Nokia/Trolltech An application framework to develop native applications

Silverlight Microsoft An API primarily for Web Application programming, but also for native

.NET Microsoft An API for application programming, with C# language

Surfaceflinger Android/Google Framework for surface creation and managing drawing

DirectFB DirectFB Framework for surface creation and managing drawing, mostly using HW support

X11 X org A windowing system/Manager very common on Linux system

DirectX Microsoft A collective name for interfaces like DirectDraw etc. Currently in DirectX11 version. Handles 3D and 2D

GDI Microsoft Native API for Win32 programming

Opera Opera Optimised Browser for embedded platforms

Mesa3D/Gallium Mesa An Opensource implementation of OpenGL API on Linux, can use HW if present. Moving to Gallium3D

Vincent3D Vincent An Opensource pure software implementation of OpenGL API on WinCE/Mo

SDL SDL A UI API that is cross platform, primarily meant for Games but also used as backend for rendering

GLEW - Linux library that makes it easier for an OpenGL application to use GL extensions

GLUT - Cross platform library, having window system and GL utility funcs on different platforms

Introduction to 2D/3D Graphics

Technology

varying mediump vec2texcoord

customer centric strategies

dst src amp

pixel throughput

surface creation

vertex throughput

vertex shader

3d graphics