1/4 Featured Sessions, Kandroid S/W Fundamentals Study Group Android Graphics and Android EGL www.kandroid.org 운영자 양정수 ([email protected])
1/4 Featured Sessions,
Kandroid S/W Fundamentals Study Group
Android Graphics
and Android EGL
www.kandroid.org 운영자
양정수 ([email protected])
목 차
Why review Android EGL?
• Google IO 2012 Project Butter
• Change history of Android Internal graphics architecture
• TLS based EGLContext issues
Android Graphics and EGL
• Understanding Android Graphics Internals
• EGLSurface and ANativeWindow
• EGLContext and Thread Local Storage
• EGL Implementation : SurfaceFlinger & HardwareRenderer
Useful Topics
• Performance Analysis Knowledge : Systrace, dumpsys
• OpenGL|ES Context Management in MT Environment
• Unified Memory Management
3 2014, 13th Kandroid minmax - www.kandroid.org
For Butter or Worse
Smoothing Out Performance in Android UIs
Why review Android EGL?
4 2014, 13th Kandroid minmax - www.kandroid.org
For Butter or Worse
Smoothing Out Performance in Android UIs
Why review Android EGL?
5 2014, 13th Kandroid minmax - www.kandroid.org
Why review Android EGL?
GLES2.0 SurfaceFlinger
Android 4.4 upgrades its
SurfaceFlinger from OpenGL ES
1.0 to OpenGL ES 2.0. This
boosts performance by using
multi-texturing, and it improves
color calibration and supports
more advanced special effects.
New Hardware Composer
support for virtual displays
The latest version of Android
Hardware Composer,
HWComposer 1.3, supports
hardware composition of one
virtual display in addition to the
primary, external (e.g. HDMI)
display, and has improved
OpenGL ES interoperability.“
https://groups.google.com/forum/
#!topic/android-
developers/RQTTqLTJfUs
ADF : http://lwn.net/Articles/565422/
EGL_Android _native_fence_sync :
http://www.khronos.org/registry/egl/extensi
ons/ANDROID/EGL_ANDROID_native_fen
ce_sync.txt
Android 4.4 (KitKat) Design
Pattern-Graphics Subsystem
http://blog.csdn.net/jinzhuojun/art
icle/details/17427491
Android 4.4 (KitKat) in
virtualization VSync signal
http://blog.csdn.net/jinzhuojun/art
icle/details/17293325
./frameworks/native/services/surfaceflinger
6 2014, 13th Kandroid minmax - www.kandroid.org
Why review Android EGL?
Android 4.4 & rendering pipeline improvements
• Shared assets texture • Better merging of drawing operations • Asynchronous texture uploads • Improved GPU state management • Automatic overdraw avoidance • Software v-sync • OpenGL drivers pre-loading
Source :
https://plus.google.com/+RomainGuy/posts/9QSTyVCSoz3
./frameworks/base/libs/hwui
7 2014, 13th Kandroid minmax - www.kandroid.org
Why review Android EGL?
./frameworks/native/libs/gui
./frameworks/native/libs/ui
8 2014, 13th Kandroid minmax - www.kandroid.org
./frameworks/base/core/java/android/view/SurfaceView.java
public class SurfaceView extends View {
final Surface mSurface = new Surface();
private SurfaceHolder mSurfaceHolder = new SurfaceHolder() {
public void addCallback(Callback callback) { }
public void setType(int type) {
// SURFACE_TYPE_NORMAL (SURFACE_TYPE_HARDWARE, SURFACE_TYPE_GPU)
// SURFACE_TYPE_PUSH_BUFFERS
}
public Canvas lockCanvas() { }
public void unlockCanvasAndPost(Canvas canvas) { }
public Surface getSurface() { }
}
}
Deprecated in API Level 11 (H)
Deprecated in API Level 5 (D)
Why review Android EGL?
9 2014, 13th Kandroid minmax - www.kandroid.org
Why review Android EGL?
./frameworks/base/core/java/android/view
Layout
Object
Adapter
View
Widget
SurfaceView
View
View
Group
TextureView
A TextureView can be used to display a
content stream. Such a content stream
can for instance be a video or an
OpenGL scene. The content stream can
come from the application's process as
well as a remote process.
TextureView can only be used in a
hardware accelerated window. When
rendered in software, TextureView will
draw nothing.
Unlike SurfaceView, TextureView does
not create a separate window but
behaves as a regular View. This key
difference allows a TextureView to be
moved, transformed, animated, etc.
10 Korea Android - www.kandroid.org
Why review Android EGL?
If a green dotted area each have GLSurfaceView in the Activity above,
is there any problem?
GLThread GLThread GLThread GLThread
EGLContext
TLS
EGLContext
TLS
EGLContext
TLS
EGLContext
TLS
Why review Android EGL?
What is the Direction?
SW : Standard API (Java, NDK Stable API)
HW : OpenGLES, OpenSLES, OpenMAX
EGL™ is an interface between Khronos rendering
APIs such as OpenGL ES or OpenVG and the
underlying native platform window system
Passion is not about speed,
but DIRECTION !
목 차
Why review Android EGL?
• Google IO 2012 Project Butter
• Change history of Android Internal graphics architecture
• TLS based EGLContext issues
Android Graphics and EGL
• Understanding Android Graphics Internals
• Android EGL Overview
• EGLSurface and ANativeWindow
• EGLContext and Thread Local Storage
• EGL Implementation : SurfaceFlinger & HardwareRenderer
Useful Topics
• Performance Analysis Knowledge : Systrace, dumpsys
• OpenGL|ES Context Management in MT Environment
• Unified Memory Management
13 2014, 13th Kandroid minmax - www.kandroid.org
Graphics Basics • Surfaces and centralized surface composition
• What is a window on android? (ANativeWindow)
• The native graphic buffer handle
• The native graphic buffer ANativeWindowBuffer
• Flattenable
• GraphicBuffer class
• Surface class
The Graphics Surface Service Interfaces
• ISurfaceComposer
• ISurfaceComposerClient
Graphics Buffer Handling Interfaces
• IGraphicBufferProducer
• IGraphicBufferAlloc
Gralloc and HWComposer
• gralloc
• hwcomposer
SurfaceFlinger
• Periodical vsync event sources
• Spontaneous vsync event sources
• BufferQueue
• BufferQueue::BufferSlot and BufferQueue::BufferSlot::BufferState
• SurfaceFlingerConsumer and GLConsumer class
• Layer Class
• FramebufferSurface and DisplayDevice
• HWComposer
• SurfaceFlinger
Source : https://charleszblog.wordpress.com/category/android-2/graphics-android/
Understanding Android Graphics Internals
Android 4.4 (KitKat) Design Pattern-Graphics Subsystem
http://blog.csdn.net/jinzhuojun/article/details/17427491
Android 4.4 (KitKat) in virtualization VSync signal http://blog.csdn.net/jinzhuojun/article/details/17293325
http://blog.csdn.net/yangwen123/article/details/22647255
14 Korea Android - www.kandroid.org
EGL
Android Native Window System
Display
• eglGetDisplay()
Surface
• eglCreateWindowSurface()
• eglCreatePBufferSurface()
• eglCreatePixmapSurface()
Context
• eglCreateContext()
• eglInitialize()
• eglChooseConfig()
• eglGetConfigAttrib()
• eglQuerySurface() • eglMakeCurrent()
• eglSwapBuffers()
EGLNativeWindowType EGLNativePixmapType
D D S D S C
D D
Android EGL Overview
S
OpenGL|ES
15 Korea Android - www.kandroid.org
Android EGL Overview
• eglGetDisplay()
• eglInitialize()
• eglChooseConfig()
• eglCreateContext()
• eglCreateWindowSurface()
eglCreatePBufferSurface()
eglCreatePixmapSurface()
• eglMakeCurrent()
• eglSwapBuffers()
draw()
|-- egl
| |-- egl.cfg
| `-- libGLES_android.so
|-- libEGL.so
|-- libETC1.so
|-- libFFTEm.so
|-- libGLES_trace.so
|-- libGLESv1_CM.so
|-- libGLESv2.so
|-- libGLESv3.so
JNI
16 Korea Android - www.kandroid.org
EGLSurface and ANativeWindow
• eglGetDisplay()
• eglInitialize()
• eglChooseConfig()
• eglCreateContext()
• eglCreateWindowSurface()
eglCreatePBufferSurface()
eglCreatePixmapSurface()
• eglMakeCurrent()
• eglSwapBuffers()
draw()
EGLSurface eglCreateWindowSurface(
EGLDisplay dpy,
EGLConfig config,
NativeWindowType window,
const EGLint *attrib_list)
{ …
ANativeWindow* anw
= reinterpret_cast<ANativeWindow*>(window); anw->setSwapInterval(anw, 1);
EGLSurface surface = cnx->egl.eglCreateWindowSurface(
iDpy, config, window, attrib_list);
if (surface != EGL_NO_SURFACE) {
egl_surface_t* s = new egl_surface_t(dp.get(), config, window,
surface, cnx);
return s;
}
}
EGLSurface eglCreatePixmapSurface(
EGLDisplay dpy,
EGLConfig config,
NativePixmapType pixmap,
const EGLint *attrib_list) { … }
EGLSurface eglCreatePbufferSurface(
EGLDisplay dpy,
EGLConfig config,
const EGLint *attrib_list) { … }
libagl
17 Korea Android - www.kandroid.org
class egl_object
egl_object_tegl_display_t
egl_context_tegl_surface_t
egl_window_surface_v 2_t
+ initCheck() :bool
+ swapBuffers() :EGLBoolean
+ bindDrawSurface() :EGLBoolean
+ bindReadSurface() :EGLBoolean
+ connect() :EGLBoolean
+ disconnect() :void
+ getWidth() :EGLint
+ getHeight() :EGLint
+ getHorizontalResolution() :EGLint
+ getVerticalResolution() :EGLint
+ getRefreshRate() :EGLint
+ getSwapBehavior() :EGLint
+ setSwapRectangle() :EGLBoolean
- lock() :status_t
- unlock() :status_t
egl_pixmap_surface_t
+ initCheck() :bool
+ bindDrawSurface() :EGLBoolean
+ bindReadSurface() :EGLBoolean
+ getWidth() :EGLint
+ getHeight() :EGLint
egl_pbuffer_surface_t
+ initCheck() :bool
+ bindDrawSurface() :EGLBoolean
+ bindReadSurface() :EGLBoolean
+ getWidth() :EGLint
+ getHeight() :EGLint
libagl
implementation
EGLSurface and ANativeWindow
18 2014, 13th Kandroid minmax - www.kandroid.org
Source : https://wiki.mozilla.org/Platform/GFX/Gralloc
EGLSurface and ANativeWindow
EGLSurface
ANativeWindow
FramebufferNativeWindow Surface
GraphicBuffer
EGL world
Android world
C++ classes
19 2014, 13th Kandroid minmax - www.kandroid.org
EGLContext and Thread Local Storage
• eglGetDisplay()
• eglInitialize()
• eglChooseConfig()
• eglCreateContext()
• eglCreateWindowSurface()
eglCreatePBufferSurface()
eglCreatePixmapSurface()
• eglMakeCurrent()
• eglSwapBuffers()
draw()
EGLBoolean eglMakeCurrent(
EGLDisplay dpy,
EGLSurface draw,
EGLSurface read,
EGLContext ctx)
{ …
EGLBoolean result = dp->makeCurrent(c, cur_c, draw, read, ctx,
impl_draw, impl_read, impl_ctx);
if (result == EGL_TRUE) {
if (c) {
setGLHooksThreadSpecific(c->cnx->hooks[c->version]);
egl_tls_t::setContext(ctx); _c.acquire();
_r.acquire();
_d.acquire();
} else {
setGLHooksThreadSpecific(&gHooksNoContext);
egl_tls_t::setContext(EGL_NO_CONTEXT);
}
}
… return result;
}
20 Korea Android - www.kandroid.org
EGLContext and Thread Local Storage
Source : http://www.roguewave.com/portals/0/products/legacy-hpp/docs/thrug/3-9.html
pthread_internal_t
TLS area [TLS_SLOT]
stack area
guard page
Source : bionic/libc/bionic/pthread.c
The layout of the thread’s stack
enum {
TLS_SLOT_SELF = 0,
TLS_SLOT_THREAD_ID,
TLS_SLOT_ERRNO,
TLS_SLOT_OPENGL_API = 3,
TLS_SLOT_OPENGL = 4,
TLS_SLOT_BIONIC_PREINIT
= TLS_SLOT_OPENGL_API,
TLS_SLOT_STACK_GUARD = 5,
TLS_SLOT_DLERROR,
TLS_SLOT_FIRST_USER_SLOT
}; // bionic_tls.h
Heap
21 2014, 13th Kandroid minmax - www.kandroid.org
Source : https://source.android.com/devices/graphics.html
EGLImplementation : HWRenderer and SurfaceFlinger
22 Korea Android - www.kandroid.org
EGLImplementation : HWRenderer and SurfaceFlinger
SurfaceTextureLayer Surface Layer
BufferQueue
IGraphicBuffer
Producer
IGraphicBuffer
Consumer
SurfaceView
Surface
SurfaceHolder
GLSurfaceView TextureView
SurfaceTexture
SurfaceTexture
23 Korea Android - www.kandroid.org
screensavers : com.android.dreams.basic.ColorsGLRenderer : SurfaceTexture
Gallery2 : com.android.photos.views.BlockingGLTextureView : SurfaceTexture
Camera2 : com.android.camera.SurfaceTextureRenderer : SurfaceTexture
Nfc : com.android.nfc.FireflyRenderer : SurfaceTexture
Camera : com.android.camera.MosaicPreviewRenderer : SurfaceTexture
Launcher3 : com.android.photos.views.BlockingGLTextureView : SurfaceTexture
frameworks : com.android.systemui.ImageWallpaper : SurfaceHolder
frameworks : com.android.photos.views.BlockingGLTextureView : SurfaceTexture
frameworks :com.android.server.power.ElectronBeam : Surface
frameworks : android.view.HardwareRenderer.GlRenderer : Surface
frameworks : android.opengl.GLSurfaceView.DefaultWindowSurfaceFactory : SurfaceHolder
frameworks : android.opengl.EGLLogWrapper : Surface
EGLImplementation : HWRenderer and SurfaceFlinger
EGLSurface surface;
if (sur != null) {
surface = _eglCreateWindowSurface(dpy, config, sur, attrib_list, offset);
}
else if (win instanceof SurfaceTexture) {
surface = _eglCreateWindowSurfaceTexture(dpy, config, win, attrib_list,
offset);
}
eglCreateWindowSurface()
• android.opengl.EGL14
• com.google.android.gles_jni.EGLImpl implements EGL10
목 차
Why review Android EGL?
• Google IO 2012 Project Butter
• Change history of Android Internal graphics architecture
• TLS based EGLContext issues
Android Graphics and EGL
• Understanding Android Graphics Internals
• EGLSurface and ANativeWindow
• EGLContext and Thread Local Storage
• EGL Implementation : SurfaceFlinger & HardwareRenderer
Useful Topics
• Performance Analysis Knowledge : Systrace, dumpsys
• OpenGL|ES Context Management in MT Environment
• Unified Memory Management
25 Korea Android - www.kandroid.org
http://developer.android.com/tools/debugging/systrace.html
Performance Analysis Knowledge : systrace
26 Korea Android - www.kandroid.org
systrace
(python)
adb adbd
atrace
• atrace_args = ['adb', 'shell', 'setprop', 'debug.atrace.tags.enableflags', hex(flags)]
• atrace_args = ['adb', 'shell', 'atrace', '-z'] + args
start
Trace
stop
Trace
dump
Trace
Linux Kernel (ftrace feature)
android.os.Trace ATRACE_CALL()
ATRACE_INT()
ScopedTrace
Tracer
Tracing Point
Java Native
https://code.google.com/p/trace-viewer/
android_os_Trace
jni
trace_marker trace tracing_on …
Performance Analysis Knowledge : systrace
27 Korea Android - www.kandroid.org
ATRACE_TAG
• GRAPHICS
• INPUT
• VIEW
• WEBVIEW
• WINDOW_MANAGER
• ACTIVITY_MANAGER
• SYNC_MANAGER
• AUDIO
• VIDEO
• CAMERA
• HAL
• APP
• RESOURECS
• DALVIK
• RS
0
50
100
150
200
250
Java
Native
Total
Performance Analysis Knowledge : systrace
28 Korea Android - www.kandroid.org
Activity
Thread
Hello
Android .apk Activity
Looper
Message
Queue
Service
BroadcastReceiver
ContentProvider
H
dispatchMessage()
TLS
Threaded
Task
Binder
Object
Storage
(File,Prefs,DB,Network)
Android
Manifest.xml
Resources
Intent
Application
Instru
me
nta
tion
test.apk Test Case Application
Thread
Binder
Thread
Pool
AsyncTask
Thread
Pool
Choreographer
$FrameHandler
dispatchMessage()
Window
Views Views
Views
ViewRootImpl
Performance Analysis Knowledge : systrace
Drawable
Bitmap
Input
(4)
AM
(30)
View
(5)
AM
(3)
Res
(3)
Res
(3)
Graphics
(3)
29 Korea Android - www.kandroid.org
Layout
Inflation Adapter
Adap
terV
iew
ConvertView
Dumb
Recycling
Views
View
Holder
Vie
w H
old
er
getView(pos,convertView, parent)
View getView(pos,convertView, parent)
Performance Analysis Knowledge : systrace
View
(3)
View
(2)
30 Korea Android - www.kandroid.org
Activity
SurfaceFlinger
Event
Set
Property
Value
Invalidate
Measure
&
Layout
Prepare
Draw
Update
DisplayList
Draw
DisplayList
Swap
Buffers
Display
List
Dequeue
Buffer
Composite
Windows
Post
Buffer
queue
Buffer
Something
Happens
Draw
Display
Performance Analysis Knowledge : systrace
31 2014, 13th Kandroid minmax - www.kandroid.org
Performance Analysis Knowledge : dumpsys
Phone & Nfc(JB) App
DRMServer (JB)
MediaServer
Kernel Binder
/dev/binder
Service
Manager
Binder
Context
Manager
SurfaceFlinger
SystemServer
Dalvik VM
Native Service
• Sersor Service
• Surface Flinger
Framework Service
• System Services
• Hardware Services
libbinder.so (libcutils.so, libutils.so)
dumpsys
addService listService
checkService
dumpService
1 2
3
32 Korea Android - www.kandroid.org
SurfaceFlinger
accessibility
account
activity
alarm
android.security.keystore
appops
appwidget
assetatlas
audio
backup
battery
batterypropreg
batterystats
clipboard
commontime_management
connectivity
consumer_ir
content
country_detector
cpuinfo
dbinfo
device_policy
devicestoragemonitor
diskstats
display
dreams
drm.drmManager
dropbox
entropy
gfxinfo
hardware
input
input_method
iphonesubinfo
isms
location
lock_settings
media.audio_flinger
media.audio_policy
media.camera
media.player
media_router
meminfo
mount
netpolicy
netstats
network_management
notification
package
permission
phone
power
procstats
samplingprofiler
scheduling_policy
search
sensorservice
serial
servicediscovery
simphonebook
statusbar
telephony.registry
textservices
uimode
updatelock
usagestats
usb
user
vibrator
wallpaper
wifi
wifip2p
window
Performance Analysis Knowledge : dumpsys
33 2014, 13th Kandroid minmax - www.kandroid.org
Source : http://malideveloper.arm.com/downloads/deved/tutorial/SDK/android/thread_sync.html
OpenGL|ES Context Management in MT Environment
OpenGL ES 3.0 sync objects
• GLsync glFenceSync(GLenum condition, GLbitfield flags);
• GLenum glClientWaitSync(GLsync sync, GLbitfield flags, GLuint64 timeout);
• void glWaitSync(GLsync sync, GLbitfield flags, GLuint64 timeout);
• void glDeleteSync(GLsync sync);
• void glGetSynciv(GLsync sync, GLenum pname, GLsizei bufSize,
GLsizei *length, GLint *values);
fencing on vs. fencing off
OpenGL ES Context Management in Multi-threaded Environment
Working in OpenGL ES within a multi-threaded environment requires some minor additional work in your
application. In order to work with multiple threads in OpenGL ES, we must follow these rules:
• Only one rendering context can be current per thread.
• A given context can be current to only one thread.
In the case of multiple threads sharing a single rendering context, we must unbind the context from the thread
where it is current before making it current to any other thread. The order of operations is as follows:
Thread 1 Thread 2
eglMakeCurrent(display, surface, surface, context);
Perform OpenGL ES operations
eglMakeCurrent(display, EGL_NO_SURFACE,
EGL_NO_SURFACE, EGL_NO_CONTEXT);
eglMakeCurrent(display, surface, surface, context);
Perform OpenGL ES operations
34 2014, 13th Kandroid minmax - www.kandroid.org
Unified Memory Management
Source : http://people.linaro.org/linaro-connect/lcu13/presentations/World-of-UMM.pdf
Ref : http://www.slideshare.net/linaroorg/android-graphicsupstreamingv2
35 Korea Android - www.kandroid.org
Source : http://people.linaro.org/linaro-connect/lcu13/presentations/World-of-UMM.pdf
Ref : http://www.slideshare.net/linaroorg/android-graphicsupstreamingv2
Unified Memory Management
1/4 Featured Sessions,
Kandroid S/W Fundamentals Study Group
Q & A
www.kandroid.org 운영자
양정수 ([email protected])