20090114_UniPV_Nomadik

Nomadik Application Processor

Andrea GalloGiancarlo Asnaghi

January 14, 2009

MP3 PortablePlayer

Set Top Box DVD Player

ST is #1 world-wide leader inDigital TV and Consumer Audio

Digital Satellite RadioDigital Car Radio

MMDSP+ inside more than 200 million produced chips

January 14, 2009

ST leader in mobile phone chips

January 14, 2009

Nomadik

Nomadik is based on this heritage providing:

– Unrivalled multimedia performances– Very low power consumption– Scalable performances

January 14, 2009

Best Application Processor 2004Best Application Processor 2004

Nominees:

Intel XScale PXA260, NeoMagicMiMagic 6, Nvidia MQ-9000, STMicroelectronics Nomadik STn8800, Texas Instruments OMAP 1611

Lowest power consumptionScalable performanceVideo/Audio qualityCost-effective

January 14, 2009

Nomadik

Nomadik is a family of Application Processors

– Distributed processing architecture ARM9 + multiple Smart Accelerators

– Support of a wide range of OS and applications

– Seamless integration in the OS through standard API drivers and MM framework

January 14, 2009

roadmap

...

January 14, 2009

Some Nomadik products on the market...

January 14, 2009

STn8815 block diagram

January 14, 2009

Nomadik : a true real time multiprocessor platform

SDRAM

•Unlimited Space•Limited Bandwidth

NAND Flash

•Unlimited Space•“No” Bandwidth•Mass storage

Mem

ory Controller

GeneralPurposeSystemDMA

ARM926(L1 + L2)

Master OS

MMDSP+ Audio133 MHz, 24-bit

RTOSMulti-thread

MMDSP+ Video66 MHz, 16-bit

RTOS(Scheduler FSM)

Peripherals

multi-layer AHB bus

SRAM

(Level 2Cache

for Video)

HW Acceleration

Audio Peripherals Video Peripherals

January 14, 2009

Context & task relation: Logical view

User side

System side (kernel)System calls

Drivers &

Kernel components

GUI, User Applications

Application engines

& serversMultiple

Address Space

HW/IO process

ARM boundary (WinCE & Linux OS type)

SAA boundarySVA boundary

Single

Address Space

Flat

Address Space

Virtual world

Physical world

IO process

IO

IO

January 14, 2009

Picture Post Processing

January 14, 2009

Resized: zoom out (preview)

Examples of Video Resize filters targeting Mobile TV

Regular: display 1:1 scale

Wide: zoom to full height

Panoramic crop and zoom to full width

Stretch zoom to full width and height

January 14, 2009

Rotate

January 14, 2009

Hardware/Software partitioning

Handled by Hardware.No flexibility, high computation

Handled by Software on the host CPU.High flexibility, low computationOS friendly

Block

Macro-Block

FirmwareGroup of Macro-Block

Picture

Group of picture

Sequence

MPEG4 software stack

AR

M S

WH

W/F

W

January 14, 2009

MM integration in an open OS

Application (Player,…)

SourceFilter

Codec

Audio

Sink

MM Framework(plug-ins based)

Audio

Filter/Codec

Video

Filter/CodecVideo

Sink Virtual Media pipeline

Integration/Adaptation layer

SAA HW Driver (control path)

SAA

SVA HW Driver (control path)

SVA

User side

Kernel side

Video

Codec

Post-

Processing

Audio

Codec

Display FrameBuffer

PCM if

January 14, 2009

t1 t2 t3 t4 t5

Concurrent execution

ARM

SAA

SVA

Read & Parse

Audio Decode

Video Decode

Audio Playback

Video Display

Read & Parse

Audio Decode

Video Decode

Audio Playback

Video Display

Read & Parse

Audio Decode

Video Decode

Audio Playback

Video Display

Read & Parse

Audio Decode

Video Decode

Read & Parse

January 14, 2009

Power Management support

~200mA

~mA

~µA

Standby mode: ARM in WFI mode (Wait For Interrupt)

• PLL1 & 2 are running

• All peripheral clocks are gated

Power Save mode: • ARM in WFI mode

• PLL1 OFF, PLL2 Running

• CPU & Bus clocked at XTAL frequency

• All peripheral clocks unused are gated

RUN mode:

Full clock speed Doze mode: • ARM in WFI mode

• PLL1 & 2 OFF, clocks = 32 kHz

• All peripheral clocks are gated

Sleep mode: • ARM in WFI mode

• PLL1 & 2 OFF, clocks are stopped except the non-switchable part @ 32 kHz

• All peripheral clocks are gated

Deep-Sleep mode:- Main part of the logic is no more powered- The non switchable part is clocked @32 kHz

Wake up:3ms

Standby mode: ARM in WFI mode (Wait For Interrupt)

• PLL1 & 2 are running 64-393MHz

• All peripheral clocks are gated

Power Save mode• ARM in WFI mode

• PLL1 OFF, PLL2 Running 19MHz

• CPU & Bus clocked at XTAL frequency

• All peripheral clocks unused are gated

RUN mode:Full clock speed

64 – 393MHz Doze mode: • ARM in WFI mode

• PLL1 & 2 OFF, clocks 32 kHz

• All peripheral clocks are gated

Sleep mode: • ARM in WFI mode

• PLL1 & 2 OFF, clocks are stopped except the non-switchable part @ 32 kHz

• All peripheral clocks are gated

Deep-Sleep mode:- Main part of the logic is no more powered- The non switchable part is clocked @32 kHz

Wake up:3ms

SW architectureLinux

January 14, 2009

Screenshots and pictures

WVGA 800x480 @ 24bpp Philips TPO display

20%CPU

3%CPU

January 14, 2009

Average CPU load

VGA 30fps 2MbpsXine FFMPEG 10-20%GSTLaunch MPEG4 27%GSTLaunch H264 37%

Playbin +20pp

OpenGL-ESpanel carousel demoARM load 3%

Khronos compliance ongoingLauterbachETM traceH264 VGA

30fps 2Mbps AAC

LauterbachETM trace

MPEG4 VGA 30fps 2Mbps MP3

January 14, 2009

System overview for STn8815

Full open source Linux 2.6.20 kernel

OpenedHandPokyLinux for system test

In line with GMAE, Limo and Maemo

Power Management

cmp K+ architecture

Kernel and driver level

Middleware

Multimedia Interfaces

+ ALSA+ V4L2+ X.11+ xvideo+ EXA+ DirectFB

Multimedia Driv ers

+ SAA Driver+ SGA Lib and Driver+ SVA Driver

GStreamer

+ GStreamer+ OpenMAX-IL+ FFMPEG+ SAA Block Library

OpenGL-ES 1.1

+ Clutter+ EGL+ OpenGL Driver+ OpenGL-ES 1.1+ GLES Library

Graphic User Interface

+ Matchbox+ GTK+

Communication

+ Connection Manager+ Bluetooth+ Wi-Fi+ Ethernet+ Telephony

MEMS

+ Gesture Library+ MEMS Driver

Security

+ Cryptoengine driver+ IPSec+ OpenSSL+ Security

Camera

+ Camera ISP+ ISP Driver

Application Framework

+ Application Framework+ Azingo+ Google Android+ Access ALP+ Trol ltech Qtopia+ OpenMoko+ Poky

Kernel and Driv ers

+ DMA+ I2C+ Linux kernel 2.6.20+ MSP+ SPI+ UART

Multimedia HCL

+ SAA HCL+ SGA HCL+ SVA HCL

Power Management

+ PM framework+ Governor+ Battery Manager+ CPUFreq+ Linux DPM+ APM+ LiMo PM

Connectiv ity

+ SD/MMC+ USB Hi-Speed

File systems

+ JFFS2+ YAFFS+ CRAMFS+ RAMFS+ MTD

January 14, 2009

A/V features at a glanceVideo

– MPEG4 / H.264 decoder up to VGA 30fps 2Mbps– MPEG4 encoder up to VGA 30fps– H.263 codec– JPEG codec

Audio– MP3, AAC / eAAC+, WMA– G.711, G.723, G.726, G.729– AMRnb / AMRwb– Full duplex noise and acoustic echo cancelling– Effects, e.g. equaliser, reverberation, spatial sound

January 14, 2009

Multimedia interfaces

cmp Multimedia Interfaces

V4L2

xvideo

ALSA X.11

EXA

DirectFB

Multimedia Drivers::SGA Lib and Driver

Multimedia Drivers::SVA Driver

Kernel and Drivers::MSP

«use»«flow»

«use» «use»

«use»«use»

Standard Linux interfaces

Nomadik drivers

January 14, 2009

Graphic and video subsystem2D Graphics

– X.11 EXA for copy, solid fill, compositing

– Good acceleration up to 3x on big pixmaps

– Penalties for smaller primitives ARM + L2C is much faster

3D Graphics– ARM for EGL and

parsing– SVA+FPU for TnL– SGA for rendering– Good fit for smart GUI,

not for 3D gamesVideo rendering

– Xvideo extension with hw acceleration in SVA

– Resize, rotate, colourconversion and blitting

January 14, 2009

Multimedia with FFMPEGcmp Karakum

Matchbox

SVA driverSAA Driver

SGA Driver

BlockLibSGA user lib

PMP application Xine

ALSA

Xvideo

EXA

Xserver

GTK+

Web Browser -Minimo

FFMPEG video codecs

FFMPEG audio codecs

AudioCodec

Linphone

V4L2 API

frame buffer

«flow»

«use» «use»«use»

«use»

«use»

«use»

«flow»

«use»«flow»

«flow»

«use»

«flow»

«flow»

«flow»

«flow»

«flow»«flow»

«flow»

«flow»

«flow»

«flow» «flow» «flow» «flow»

«use»

January 14, 2009

cmp Static View

INTEGRATION

FRAMEWORK

APPLICATION

NOMADIK HW

KERNEL

XVideo

«Gstreamer Based»Player App

OMX::OMX_Core

OMX::OMX_AudioDecComponent

OMX::OMX_VideoDecComponent

GSTREAMER_OMX::GstOmxMp3Dec

GSTREAMER_OMX::GstOmxMpeg4Dec

GSTREAMER::GstXVImageSink

GSTREAMER::GstAlsaSink

SAA User Lib::Blocklib

Linux DDrivers::SAA driverLinux DDrivers::ALSA driver Linux DDrivers::SVA driver

ST_FFmpeg::ffmpeg libavcodec

Nomadik HW::SAA

Nomadik HW::SVA

Nomadik HW::Audio Codec (ex. 5095)

Nomadik HW::Frame Buffer

«instantiate» «instantiate» «instantiate» «instantiate»

PCM_Data_Buffer

«flow»«flow»

GStreamer and OpenMAX layers

January 14, 2009

Google Android on Nomadik

LIBRARIES

Media Framework

FreeType

SQLite

WebKit

SSL libc

APPLICATION FRAMEWORKActivity

Manager

Package Manager

WindowManager

Telephony Manager

Content Providers

Resource Manager

View System

Location Manager

Notification Manager

XMPP Service

APPLICATIONS

Home Contacts Phone Browser . . .

LINUX KERNEL

Display Driver

USB Driver

Camera Driver

Keypad Driver

BluetoothDriver

WiFi Driver

Flash Memory Driver

A/V Driver

Binder (IPC) Driver

Power Management

ANDROID RUNTIME

Dalvik Virtual Machine

Core libraries

OpenGL|ES

SGL

Surface Manager

ModemDriver

Crypto Engine Driver

January 14, 2009

Multiple clipping video rendering

Minimise bandwidth usage when not needed!!!

R1

R2 R3

January 14, 2009

Thread Model

omx mpeg4 msg thread omx mp3 msg threadOMX mp3 dec threadOMX mpeg4 dec thread

Play bin thread

gst player

gst media parser

gst-omx mpeg4 dec plugin

gst-omx mp3 decplugin

ffmpeg mpeg4 dec ffmpeg mp3 dec

SAA block lib

Linux SAA driverLinux SVA driver

ALSA libXVLib

gst XVImageSink gst ALSA sink

Input file

omx mpeg4 deccomponent

omx mp3 deccomponent

omx mpeg4 decmsg handler

omx mpeg4 decbuf manager

omx mp3 dec msghandler

omx mp3 dec bufmanager

January 14, 2009

Power ManagementCPUfreq to scale voltage and frequencyCPUIdle to switch amongst running, idle, sleep and deep sleepCustom API to control clock gating to all internal IP’sFeasibility on-going with DPM

Kernel Space

Constraint Manager

/sysfs

PowerTop

Drivers and Bus Drivers

Driver Interface

CpuIdle

/sysfs

Dynamic Tick (Wakeup event)

cpufreq

/sysfs

Governor

«flow»

«use»

January 14, 2009

DVFS – Dynamic Voltage Frequency Scaling

stm Operating Points PH1

489.6/163.2

Initial

393.6/131.2

326.4/163.2

264.0/132.0

100.8/100.8201.6/100.8

Boot

(lab only)

1.0V

1.2V

1.2V

1.2V

1.26V

1.45V

MHzMHz

MHz

MHz

MHz

MHz

January 14, 2009

PM Dump Tool

PM monitor to optimise operating modes and ULP, estimate consumption

– (Under development)

UARTSDMMCMCDETVOUTAUDIOSAASVABTWIFILANCPU 0CPU 1

I (mA) 100 95 20 50 10 100 95 20 50 10 100 95 20V 1.2 1.2 0.9 1 0.8 1.2 1.2 0.9 1 0.8 1.2 1.2 0.9Clock 393 393 64 166 19 393 393 64 166 19 393 393 64

35510ms 35530ms 35550ms 35570ms 35590ms 35610ms 35630ms

Thank you!