MediaPlayer Playing Flow

Confidential

Media Playback Flow

Javid Hsu

Confidential

Outline

MediaPlayer Subsystem

Related Files

MediaPlayer Frame of Playing Flow

─ StageFright and AwesomePlayer Relatin

─ AwesomePlayer Frame and Playing Flow

Simple Playing Implement

Confidential

Binder

MediaPlayer Subsystem

Java MediaPlayer

(frameworks/base/media/java/android/media)

libmedia_jni.so

JNI

libmedia.so

(frameworks/av/media/libmedia)

libmediaplayerservice.so

(frameworks/av/media/

libmediaplayerservice)

libstagefright.so

libstagefright_omx.so

libstagefright_soft_XXX.so

(codec)

(frameworks/av/media/

libstagefright/*)

(frameworks/base/media/jni)

libaudioflinger.so

(frameworks/av/services/

audioflinger)

libstagefrighthw.so

libOMXVideoEncoderXX.so

libOMXVideoDecoderXX.so

(vendor codec (hw codec))

Confidential

Related Files

libmedia_jni.so (frameworks/base/media/jni) libmedia.so (frameworks/av/media/libmedia)

Confidential

Related Files

libmediaplayerservice.so

(frameworks/av/media/libmediaplayerservice)

libstagefright.so

(frameworks/av/media/libstagefright)

Confidential

Related Files

libaudioflinger.so

(frameworks/av/services/audioflinger)

Confidential


Stagefright and AwesomePlayer Relation

Confidential


Ref: http://blog.csdn.net/tx3344/article/details/8084912

SurfaceFlinger

AudioFlinger

AwesomePlayer Frame and Playing Flow


http://blog.csdn.net/tx3344/article/details/8084912


Confidential


//For example:

//Simple Playing Implement:

MediaPlayer mp = new MediaPlayer();

mp.setDataSource(PATH_TO_FILE); mp.prepare();

mp.start();

Ref: http://developer.android.com/reference/android/media/MediaPlayer.html

MediaPlayer State Diagram

http://developer.android.com/reference/android/media/MediaPlayer.html

Confidential


//For example: //Simple Playing Implement:


mp.setDataSource(PATH_TO_FILE);

mp.prepareAsync();

mp.start();

Confidential

New MediaPlayer

Applications

Applications Framework

• Initiation and Setup

Media Player

Application

android_media_MediaPlayer MediaPlayer.java

android_media_MediaPlayer_native_init()

android_media_MediaPlayer_native_setup()

Static{ System.loadLibrary("media_jni"); native_init(); }

static void android_media_MediaPlayer_native_init (JNIEnv *env){ clazz = env->FindClass("android/media/MediaPlayer"); : fields.post_event = env->GetStaticMethodID(clazz, "postEventFromNative", "(Ljava/lang/Object;IIILjava/lang/Object;)V"); : fields.context = env->GetFieldID(clazz, "mNativeContext", "I"); : }

static void android_media_MediaPlayer_native_setup (JNIEnv *env, jobject thiz, jobject weak_this) { : sp<MediaPlayer> mp = new MediaPlayer(); // create new listener and give it to MediaPlayer sp<JNIMediaPlayerListener> listener = new JNIMediaPlayerListener(env, thiz, weak_this); mp->setListener(listener); // Stow our new C++ MediaPlayer in an opaque field in the Java object. setMediaPlayer(env, thiz, mp); }

static sp<MediaPlayer> setMediaPlayer (JNIEnv* env, jobject thiz, …) { Mutex::Autolock l(sLock); : env->SetIntField(thiz, fields.context, (int)player.get()); : }

Confidential





mp.prepareAsync();

mp.start();

Confidential

Libraries

(libmedia.so)

MediaPlayer player->setDataSource

IMediaPlayer

Media Player

Application

android_media_MediaPlayer

MediaPlayer.java

Applications


MediaPlayerService::

Client

Media Set Data Source Flow (setDataSource)

SetDataSource

Using file type to choose

setDataSource_pre(player_type

playerType)…..

Using real player to set data source

setDataSource()

android_media_MediaPlayer_setDataSourceFD()

setDataSource()

status_t MediaPlayerService::Client::setDataSource(int fd, int64_t offset, int64_t length) { : player_type playerType = MediaPlayerFactory::getPlayerType(this, fd, offset, length); sp<MediaPlayerBase> p = setDataSource_pre(playerType); if (p == NULL) { return NO_INIT; } : // now set data source setDataSource_post(p, p->setDataSource(fd, offset, length)); return mStatus; }

Confidential


Get Player Type

Create Player Instance

Player Set Data Source

Using file type to get player type

get real player instance

• Load software and hardware codec

• Register default sniffers

• Create AwesomeEvents of onVideoEvent, onStreamDone, onBufferingUpdate,

onVideoLagUpdate, onThumbnailEvent, onCheckAudioStatus and

onAudioTearDownEvent.

• setNotifyCallback


• new FileSource(fd, …) insert to MediaExtractor::Create()

─ Sniff DataSource and get mine type

─ Get extractor with mime type

• Separate video (sp<MediaSource> mVideoTrack) and

audio (sp<MediaSource> mVideoSource) track

Confidential
















Get Player Type



Confidential

Libraries



MediaPlayerFactory.cpp

-factory: IFactory*

-sFactoryMap

NuPlayerFactory

+createPlayer(): return new NuPlayerDriver

SonivoxPlayerFactory

+createPlayer(): return new MidiFile

TestPlayerFactory

+createPlayer(): return new TestPlayerStub

StagefrightPlayerFactory

+createPlayer(): return new StagefrightPlayer

(libmedia.so)

MediaPlayer

player->setDataSource

MediaPlayerFactory

IMediaPlayer

Media Player

Application


Applications



Client

a. playerType:getPlayerType()

MediaPlayerFactory:

StagefrightPlayerFactory

createPlayer() return new StagefrightPlayer();

setDataSource()


setDataSource()

b. sp<MediaPlayerBase>: createPlayer(playerType)

a. getPlayerType:

1. sFactoryMap add all player factory while MediaPlayerService

started.

2. scoreFactory function using compare file

to get corresponding player type:

a. NuPlayerFactory

if(!strncasecmp("http://", url, 7)

|| !strncasecmp("https://", url, 8)

|| !strncasecmp("file://", url, 7))

b. SonivoxPlayerFactory

if (!strncasecmp(url + start, FILE_EXTS[i], len))

static const char* const FILE_EXTS[] = { ".mid",

".midi",

".smf",

".xmf",

".mxmf",

".imy",

".rtttl",

".rtx",

".ota" };

c. StagefrightPlayerFactory

1. compare raw data header if is equal to 0x5367674f

2. Default Player Type

Confidential


Get Player Type Using file type to get player type














Get Player Type



Confidential


Media Player

Application

Applications



MediaPlayer.java

Libraries


StagefrightPlayer MediaPlayerFactory

new StagefrightPlayer()

connect()

OMXClient

OMX

OMXMaster

OMX()

mMaster(new OMXMaster)

Software Codec Hardware Codec

SoftOMXPlugin

AwesomePlayer

new AwesomePlayer()

libstagefrighthw.so

Confidential


Media Player

Application

Applications



MediaPlayer.java

Libraries StagefrightPlayer MediaPlayerFactory

new StagefrightPlayer()

RegisterDefaultSniffers()

AwesomePlayer

new AwesomePlayer()

RegisterSniffer_l(SnifferFunc func)

MPEG4Extractor.cpp

MP3Extractor.cpp

bool SniffMP3(...)

:

:

1. SniffMPEG4 <--> MPEG4Extractor

2. SniffMatroska <--> MatroskaExtractor

3. SniffOgg <--> OggExtractor

4. SniffWAV <--> WAVExtractor

5. SniffFLAC <--> FLACExtractor

6. SniffAMR <--> AMRExtractor

7. SniffMPEG2TS <--> MPEG2TSExtractor

8. SniffMP3 <--> MP3Extractor

9. SniffAAC <--> AACExtractor

10. SniffMPEG2PS <--> MPEG2PSExtractor

11. SniffWVM <--> WVMExtractor

12. SniffAVI <--> AVIExtractor

All Sniff register in gSniffers

(List<SnifferFunc> gSniffers)

bool SniffMPEG4(...)

typedef bool (*SnifferFunc)(

const sp<DataSource> &source, String8 *mimeType,

float *confidence, sp<AMessage> *meta);

DataSource


FileSource

DataSource

- static List<SnifferFunc> gSniffers

+static sp<DataSource> CreateFromUR()

+static void RegisterDefaultSniffers()

+static void RegisterSniffer_l()

+typedef bool (*SnifferFunc)()

+bool sniff()

+bool getUInt16()

+bool getUInt64()

+virtual ssize_t readAt()

+virtual String8 getMIMEType() const()

HTTPBase NuCachedSource2

:

Confidential


Media Player

Application

Applications



MediaPlayer.java

Libraries StagefrightPlayer MediaPlayerFactory

new StagefrightPlayer() new AwesomePlayer()

AwesomePlayer

mVideoEvent = new AwesomeEvent(this, &AwesomePlayer::onVideoEvent); mStreamDoneEvent = new AwesomeEvent(this, &AwesomePlayer::onStreamDone); mBufferingEvent = new AwesomeEvent(this, &AwesomePlayer::onBufferingUpdate); mVideoLagEvent = new AwesomeEvent(this, &AwesomePlayer::onVideoLagUpdate); mThumbnailEvent = new AwesomeEvent(this, &AwesomePlayer::onThumbnailEvent);

• Create AwesomeEvents of onVideoEvent …



Confidential

Applications


Libraries

• set Notify Callback (before new MediaPlayer()) : AP -> HAL

APP : setOn##Listener(m##Listener);

MediaPlayer.java : setOn##Listener(listener){mOn##Listener=listener}

Media Player

Application

android_media_MediaPlayer_native_init(){ : fields.post_event = env->GetStaticMethodID(clazz, "postEventFromNative", "(Ljava/lang/Object;IIILjava/lang/Object;)V"); : }


Static{ System.loadLibrary("media_jni"); native_init(); }


Confidential


(libmedia.so)

MediaPlayer

android_media_MediaPlayer_native_setup()

• set Notify Callback (new MediaPlayer()) : HAL -> C++

Applications


Libraries

Media Player

Application

new MediaPlayer()

setListener(const sp<MediaPlayerListener>& listener) {mListener = listener;}

listener = new JNIMediaPlayerListener(env, thiz, weak_this);


Confidential

Media Player

Application


MediaPlayerService (libmedia.so)

MediaPlayer

player->setDataSource


Client

IMediaPlayer

• set Notify Callback (setDataSource(…)) : client -> service

Applications


sp<Client> c = new Client( const sp<MediaPlayerService>& service, pid_t pid, int32_t connId, const p<IMediaPlayerClient>& client, int audioSessionId, uid_t uid) { : mClient = client; : }

service->create(this, mAudioSessionId)

Libraries

setDataSource()


setDataSource()


Confidential

Libraries

MediaPlayerFactory a. player_type:getPlayerType()

b. sp<MediaPlayerBase>:createPlayer(playerType, this, notify)


Client

• set Notify Callback (setDataSource(…)) : service -> player

StagefrightPlayer:public MediaPlayerInterface:

public MediaPlayerBase

void setNotifyCallback( void* cookie, notify_callback_f notifyFunc) { Mutex::Autolock autoLock(mNotifyLock); mCookie = cookie; mNotify = notifyFunc; } void sendEvent(int msg, int ext1=0, int ext2=0, const Parcel *obj=NULL) { Mutex::Autolock autoLock(mNotifyLock); if (mNotify) mNotify(mCookie, msg, ext1, ext2, obj); }


Media Player

Application

Applications



MediaPlayer.java

StagefrightPlayer AwesomePlayer

mPlayer(new AwesomePlayer)

mPlayer->setListener(this)

Confidential

Libraries

(libmedia.so)

MediaPlayer

IMediaPlayer

Media Player

Application

android_media_MediaPlayer::

JNIMediaPlayerListener MediaPlayer.java

Applications



Client

• set Notify Callback (setDataSource(…)) : send message

AwesomePlayer.cpp

notifyListener_l(int msg, int ext1, int ext2) { : listener->sendEvent(msg, ext1, ext2); : }

StagefrightPlayer:

public MediaPlayerInterface:

public MediaPlayerBase

Notify(int msg, …)

sendEvent(int msg, …)

Notify(int msg, …)

Notify((int msg, …) postEventFromNative(int what, …)

• Media Event Type:


Confidential
















Get Player Type Get Player Type



Confidential


AwesomePlayer.cpp

setDataSource_l(dataSource)

MediaExtractor.cpp

Create(dataSource)

if (!strcasecmp(mime, MEDIA_MIMETYPE_CONTAINER_MPEG4) || !strcasecmp(mime, "audio/mp4")) { ret = new MPEG4Extractor(source); : : return ret;

DataSource.cpp

sniff(mimeType,…)

setDataSource_l(extractor)

extractor

Using Real Player Set Data Source • MediaExtractor::Create(const sp<DataSource> &source,...)


Read at filesource

Compabile ?

Write mine type


Confidential



Read at filesource

Compabile ?

Write mine type

─ Get extractor with mime type Compabile ?

Read at filesource: Ex: Golden Flower mp4

Write mine type

Confidential


Using Real Player Set Data Source • Separate video and audio track

Video track

Audio track

Confidential





mp.prepareAsync();

mp.start();

Confidential

AwesomePlayer.cpp

status_t prepareAsync(){

return prepareAsync_l(); }

status_t prepareAsync_l(){

new AsyncPrepareEvent;

postEvent();

return OK;

}

onPrepareAsyncEvent()

initVideoDecoder

initAudioDecoder

notifyVideoSize_l

notifyListener_l

AwesomePlayer.cpp

status_t prepare(){

prepareAsync_l(…)

condition.wait(Mutex)

}

status_t prepareAsync_l(){

:

:

}

onPrepareAsyncEvent()

initVideoDecoder initAudioDecoder

condition.broadcast();

• prepare • prepareAsync

prepare V.S. prepareAsync (AwesomePlayer)


Confidential

Libraries MediaPlayerService::

Client

(libmedia.so)

MediaPlayer

IMediaPlayer

Media Player

Application


Applications



prepareAsync

prepareAsync()

android_media_MediaPlayer_prepareAsync()

prepareAsync()

mPlayer->prepareAsync()

StagefrightPlayer

prepareAsync() prepareAsync()

AwesomePlayer

AwesomePlayer::

initVideoDecoder() AwesomePlayer::

initAudioDecoder()

TimedEventQueue::

start()

Confidential


initVideoDecoder • How to choose codec

AwesomePlayer

mVideoSource

initVideoDecoder()

findMatchingCodecs(mime, componentName, …)

MediaCodecList : cat /etc/media_codecs.xml

OMXCodec

create(…MetaData, MediaSource, …)

makeComponentInstance(name, callbacks…)

Load the corresponding module and register callback functions. Callback Function: OMX_CALLBACKTYPE OMXNodeInstance::kCallbacks = { &OnEvent, &OnEmptyBufferDone, &OnFillBufferDone };

libstagefrighthw.so

Confidential


Video Playback Flow

Ref: http://www.cnblogs.com/shakin/p/4729534.html

http://www.cnblogs.com/shakin/p/4729534.html

Confidential





mp.prepareAsync();

mp.start();

Confidential

Libraries MediaPlayerService::

Client

(libmedia.so)

MediaPlayer

IMediaPlayer

Media Player

Application


Applications


MediaPlay Start

Start()

start() android_media_MediaPlayer_start()

start()

mPlayer->start ()

StagefrightPlayer

start()

play()

AwesomePlayer AudioPlayer start()

postVideoEvent_l()

AwesomePlayer::

onVideoEvent()

Confidential

Video Audio Synchronization

Playing point

Audio Samples

Video Samples

Start Point

The Time in AudioPlayer Real Playing

The Time in Audi Real Playing out

Playback Time Line:

Playing point

latenessUs (VideoPlayer Lateness Time)

Confidential

The End