Multimedia on android

Multimedia on Android

Types of Multimedia PlaybackPlayback from File system - File formats, codecs

Streaming from network - Protocols, formats, codecs

Real time Conversational - Latency, Protocols, formats, codecs

Playback from File SystemHigh level APIs

MediaPlayer

VideoView

No access to the decoded data - post processing (Audio mixing, Image processing, cropping) etc

Low Level Media APIs - MediaCodecIntroduced in Android 4.1 (API Level 16)

Access to low level media codecs for encoding/decoding of media

Have access to compressed data as well as decompressed data

Compressed data can be used for recording

Decompressed data can be used for post processing

Need to understand codecs and file formats

Data Flow Java Application

Application Framework

Java Runtime

JNI

Native Libs

Kernel

Device Drivers

Hardware

Workflow1. createEncoderByType()/createDecoderByType()

a. mime type "video/3gpp" or "audio/vorbis"

2. configure() - properties eg encryption

3. getInputBuffers() and getOutputBuffers() - get buffers

4. dequeueInputBuffer() - index position of buffer to fill

5. queueInputBuffer() - release buffer

4. dequeueOutputBuffer() - index position of buffer data

5. releaseOutputBuffer() - release buffer

Low Level Media APIs - MediaExtractor

Android 4.2 - API Level 17

MediaExtractor facilitates extraction of demuxed, typically encoded, media data from a data source.

Low Level Media APIs - MediaMuxerAndroid 4.3 - API Level 18

MediaMuxer facilitates muxing elementary streams.

Currently supports mp4 or webm file as the output

At most one audio and/or one video elementary stream.

MediaMuxer does not support muxing B-frames.

ReferencesMore details -

https://developer.android.com/about/versions/android-4.1.html#Multimedia

https://developer.android.com/about/versions/android-4.3.html#Multimedia

https://developer.android.com/reference/android/media/MediaCodec.html

https://developer.android.com/reference/android/media/MediaExtractor.html

https://developer.android.com/reference/android/media/MediaMuxer.html

Case Study

1080p 60 fps250 ms LatencyPlayback + Recording

StreamingRTSP/RTCP/RTP

HTTP Live Streaming (HLS)

MPEG - Dynamic Adaptive Streaming over HTTP (DASH)

RTSP/RTCP/RTPRTSP

- TCP - connection setup and teardown

RTCP

- UDP- Connection feedback

RTP

- UDP- Compressed Audio/Visual Data

RTSP/RTCP/RTPLow latency, suitable for realtime, conversational applications

As it uses UDP, data losses happen causing corrupted playback

Uses custom ports, blocked by firewalls

Custom servers required - Wowza, Real, Darwin

HLSImplemented by Apple, non-standard, defacto standard

HTTP based, uses TCP, no losses, adaptive

Near realtime, not suitable for realtime/conversational applications

Suitable for Live broadcasts (one way), VoD

Uses manifest with links to the media segments, .m3u8 extension

Commodity HTTP infrastructure, caching, CDN (Scale)

Facebook live, Periscope etc

M3u8 File Structure#EXTM3U

#EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=200000

gear1/prog_index.m3u8

#EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=311111

gear2/prog_index.m3u8

#EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=484444

gear3/prog_index.m3u8

#EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=737777

gear4/prog_index.m3u8

M3u8 File Structure#EXTM3U#EXT-X-MEDIA-SEQUENCE:1#EXT-X-TARGETDURATION:10#EXT-X-ALLOW-CACHE:YES#EXTINF:10,130130211307_1.ts#EXTINF:10,130130211307_2.tsEXT-X-DISCONTINUITY#EXTINF:10,130130211307_3.ts#EXTINF:10,130130211307_4.ts

#EXT-X-ENDLIST

#EXTM3U#EXT-X-MEDIA-SEQUENCE:1#EXT-X-TARGETDURATION:10#EXT-X-ALLOW-CACHE:YES#EXTINF:10,130130211307_1.ts#EXTINF:10,130130211307_2.ts#EXTINF:10,130130211307_3.ts

#EXTM3U#EXT-X-MEDIA-SEQUENCE:2#EXT-X-TARGETDURATION:10#EXT-X-ALLOW-CACHE:YES#EXTINF:10,130130211307_2.ts#EXTINF:10,130130211307_3.ts#EXTINF:10,130130211307_4.ts

MPEG - DASHSimilar to HLS, Adobe HDS, Microsoft Smooth Streaming but more flexible

International standard, XML based manifest with .mpd extension

Codec agnostic, HTML5 support

Supports multiple use cases like Ad insertion

More efficient than HLS

Used by Youtube, Netflix etc

Limitation of MediaPlayerSupport for RTSP/RTP/RTCP is good but support for HLS is flaky

Playback issues

No current playback time

No support for discontinuity

No support for ID3

Support across different versions inconsistent, new versions not necessarily better

No support for DASH, DRM etc

ExoplayerThe Android framework provides MediaPlayer as a quick solution for playing media with minimal code

MediaCodec and MediaExtractor classes are provided for building custom media players.

The open source project, ExoPlayer, is a solution between these two options, providing a pre-built player that you can extend.

ExoPlayer supports features not currently provided by MediaPlayer, including Dynamic adaptive streaming over HTTP (DASH), HLS, SmoothStreaming, DRM - Common Encryption, Widevine, Playready.

It's designed to be easy to customize and extend, allowing many components to be replaced with custom implementations.

ExoPlayer is a library that you include in your application, it can be easily updated along with your app.

https://developer.android.com/guide/topics/media/exoplayer.html

Thank you!ramesh130@gmail.com