Multimedia on Android
Multimedia on Android
Types of Multimedia PlaybackPlayback from File system - File formats, codecs
Streaming from network - Protocols, formats, codecs
Real time Conversational - Latency, Protocols, formats, codecs
Playback from File SystemHigh level APIs
MediaPlayer
VideoView
No access to the decoded data - post processing (Audio mixing, Image processing, cropping) etc
Low Level Media APIs - MediaCodecIntroduced in Android 4.1 (API Level 16)
Access to low level media codecs for encoding/decoding of media
Have access to compressed data as well as decompressed data
Compressed data can be used for recording
Decompressed data can be used for post processing
Need to understand codecs and file formats
Data Flow Java Application
Application Framework
Java Runtime
JNI
Native Libs
Kernel
Device Drivers
Hardware
Workflow1. createEncoderByType()/createDecoderByType()
a. mime type "video/3gpp" or "audio/vorbis"
2. configure() - properties eg encryption
3. getInputBuffers() and getOutputBuffers() - get buffers
4. dequeueInputBuffer() - index position of buffer to fill
5. queueInputBuffer() - release buffer
4. dequeueOutputBuffer() - index position of buffer data
5. releaseOutputBuffer() - release buffer
Low Level Media APIs - MediaExtractor
Android 4.2 - API Level 17
MediaExtractor facilitates extraction of demuxed, typically encoded, media data from a data source.
Low Level Media APIs - MediaMuxerAndroid 4.3 - API Level 18
MediaMuxer facilitates muxing elementary streams.
Currently supports mp4 or webm file as the output
At most one audio and/or one video elementary stream.
MediaMuxer does not support muxing B-frames.
ReferencesMore details -
https://developer.android.com/about/versions/android-4.1.html#Multimedia
https://developer.android.com/about/versions/android-4.3.html#Multimedia
https://developer.android.com/reference/android/media/MediaCodec.html
https://developer.android.com/reference/android/media/MediaExtractor.html
https://developer.android.com/reference/android/media/MediaMuxer.html
Case Study
1080p 60 fps250 ms LatencyPlayback + Recording
StreamingRTSP/RTCP/RTP
HTTP Live Streaming (HLS)
MPEG - Dynamic Adaptive Streaming over HTTP (DASH)
RTSP/RTCP/RTPRTSP
- TCP - connection setup and teardown
RTCP
- UDP- Connection feedback
RTP
- UDP- Compressed Audio/Visual Data
RTSP/RTCP/RTPLow latency, suitable for realtime, conversational applications
As it uses UDP, data losses happen causing corrupted playback
Uses custom ports, blocked by firewalls
Custom servers required - Wowza, Real, Darwin
HLSImplemented by Apple, non-standard, defacto standard
HTTP based, uses TCP, no losses, adaptive
Near realtime, not suitable for realtime/conversational applications
Suitable for Live broadcasts (one way), VoD
Uses manifest with links to the media segments, .m3u8 extension
Commodity HTTP infrastructure, caching, CDN (Scale)
Facebook live, Periscope etc
M3u8 File Structure#EXTM3U
#EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=200000
gear1/prog_index.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=311111
gear2/prog_index.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=484444
gear3/prog_index.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=737777
gear4/prog_index.m3u8
M3u8 File Structure#EXTM3U#EXT-X-MEDIA-SEQUENCE:1#EXT-X-TARGETDURATION:10#EXT-X-ALLOW-CACHE:YES#EXTINF:10,130130211307_1.ts#EXTINF:10,130130211307_2.tsEXT-X-DISCONTINUITY#EXTINF:10,130130211307_3.ts#EXTINF:10,130130211307_4.ts
#EXT-X-ENDLIST
#EXTM3U#EXT-X-MEDIA-SEQUENCE:1#EXT-X-TARGETDURATION:10#EXT-X-ALLOW-CACHE:YES#EXTINF:10,130130211307_1.ts#EXTINF:10,130130211307_2.ts#EXTINF:10,130130211307_3.ts
#EXTM3U#EXT-X-MEDIA-SEQUENCE:2#EXT-X-TARGETDURATION:10#EXT-X-ALLOW-CACHE:YES#EXTINF:10,130130211307_2.ts#EXTINF:10,130130211307_3.ts#EXTINF:10,130130211307_4.ts
MPEG - DASHSimilar to HLS, Adobe HDS, Microsoft Smooth Streaming but more flexible
International standard, XML based manifest with .mpd extension
Codec agnostic, HTML5 support
Supports multiple use cases like Ad insertion
More efficient than HLS
Used by Youtube, Netflix etc
Limitation of MediaPlayerSupport for RTSP/RTP/RTCP is good but support for HLS is flaky
Playback issues
No current playback time
No support for discontinuity
No support for ID3
Support across different versions inconsistent, new versions not necessarily better
No support for DASH, DRM etc
ExoplayerThe Android framework provides MediaPlayer as a quick solution for playing media with minimal code
MediaCodec and MediaExtractor classes are provided for building custom media players.
The open source project, ExoPlayer, is a solution between these two options, providing a pre-built player that you can extend.
ExoPlayer supports features not currently provided by MediaPlayer, including Dynamic adaptive streaming over HTTP (DASH), HLS, SmoothStreaming, DRM - Common Encryption, Widevine, Playready.
It's designed to be easy to customize and extend, allowing many components to be replaced with custom implementations.
ExoPlayer is a library that you include in your application, it can be easily updated along with your app.
https://developer.android.com/guide/topics/media/exoplayer.html
Thank [email protected]