Composing and Editing Media with AV Foundation

http://bobmccune.com

Composing & Editing Media with

AV Foundation

Bob McCuneAbout...

‣MN Developer and Instructor‣Owner of TapHarmonic, LLC.‣Founded Minnesota CocoaHeads in 2008

What will I learn?Agenda

‣ AV Foundation Overview‣ Media Playback‣ Media Editing‣ Composing Media‣ Mixing Audio‣ Building Video Transitions‣ Layering Visual Content

OverviewAV Foundation

‣ Apple’s advanced Objective-C framework for working with timed-media‣ High performance, asynchronous processing‣ Hardware accelerated handling of AV media

‣ Available in its current form since iOS 4‣ Signi!cant additions and enhancements iOS 5 and 6‣ Default media framework on Mac OS X since 10.7 Lion

‣ Apple’s focus for media apps on both iOS and Mac

iOS Media OptionsWhere does it !t?

MediaPlayer

UIKit

Simple

CoreAudio

CoreMedia

CoreVideo

CoreAnimation

Complex

AVFoundation

Awesome

AV Essentials

Understanding AssetsMedia Assets

‣ AVAsset is an abstract representation of media resource modeling the static aspects of the media.‣ Abstracts away the type and location

‣ AVAssetTrack models the static aspects of the individual media streams within an asset‣ Tracks are of a uniform type (video, audio, etc.)

AVAssetTrack (Video)

AVAssetTrack (Audio)

What can I do with an asset?Using AVAssets

‣ Inspect‣ Generate Images‣ Transcode and Export‣ Playback

Media Playback

Playback ControllerAVPlayer

‣ AVPlayer is a controller for managing playback‣ play‣ pause‣ rate‣ seekToTime:

‣ Use KVO to observe playback readiness and state‣ status

‣ Timed Observations‣ addPeriodicTimeObserverForInterval:queue:usingBlock‣ addBoundaryTimeObserverForInterval:queue:usingBlock

Static

Static vs Dynamic ModelsPlaying Media

‣ AV Foundation distinguishes between static and dynamic aspects of media

D ynamic

AVPlayerItemAVPlayerItemTrack

AVPlayerItemTrackAVPlayerItemTrack

AVAssetAVAsset

AVAssetAVAssetTrack

Core Media EssentialsUnderstanding Time

CMTime‣ Rational number representing time‣ 64-bit integer time value (numerator)‣ 32-bit integer time scale (denominator)

CMTime fiveSeconds = CMTimeMake(5, 1);CMTime oneSample = CMTimeMake(1, 44100);CMTime zeroTime = kCMTimeZero;

‣ Large number of utility functions in Core Media:‣ CMTimeAdd,CMTimeSubtract,CMTimeCompare, etc.

Core Media EssentialsUnderstanding Time

CMTimeRange‣ Core Media struct containing start time and duration

‣ Like CMTime, there are many Core Media functions:‣ CMTimeRangeEqual, CMTimeRangeContainsTime, CMTimeRangeGetEnd, CMTIMERANGE_ISVALID, etc.

CMTimeRange zeroRange = kCMTimeRangeZero;CMTimeRange assetRange = CMTimeRangeMake(kCMTimeZero, asset.duration);

AVPlayerLayerVideo Playback

AVPlayerAVPlayerItem

AVPlayerItemTrackAVPlayerItemTrackAVPlayerItemTrack

AVAsset

AVAssetAVAssetAVAssetTrack

AVPlayerLayerVideo Playback

AVPlayerAVPlayerItem

AVPlayerItemTrackAVPlayerItemTrackAVPlayerItemTrack

AVPlayerLayer

Demo

Composing Media

AVCompositionComposing Assets

‣Concrete extension of AVAsset‣Composes asset segments on a timeline

Tracks and SegmentsComposing Assets

AVComposition

AVMutableComposition *composition = [AVMutableComposition composition];

AVComposition

Tracks and SegmentsComposing Assets

CMPersistentTrackID trackID = kCMPersistentTrackID_Invalid;

AVMutableCompositionTrack *videoTrack = [composition addMutableTrackWithMediaType:AVMediaTypeVideo preferredTrackID:trackID];

AVMutableCompositionTrack *audioTrack = [composition addMutableTrackWithMediaType:AVMediaTypeAudio preferredTrackID:trackID];

AVCompositionTrack (Video)

AVCompositionTrack (Audio)

AVCompositionTrack (Video)

AVCompositionTrack (Audio)

AVComposition

Tracks and SegmentsComposing Assets

AVCompositionTrack (Video)

AVCompositionTrack (Audio)

AVCompositionTrackSegment

Seconds 10-30 of “redpanda.m4v”

AVCompositionTrackSegment

Seconds 20-60 of “waves.m4v”

AVCompositionTrackSegment

Seconds 0-60 of “soundtrack.mp3”

AVAssetTrack *srcVideoTrack1 = // source video track 1[videoTrack insertTimeRange:timeRange ofTrack:srcVideoTrack1 atTime:startTime error:&error];

AVAssetTrack *srcVideoTrack2 = // source video track 2[videoTrack insertTimeRange:timeRange ofTrack:srcVideoTrack2 atTime:startTime error:&error];

AVAssetTrack *srcAudioTrack = // source audio track[audioTrack insertTimeRange:timeRange ofTrack:srcAudioTrack atTime:startTime error:&error];

Demo

Mixing Audio

AVAudioMixAudio Mixing

‣ Composition tracks play at their natural volume‣ AVAudioMix applies track-level volume adjustments

‣ Composed of AVAudioMixInputParameters‣ Parameters control individual track volume over time

CMTime CMTimeRange

CMTime fadeTime = CMTimeMake(5, 1);CMTime fadeInStartTime = kCMTimeZero;CMTime fadeOutStartTime = CMTimeSubtract(asset.duration, fadeTime);

CMTimeRange fadeInRange = CMTimeRangeMake(fadeInStartTime, fadeTime);CMTimeRange fadeOutRange = CMTimeRangeMake(fadeOutStartTime, fadeTime);

AVMutableAudioMixInputParameters *parameters = [AVMutableAudioMixInputParameters audioMixInputParameters];parameters.trackID = assetTrack.trackID;[parameters setVolumeRampFromStartVolume:0.0 toEndVolume:1.0 timeRange:fadeInRange];[parameters setVolumeRampFromStartVolume:1.0 toEndVolume:0.0 timeRange:fadeOutRange];

AVMutableAudioMix *audioMix = [AVMutableAudioMix audioMix];audioMix.inputParameters = @[parameters];playerItem.audioMix = audioMix;

ExampleAVAudioMix

Demo

Video Transitions

AVVideoCompositionVideo Transitions

AVVideoCompositionDefines how two or more video tracks are composited together

Configured through collection of composition instructions describing compositing behavior

AVVideoCompositionInstructionAVVideoCompositionInstructionAVVideoCompositionInstruction

AVVideoCompositionInstructionVideo Transitions

AVVideoComposition

AVVideoCompositionInstructionAVVideoCompositionInstructionAVVideoCompositionInstructionDefines the time range of compositing behavior

Composed of layer instructions describing compositing behavior

AVAssetAVAssetAVVideoCompositionLayerInstruction

AVVideoCompositionLayerInstructionVideo Transitions

AVVideoComposition

AVAssetAVAssetAVVideoCompositionLayerInstruction

AVVideoCompositionInstructionAVVideoCompositionInstructionAVVideoCompositionInstruction

Defines the transform and opacity ramps of input layers

Transform and opacity changes modified over given time range

Conceptual StepsBuilding Transitions

Stagger LayoutBuilding Transitions 1

A

B

De!ne Overlapping RegionsBuilding Transitions 2

A

B

De!ne Time RangesBuilding Transitions

A

B

Transition

Passthrough

Transition

Passthrough Passthrough

* Time ranges must not have gaps or overlap* Total duration must not be shorter than composition

3

Con!gure InstructionsBuilding Transitions 4

// Build transition instructionsAVMutableVideoCompositionInstruction *transitionInstruction = ...;transitionInstruction.timeRange = transitionTimeRange;

AVMutableVideoCompositionLayerInstruction *fromLayerInstruction = ...;AVMutableVideoCompositionLayerInstruction *toLayerInstruction = ...;

// Cross Disolve[fromLayerInstruction setOpacityRampFromStartOpacity:1.0 toEndOpacity:0.0 timeRange:transitionTimeRange];

NSArray *instructions = @[fromLayerInstruction, toLayerInstruction];transitionInstruction.layerInstructions = instructions;[instructions addObject:transitionInstruction];

Set sizes and applyBuilding Transitions 5

AVMutableVideoComposition *videoComposition = [AVMutableVideoComposition videoComposition];

// Set instructions on AVVideoComposition instancevideoComposition.instructions = instructions;videoComposition.frameDuration = FRAME_RATE;videoComposition.renderSize = RENDER_SIZE;

AVPlayerItem *playerItem = [AVPlayerItem playerItemWithAsset:[composition copy]];playerItem.videoComposition = videoComposition;

#define FRAME_RATE CMTimeMake(1, 30)#define RENDER_SIZE CGSizeMake(1280, 720)

videoCompositionWithPropertiesOfAsset:

New in iOS 6AVVideoComposition

• Calculates all required passthrough and transition time ranges

• Builds appropriate composition and layer instructions for time ranges

• Sets the appropriate render size

• Sets the appropriate frame rate

Automagic Setup:

Demo

Layering Content

Core AnimationLayering Content

Core Animation a natural choice‣ High performance, inherently time-based‣ CALayer subclasses used for all video rendering

CALayer: used to layer images and text

CAAnimation: used to animate layered content

CABasicAnimation

CAKeyframeAnimation

AVSynchronizedLayerAnimation Timing

‣ Core Animation operates on host time‣ Starts at boot, marches towards in!nity

‣ Timeline animations need to use movie time‣ Starts at kCMTimeZero and runs to duration‣ Can be started, stopped, rewound, etc.

‣ Use AVSynchronizedLayer to use movie time‣ Confers player item timing on to its sublayer tree

AVSynchronizedLayer

CATextLayer CABasicAnimation

AVPlayerItem

Timeline vs Realtime AnimationsCore Animation

‣ Exactly the same, but different:‣ Animations with zero beginTime won’t be seen‣ Set beginTime = AVCoreAnimationBeginTimeZero

‣ Animations removed by default‣ Set removedOnCompletion = NO‣ Unable to use CAAnimationGroup?

Natural Choice, Awkward ImplementationCore Animation

‣ Different conceptual models for timeline editing‣ CMTime and CMTimeRange for asset items‣ Seconds and milliseconds for layers‣ Build abstraction to help bridge the gap

‣ Usage differs in playback and export scenarios‣ AVSynchronizedLayer for playback‣ Attach to player’s view hierarchy/layer tree

‣ AVVideoCompositionCoreAnimationTool for export

Demo

AV Foundation Rocks!Summary

‣ Powerful tools for audio and video playback‣ AVPlayer, AVPlayerItem, AVPlayerLayer

‣ Powerful tools for composing/editing media:‣ AVComposition, AVAudioMix, AVVideoComposition, AVSynchronizedLayer

‣ Powerful utility classes:‣ AVAssetImageGenerator‣ AVExportSession

‣ Steep learning curve, but worth the investment!

ResourcesPresentation Materialshttp://www.speakerdeck.com/bobmccune/http://www.slideshare.net/bobmccune/https://github.com/tapharmonic/AVFoundationEditor

Learning AV Foundationhttp://www.speakerdeck.com/bobmccune/https://github.com/tapharmonic/AVFoundationDemos

WWDC 2011: Exploring AV Foundationhttps://developer.apple.com/videos/wwdc/2011/?id=405

WWDC 2011: Working with Media in AV Foundationhttps://developer.apple.com/videos/wwdc/2011/?id=415

BobMcCune.com @bobmccune