Core Image

Core Image

Mark PavlidisCo-founderFlixel Photos Inc.

Toronto Cocoa and WebObjects Developers Group2013/01/08

Motivation Why use Core Image?

• Image processing and analysis

• Fast, efficient filters

• Auto enhancement

• Feature detection

• Still and “real-time” video

• Simple Objective-C API

Definitions• Image Filter

• a single transform or effect

• built-in or loaded from an Image Unit plugin (OS X only)

• Framework

<CoreImage/CoreImage.h>

• Key Classes

CIImage, CIFilter, CIContext

Overview

• Pixel Accurate

• Non-destructive

• Filters can be chained together

MonochromeFilter

Apply an image filter to the source pixel data

Core Image RuntimeHow it works

Core Image RuntimeHow it works

• Operates on image data types from:• Core Graphics• Core Video• Image I/O

Core Image RuntimeHow it works

• Built-in or plugin filters apply effects• Written in the Core Image Kernel Language• JIT complier assembles instruction pipeline

Core Image RuntimeHow it works

• Executed on the GPU or CPU• Low-level graphics processing encapsulated by CI• Performance: GLSL capabilities of the GPU or

processing power of the CPU

Core Image Inputs• Photo Library and Files

imageWithContentsOfURL:

• Live Video CaptureimageWithCVPixelBuffer:

imageWithCVImageBuffer:

• In MemoryimageWithCGImage:

•GL TextureimageWithTexture:size:flipped:colorSpace:

Core Image Outputs • CGImageRef

• Output to UIImage, ImageIO, ALAssetLibrarycreateCGImage:fromRect:

• CAEAGLLayerdrawImage:inRect:fromRect:

• CVPixelBufferRefrender:toCVPixelBuffer:

• Bitmaprender:toBitmap:rowBytes:bounds:format:colorSpace:

Simple Filter

// Create a CIImage CIImage *ciImage = [CIImage imageWithContentsOfURL:myURL];

// Create a CIFilterCIFilter *filter = [CIFilter

filterWithName:@”CIColorMonochrome”];[filter setValue:ciImage forKey:kCIInputImageKey];[filter setValue:@(0.5) forKey:@”inputIntensity];

// Render the filter output image into a UIImageUIImage *uiImage = [UIImage

imageWithCIImage:filter.outputImage];

Filter Chains// Filter 1CIImage *output = [CIFilter

filterWithName:@”CIColorMonochrome”] keysAndValues: kCIInputImageKey, ciImage, @”inputIntensity, @(0.5), nil].outputImage;

// Filter 2output = [CIFilter

filterWithName:@”CIVignette”] keysAndValues:kCIInputImageKey, output,@”inputIntensity, @(0.5), @”inputRadius, @(1.5), nil].outputImage;

•CI defers pixel processing until render is requested•CI optimizes the render graph (e.g., sepia then scale)

Math!• An image filter is kernel

•Output is the convolution of the source & kernel

• Commutative, associative, distributive, associative with scalar multiplication, multiplicative identity

1 1 1

-1 2 1

-1 -1 1

2 2 2

2 1 3

2 2 1

6

SourceKernel

Output

Demo - QuartzComposer

Demo - Simple filter

Tips and Best Practices• CIImage and CIFilter are autoreleased

• Use autorelease pools

• CIImage, CIContext are immutable and thread-safe, CIFilter is not

• CIContext is expensive

• Stores a lot of state information so reuse them

• Avoid CA animations when rendering on GPU

•Use smaller images when possible

• Use Core Animation to upscale view, texture, framebuffer

•Disable colour management

• If real-time performance and/or reduced quality is unnoticeable

Face Detection• Identifies rectangles that contain human faces

• Feature detection

• eyes & mouth position, tracking ID and frame count (video)

CIContext *context = [CIContext contextWithOptions:nil];

NSDictionary *opts = @{CIDetectorAccuracy, ! ! ! ! ! ! ! CIDetectorAccuracyHigh};

CIDetector *detector = [CIDetector ! ! ! ! ! ! detectorOfType:CIDetectorTypeFace! ! ! ! ! ! ! ! context:context ! ! ! ! ! ! ! ! options:opts];

NSArray *features = [detector featuresInImage:myImage ! ! ! ! ! ! options:opts];

CIDetector

Demo - Redacted Faces

OS X and iOS Differences• Some CIImage create methods differ

•Has more built-in filters

• Permits custom filter plugins

• filter.outputImage is iOS only, use [filter valueForKey:kCIOutputImage]

• Remember to [filter setDefaults]

• All input parameter keys have strings defined

Portability

Alternatives•GPUImage (iOS)

✓Hand-tuned OpenGL ES shaders that are much faster than CI

✓Custom filters

- Lacks advanced features of Core Image

• Aviary

✓ Everyone is using it (Twitter, Flickr)

- Everyone is using it

• Only still images

•Hardcore - Write your own GL shaders

• Don’t be silly

Summary

• Fast and efficient image processing framework

• Simple Objective-C API

• Encapsulates the low-level instruction details

•Many built-in filters

• Advanced processing features:

• Face Detection

• Auto Enhancement

• Real-time video filters

Core Image

More Information• Core Image Programming Guide

•WWDC 2012 Videos • Session 510 - Getting Started with Core Image

• Session 511 - Core Image Techniques

• Sample Code

• https://github.com/mhpavl/FilterDemo

• https://github.com/mhpavl/RedactedFaces

• Core Image Funhouse (TBD)

[self release];

Mark Pavlidis

[email protected]

Twitter: @mhp

ADN: @mhp