OpenIMAJ and ImageTerrier: Java Libraries and Tools for Scalable Multimedia Analysis and Indexing of Images

ImageTerrierwww.imageterrier.org

OpenIMAJIntelligent Multimedia Analysis In Java

www.openimaj.org

OpenIMAJ and ImageTerrierJava Libraries and Tools for Scalable Multimedia

Analysis and Indexing of Images

Jonathon Hare, Sina Samangooei & David Dupplaw The University of Southampton

What is OpenIMAJ?

Scalable Clustering

Scalable KNN

OpenIMAJIntelligent Multimedia Analysis In Java

Image

Analysis

LibrariesCommand-line Tools

Hadoop Map-Reduce Implementations

HardwareMachine Learning

Video

Web Audio

Analysis

AnalysisProcessing

ProcessingProcessing

Analysis Rendering

Kinect Audio & Video

CaptureExotic

Hardware

LibrariesCommand-line Tools

Hadoop Map-Reduce Implementations

Efficient, Compressed

Inverted Index

Query Engine

Term Metadata

IntegrationBased on

Terrier Constructable from Images

Distance Metrics

scaleRe-Ranking affine

orientation

Compressed

Extensible

OpenIMAJIntelligent Multimedia Analysis In Java

Scalable

e.g SIFT BoVW (x, y, scale, etc)

What is ImageTerrier?Efficient inverted index construction for image/video features (and potentially audio features too)

Augmented inverted index, capable of storing almost any kind of meta information per term

Easily extensible

Provided implementations contain things like (x,y,scale) for SIFT BoWV

Indexing is scalable across machines using Hadoop

Complete OpenIMAJ integration

Included tools can build an index straight from a set of images using common techniques like SIFT BoVW or read features extracted from other tools.

This can even be done on Hadoop!

Easy to define searching and re-ranking strategies

Various distances, weightings and constraints (such as Affine & Homographic) are provided

Tested with corpuses in excess of 10M images

A Little history...IAMMediaEngine

c. 2005

MapSnapc. 2006

JCVMLTKc. 2009

OpenIMAJ & ImageTerrier

May 2011

MoVEc. 2010

Pure-Java SIFT & Matching

Extensible Java (& JNI) Content-

Based Retrieval System

Refactoring of older code, addition of face detection

and machine learning components

Initial attempts at scalable BoWV indexing using terrier,

shape analysis, etcMany new features added.

Improvements to ImageTerrier, face pipeline, demos, video capture and

hardware, ... PhotoCopainc. 2005

Image classifiers

SMS-NLPc. 2010

Machine learning, clustering

Massive code cleanup and documentation effort. First

Open Source Release

It’s pure Java... (with the exception of thin interfaces for hardware)

Isn’t it slow?

Why another set of libraries?Scalability + Portability. OpenIMAJ projects work on:

Android mobile devices

Hadoop clusters

...without recompilation!

Easy to use and extend

Implement many important Multimedia algorithms

While allowing easy adaption and extensions.

Software engineering principles adopted

clear separations of concerns

maintainable, understandable

We simplify data representations:Images are 2D-arrays of pixels;

Videos are iterable streams of Images

Audio is an array of samples

Both OpenIMAJ and ImageTerrier have been designed from the ground-up to be easily extensible.

We’ve made it easy to modify and understand algorithms

and techniques by separating different components.

Dominant(orienta+on(of(an(orienta+on(histograms(

Replace'with'a'Null'orienta1on'extractor.'

Regular(SIFT(feature(provider(

Replaced(with(Irregular(Binning(

Rapid Application Development

“Make it easy to use”

This philosophy lets researchers build complex applications quickly

Example: Southampton Goggles

synchronised: 6 cameras, GPS device, compass

feature extraction and index construction

mobile integration

…all programmed over 2 days!

Find out more at: http://bit.ly/SOTONGOG

Student projects

Our students use OpenIMAJ (…for 2 years now!)

OpenIMAJ lets them:

Get something basic working quickly

Understand the algorithms they are using

Implement new algorithms in a structured way

Contribute to a living, open source project

To support our students we wrote a comprehensive OpenIMAJ tutorial.

Research applications

Through combination of OpenIMAJ and ImageTerrier various multimedia research applications can be addressed

- We have done so for over 6 years!

1)Guess The Building - part of the LiveMemories project

Estimate geolocation and semantic information for an un-annotated image of a landmark

2)Stock Photo Finder - part of the Living Knowledge project

Find the source of a web image in a stock photo database

Any Questions?Try the OpenIMAJ tutorial for yourself:

http://openimaj.org/tutorial.pdf