Smart image and video search presentation

Smart Image And Video SearchEnterprise Image and Video Analytics

The problem

• Government has piles of video, analysts miss things after looking at it for 20 minutes !

• What to do?

Help!

We are collecting video and images EVERYWHERE !!!!!!!!

How to search image/video for what you are looking for

identify people suspected IED sites

changes in geospatial data

geographic features

IED circuit boardsyellow ships

trains

planesmini vans

known terrorists

activity …

The Enterprise Solution• Only piXserve by piXlogic offers a generalized enterprise

class image/video search capability• Enterprise class 64bit Java application for Windows Server

2003+• Indexes all objects in images and frames of video and also

attempts to classify those objects• Supports 90+ image or video formats• Generates a light weight XML index and leaves the original

information where it lives and can place index in RDBMS• Special focus on text in image and facial recognition• Offers a JMS based event and alerting system• Supports a web services REST API

piXlogic A New Approach

Background

Foreground

Connected Background

Face

Tree

Sky

“Seeing”

Synthesize descriptions

Discern shapes

Vectorize shapes

LocationSizeColorGeometryRelationshipsEtc.

“Comparing” Direct ComparisonsRelationshipsContext

“Understanding”

Think About Text Indexing

By three methods we may learn wisdom: First, by reflection, which is noblest; Second, by imitation, which is easiest; and third by experience, which is the bitterest. Confucius

At their most Basic they Index Text ….

By three methods we may learn wisdom….

Confucius

<Keywords>

Enter Query:

Search

Common Text Indexing Engines:

<Phrases>“three methods”“learn wisdom”“by imitation”“by experience”

</Keywords>

</Phrases>

Its All About Metadata

Text Result Set

Rank Date Description . 98% 01/01/2011 Confucius dreams .. 87% 04/12/2010 Eastern philosophy and culture84% 07/23/2009 Harvard study on ancient China 73% 02/12/2009 Fortune cookies run amok65% 03/01/2008 Medical studies calm fears in …58% 01/01/2007 Confucius dreams .. 47% 04/12/2007 Eastern philosophy and culture44% 07/23/2004 Harvard study on ancient China 43% 02/12/2000 Fortune cookies run amok35% 03/01/1999 Medical studies calm fears in …

Image/Full Motion Video Indexing

Foreground

Background

Background

WE LOOK FOR SHAPES AT THE PIXEL LEVEL

How Pixel Indexing WorksEnterprise Pixel Indexing Engine:

Indexes Key Objects …

…

Key Objects

people" id="61" <ImageIndex ImageURL="C:\Data\Sample\quebec2008-1.jpg" ImageDate=" 2 6 11016" ImageSize="57489323"><ImageText Name="1" Sz="4" X="66" Y="1" W="2" H="2" L="0" U="0" V="0" R="400852085d086608" FC="41002e002e005600" FL="2e002e002e007400" FN="2e00310031003400" <ImageText Name="2" Sz="4" X="67" Y="1" W="2" H="2" L="0" U="0" V="0" R="7c08890891089808" FC="52002e0055002e00" FL="2e0074002e002e00" FN="2e0031002e003100" <ImageText Name="3" Sz="3" X="11" Y="2" W="4" H="3" L="0" U="0" V="0" R="65018801a401" FC="48004c005a00" FL="6e002e007a00" FN="2e0031003200" …

Synthetic Metadata

Enter Query:

Search

Query By ImageEnter Text

Date Result 01/01/2011

04/12/2010

07/23/2009

02/12/2009

03/01/2008

01/01/2007

08/14/2005

Image/Video Result Set

04/14/2009

How Do We Search

• Select a database or databases to search• Select search term type– Image only– Keyword– Text in image– Text and image– Coordinates– Faces

How Does It Work

Segment Images

…

Break up images into smaller and smaller pieces

Fill In Missing Information• Uses an ontology called Notions:– Apriori information about the world– Attempts to classify objects in an image or frame

of video.– Reasons using the notions to identify objects– System has over 70 notions

PeopleTree

Helicopter

Beach

Sky

Auto-Tagging in piXserve• Add as much meaningful metadata as possible at the

time of indexing (“Auto Tagging”)• Several mechanisms:– “Name Faces”: name a person and autotag when the

person is seen in the image/video frame– “Keyword Recommendations”: add keywords based on

how similar images have been tagged in the past– “Objects-2D”: autotag for a specific item based on a

single example (can use multiple views for 3D objects) – “Notions”: autotag generic classes of items (conceptual

tagging)

Advanced Capabilities

• Text in image or video– Not OCR– Uses the same techniques as we use to find any

object in image or video frames– Tuned for languages – English, European, Some

Chinese, Japanese, Korean, Russian• Faces (biometric)– No pre-registration – just uses images– Very high precision and recall using the NIST standard

test >98%

2D Objects Library

• Topic specific filtering– Specific library can be built to filter for specific

information – as images/frames of video arrive they are

compared to this information and specific tagging / matching occurs

– E.g. light arms, ships, clothing, hair styles etc.

Text In ImageEnter Query:

Search

Query By Text in Image

HEALTH

Enter Text

RESTful API

• All Server features & functions • All User interface functions• Data access•Machine to machine integration• API Guide

Alerting Engine

• Create standing queries• When an event in an image triggers an alert

two things happen:– A JMS message is sent – to whatever program or

system designated– An email message is sent to whomever has been

designated to receive the alert

END

gpepus@flexanalytics.com(301) 787-2989www.pixlogic.com

SUPPORTING MATERIALS

The Industry

• Video monitoring software packages integrated with DVR support such as Object Video, Image IQ, 3VR, Virage and others can recognize certain objects, perimeters, and activities ..

• They are security focused .. Often with some ability to search faces and possibly license tags

The Industry (Cont’d)• Other packages (COTS & GOTS) are starting to address generalized image and

video search• Numenta – offers an image search/pattern matching toolkit (not enterprise .. and

each solution must be hand crafted)• Pixsy – offers image search to match images – for internet advertising and

shopping online• IDEE Tin-Eye – offers image on the internet – more advertising based approaches• KEVA (Sierra Nevada Corp/PNNL) is a GOTS parallel processing engine for

searching video – government only and fairly early stage• Google and Facebook – Both have image and video tagging algorithms in their

general search and/or analytics suites, with some specific object search capabilities – but web focused only

• There are others .. But NONE are enterprise class generalized Google like search engines

• NERVVE

Notions/Classesartificial" id="1" white" id="2" flower" id="3" sky" id="4" gray" id="5" vegetation" id="6" lawn" id="7" skin" id="8" road" id="9" vertical" id="10" landscape" id="11" red" id="12" green" id="13" blue" id="14" aqua" id="15" yellow" id="16" gun" id="65"

smooth" id="19" circle" id="20" polygonal" id="21" sparse" id="22" face" id="23" horizontal" id="24" document" id="25" map" id="26" diagram" id="27" text" id="28" bikini" id="29" exposed body" id="30" line" id="31" adult" id="32" mountain" id="60" people" id="61" fire" id="62"

rectangle" id="37" pentagon" id="38" clutter" id="39" star" id="40" zebra" id="41" silver" id="42" pumpkin" id="43" bride" id="44" sea" id="45" snow" id="46" sand" id="47" beach" id="48" fireworks" id="49" sunset" id="50" soccer ball" id="59" airplane" id="66"

purple" id="17" orange" id="18" child" id="33" drawing" id="34" flat" id="35" triangle" id="36"flash" id="51" indoor" id="52" outdoor" id="53" near" id="54" dark" id="55" helicopter" id="56" building" id="57" car" id="58" crowd" id="63" 3D drawing" id="64"

Segment the Image or Video Frame

Indexing breaks the image up into smaller and smaller pieceswhich are more easily mathematically described

Medical Uses

Ductal CarcinomaPerspective 1

Ductal CarcinomaPerspective 2

Medical Uses

Frontal Lobe InjuryPerspective 1

Frontal Lobe InjuryPerspective 2

Physical CONOP

video/imagesource

continuousstreams or files

contentStore

db

Store inDAM system

indexingservers

direct indexing from source

indirect indexingfrom DAM system

Web Server

Web Server

Web Server

analyst

analyst

analyst

analyst

analystnetwork

Index held in memory

searching

downstreamanalytical systems

analysis

Alert Server

application

faces

Logical CONOPS 1Watching The Border

continuousvideo/image

source

automatic continuous

indexing

indexal

erts

reporting

metadata• name• watch list• vehicle make• vehicle model• tag state• tag number

analysts

downstreamanalyticalsystems

correlationFilter

indexes

vehicles

license plates

Filter indexesFilter

indexes

Conceptual CONOPS 1Watching The Border

vehicle border crossing

cameras

Tag - state & text extracted Vehicle - make/model extractedFace - biometrics

vehicle

tag face

Indexes-watch list-vehicle types-state license plates

-State: Florida-Tag: SAMPLE

-Daniel San Diego-U.S. Most Wanted

-Make: Honda-Model: Civic

NET EMULATED UAV

use video filespiXlogic index

video and searches

piXlogic matches and

clips video on target

event

piXlogic apitransfers clip

to flumedirectory

flume transmits matches over

MITRE network emulator

1

activity

2 3 4 5

RADAR INFRARED VIDEO VIDEO IMAGERY

Query:• Buildings• Ships• Aircraft

TNT - CONOPS

Alert Results

network

pixserve

flume

net emulator

commsLo

cal N

etw

ork

FIELD ANALYSTS