T OPIC 5: L EVEL 0 David L. Hall. T OPIC O BJECTIVES Introduce concepts of level 0 processing Identify categories of techniques for level 0 processing.

TOPIC 5: LEVEL 0

David L. Hall

TOPIC OBJECTIVES

Introduce concepts of level 0 processing Identify categories of techniques for level 0

processing (e.g., signal and image processing) Extend the concept of traditional signal and

image processing and conditioning to meta-data generation

LEVEL 0 PROCESSING

COMMENTS ON LEVEL 0 PROCESSING

Level 0 processing concerns processing each source of data independently to obtain the most useful information possible; we want to “squeeze out” useful information

In general, input data to a fusion system may include scalar or vector data, signal data, image data, or textual information. Each of these classes of information may entail an entire range of potentially applicable techniques (e.g., signal processing, image processing, text-based processing)

It is beyond the scope of this lecture (and indeed this entire course) to address these subjects – however, we seek to make the student aware of the need to consider such processing

Finally, an emerging new area is the generation of semantic meta-data (e.g., automatic semantic labeling of images) as a new powerful process to improve data fusion.

METASENSOR PROCESSING

METASENSOR PROCESSING

CA

TEG

OR

YFU

NC

TIO

N

TEC

HN

I QU

E

Sensor Validation

and Calibration

Parametric

Modeling

• Spectral analysis• Wavelet processing• Short-time Fourier transform• AR modeling• System I/D analysis

Entity

Detection

• Array processing - Synthetic aperture (SA) - Inverse SA• Moving target indicator (MTI) • Hypothesis testing• Pre-detection fusion• Distributed fusion

• Phase and gain calibration• Sensor data validation

Non-Parametric

Analysis

• Time-series analysis• Dynamical systems analysis• Image analysis• Neural networks• Trend analysis• Thresholding and integration• Energy detection

LEVEL 0 PROCESSING BY SOURCE TYPE

Scalar orVectorSensor

Signalsensor

ImagingSensor

Text-based information

yt

Semantic processing

Signal processing

Image processing

Data Conditioning

Original Source Data plus transformed & added meta-data

SOURCE TYPE: SCALAR & VECTOR

DATA Representative techniques

Scaling methods and transformations Coordinate transformations and rotations Smoothing, filtering, averaging Thresholding Feature extraction and representations

y y + z Transformation

Original data Original data Alternate representations

SOURCE TYPE: SIGNAL DATA Representative Parametric

Modeling Techniques Spectral analysis Wavelet processing Short-time Fourier transform Auto-Regression modeling System Identification analysis

Representative Non-Parametric Modeling Techniques

Time-series analysis Dynamical systems analysis Neural networks Trend analysis Threshold & integration Energy detection

Transform + characterization vector, y

Original signalTransformed signal

EXAMPLES OF SIGNAL PROCESSING

Signal processing and enhancement are standard on most radios & receivers High & low frequency filtering Boost of selected frequency ranges “Coloring” of signals

Active Noise Cancelation devices Active cancellation of background

noise (e.g., airline engine noise)

Extensive commercial tools are available such as MATLAB, Mathematica, etc

SOURCE TYPE: IMAGE DATA Classification Feature Extraction Pattern Recognition Projection Multi-Scale Signal Analysis

Principal Components Analysis Independent Component Analysis Self-Organizing Maps Hidden Markov Models Neural Nets

+ Meta-Data

Original Image Transformed image

ImageProcessing

An enormous amount of methods exist to process and transform image data from the individual pixel level, to object level to complete image level. Commercial tools such as Photoshop Pro make many of these available to casual users

EXAMPLES OF IMAGE PROCESSING

http://web.uct.ac.za/depts/physics/laser/hanbury/intro_ip.html

Contrast enhancement

Image sharpening using wavelets

http://www.phasespace.com.au/wavelet_ex.htm

Deconvolution

http://www.phasespace.com.au/wavelet_ex.htm

SOURCE TYPE: TEXT DATA

Representative techniques Key word extraction Name disambiguation Link of semantics to parametric data (e.g. location) Syntactic processing Thesaurus processing Semantic distance calculations Links to other documents/

Textual input via human reports or web info.

Textual input via human reports or web info.

Meta Data - key words - lexicon data - identified parameters - links to related documents

+TextualProcessing

CONCEPT OF SEMANTIC META DATA GENERATION

Signal & images represent a data level view of patterns, features & characteristics

In human pattern recognition, we identify labels to represent signals & data

New techniques are being developed to train computers to perform a similar function

PROBLEM: HOW TO EFFECTIVELY QUERY ON A VERY LARGE COLLECTION OF SATELLITE IMAGERY FOR

SEMANTICALLY MEANINGFUL REGIONS ?

Use semantic categorization combined with CBIR on small “patches” of the images (Dr. James Wang)

QUERY FORMULATION

Query “patch” – pertaining to some semantics, e.g. mountains

Satellite Image Database Ranked

ResultsPurpose ?

Geography - Find mountainous regions with snow-caps (low-level semantics).

Forestry – Find forests of a certain density, analyze deforestation (mid-level semantics).

Military – Find air-bases in certain regions of the world (high-level semantics).

LIMITATIONS OF CURRENT APPROACHES

Difficulty in defining classes Continuum of variety Subjectivity

No major improvement in classification accuracy G. G. Wilkinson, “Results and Implications of a Study of Fifteen Years of

Satellite Image Classification Experiments,” IEEE Trans. On Geoscience and Remote Sensing, 2005.

Recognition of higher-order semantics More focus on spectral than spatial dimensions

Scalability of complex querying and browsing

Flexibility to handle diverse applications

AUTOMATIC SEMANTIC CATEGORIZATION

SUMMARY OF APPROACH

Practical implementation on large-scale archives

Flexibility to adapt to various applications of satellite imagery, e.g., Geography, Military, Metallurgy, Agriculture etc.

Exploiting spectral and spatial information using a generative model for semantic categorization (supervised learning)

Handling of untrained classes (supervised learning)

Using a scalable CBIR system for efficient querying and browsing (unsupervised)

OVERVIEW OF THE SYSTEM

Classification Generative Classifier: Two-dimensional Multi-resolution Hidden Markov Models

(2-D MHMM) Handling untrained classes

Discriminative Classifier: Support Vector Machines Querying for fast retrieval

Integrated Region Matching (IRM) measure used in SIMPLIcity system

K trained 2D-MHMMs

40 x K training patches

Query patch

Patch Class

Random samples from trained and untrained classes

K Likelihood scores Trained

biased SVM

Database search using IRM

Query Results

DETAILS OF THE ARCHITECTURE

Efficient database design

Pre-computation of classes using 2D-MHMM for faster response

Suitable user interface and visualization (Common CBIR issue)

TRAINING PROCESS

AUTOMATIC SEMANTIC CATEGORIZATION

RETRIEVAL RESULTS

http://riemann.ist.psu.edu/image

RETRIEVAL RESULTS

http://riemann.ist.psu.edu/image

TOPIC 5 ASSIGNMENTS

Preview the on-line topic 5 materials Read Farid paper (2008)

DATA FUSION TIP OF THE WEEK

Level 0 processing is a vital part of the data fusion process - in essence pre-processing or conditioning data from individual sources and sensors. This kicks off all subsequent processing. It is necessary to do the best job possible at this stage without either introducing spurious information into the observed data, nor failing to extract and enhance the data “at the source”. It is not possible to use “down-stream” data fusion processing to make up for failures at the level 0 stage.