Army Research Office Multidisciplinary University Research Initiative Unified Research on Network-based Hard/Soft Information Fusion Third Annual Review Period August 1, 2011 thru July 31, 2012 Drs. Rakesh Nagi (PI), Moises Sudit (co-PI), James Llinas (Emeritus PI) Center for Multisource Information Fusion State University of New York at Buffalo Buffalo, New York, USA {nagi, sudit, llinas}@buffalo.edu Dr. John Lavery, ARO PM Army Research Office Research Triangle Park, NC 27709 [email protected]
42
Embed
Army Research Office Multidisciplinary University Research ...
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Army Research Office Multidisciplinary University Research Initiative
Unified Research on Network-based Hard/Soft Information Fusion
Third Annual Review Period August 1, 2011 thru July 31, 2012
“The attacks of September 11th 2001 killed 2,996 people. Despite the subsequent declaration of a war on terror, over the past ten years thousands more have been killed by terrorists of all hues.”
Some Distinctions in Hard and Soft Observational Data
Totally distinct from Hard Sensors Philosophy: Relations not directly
observable—require reasoning over properties of entities*
* Brower, J., (2001) "Relations without Polyadic Properties: Albert the Great on the Nature and Ontological Status of Relations." Archiv für Geschichte der Philosophie 83: 225–57.
Humans can also judge intangibles --emotional state
• Hard-Soft Fusion Architecture (what/when to fuse?) – Data-level Fusion (Raw Fusion) – Feature-level Fusion (Measurement or Early Fusion) – Estimate-level Fusion (Late Fusion)
• Nature of Soft Information – Symbolically expressed (symbolic ambiguity) – Human observations have bias and (seldom characterized) uncertainty
• Uncertainty representation and transformation
– Results of cognitive fusion are unavailable (feature to symbol) – Natural language processing difficulty
• Contextual Information – Observational gap-filling, interpretation
• Association of Hard and Soft Information – Symbolic versus Numerical (Baghdad vs. 33°20’ N, 44°26’ E) – Asynchronous (out of sequence) information arrival
• Processing of Hard or Soft Information – How does the one influence the parameters, biases or algorithms for the processing of the other
• Lack of Datasets for Technology Development and Verification • Traditional Challenges in Hard Fusion • High-level Fusion (Situation Understanding) • Concepts of Employment (and Transition) 9
• Hard-Soft Fusion Architecture (what/when to fuse?) [T1] – Feature-level Fusion (Measurement or Early Fusion) – Estimate-level Fusion (Late Fusion)
• Nature of Soft Information – Symbolically expressed (symbolic ambiguity) – Human observations have bias and (seldom characterized) uncertainty [T5]
• Uncertainty representation and transformation [T4]
– Results of cognitive fusion are unavailable (feature to symbol) – Natural language processing difficulty [T2]
• Contextual Information – Observational gap-filling, interpretation [T3]
• Association of Hard and Soft Information [T8] – Symbolic versus Numerical (Baghdad vs. 33°20’ N, 44°26’ E) [T7] – Asynchronous (out of sequence) information arrival [T6]
• Processing of Hard or Soft Information [T10] – How does the one influence the parameters, biases or algorithms for the processing of the
other
• Lack of Datasets for Technology Development and Verification [T11] • Traditional Challenges in Hard Fusion [T12] • High-level Fusion (Situation Understanding) [T9] • Concepts of Employment (and Transition) [T13]
Soft Messages • Developed SYNCOIN, including interlaced scenarios,
600 text messages and synthetic hard data
Hard Sensor Data Collect
Combined Hard and Soft • Realistic COIN situations • Associable (entities, locations, etc.) • Complementary • Insightful and non-trivial •Non-linear (intertwined, out-of-sequence data) • Confounding elements (background noise) • Linguistic and human observational uncertainty
• Hard-Soft Fusion Architecture (what/when to fuse?) [T1] – Feature-level Fusion (Measurement or Early Fusion) – Estimate-level Fusion (Late Fusion)
• Nature of Soft Information – Symbolically expressed (symbolic ambiguity) – Human observations have bias and (seldom characterized) uncertainty [T5]
• Uncertainty representation and transformation [T4]
– Results of cognitive fusion are unavailable (feature to symbol) – Natural language processing difficulty [T2]
• Contextual Information – Observational gap-filling, interpretation [T3]
• Association of Hard and Soft Information [T8] – Symbolic versus Numerical (Baghdad vs. 33°20’ N, 44°26’ E) [T7] – Asynchronous (out of sequence) information arrival [T6]
• Processing of Hard or Soft Information [T10] – How does the one influence the parameters, biases or algorithms for the processing of the
other
• Lack of Datasets for Technology Development and Verification [T11] • Traditional Challenges in Hard Fusion [T12] • High-level Fusion (Situation Understanding) [T9] • Concepts of Employment (and Transition) [T13]
• Hard-Soft Fusion Architecture (what/when to fuse?) [T1] – Feature-level Fusion (Measurement or Early Fusion) – Estimate-level Fusion (Late Fusion)
• Nature of Soft Information – Symbolically expressed (symbolic ambiguity) – Human observations have bias and (seldom characterized) uncertainty [T5]
• Uncertainty representation and transformation [T4]
– Results of cognitive fusion are unavailable (feature to symbol) – Natural language processing difficulty [T2]
• Contextual Information – Observational gap-filling, interpretation [T3]
• Association of Hard and Soft Information [T8] – Symbolic versus Numerical (Baghdad vs. 33°20’ N, 44°26’ E) [T7] – Asynchronous (out of sequence) information arrival [T6]
• Processing of Hard or Soft Information [T10] – How does the one influence the parameters, biases or algorithms for the processing of the
other
• Lack of Datasets for Technology Development and Verification [T11] • Traditional Challenges in Hard Fusion [T12] • High-level Fusion (Situation Understanding) [T9] • Concepts of Employment (and Transition) [T13]
Objectives: • Support development of hard/soft information fusion • Develop methods for the aggregation of uncertain information • Provide formalisms for the representation and modeling of soft information
DoD Benefit: • Better use of available information
Scientific/Technical Approach
• Fuzzy Set Theory
• Monotonic Set Measure
• Dempster Shafer Theory
• Mathematical theory of aggregation
• Computing with Words
Accomplishments • Set measure model of fusion
• Role of expert instructions
• Nature of Soft information
• Modeling different uncertainty modes
Challenges • Mixed uncertainty mode fusion • Complexity of Soft information
• Hard-Soft Fusion Architecture (what/when to fuse?) [T1] – Feature-level Fusion (Measurement or Early Fusion) – Estimate-level Fusion (Late Fusion)
• Nature of Soft Information – Symbolically expressed (symbolic ambiguity) – Human observations have bias and (seldom characterized) uncertainty [T5]
• Uncertainty representation and transformation [T4]
– Results of cognitive fusion are unavailable (feature to symbol) – Natural language processing difficulty [T2]
• Contextual Information – Observational gap-filling, interpretation [T3]
• Association of Hard and Soft Information [T8] – Symbolic versus Numerical (Baghdad vs. 33°20’ N, 44°26’ E) [T7] – Asynchronous (out of sequence) information arrival [T6]
• Processing of Hard or Soft Information [T10] – How does the one influence the parameters, biases or algorithms for the processing of the
other
• Lack of Datasets for Technology Development and Verification [T11] • Traditional Challenges in Hard Fusion [T12] • High-level Fusion (Situation Understanding) [T9] • Concepts of Employment (and Transition) [T13]
• Hard-Soft Fusion Architecture (what/when to fuse?) [T1] – Feature-level Fusion (Measurement or Early Fusion) – Estimate-level Fusion (Late Fusion)
• Nature of Soft Information – Symbolically expressed (symbolic ambiguity) – Human observations have bias and (seldom characterized) uncertainty [T5]
• Uncertainty representation and transformation [T4]
– Results of cognitive fusion are unavailable (feature to symbol) – Natural language processing difficulty [T2]
• Contextual Information – Observational gap-filling, interpretation [T3]
• Association of Hard and Soft Information [T8] – Symbolic versus Numerical (Baghdad vs. 33°20’ N, 44°26’ E) [T7] – Asynchronous (out of sequence) information arrival [T6]
• Processing of Hard or Soft Information [T10] – How does the one influence the parameters, biases or algorithms for the processing of the
other
• Lack of Datasets for Technology Development and Verification [T11] • Traditional Challenges in Hard Fusion [T12] • High-level Fusion (Situation Understanding) [T9] • Concepts of Employment (and Transition) [T13]
Genlock Multiple Hypothesis Particle Filter Color Tracker
Multiple Hypothesis Lidar Range Segmentation Tracker
Fusi
on
En
gin
e
3D
2D/3D
• The objective of the implementation is to incorporate range data from the lidar which will be used to guide the color tracker in cases where track is lost due to occlusion.
Acoustic Signature Analysis Based on STFT Spectrogram
(PSU)
Short-Time Fourier Transform (STFT)
Discrete Wavelet Transform (DWT)
Kernel-Based Gaussian Mixture Model (K-GMM)
Alkilani, A., Shirkhodaie, A., “Vibro-Acoustic Analysis of Contented Containers,” in review for publication in SPIE 2012 Defense and Security Conference, April 2012, Orlando, FL.
• Hard-Soft Fusion Architecture (what/when to fuse?) [T1] – Feature-level Fusion (Measurement or Early Fusion) – Estimate-level Fusion (Late Fusion)
• Nature of Soft Information – Symbolically expressed (symbolic ambiguity) – Human observations have bias and (seldom characterized) uncertainty [T5]
• Uncertainty representation and transformation [T4]
– Results of cognitive fusion are unavailable (feature to symbol) – Natural language processing difficulty [T2]
• Contextual Information – Observational gap-filling, interpretation [T3]
• Association of Hard and Soft Information [T8] – Symbolic versus Numerical (Baghdad vs. 33°20’ N, 44°26’ E) [T7] – Asynchronous (out of sequence) information arrival [T6]
• Processing of Hard or Soft Information [T10] – How does the one influence the parameters, biases or algorithms for the processing of the
other
• Lack of Datasets for Technology Development and Verification [T11] • Traditional Challenges in Hard Fusion [T12] • High-level Fusion (Situation Understanding) [T9] • Concepts of Employment (and Transition) [T13]
• Hard-Soft Fusion Architecture (what/when to fuse?) [T1] – Feature-level Fusion (Measurement or Early Fusion) – Estimate-level Fusion (Late Fusion)
• Nature of Soft Information – Symbolically expressed (symbolic ambiguity) – Human observations have bias and (seldom characterized) uncertainty [T5]
• Uncertainty representation and transformation [T4]
– Results of cognitive fusion are unavailable (feature to symbol) – Natural language processing difficulty [T2]
• Contextual Information – Observational gap-filling, interpretation [T3]
• Association of Hard and Soft Information [T8] – Symbolic versus Numerical (Baghdad vs. 33°20’ N, 44°26’ E) [T7] – Asynchronous (out of sequence) information arrival [T6]
• Processing of Hard or Soft Information [T10] – How does the one influence the parameters, biases or algorithms for the processing of the
other
• Lack of Datasets for Technology Development and Verification [T11] • Traditional Challenges in Hard Fusion [T12] • High-level Fusion (Situation Understanding) [T9] • Concepts of Employment (and Transition) [T13]
• Hard-Soft Fusion Architecture (what/when to fuse?) [T1] – Feature-level Fusion (Measurement or Early Fusion) – Estimate-level Fusion (Late Fusion)
• Nature of Soft Information – Symbolically expressed (symbolic ambiguity) – Human observations have bias and (seldom characterized) uncertainty [T5]
• Uncertainty representation and transformation [T4]
– Results of cognitive fusion are unavailable (feature to symbol) – Natural language processing difficulty [T2]
• Contextual Information – Observational gap-filling, interpretation [T3]
• Association of Hard and Soft Information [T8] – Symbolic versus Numerical (Baghdad vs. 33°20’ N, 44°26’ E) [T7] – Asynchronous (out of sequence) information arrival [T6]
• Processing of Hard or Soft Information [T10] – How does the one influence the parameters, biases or algorithms for the processing of the
other
• Lack of Datasets for Technology Development and Verification [T11] • Traditional Challenges in Hard Fusion [T12] • High-level Fusion (Situation Understanding) [T9] • Concepts of Employment (and Transition) [T13]
1. Created the first known Hard and Soft "SYNCOIN" dataset for fusion technology and analytics development.
This counter insurgency dataset has about 600 natural language messages with interleaved vignettes for the development and testing of natural language processing methods, and hard sensor associable counterparts for developing semantic extraction and association technologies. This unique data sets is inspired by a Counter-Insurgency (COIN) scenario in Bagdad and contains synthetic soft (human report) data, synthetic hard (physical sensor) data, and real hard data collected using human-in-the-loop vignettes collected at a special facility in central Pennsylvania. The data are augmented by extensive “ground truth” information including, scene setter descriptions, identification and location of all events and activities, social network information, database schema, reference maps, and word cloud diagrams.
2. First ever comprehensive characterization of human observations that are context sensitive. Results published in a submitted paper for the Information Fusion journal, and transition to ARL pending through the ARL infrastructure initiative. Characterizing the human observer is an essential “human” source characterization and common referencing step for hard-soft fusion.
3. An novel approach to Natural Language Processing through "Tractor" and context overlay so that no information is lost and all semantically meaningful information is extracted. Relevant contextual information (as a human would have when reading text) is added so that machine reasoning is feasible. This is a goal, but it is an important discriminator from NLP engines that are simply interested in "information retrieval" of entities and relationships.
4. Best in class graph association engine. First Map-Reduce implementation for distributed computing. The event association engine, which includes location normalization has been compared to others in the literature. Paper under revision for ACM trans on Info Sys. Association results provide better Precision, Recall and F-measure than other approaches in the literature. Results to be published [Note here that since there is no agreed corpus, a direct comparison with other authors is sometime not accurate.] Journal paper revised for Naval Research Logistics establishes a Lagrangian approach to Graph Association. Results within 5% of the optimal are produced.
5. Graph Matching: Most comprehensive suite of graph matching engines to include uncertainty, AND-OR template graphs (which are suitable to model PIRs), and incremental graph information.
6. 2D to 3D image fusion. We have developed advanced algorithms for the fusion of hard sensor data that include Lidar, SWIR, LWIR, visual video and acoustic sensors. The algorithms perform fusion at the data level – mapping 2-D image data to 3-D Lidar data, creating a “smart pixel” 3-D point cloud that allows multi-spectral feature extraction, hierarchical object classification (based on object size, shape and spectral characteristics), and very accurate target location. New algorithms for multiple hypothesis tracking provide the capability to track dynamic objects in complex environments including tracking of multiple people through crowds including obscurations.
7. Creation and demonstration of a distributed cyber infrastructure for fusion. We have explored the design and implementation of an infrastructure to support distributed information fusion. The toolbox of techniques include; communication methods and protocols, extensions of Service Oriented Architecture (SOA) and Message Oriented Middleware (MOM) paradigms, optimized information flow and tasking, complex event processing, and utilization of community standard data representations. Demonstrations of this infrastructure have been created using tools such as GeoSuite.