Research and Development of Multimedia Data Management ...users.cis.fiu.edu/~luiss/IAB-20091204/FIU_DMIS_Shu-Ching_Chen_2009_IAB.pdfThree Research issues in Multimedia Database Three

ShuShu--ChingChing ChenChenProfessor

Distributed Multimedia Information Systems LaboratoryFlorida International University

(The State University of Florida at Miami)

Contact: [email protected]://www.cs.fiu.edu/~chens/

Research and Development of Multimedia Data Management System (MDBMS)

AgendaAgendaMotivation and ChallengesMotivation and ChallengesPrototype MDBMS SystemPrototype MDBMS SystemApplicationsApplications

Social NetworkingSocial NetworkingHighHigh--level Feature Extractionlevel Feature ExtractionStorm Surge Animation ModelStorm Surge Animation ModelFlorida Public Hurricane Loss ModelFlorida Public Hurricane Loss Model

Motivation and ChallengesMotivation and ChallengesWhy multimedia data popular?

Attractive InformativeCompactCheap memory makes storage easy

Why handling and representing multimedia data challenging?Huge size (a typical 10 sec MPEG video ~4M)Temporal and spatial informationSemantic meaningMultidimensional representation

Why a separate database management system for multimedia data?Traditional database incapable of accommodating above characteristics efficiently

Motivation and Challenges (Cont.)Motivation and Challenges (Cont.)Three Research issues in Multimedia Database Three Research issues in Multimedia Database Management System (1)Management System (1)A. Storage and Representation: query efficiency and accuracy depends on optimum storage and representation

Leads to investigation in feature selection and indexingConflicting Issues

Greater the feature space, more accurate the query resultGreater the feature space, greater the computation overhead

Thus, we needEfficient Indexing supporting varied multimedia retrieval Optimum feature selection to represent semantic, temporal and spatial information

Motivation and Challenges (Cont.)Motivation and Challenges (Cont.)Three Research issues in Multimedia Database Three Research issues in Multimedia Database Management System (2)Management System (2)B. Query and Retrieval: temporal, spatial and semantic content should be considered during query of multimedia data

Leads to investigation in Image and Video retrieval strategiesLeads to investigation in concept identification and event detection via multimedia data mining

Eg. Key word based query: SunsetSunset

KeokiSunsetBottled

SunsetView

Driving directions to Sunset

Query By ExampleSimilarity MeasurementContent InterpretationUser FeedbackEtc.

Query By ExampleSimilarity MeasurementContent InterpretationUser FeedbackEtc.

Motivation and Challenges (Cont.)Motivation and Challenges (Cont.)Three Research issues in Multimedia Database Three Research issues in Multimedia Database Management System (3)Management System (3)

C. Multimedia System: integrated and robust system should be developed to address the nuances of multimedia data

Leads to investigation in developing an unique prototype to integrate the complex characteristics of multimedia data

PresentationAuthoringUser FeedbackData SecurityWeb-Based Multimedia Search


Social NetworkingSocial NetworkingHighHigh--level Feature Extractionlevel Feature ExtractionSoccer Event DetectionSoccer Event DetectionStorm Surge Animation ModelStorm Surge Animation ModelFlorida Public Hurricane Loss ModelFlorida Public Hurricane Loss Model

Prototype MDBMS SystemPrototype MDBMS SystemMotivation: The distributed multimedia management system design requires a comprehensive and integrated framework, which

Employs distributed architecture for the large-scale multimedia management systemOffers a database modeling solution to manage the complicated multimedia databasesFacilitates content based retrievalIncorporates the innovative techniques to further refine the retrieval performanceDelivers appealing experiences on multimedia presentations to all the usersEnsures the security assurance for multimedia data access

Prototype MDBMS Prototype MDBMS System (Cont.)System (Cont.)

Client SideClient Side

Security Management Module

Security Management Module

Multimedia Browsing and Retrieval Module

Multimedia Browsing and Retrieval Module

Presentation Authoring and Rendering Module

Presentation Authoring and Rendering Module

Image RetrievalImage

Retrieval

Video BrowsingVideo

BrowsingVideo

RetrievalVideo

Retrieval

Online Media Retrieval

Online Media Retrieval

Environment Role ManageEnvironment Role Manage

User Roles Manage

User Roles Manage

Object Role Manage

Object Role Manage

Security Rule Manage

Security Rule Manage

Java Media Frame PlayerJava Media Frame Player

MATN Model Design

MATN Model Design

SMIL GeneratorSMIL

Generator

Web based Player

Web based Player

User Log InUser Log InAdministratorAdministratorGeneral UserGeneral User

Security CheckingSecurity Checking

Request HandlerRequest HandlerRequest PackagerRequest Packager Request SenderRequest Sender Information ReceiverInformation Receiver

Network LayerNetwork Layer TCP/IPTCP/IP UDPUDP RTPRTP HTTPHTTP ……

Server SideServer Side

Proposed Framework


Network LayerNetwork Layer

Server SideServer Side

PostgreSQL: Multimedia Database Management (Relational Object Database )PostgreSQL: Multimedia Database Management (Relational Object Database )

Security Policy Rules

Environmental Roles

Object Roles

User Roles

Permission Roles

Affinity Relationships

Visual/Audio Features

Multimedia Meta Data

Segments and Objects

Processed Data

Videos

Audios

Images

Textual Data

Source DataMMM & HMMMMMM & HMMM XMLXML

Multimedia Search EngineMultimedia Search Engine

SMARXO Security ControlSMARXO Security Control

Image RetrievalImage Retrieval Video RetrievalVideo Retrieval Online LearningOnline Learning Video ClusteringVideo Clustering

Security Policy Checker

Security Policy Checker

Multimedia Data Processor

Multimedia Data Processor

Multimedia Data ManagerMultimedia Data Manager

Multimedia Data Analyzer

Multimedia Data Analyzer

Multimedia Data Supplier

Multimedia Data Supplier

Request HandlerRequest HandlerReceiverReceiver

Data SenderData Sender

Multimedia Database Indexing Modeling Multimedia Data Relationships

Offline LearningOffline Learning


Multimedia Database Management ‐ PostgreSQLMultimedia Database Management ‐ PostgreSQL

Multimedia Database Indexing (AH‐tree)

Modeling Multimedia Data Relationships

Source Data + Processed Data (MMM & HMMM)Source Data + Processed Data (MMM & HMMM)

Modified PostgreSQL’s index mechanism to include AH‐TreeEfficient multimedia data retrieval in a commercial DBMSAllow nearest neighbor queries (KNN)

Modeling multimedia data relationships (Social networking)


Modification of PostgreSQL’s indexing mechanismPostgreSQL’s index algorithm based on GIST was changed to have the functionalities of AH‐Tree

Added index key of AH‐TreeModified insert algorithmModified search algorithm to allow KNN queriesAdjusted the insert and search algorithms of AH‐Tree to be robust to concurrency


Functionalities:

Content‐based image retrieval

Video browsing and retrieval

Presentation and authoring

Prototype MDBMS Prototype MDBMS System (Cont.)System (Cont.)Content Based Image Retrieval

Prototype MDBMS Prototype MDBMS System (Cont.)System (Cont.)Video Browsing and Retrieval

Prototype MDBMS Prototype MDBMS System (Cont.)System (Cont.)Presentation and Authoring


Social NetworkingSocial NetworkingHighHigh--level Feature Extractionlevel Feature ExtractionStorm Surge Animation ModelStorm Surge Animation ModelFlorida Public Hurricane Loss ModelFlorida Public Hurricane Loss Model

Social NetworkingSocial NetworkingSocial networks are huge and growing

~ 35 million

~ 200 million

~ 12 million

Visualizing Social Data1

1. http://overstated.net/2009/03/09/maintained‐relationships‐on‐facebook

Social Networking (Cont.)Social Networking (Cont.)Visualizing Large Social Networks

IssuesIssues Solution approachesSolution approaches

Social Networking (Cont.)Social Networking (Cont.)Existing Solutions

Social Networking (Cont.)Social Networking (Cont.)Discussions

Issues of Clustered Graph representation

Social Networking (Cont.)Social Networking (Cont.)Proposed Approach

Social Networking (Cont.)Social Networking (Cont.)Generated Representative Graph

The lines between images represent the similarity relationship between them.


Social NetworkingSocial Networking

HighHigh--level Feature Extractionlevel Feature ExtractionStorm Surge Animation ModelStorm Surge Animation ModelFlorida Public Hurricane Loss ModelFlorida Public Hurricane Loss Model

TRECVID HighTRECVID High--level Feature Extractionlevel Feature Extraction

TREC conferenceResearch in information retrievalTRECVID: independent track created in 2001

TRECVIDResearch in automatic segmentation, indexing, and content‐based retrievalLarge test collection (videos) and uniform scoring procedures

TRECVID HighTRECVID High--level Feature Extraction (Cont.)level Feature Extraction (Cont.)

TRECVID TasksHigh‐level feature extraction

High‐level semantic concept (e.g., people, indoor/outdoor, speech, etc) detection

Search (interactive, manually‐assisted, and/or fully automatic)Surveillance event detectionContent‐based copy detection


High‐level Feature Extraction Automatically identify the occurrence of high‐level concepts (Indoor/Outdoor, People, Speech, etc) in test video dataData

Model‐development data: 200 hours of videoTest data: 180 hours of video

Multimedia Framework Feature ExtractionNormalization, DiscretizationSubspace AnalysisClassification


Next Step:The techniques of the High‐level Feature Extraction task serve to help video searchSearch Task (interactive, manually‐assisted, and/or fully automatic)Apply features of our multimedia management system, such as event detection


Social NetworkingSocial NetworkingHighHigh--level Feature Extractionlevel Feature Extraction

Storm Surge Animation ModelStorm Surge Animation ModelFlorida Public Hurricane Loss ModelFlorida Public Hurricane Loss Model

Storm Surge Animation ModelStorm Surge Animation ModelStorm Surge Animation Environment

Storm Surge Animation Model (Cont.)Storm Surge Animation Model (Cont.)Virtual Terrain Project

SimulatorSimulator

33‐‐dimensionaldimensional

InteractiveInteractive

OpenOpen‐‐sourcesource

EnginesEngines

Storm Surge Animation Model (Cont.)Storm Surge Animation Model (Cont.)LIDAR

AirborneAirborneLight detectionLight detectionTopographic dataTopographic dataAccurateAccurateHigh resolutionHigh resolution

Morphological Morphological filteringfilteringManMan‐‐made structuresmade structuresFootprintsFootprintsProcedural Procedural constructionconstruction

Storm Surge Animation Model (Cont.)Storm Surge Animation Model (Cont.)Engines

SkySky Wind & RainWind & Rain MarineMarine

VegetationVegetation TrafficTraffic Storm SurgeStorm Surge

Storm Surge Animation Model (Cont.)Storm Surge Animation Model (Cont.)Storm Surge Simulator

Simulate surge flooding effect by manipulating water mesh plane.Water height are taken from NOAA hourly numerical surge estimation.Displaying a new height every 10 seconds.

…1,3,6,7,8,5,4,8…

NOAA hourly water height estimation

Storm surge engine

Adjusting water mesh height

3

Time Engine

Synchronized sunlight


Social NetworkingSocial NetworkingHighHigh--level Feature Extractionlevel Feature ExtractionStorm Surge Animation ModelStorm Surge Animation Model

Florida Public Hurricane Loss Florida Public Hurricane Loss ModelModel

Florida Public Hurricane Loss ModelFlorida Public Hurricane Loss Model

Introduction Florida Public Hurricane Loss Model is funded for $4.3 million by Florida Office of Insurance Regulation.Approved by Florida Commission on Hurricane Loss Projection Methodology in 2007, 2008, and 2009. The objective of the project is to develop and maintain a computer model to assess hurricane risk and to project insured losses due to the occurrence of hurricanes in Atlantic Basin.

Florida Public Hurricane Loss Model (Cont.)Florida Public Hurricane Loss Model (Cont.)FPHLM Framework

Wind ModelWind Model

Vulnerability Model

Vulnerability Model

Insured Loss Model

Insured Loss Model

Output

Met Data

Bldg Stock Data

Engr Data

Insurance Data

Terrain Data

GIS Data

Exposure Data

Statistical Program

Statistical Program

Com

puter Platform

Com

puter Platform

C++C++

JavaJava

IDLIDL

MatlabMatlab

ArcGISArcGIS

Oracle 9iOracle 9i

FortranFortran

DMIS (Distributed Multimedia Information DMIS (Distributed Multimedia Information Systems) LaboratorySystems) Laboratory

Currently 9 Ph.D. students and several undergraduate students 4 Ph.D. students graduated. Two obtained tenure track assistant professor positions at reputed US schools (U. of Alabama at Birmingham and U. of Montana). One works at eBay (first job after graduation) and another one works at State Street (Assistant Vice President), USA18 MS students graduated with placements at Microsoft, IBM etc.Research areas:

Distributed Multimedia Database Systems Multimedia Data Mining GIS and Remote Sensing

DMIS (Distributed Multimedia Information DMIS (Distributed Multimedia Information Systems) Laboratory (Cont.)Systems) Laboratory (Cont.)

Dr. Chen is the Editor-in-Chief of International Journal of Multimedia Data Engineering and Management (IJMDEM)

Most Active SMC Technical Committee Award, IEEE Systems, Man, and Cybernetics Society, October 2006Outstanding Contribution Award, IEEE Systems, Man, and Cybernetics Society, August 2005The Inaugural Florida International University Excellence in Graduate Mentorship Award, 20062004 Florida International University Outstanding Faculty Research AwardSCIS Faculty Research/Service/Mentorship AwardsSIRI Fellow

Best Paper Award, IEEE International Symposium on Multimedia (ISM2006), December 11-13, 2006, San Diego, CA, USA.

Dr. Shu-Ching Chen

e-mail: [email protected]

Website: http://www.cs.fiu.edu/~chens

Research Lab: http://dmis.cs.fiu.edu/

Dr. Dr. ShuShu--ChingChing ChenChen

e-mail: [email protected]

Website: http://www.cs.fiu.edu/~chens

Research Lab: http://dmis.cs.fiu.edu/

Research and Development of Multimedia Data Management ...users.cis.fiu.edu/~luiss/IAB-20091204/FIU_DMIS_Shu-Ching_Chen_2009_IAB.pdfThree Research issues in Multimedia Database Three

Documents