A Continuous Media Data Rendering System For
Visualizing Psychological Impression-Transition
† Fujiko Yara ([email protected]) ‡ Naofumi Yoshida ([email protected])
‡ Shiori Sasaki ([email protected]) † Yasushi Kiyoki ([email protected])
† Faculty of Environmental Information, Keio University‡ Graduate School of Media and Governance, Keio University
Application Example and Demonstration
• Our system enables us to render the continuous impression-transition for making user’s feeling change.
• Our system calculates the relationship between media data and impression.• And then renders set of media data continuously.• The demonstration shows an example of rendering media data.
Demonstration
STRTFINISHconfuse comfortable
Input : from “confuse” to “comfortable”
Output : Sequence of colors by emotionally continuous transition
t
( ゚゚ ;)( 。。 ;))((; ゚゚ )(; 。。 )
(*´ ェ `*)
~~~Smell
♪♪
♪ ♪Music
Picture
Lighting
The statement of user is “confuse”. The statement of user is “comfortable”.
Demonstration
Three Main Features in our system
1. Our system enables us to render media data with continuous impression-transition to guide user’s feelings by psychological models.
2. There are a lot of variety of routes. Because of many models and routes, our system renders the appropriate set of media data in the several scenes for users.
3. Our system use the psychological impression-transition models that can be applied to render various media continuously.
Our Understanding of the Impression-Transition Models
We believe that it is necessary to visualize psychological impression-transition for rendering continuous media data emotio
nally. • In psychological impression-transition models, many elements (for exa
mple, color data, impression-words) are continuously allocated by something to define.
• CIS (Color Image Scale) model allocates color data into 2 dimensional manner, and Hevner-model allocates impression-words into 1 dimensional manner.
• We think the continuous relationships between each elements in these models show emotionally continuous transition.
CIS-model
Allocated by color gradation
Allocated by impression-words
Hevner-model
Overview• In this presentation, we show an
implementation method and effectiveness of our method .
• We demonstrate several experiments for the feasibility.
• We explain about our understanding of Impression-Transition Model and how to apply to our system.
Background Issues
Retrieval ResultsDB
Design
Rendering media data with emotionally
sequence
Our target
•Multi Media Data Base
•Meta Data Base System [KK_1993]
•Data Base Management system
•Metadata of Multimedia management system
•Media data searching method and system [IK_2005] [KK_1994]
•Metadata extraction method and system [IKNS_2003]
•Psychological Field
•Image transaction technology
•Psychological Field
•[KK_1993] Kitagawa, T. and Kiyoki, Y. :The mathematical model of meaning and its application to multidatabase systems, Proceedings of 3rd IEEE International Workshop on Research Issues on Data Engineering Interoperability in Multidatabase Systems, April 1993, 130-135•[KK_1994] Kiyoki, Y. Kitagawa, T. and Hayama, T. : A metadatabase system for semantic image search by a mathematical model of meaning, ACM SIGMOD Record, Vol.23, No.4, 1994, 34-41.•[IKNS_2003] shibashi, N. Kiyoki, Y. Nakagami, Y. and Sato, A. : An Impressionistic Metadata Extraction Method for Music Data with Multiple Note Streams, DBSJ Letters, Vol.2, No.2, October 2003, pp.61-64 .•[IK_2005] Ijichi, A. and Kiyoki, Y.:A Kansei Metadata Generation Method for Interpretation, Information Modeling and Knowledge Bases, 16, 170-182, 2005.
Retrieval technology
Rendering technology
System Overviewresult 1
・・・
Impression-Transition DB
CIS-model
Hevner-model
Metric for rendering emotionally continuous transition
Start-point-query
Goal-point-query
Start-media data
Goal-media data
Metric for distances on vector spaces
Multi Media DBColor DB Music DBSensor DB
start-query
goal-query
terminal-impression word
starting-impression word
▲
▲
▲
▲
▲ ▲▲
▲■
■■■
■
■■■
■
▲■
starting-point query◆ ◆
terminal-point query
▲▲
starting-media data
terminal-media data
Vector Space
Basic Method (1/2)• Calculation in Metric for distances on vector space.
■ Impression-Words in Longman Dictionary ▲ Impression-Words expressing Media data◆ Query words
terminal-impression word
starting-impression word
▲
▲
▲
▲
▲ ▲
▲
▲■
■
■■
■
■■
■
■
▲■
starting-point query◆ ◆
terminal-point query
▲
▲
starting-media data
terminal-media data
1. Creating an impression-words vector space to calculate the impression retrieval.
2. Impression-words and media data are mapped into this vector space.
3. Converting 2-queries into 2-media-data.
Basic Method (2/2)• Calculating in Metric for rendering
emotionally continuous transition.
・・・・・・・●
●● ●
●●
●●●
● ● ●
●●●●
・・・・・・
W (1)W (2) W (3)
W (n)W (n-1)
W (1)W (2)W (n)
W (n-1)W (n-2)
・・・・・・
Based Point
Based Point
・・・●● ● ●●
● ●●● ●
●● ●●● ● ●・・・
・・・
Sets of rendering output media data
CIS-model
Hevner-model
1. Using transition model representing something to meant.
2. Creating database of transition’s routes.
3. Defining many parameters to decide the route.
4. Mapping 2-media data (start-media data and terminal-media data) into transition model.
5. Converting data on transition model into output media.
Implementation Method (1/8)• Step 1 : Creating an impression-words vector space
of media data and mapping the impression-words of media contents into it.
▲
▲
▲
▲
▲
▲
▲▲
▲
▲■
■■
■
■■
■
■
■
■■
■
■
▲
■ Impression-Words in Longman Dictionary ▲ Impression-Words expressing Media data
1. We have implemented this system using the Mathematical Model of Meaning (MMM) [KK_1994].
2. The MMM search space is created by using the Longman Dictionary of Contemporary English.
3. The impression-words of media data are mapped into this impression-word vector space.
•[KK_1994] Kiyoki, Y. Kitagawa, T. and Hayama, T. : A metadatabase system for semantic image search by a mathematical model of meaning, ACM SIGMOD Record, Vol.23, No.4, 1994, 34-41.
•Longman Dictionary of Contemporary English, Longman, 1987
途中!
Implementation Method (2/8)• Step 2 : Creating database representing the
route of the impression-transition.
1. By using psychological models as impression-transition models, we create databases which express the route of the impression-transition.
2. Using the Hevner-model and the CIS-model (Color Image Scale model) as impression-transition.
途中!
Implementation Method (3/8)• Step 3 : Submitting two query words (starting-query
and terminal-query) into the impression space.
■ Impression-Words in Longman Dictionary ▲ Impression-Words expressing Media data● Query words
1. A user submits two query words into MMM search space created in step1.
2. Two query are not always words used in Longman Dictionary, so in next step, two query words are converted into the words used in Longman Dictionary.
terminal-impression
start-impression▲
▲
▲
▲
▲ ▲▲
▲
▲■
■■
■
■
■
■■■
■
start-query
terminal-query・ ・
■
▲■
途中!
Implementation Method (4/8)• Step 4 : Converting two query words as two
impression words (starting-impression and terminal-impression).
■ Impression-Words in Longman Dictionary ▲ Impression-Words expressing Media data● Query words
terminal-impression
start-impression▲
▲
▲
▲
▲ ▲▲
▲
▲■
■■
■
■
■
■■■
■
starting-query
terminal-query・ ・
■
▲■
1. The starting-query is converted into the semantically closest word (starting-impression) within the impression words included in the route representing the impression transition (Step2).
2. The terminal-query is also converted into the semantically closest word (terminal-impression).
途中!
Implementation Method (5/8)• Step 5 : Converting impression-words to media data
(starting-media data and terminal-media data)
start-impression
terminal-media data
start-media data
▲terminal-
impression
▲
▲
▲
▲
▲
▲
▲
■
■■
■
■
■
■■■
■
▲
■ Impression-Words in Longman Dictionary ▲ Impression-Words expressing Media data
途中!
Implementation Method (6/8)• Step 6 :Extracting two media data from vector
space and mapping into the impression-transition models respectively.
Hevner-model
CIS-model
1. Two media data (Starting-media data and terminal-media data) are mapped into the database based on the impression-transition models.
2. The appropriate route is chosen for connecting starting-media data and terminal-media data continuously.
途中!
Implementation Method (7/8)• Step 7 : Choosing the route on the impression-transition
model for rendering media data for visualization by color data.
CIS-model
Allocated by color gradation
Hevner-model
Allocated by impression-words
1. To realize rendering the media data continuously, the route from starting-media data to terminal-media data is chosen.
2. According to taking time, the way how to walk on the impression-transition is decided.
3. The way how to walk must be uniformed like chess.
途中!
Implementation Method (8/8)• Step 8 : Rendering the sequence of output media
data generated in Step7.
1. We render the set of color data generated in Step7 along the selected route using the impression-transition model.
2. A personal computer is used to display the rendering of the set of color data.
途中!
Results Example
Experimental Results (1/3)[confuse → comfortable]
• These results of the set of color data show the feasibility of our continuous media data rendering system for visualizing the change of psychological impression-transition.
(start-point-query=“confuse”, goal-point-query=”comfortable” R=shortest (右回り ))
(start-point-query=“confuse”, goal-point-query=”comfortable”, R=longest (左回り ))
(start-point-query=“confuse”, goal-point-query=”comfortable” R=第4節 5(B))
Hevner-model CIS-model
Results Example
Experimental Results (2/3)[merry → calm]
• These experiments have shown the applicability of our method for user’s various requirement of impression-transition.
(start-point-query=”merry”, goal-point-query=”calm” R=shortest (右回り ))
(start-point-query=”merry”, goal-point-query=”calm” R=longest (左回り ))
(start-point-query=“merry”, goal-point-query=“calm” R=longest (Right and Down))
(start-point-query=“merry”, goal-point-query=“calm” R=longest (Down and Right))
(start-point-query=“merry”, goal-point-query=“calm” R=shortest (Left and Down))
(start-point-query=“merry”, goal-point-query=“calm” R=shortest (Down and Left))
Hevner-model
CIS-model
Results Example
Experimental Results (3/3)[simple → graceful]
• Our method has applicability for various strength of relationship between two query words, even if the two impression-words have a weak relationship in the vector space.
(start-point-query=“simple”, goal-point-
query=“graceful”)
(start-point-query=“simple” goal-point-query=“graceful”, R=longest (Right and Down))
(start-point-query=“simple” goal-point-query=“graceful”,
R=longest (Right and Down))
(start-point-query=“simple” goal-point-query=“graceful”, R=longest (Right and Down))
(start-point-query=“simple” goal-point-query=“graceful”, R=longest (Right and Down))
Hevner-model
CIS-model
Summary and Future Work• Our method makes it possible to implement visualizations
of the continuous change of impression-transition, according to impression-words expressed for starting point and terminal point.
• By the implementation and experiments using color data as output media data, we have clarified the feasibility of our method for visualizing the change of impression-transition from starting point to terminal point by using the research results of musical psychology and color psychology.
• We will design aggregate functions for color data expression in the experiments using psychological word groups by the Hevner-model.
• We will approach to the computation mechanisms of continuous transition of impression.
References• [KK_1993] Kitagawa, T. and Kiyoki, Y. :The mathematical model of meaning and its application t
o multidatabase systems, Proceedings of 3rd IEEE International Workshop on Research Issues on Data Engineering Interoperability in Multidatabase Systems, April 1993, 130-135
• [KK_1994] Kiyoki, Y. Kitagawa, T. and Hayama, T. : A metadatabase system for semantic image search by a mathematical model of meaning, ACM SIGMOD Record, Vol.23, No.4, 1994, 34-41.
• [KKH_1995] Kiyoki, Y. Kitagawa, T. and Hitomi, Y. : A fundamental framework for realizing semantic interoperability in a multidatabase environment, Journal of Integrated Computer-Aided Engineering, Vol.2, No.1, Jan.1995, 3-20.
• [AS_1994] Aiello, R. and Slobada, J.A.: Musical perceptions, Oxford University Press, 1994.• [IK_2005] Ijichi, A. and Kiyoki, Y.:A Kansei Metadata Generation Method for Interpretation, Infor
mation Modeling and Knowledge Bases, 16, 170-182, 2005.• Longman Dictionary of Contemporary English, Longman, 1987• [H_1937] Hevner, K. : The affective value of pitch and tempo in music, American Journal of psyc
hology, 49, 621-630.• [IKNS_2003] shibashi, N. Kiyoki, Y. Nakagami, Y. and Sato, A. : An Impressionistic Metadata Ex
traction Method for Music Data with Multiple Note Streams, DBSJ Letters, Vol.2, No.2, October 2003, pp.61-64 .
Thank you for your attention.
Derivation of Kansei• 印象遷移モデルは一定を約束している• 別のストーリを持つモデルは多く存在し、• 微分係数が一定でないものが多い。
・・・・・・・●
●● ●
●●
●●●
● ●●
●●●●
・・・
・・・W (1)
W (2) W (3)
W (n)W (n-1)W (1)
W (2)W (n)
W (n-1)W (n-2)
・・・・・・
Based Point
Based Point
terminal-impression word
starting-impression word
▲
▲
▲
▲
▲ ▲▲
▲■
■■■
■
■■■
■
▲■
starting-point query◆ ◆
terminal-point query
▲▲
starting-media data
terminal-media data
Vector Space
c1
sober
serioussacred
dignifiedinspiring
awe
loftyspiritual
solemn
c2
dark
depressing
dolefulfrustrated
sad
melancholy
heavygloomy
tragicmournful
pathetic
c3
dreamy longingplaintive sentimental
yielding
yearningtender
c4calm leisurely
tranquilsoothingserenesatisfying
quietlyricalc5
playfullight
humorousgraceful fancifuldelicate
whimsicalsprightly
quaint
c6
merryjoyousbrighthappygaycheerful
c7
passionateimpetuous
exhilaratedexciting dramaticagitated
sensationalrestless
triumphant
soaring
c8
emphatic exaltingvigorous
robustponderousmartialmajestic
始点の印象語S:Start-query
終点の印象語t:goal-query
Hevner’s model[4,5]
CIS model[6]
Neuton’s model
○
○○
○
○○
●● ●● ●● ●●・・・・・・・・ Hevner’ modelでのルート群RHev・・・・・・・・・・・・・・・・
n
●● ● ・・・・・・・・●● CIS model でのルート群RCIS・・・・・・・・・・・・・・・・
n
○○○○○○○
●
○
○○
○
Mansel’s model
メディアデータMDで出力 メディアデータMDで出力
・・・・・・
心理学的印象モデル Co群
○ :印象語● :MD
●● ●●●● ● ●●● ●● ●●●● ● ●●● ●● ●●
●● ● ●●● ●● ●●●● ● ●●● ●● ●●●● ● ●●● ●● ●●
c1
sober
serioussacred
dignifiedinspiring
awe
loftyspiritual
solemn
c2
dark
depressing
dolefulfrustrated
sad
melancholy
heavygloomy
tragicmournful
pathetic
c3dreamy longingplaintive sentimental
yielding
yearningtender
c4calm leisurely
tranquilsoothingserenesatisfying
quietlyricalc5
playfullight
humorousgraceful fancifuldelicate
whimsicalsprightly
quaint
c6
merryjoyousbrighthappygaycheerful
c7
passionateimpetuous
exhilaratedexciting dramaticagitated
sensationalrestless
triumphant
soaring
c8emphatic exalting
vigorous
robustponderousmartialmajestic