Fuzzy Database & Information Retrieval

Fuzzy Database & Information Retrieval

Similarity relation defined for the domain opinion

18.06.02.0008.018.06.02.006.08.018.06.02.02.06.08.018.06.0

02.06.08.018.0002.06.08.01HNNSNSFFHF

HNNSNSFFHF

Query:which sociologists are in considerable agreement with Kass

concerning policy Y?

Fuzzy Relational Data Base: Buckles, Petry(1) Elements of the tuples contained in the relations may be

subsets of the domain universal set.(2) A similarity relation is defined on each domain universal set.

Fuzzy Data Base1. (Project (select Assessment where Name = Kass and Option =

Y) over Opinion) giving R1

Relation : R1 Option

favorable

Retrieve the opinion of Kass concerning option Y

2. (Project (select Expert where Field = Sociologist) over Name) giving R2

Relation : R2 Name

Osborn

Schreiber

Cohen

Specterman

Select all sociologists from the table of experts

3. (Project (select (Join R2 and Assessment over Name) where Opinion = Y) over Name, Opinion) giving R3

List the opinions of the sociologists

4. (Join R3 and R1 over Opinion) with THRES (Opinion) 0.75 ≧and THRES (Name) 0≧

Relation : R3Name　　　　　 Opinion

Obsorn Slightly favorable

Schreiber Favorable

Cohen Slightly negative

Specterman Highly favorable

Name　　　　　 Opinion

{Obsorn, Schreiber Specterman}

Slightly favorable, favorable, highly favorable}

Information Retrieval

1. IR process can be viewed as a knowledge communication process, which involves learning and problem solving strategies.

2. IR system controls the knowledge flow between documents and the user.

3. An IR system is a computer system that allows users to retrieve information from a document connection stored in a data base.

4. Classical IR is built on Boolean algebra framework.

Fuzzy Information Retrieval

1. Fuzzy relation for the grade of relevance between index terms and documents:

– R: X x Y [0,1]– Determined subjectively or objectively– Number of occurrences; publication dates,

document types

2. Fuzzy thesaurus– T: X x X [0,1]

Fuzzy Information Retrieval

1. Information retrieval model

2. Advantages:– R and T are more expressive and their

construction is more realistic– Fuzzy inquiry provides greater flexibility

RBD

xxTxAxB

TAB

jiiXx

ji

)],(,min[ max

Information retrieval based on fuzzy associations

1. Introduction

2. Three components in information retrieval

3. Fuzziness in a thesaurus: first component

4. Fuzziness in retrieval: second component

5. Fuzziness on output: third component

6. Classification of output

7. Conclusion

2. Three components in information retrieval D = {d1,d2,…,dn} be a finite set of documents for

retrieval W = {w1,w2,…,wm} denote a set of descriptors T : D ─>[0,1]w ． T(d): a subset of descriptors in W

indexed to the document d. U(U = T-1) ． U(w): documents have keyword w.

F

Information retrieval based on fuzzy associations

U P rr’

q

3. Fuzziness in a thesaurus: first componentThree type thesaurus (represented as binary relation)RT: related termsNT: narrower termsBT: broader termsB(v,w) = N(w,v) R(v,w) = R(w,v)Method of automatic generation of thesauri:1. Typical:counting frequencies of simultaneous occurrences

of pairs of keywords in a set of documents.2. Fuzzy set model:C = {c1,c2,…,cp} be a finite set of concepts where each ci, i=1,

…p represents a unit of conceptH:W ─>[0,1]p a fuzzy set valued function which maps each

keyword to it’s corresponding concepts as a fuzzy set in C.Wwwh :)( is concept of the word w.

)(

)()(),(

)()(

)()(),(

vh

whvhwvN

whvh

whvhwvR

Even by present computers, it’s difficult to calculate values of the fuzzy relation above using array in straightforward way, since the numbers of elements in W and D are very large(103 ｘ 105). Although techniques to handle sparse matrices may be applied, there is another method for generation R and N based on manipulation of sequential files. The principle tool for this is sorting.

(a,b,c) means a record in which field are a,b and c.{(a,b,c)} means a set of records such as (a,b,c).

Input: a set D of documents, Each document d ∈ D has a number of keywords in W.A keyword may occur twice or more in a document. The frequency of occurrence of wi in dk is denoted by hik.

Output: a set of records {(wi,wj,R(wi,wj)]} for all pairs R(wi,wj)<>0

Algorithm GFT (generation of a fuzzy thesaurus).// Find pairs of keywords in every document.//For all dk D do∈　 find all keywords wi W and calculate h∈ jk

　 for all (wi,wj),wi<wj, that are found in dk domake record (wi,wj, min(hik,hjk))output (wi,wj,min(hik,hjk) to WORK1

　 repeat　 for all wi that are found in dk do

make record (wi, hjk)output (wi,hjk) to WORK2

　 repeatrepeat//sort WORK1 and WORK2.//sort WORK1 into increasing order of the key (wi,wj)sort WORK2 into increasing order of the key wi

//Calculate R.Scan WORK1 and WORK2.//for all (wi,wj) in WORK1 do　 find all record for (wi,wj) in WORK1

and all records for wi, and wj in WORK2　 R (wi,wj)←∑k min(hik,hjk)/(∑k hik+ ∑k hjk- ∑kmin(hik,hjk))　 output (wi,wj,R(wi,wj)) to an output filerepeatend-of algorithm GFT

In a foregoing paper an experimental calculation on three thousand documents and thirty thousand keywords was carried out using GFT based on sorting shows a reasonable amount of 800 sec of CPU time.

//record (di,pi)////before another record (di,pi) satisfies either di<dj or////di = dj, pi > pj//Take the first record (d1,p1) in work(D,P)<-(d1,p1)for all dj in WORK do//the dj’s are sequentially examined.//　 if D <> dj then

output (D,P) to to an output file OUT(D,P)<-(di,pj)

　 endifrepeatoutput(D,P) to OUT//OUT contains exactly those records that represent P=Uf(d,w) define by above//

//Third step: if necessary sort again.//sort OUT into the decreasing order of the key p and print OUT

4. Fuzziness in retrieval: second component

For the crisp case a retrieval through a thesaurus

given a keyword w is as follows.

(a) Examine the thesaurus F and find all associated

terms v11,v12,…,v1p.

(b)Find subsets U(v11),U(v12),…,U(v1p)．(c) Establish the retrieved set of documents as the

union of U(v11),U(v12),…,U(v1p): ∪1 i p≦ ≦ U(v1i)

Uf(d,w) = 1 iff d U(v∈ 1) for some v1 such that

F(v1,w) = 1,

0 otherwise.

When the thesaurus F is fuzzy and U is crispUf(d,w) = max v W∈ min [U(d,v),F(v,w)].This equation is valid also for a fuzzy relation U(d,v).Algorithm FR(Fuzzy Retrieval).//First step: Find all records.//for all v such that F(v,w) <> 0 in FT do　 for all d U(v) do∈

p(d,v)<-min[U(d,v),F(v,w)]output record (d,p(d,v)) to a work file WORK

　 repeatrepeat//second step: Find values of Uf.//sort WORK into increasing order of the first key d　 and into decreasing order of the second key p//the above sorting means that in the resulting sequence,a//

End-of FR5. Fuzziness on output: third component

Fuzzy filter. EX:(a) Find recent documents that have keyword w.(b) Find documents that have keywords w and are

relevant to one’s field of interestr = r’ ∩ g

6. Classification of output1) Decreasing of membership2) Divide into layers

7. ConclusionProblem for further studies1) Discussion of crisp techniques of advanced

indexing and retrieval using a fuzzy set model,

2) Studies of efficient algorithms for large scale database. In particular, development of hardware for information retrieval should be taken into account.

3) Application of methods in fuzzy information retrieval to related areas.

個人化服務元件技術之研究

郭耀煌教授成功大學資訊工程系

成功大學數位生活科技研究中心