Graduate School of Informatics Kyoto University, November 21, 2001 Technologies of the Interspace Peer-Peer Semantic Indexing Bruce Schatz CANIS Laboratory.

Graduate School of InformaticsGraduate School of InformaticsKyoto University, November 21, 2001Kyoto University, November 21, 2001

Technologies of the InterspaceTechnologies of the Interspace Peer-Peer Semantic IndexingPeer-Peer Semantic Indexing

Bruce SchatzCANIS Laboratory

Graduate School of Library and Information ScienceUniversity of Illinois at Urbana-Champaign

www.canis.uiuc.edu, [email protected]

THE THIRD WAVE OF NET EVOLUTIONTHE THIRD WAVE OF NET EVOLUTION

PACKETS

OBJECTS

CONCEPTS

SCALABLE SEMANTICSSCALABLE SEMANTICS

Automatic indexing Domain-Independent indexing Statistical clustering

Compute Context of

concepts within documents documents within repositories

CROSS-OVERS IN SEMANTIC INDEXINGCROSS-OVERS IN SEMANTIC INDEXING

1992 1993 1995 1996 1998

COMPUTING CONCEPTSCOMPUTING CONCEPTS

‘92: 4,000 (molecular biology)

‘93: 40,000 (molecular biology)

‘95: 400,000 (electrical engineering)

‘96: 4,000,000 (engineering)

‘98: 40,000,000 (medicine)

SIMULATING A NEW WORLDSIMULATING A NEW WORLD Obtain discipline-scale collection

MEDLINE from NLM, 10M bibliographic abstracts human classification: Medical Subject Headings

Partition discipline into Community Repositories 4 core terms per abstract for MeSH classification 32K nodes with core terms (classification tree)

Community is all abstracts classified by core term 40M abstracts containing 280M concepts concept spaces took 2 days on NCSA Origin 2000

Simulating World of Medical Communities 10K repositories with > 1K abstracts (1K w/ > 10K)

COMMUNITY PROCESSINGCOMMUNITY PROCESSING

Existing TechnologiesExisting Technologies Extracting Concepts (AI)

Canonical noun phrases Generic statistical parser

Computing Context (IR) Co-occurrence frequency, in collection Useful interactively, not strict ordering

CONCEPT NAVIGATIONCONCEPT NAVIGATION

Semantic Indexes for Community Repositories

Navigating Abstractions within Repository concept space category map

Interactive browsing by Community experts

Category MapCategory Map

Category Navigation

Category Navigation

Concept NavigationConcept Navigation

CONCEPT SWITCHINGCONCEPT SWITCHING

“Concept” versus “Term” set of “semantically” equivalent terms

Concept switching region to region (set to set) match

term

Semantic region

Concept SpaceConcept Space

Medicine SessionMedicine Session

Categories and ConceptsCategories and Concepts

Concept SwitchingConcept Switching

Document RetrievalDocument Retrieval

Future TechnologiesFuture Technologies Concept Switching

Spreading activation, similarity clusters

Path Matching Aggregating indexes, many repositories

Dynamic Indexing On-the-fly collections, during session

Peer-Peer ComputationsPeer-Peer Computations Local Interaction

Your PC does small computations e.g. screensaver for SETI

Global Merging Partition computation into small parts Each local forms part of global whole

Large-Scale Distribution 3M users of SETI@Home Public Health. www.intel.com/cure

THE NET OF THE 21st CENTURYTHE NET OF THE 21st CENTURY

Beyond Objects to Concepts Beyond Search to Analysis Problem Solving via Cross-Correlating

Multimedia Information across the Net

Every community has its own special library Every community does semantic indexing

Zen of Information RetrievalZen of Information Retrieval Searching without Searching

Navigate concepts into documents Based on interactive recognition

Indexing without Indexing Compute context on dynamic collections Based on distributed extraction

Sharing without Sharing Record paths during user sessions Based on community practices