Duane Searsmith Automated Learning Group National Center for Supercomputing Applications University of Illinois [email protected]Office: (217) 244-9129 http://alg.ncsa.uiuc.edu Michael Welge, Director, [email protected]Loretta Auvil, Project Manager, lauvil @ncsa.uiuc.edu , (217) 265-8021 July 9, 2004 Text Mining with D2K/T2K
Text Mining with D2K/T2K. Outline. Text Mining Brief Intro Unsupervised Supervised Information Extraction … ALG Technology Pieces Demonstrations Discussion. What is text mining?. - PowerPoint PPT Presentation
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Duane SearsmithAutomated Learning GroupNational Center for Supercomputing ApplicationsUniversity of [email protected]: (217) 244-9129http://alg.ncsa.uiuc.edu
• Agglomerative (bottom up)• Quadratic time complexity• Sampling
•Random•Partition
• Hard vs. Soft
• Unsupervised method
• Basic notion to all of these approaches is some heuristic for measuring similarity between documents and document groups (term co-occurrence)
Strongly Similar Arcs
Kept
Weakly Similar Arcs
Broken
Clustering: Document Self-Organization
alg | Automated Learning Group
How to Recognize a Needle• To classify your data you often need to build a model.
• To build a model you typically need examples from a “teacher” – metaphorically speaking.
• Finding good examples can be hard.
• T2K can also use active learning to help find good examples faster making model building easier.
alg | Automated Learning Group
Pattern Mining
• Finding frequent item sets -> Rule Discovery
• Many methods: Apriori, Charm, FPGrowth, CLOSET
• Working with Jiawei Han and students -- Hwanjo Yu and Xiaolei Li
• Application: topic tree construction
alg | Automated Learning Group
Just the Facts Please
• Finding a document that has the information you need is often not the end goal.
• To extract information you must first recognize it – you need to build a model, and that means you need to have examples.
• Levels of IE: What’s hard and what’s harder?
alg | Automated Learning Group
D2K
alg | Automated Learning Group
D2K Features
• Extension of existing API• Provides the capability to programmatically connect modules and set properties.• Allows D2K-driven applications to be developed.• Provides ability to pause and restart an itinerary.
• Enhanced Distributed Computing• Allows modules that are re-entrant to be executed remotely.• Uses Jini services to look up distributed resources.• Includes interface for specifying the runtime layout of a distributed itinerary.
• Processor Status Overlay • Shows utilization of distributed computing resources.
• Distributed Checkpointing• Resource Manager
• Provides a mechanism for treating selected data structures as if they were stored in global memory.
• Provides memory space that is accessible from multiple modules running locally as well as remotely.
• Batch Processing / Web Services
D2K Overview
alg | Automated Learning Group
D2K/T2K/I2K - Data, Text, and Image Analysis
Information Visualization
alg | Automated Learning Group
• The Engine (distributed, parallelized, persistent)• Core Modules (building blocks)• T2K is a specialized set of modules for text
analysis• I2K is a specialized set of modules for image
analysis• D2K Toolkit (rapid development environment)• ThemeWeaver is an independent application that
uses the D2K engine to run algorithms constructed from T2K modules. It is a demonstration platform
• Other D2K driven applications (StreamLined, EMO, …)
D2K Engine Core Modules T2K Applications
The Technology Pieces
I2K Toolkit
alg | Automated Learning Group
T2K Core
• Tokenization• POS Tagging• Stemming• Chunking• Filters• Term