WebMiningResearchASurvey Web Mining Research: A Survey Raymond Kosala and Hendrik Blockeel ACM SIGKDD, July 2000 Presented by Shan Huang, 4/24/2007 Revised.

WebWeb MiningMining ResearchResearch: AA SurveySurvey

Raymond Kosala and Hendrik Blockeel

ACM SIGKDD, July 2000

Presented by Shan Huang, 4/24/2007

Revised and presented by Fan Min, 4/22/2009

1

Outline

Introduction Web Mining Web Content Mining Web Structure Mining Web Usage Mining Conclusion & Exam Questions

2

Four Problems

Finding relevant information Low precision and unindexed information

Creating new knowledge out of available information on the web

Personalizing the information Catering to personal preference in content and presentation

Learning about the consumers What does the customer want to do? Using web data to effectively market products and/or services

3

Other Approaches

Web mining is NOT the only approach Database approach (DB) Information retrieval (IR) Natural language processing (NLP)

In-depth syntactic and semantic analysis

Web document community Standards, manually appended meta-information,

maintained directories, etc

4

Direct vs. Indirect Web Mining

Web mining techniques can be used to solve the information overload problems: Directly

Attack the problem with web mining techniquesE.g. newsgroup agent classifies news as relevant

IndirectlyUsed as part of a bigger application that addresses

problemsE.g. used to create index terms for a web search service

5

The Research

Converging research from: Database, information retrieval, and artificial intelligence (specifically NLP and machine learning)

Focusing on research from the machine learning point of view

6

Outline

Introduction Web MiningWeb Mining Web Content Mining Web Structure Mining Web Usage Mining Conclusion & Exam Questions

7

Web Mining: Definition

“Web mining refers to the overall process of discovering potentially useful and previously unknown information or knowledge from the Web data.” Can be viewed as four subtasks Not the same as Information Retrieval Not the same as Information Extraction

8

Web Mining: Subtasks

Resource finding Retrieving intended documents

Information selection/pre-processing Select and pre-process specific information from selected

documents Generalization

Discover general patterns within and across web sites Analysis

Validation and/or interpretation of mined patterns

9

Web Mining: Not IR

Information retrieval (IR) is the automatic retrieval of all relevant documents while at the same time retrieving as few of the non-relevant documents as possible

Web document classification, which is a Web Mining task, could be part of an IR system (e.g. indexing for a search engine)

10

Web Mining: Not IE

Information extraction (IE) aims to extract the relevant facts from given documents IE systems for the general Web are not feasible Most focus on specific Web sites or content

11

Web Mining and Machine Learning

Machine learning is concerned with the development of algorithms and techniques that allow computers to "learn".

Web mining is NOT learning from the Web. Some applications of machine learning on the web

are NOT Web Mining Methods used for Web Mining are NOT limited to

machine learning Oops, there is a close relationship between web

mining and machine learning

12

Web Mining: The Agent Paradigm

User Interface Agents information retrieval agents, information

filtering agents, & personal assistant agents. Distributed Agents

distributed agents for knowledge discovery or data mining.

Problem solving by a group of agents Mobile Agents

13

Web Mining: The Agent Paradigm

Content-based approach The system searches for items that match based

on an analysis of the content using the user preferences.

Collaborative approach The system tries to find users with similar

interests Recommendations given based on what similar

users did

14

Outline


15

Web Mining Categories

Web Content Mining Discovering useful information from web

contents/data/documents.

Web Structure Mining Discovering the model underlying link structures (topology)

on the Web. E.g. discovering authorities and hubs

Web Usage Mining Make sense of data generated by surfers Usage data from logs, user profiles, user sessions, cookies,

user queries, bookmarks, mouse clicks and scrolls, etc.

16

Web Content Data Structure

Unstructured – free text Semi-structured – HTML More structured – Table or Database

generated HTML pages Multimedia data – receive less attention than

text or hypertext

17

Outline


18

Web Content Mining: IR View

Unstructured Documents Bag of words, or phrase-based feature

representation Features can be boolean or frequency based Features can be reduced using different feature

selection techniques Word stemming, combining morphological

variations into one feature

19

Web Content Mining: IR View

Semi-Structured Documents Uses richer representations for features, based on

information from the document structure (typically HTML and hyperlinks)

Uses common data mining methods (whereas unstructured might use more text mining methods)

20

Web Content Mining: DB View

Tries to infer the structure of a Web site or transform a Web site to become a database Better information management Better querying on the Web

Can be achieved by: Finding the schema of Web documents Building a Web warehouse Building a Web knowledge base Building a virtual database

21

Web Content Mining: DB View

Mainly uses the Object Exchange Model (OEM) Represents semi-structured data (some structure, no rigid

schema) by a labeled graph

Process typically starts with manual selection of Web sites for content mining

Main application: building a structural summary of semi-structured data (schema extraction or discovery)

22

Outline


23

Web Structure Mining

Interested in the structure between Web documents (not within a document)

Inspired by the study of social networks and citation analysis

Example: PageRank – Google Application: Discovering micro-communities in the

Web Measuring the “completeness” of a Web site

24

Outline


25

Web Usage Mining

Tries to predict user behavior from interaction with the Web

Wide range of data (logs) Web client data Proxy server data Web server data

Two common approaches Map usage data into relational tables before using

adapted data mining techniques Use log data directly by utilizing special pre-processing

techniques

26

Web Usage Mining

Typical problems: Distinguishing among unique users, server sessions, episodes, etc in the presence of caching and proxy servers

Often Usage Mining uses some background or domain knowledgeE.g. site topology, Web content, etc

27

Web Usage Mining

Two main categories: Learning a user profile (personalized)

Web users would be interested in techniques that learn their needs and preferences automatically

Learning user navigation patterns (impersonalized)Information providers would be interested in

techniques that improve the effectiveness of their Web site or biasing the users towards the goals of the site

28

Outline


29

Conclusions

The paper tried to resolve confusion with regards to the term Web Mining Differentiated from IR and IE

Suggest three Web mining categories Content, Structure, and Usage Mining

Briefly described approaches for the three categories

Explored connection with agent paradigm

30

Exam Question #1

Question: Outline the main characteristics of Web information.

Answer: Web information is huge, diverse, and dynamic.

31

Exam Question #2

Question: How data mining techniques can be used in Web information analysis? Give at least two examples. Classification: classification on server logs using

decision tree, Naïve-Bayes classifier to discover the profiles of users belonging to a particular class

Clustering: Clustering can be used to group users exhibiting similar browsing patterns.

Association Analysis: association analysis can be used to relate pages that are most often referenced together in a single server session.

32

Exam Question #3

Question: What are the three main areas of interest for Web mining?

Answer: (1) Web Content

(2) Web Structure

(3) Web Usage

33

Thank you!

And Raymond Kosala, Hendrik Blockeel

And Shan Huang!

34

WebMiningResearchASurvey Web Mining Research: A Survey Raymond Kosala and Hendrik Blockeel ACM SIGKDD, July 2000 Presented by Shan Huang, 4/24/2007 Revised.

Documents

definition web mining

approaches web mining

web mining methods

web mining task

general web

web data

specific web sites

web search service