Web Harvesting Ppt

Web Harvest

Web Harvesting RAHUL.MADANU09BK1A0535CSE-A

1

W

Introduction -

• A Web crawler is an Internet bot that systematically browses the World Wide Web, typically for the purpose of Web indexing.

• A web search engine is software code that is designed to search for information on the World Wide Web.

• A database is an organized collection of data.

Web Harvest 2

Web Harvest 3

Existing System

Q: How does a search engine know that all

these pages contain the query terms?

A: Because all of those pages have been crawled

Web Harvest 4

Motivation for crawlers

Support universal search engines (Google, Yahoo, MSN/Windows Live, Ask, etc.)

Vertical (specialized) search engines, e.g. news, shopping, papers, recipes, reviews, etc.

Business intelligence: keep track of potential competitors, partners

Monitor Web sites of interestEvil: harvest emails for spamming, phishing…… Can you think of some others?…

Web Harvest 5

Many names• Crawler

• Spider

• Robot (or bot)

• Web agent

• Wanderer, worm, …

• And famous instances: googlebot, scooter, slurp, msnbot, …

Web Harvest 6

A crawler within a search engine

Web Harvest 7

Web

Text index PageRank

Page repository

googlebot

Text & link analysisQuery

hits

Ranker

Page Rank

Web Harvest 8

Page Rank Probability -

Web Harvest 9

Proposed System

Web Harvest 10

Aim :

Have to set higher memory range.Eliminate all file not found.

Removing negative dictionary.Need to obtain Base URL.

Multi-processing or multi-threading.

Web Harvest 11

Recovering Issues -• Don’t want to fetch same page twice or Save up in Marked list.

• Soft fail for timeout, server not responding, file not found, and other errors.• Noise words that do not carry meaning should be eliminated

(“stopped”) before they are indexed • E.g. in English: AND, THE, A, AT, OR, ON, FOR, etc…• Need to obtain Base URL from HTTP header

• Base: http://www.cnn.com/linkto/

• Relative URL: intl.html• Absolute URL: http://www.cnn.com/linkto/intl.html

• Overlap the above delays by fetching many pages concurrently• Can use multi-processing or multi-threading

Web Harvest 12

Policy -

• Coverage

• Freshness

• Trade-off (Subjective)

Web Harvest 13

Algorithm - • Algorithm for classifying a crawler data into the

database

Bayesian approaches are a fundamentally important DM technique. Given the probability distribution, Bayes classifier can provably achieve the optimal result.

Bayes Classifier is that it assumes all attributes are independent of each other.

Web Harvest 14

Basic crawlers• This is a sequential crawler

• Seeds can be any list of starting URLs

• Order of page visits is determined by frontier data structure

• Stop criterion can be anything

Graph traversal (BFS or DFS?)

• Breadth First Search• Implemented with QUEUE (FIFO) • Finds pages along shortest paths

• If we start with “good” pages, this keeps us close; maybe other good stuff…

• Depth First Search• Implemented with STACK (LIFO)

• Wander away (“lost in cyberspace”)

Web Harvest 16

Breadth-first crawlers• BF crawler tends to

crawl high-PageRank pages very early

• Therefore, BF crawler is a good baseline to gauge other crawlers

Average Number of Pages Crawled

Crawler ethics and conflicts• Crawlers can cause trouble, even unwillingly, if not

properly designed to be “polite” and “ethical”

• lexical analysis is the process of converting a sequence of characters into a sequence of tokens.

http://foo.com/woo/foo/woo/foo/woo

• For example, sending too many requests in rapid succession to a single server can amount to a Denial of Service (DoS) attack!• Server administrator and users will be upset• Crawler developer/admin IP address may be blacklisted

Web Harvest 18

Crawler etiquette (important!)

Spread the load, do not overwhelm a server• Make sure that no more than some max. number of requests to any single

server per unit time, say < 1/second

Honor the Robot Exclusion Protocol• A server can specify which parts of its document tree any crawler is or is

not allowed to crawl by a file named ‘robots.txt’ placed in the HTTP root directory, e.g. http://www.indiana.edu/robots.txt

• Crawler should always check, parse, and obey this file before sending any requests to a server

• More info at:• http://www.google.com/robots.txt• http://www.robotstxt.org/wc/exclusion.html

Web Harvest 19

A Basic crawler in Javaimport java.io.BufferedReader;

import java.io.InputStreamReader;

import java.net.URL;

public class Main {

public static void main(String[] args) {

try {

URL my_url = new URL("http://www.blogspot.com/");

BufferedReader br = new BufferedReader(new InputStreamReader(my_url.openStream()));

String strTemp = "";

while(null != (strTemp = br.readLine())){

System.out.println(strTemp);

} } catch (Exception ex) { ex.printStackTrace(); } } }

Web Harvest 20

Xml (Used) -

• Extensible Markup Language (XML) is a markup language that defines a set of rules for encoding documents in a format that is both human-readable and machine-readable.

• Many application programming interfaces (APIs) have been developed to aid software developers with processing XML data, and several schema systems exist to aid in the definition of XML-based languages.

• Any forms of Database.

Web Harvest 21

Comparison -

Fields Google Web Harvest Time out Accepted Eliminated N Dictionary Accepted Eliminated Dynamic Pages Doubled Updated URL Relative Base Table Big Table Bayes Table Search Page Rank Limit Rank

Web Harvest 22

Web Harvest 23

Conclusion -

• Web Harvesting Engine Marketing has one of the lowest costs per customer acquisition.

• Web Harvesting Engine is one of the most cost efficient ways to reach a target market for a small, medium or large business.

• Traditional marketing such as catalog mail, trade magazines, direct mail, TV or radio involves passive participation by your audience and targeting can very greatly from one medium to another.

Web Harvest 24

Queries ?

Web Harvest 25