Databases Computer Security Software Engineering Computer Graphics Networking Distributed Systems Web Search Engines: echnologies, Applications, and Opportunities Torsten Suel Associate Professor CSE Department Polytechnic Institute of NYU [email protected]
42
Embed
Databases Computer Security Software Engineering Computer Graphics Networking Distributed Systems Web Search Engines: Technologies, Applications, and Opportunities.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Databases
Computer SecuritySoftware Engineering
Computer Graphics
Networking
Distributed SystemsWeb Search Engines:
Technologies, Applications, and Opportunities
Torsten SuelAssociate ProfessorCSE DepartmentPolytechnic Institute of [email protected]
What this talk is about:
- web search engines - how they work
- underlying technologies
- applications and impact
- what is next?
- opportunities and education
The Basics: • type in some words• get back results (usually 10 at a time)
• hopefully good results among these• if not, change your query, try again
The Basics: • type in some words• get back results (usually 10 at a time)
• hopefully good results among these• if not, change your query, try again
• as the web grew, we needed a way to find sites
• has grown into many-billion-$ industry at heart of the web
• search engines: largest supercomputers of the world
• 100’s of millions of queries per day, 0.1 sec latency/query
• evaluated over tens of billions of documents
• the majors: Google, MSFT Bing, Yahoo!(?), Baidu, Yandex
• but many other businesses use similar technologies, or rely on search engines for their business
Web Search Technology
• 1. Search engines in a nutshell: - How does the web work?
- How do search engines work?
- Basic search architecture
- Historical background
• 2. Technical challenges and opportunities: - link analysis
- computational advertising
•3. Education and opportunities - the search landscape
- search engines and the curriculum
Overview of this Lecture:
1. Search Engines in a Nutshell
The Web:
text …
A lot of text …
>100 billionpages of text
and other stuff …
• pages containing (fairly unstructured) text
• images, audio, etc. embedded in (hanging off) pages
• structure defined using HTML (Hypertext Markup Language)
• hyperlinks between pages!
• over 100 billion pages
• over 3 trillion hyperlinks
a giant graph!
What is the web? (another view)
• pages reside in servers
• sites often contain related pages
• site/host structure
• local versus global links
How the web is organized: site structure
Web Server (Host)
Web Server (Host)
Web Server (Host)
www.poly.edu
www.cnn.com
www.irs.gov
How Browsing Works
Desktop(with browser)
give me the file “/world/index.html”
here is the file: “...”
Web Server
www.cnn.com
Fetching “www.cnn.com/world/index.html”
HTTP:
desktop or crawler
web server
GET /world/index.html HTTP/1.0User-Agent: Mozilla/3.0 (Windows 95/NT)Host: www.cnn.comFrom: …Referer: …If-Modified-Since: ...
• crawler: also called spider, web robot• fetches pages from the web• starts at set of “seed pages”• parses fetched pages for hyperlinks• then follows those links (e.g., BFS)
• until all pages fetches (i.e., never)
Data Mining:
• data fetched by the crawler is analyzed
• many different tasks: - link analysis (later) - detection of spam pages and dangerous pages - analyzing data about past queries (clicks etc.) - data extraction (products, people, locations, …)
• Citation Analysis and Social Network Analysis• Microfilm rapid selectors: (e.g., E. Goldberg 1931)
• Memex (Vannevar Bush, 1939/45)
IR Before 1945:
• “As We May Think”, Atlantic Monthly, 1945 (mostly written 1939)
Memex: Vannevar Bush (1890-1974)
• Querying documents by keywords• Classifying documents by topic
• Hypertext and analyzing links between documents
• Library catalogs and digital libraries
• Searching a collection of news or medical articles
• National security: - analyzing communication streams, financial networks
• Most widely used application: web search!
IR Techniques and Applications:
2. Technical Challenges and Opportunities:
• What is a graph? - nodes and edges - edges directed or undirected
• Graphs are used to model many scenarios - social relationships: who knows whom, who is friends with whom?
- citations in literature: which physicist cites which other physicist?
- email, telephone: who communicates with whom?
- follow the money: who gives money to whom?
- the web: who links to whom?
Graphs and Social Networks
• nodes are researchers• connected by edge if one cites the other
• From: University of Cottbus
Example: Scientific Literature
• nodes are employees• connected by edge if they exchanged more than 5 emails
• From: Shetty/Adibi (USC)
Example: Enron Email Network
• search engines: use hyperlink graph to improve ranking
• national security: who calls whom and what does it mean?
• social sciences: understanding societies
Social Networks: Why do we care?
• Basic idea: exploits judgments by millions of web pages
• A page that is highly referenced should be better or more important
• Pagerank (Brin&Page at Google)
“significance of a page depends on significance of those referencing it”
• s(a) = s(b)/2 + s(c)/3 + s(d)/1
• System of equations
• Unique solution under some assumptions
Link-Based Ranking Techniques
• initialize the rank value of each node to 1/n (0.2 for 5 nodes)
• a node with k outgoing links transmits a 1/k fraction of its current rank value over that edge to its neighbor
• iterate this process many times until it converges
• NOTE: this is a random walk on the link graph
• Pagerank: stationary distribution of this random walk
1/21/2
1/21/2
1/2
1/2 1
1
Pagerank
0.2
0.2
0.2
0.2
0.2
1/21/2
1/21/2
1/2
1/2 1
1 0.2
0.2
0.2
0.2
0.2
1/21/2
1/21/2
1/2
1/2 1
1 0.1
0.2
0.2
0.3
0.2
1/21/2
1/21/2
1/2
1/2 1
1 0.1
0.3
0.15
0.25
0.2
1/21/2
1/21/2
1/2
1/2 1
1 0.143
0.286
0.143
0.286
0.143
..
(1) (2)
(3) (n)
• stationary distribution: vector x with xA = x• A is primitive, and x Eigenvector of A
• computed using Jacobi or Gauss Seidel iteration
1/21/2
1/21/2
1/2
1/2 1
1
Matrix notation
0.2
0.2
0.2
0.2
0.2
0 1/2 0 0 1/2
0 0 1/2 0 1/2
0 0 0 1 0
0 0 1/2 0 1/2
1 0 0 0 0
A
• Now every engine is using link-based ranking
• But the idea is much older!
• Who is the most important physicist? - citation analysis, 1960s
• Who is the most important person in town? - social network analysis, 1950s
• Who is the most important person on Facebook? - the most friends? - or the most important friends? (recursive) - or the best friends? …. and on Twitter ?
Computational Advertising:
• New field dealing with mathematical techniques for targeting ads in electronic media• a multi-billion $ business• economic foundation of search engines and web• many startups in New York City
• ITV: TV is moving to the internet
• how to match ads with the right consumers
• scary privacy implications
Comp. Adv. Basics
• Search ads: matching ads to queries - ads on top and right hand side of search results
- based on query and past behavior (cookies etc)
- pay-per-click model with bids by advertisers
- or pay-per-action: goal is immediate action by user
• Display ads: large, shiny ads for brands - banner ads on major sites for cars, movies, etc.
- per-par-display: e.g., $100 per million impressions
- sold by contract: e.g., ads for upcoming movies
• AdSense (Google): text ads on 3rd-party pages - placing ads on pages based on content and user
- Google sharing money with site owner
- danger of manipulation by site owner
Comp. Adv. Marketplace
• Very complicated zoo of companies and roles - ad networks - ad campaign coordination/optimization - companies providing user data - arbitrage & manipulation - hundreds/thousands of companies - real-time auctions in tens of millisecs
• Emerging ads scenarios - monetizing social networks (facebook, linkedIn etc.) - mobile ads and ads in app-space (e.g., flurry) - ITV and internet radio: ads not a broadcast (hulu) - ads in games and virtual worlds
• Privacy: looking bad at the moment - (almost) everything is for sale …
• search engine positions is $$$• web pages are cheap• make lots of automatic junk …• use ads to make $ (or $$$)
Adversarial Information Retrieval
3. Education and Opportunities:
• Web search at center of the online world• interesting technical challenges• many professional opportunities• NYC is a center of this industry• Google, plus media, ads, e-commerce, mobile
• what qualifications are needed?
Perspective:
algorithms
systemsinformation retrieval
databases
machine learning
natural languageprocessin
g
AI
library &information
science
websearch
• Search is somewhat Computer Science oriented
• … it’s all software (mostly)
• also, media/design exp. becoming important
• courses/topics to learn: - algorithms
- distributed systems
- databases
- web search & information retrieval
- machine learning, data mining, and statistics
- natural language processing
In the curriculum
Course Offering & Student Activities
• CS6913: Web Search Engines (Spring Sem.)
- web and search engine architecture how does it all work?
- working with massive data sets storing and analyzing terabytes
- introduction to Information Retrieval unstructured data, text
- system building skills building distributed systems
- the Web as a social network adversarial behavior, spam, communities
• Course Objectives
• Student Activities
- course projects build your own (small-scale) search engine
- independent research projects, theses, assistantships