Spj110509

Mining the Web

Alex Johnsonmsnbc.com

http://malexjohnson.com@malexjohnson

May 9, 2011Downloadable at: http://malexj.tk/kY

Hidden Google

Hidden Google

• Advanced Search

http://www.google.com/advanced_search

Hidden Google

• Advanced Search

http://www.google.com/advanced_search• Realtime Search

http://www.google.com/realtime

Hidden Google

• Advanced Search

http://www.google.com/advanced_search• Realtime Search

http://www.google.com/realtime• Book search

http://books.google.com

Hidden Google

• Advanced Search

http://www.google.com/advanced_search• Realtime Search

http://www.google.com/realtime• Book search

http://books.google.com• Academic Search

http://scholar.google.com

Building Better Searches

Building Better Searches

Expand or restrict searches

Building Better Searches

Expand or restrict searches•+example •-example

Building Better Searches

Expand or restrict searches•+example •-example•“exact phrase”

Building Better Searches

Expand or restrict searches•+example •-example•“exact phrase”•site:

Building Better Searches

Expand or restrict searches•+example •-example•“exact phrase”•site:•combine these switches

Building Better Searches

Expand or restrict searches•+example •-example•“exact phrase”•site:•combine these switches•cache:

Building Better Searches

Expand or restrict searches•+example •-example•“exact phrase”•site:•combine these switches•cache:example.com/example•Advance operators guide:

http://www.googleguide.com/advanced_operators.html

The government is your friend

The government is your friend

• FactFinder

http://factfinder2.census.gov

The government is your friend

• FactFinder

http://factfinder2.census.gov• State & County QuickFacts

http://quickfacts.census.gov

The government is your friend

• FactFinder

http://factfinder2.census.gov• State & County QuickFacts

http://quickfacts.census.gov• FedStats

http://www.fedstats.gov

The government is your friend

• FactFinder

http://factfinder2.census.gov• State & County QuickFacts

http://quickfacts.census.gov/• FedStats

http://www.fedstats.gov• GovSpot

http://govspot.com

The government is your friend

• FactFinderhttp://factfinder.census.gov• State & County QuickFactshttp://quickfacts.census.gov• FedStatshttp://www.fedstats.gov• GovSpothttp://govspot.com

• Government Printing Officehttp://www.gpoaccess.gov/multidb.html• Transactional Records (TRAC)http://trac.syr.edu• FBI, Uniform Crime Reportshttp://www.fbi.gov/ucr/ucr.htm• Criminal Justice Statisticshttp://www.albany.edu/sourcebook

When the government stiffs you

When the government stiffs you

FOIA letter generator

http://www.splc.org/foiletter.asp

Where to find other stuff

Where to find other stuff

Businesses, nonprofits• GuideStar

http://www2.guidestar.org/AdvancedSearch.aspx• SEC Filings

http://www.sec.gov/edgar.shtml

Where to find other stuff

Businesses, nonprofits• GuideStar

http://www2.guidestar.org/AdvancedSearch.aspx• SEC Filings

http://www.sec.gov/edgar.shtml

Politics• OpenSecrets.org

http://www.opensecrets.org• PollingReport

http://www.pollingreport.com

Where to find other stuff

Businesses, nonprofits• GuideStar

http://www.guidestar.org/• SEC Filings

http://www.sec.gov/edgar.shtml

Politics• OpenSecrets.org

http://www.opensecrets.org/• PollingReport

http://www.pollingreport.com/

People• PeopleSpot

http://www.peoplespot.com• Public Records

http://www.searchsystems.net

http://www.vitalrec.com

http://www.pipl.com

Starting points

Starting points

Journalism specific

• IRE Beat Resource Guide

http://www.ire.org/resourcecenter/initial-search-beat.html

Starting points

Journalism specific

• IRE Beat Resource Guidehttp://www.ire.org/resourcecenter/initial-search-beat.html• SPJ Journalist’s Toolboxhttp://www.journaliststoolbox.org• Journalists’ Quick Reference Pagehttp://malexj.tk/kB

Starting points

Journalism specific

• IRE Beat Resource Guidehttp://www.ire.org/

resourcecenter/initial-search-beat.html

• Assignment Editorhttp://www.assignmenteditor.com• SPJ Journalist’s Toolboxhttp://www.journaliststoolbox.org• Journalists’ Quick Referencehttp://malexj.tk/kB

General portals

• LibrarySpothttp://www.libraryspot.com• The Reference Deskhttp://www.martindalecenter.com• Scholarly Resource

Collections (UC-Riverside)http://infomine.ucr.edu• Profnethttp://www.profnet.com• Help A Reporter Outhttp://www.helpareporter.com

The Hidden Web

• Much of the information on the Web is hidden from search engines. How do you get hold of it?

The Hidden Web

• Much of the information on the Web is hidden from search engines. How do you get hold of it?

• What we’re talking about is databases.

The Hidden Web

• Much of the information on the Web is hidden from search engines. How do you get hold of it?

• What we’re talking about is databases.

National Climatic Data Center (NOAA)

204 million (380 terabytes as of April 2011, and growing at 80 TB a year)

The Hidden Web

• Google

Add database to your query

The Hidden Web

• Google

Add database to your query

Add filetype:xls to your query

The Hidden Web

• Google

Add database to your query

Add xls to your query

• New York Public Library Databases

http://www.nypl.org/collections/articles-databases

The Hidden Web

• Google

Add database to your query

• New York Public Library Databases

http://www.nypl.org/collections/articles-databases

• IncyWincy

http://www.incywincy.com/search-engine/web/web?adv=1

The Hidden Web

• Google

Add database to your query

• New York Public Library Databases

http://www.nypl.org/collections/articles-databases

• IncyWincy

http://www.incywincy.com

• Europe — Europa

http://europa.eu/documentation/index_en.htm

This presentation can be downloaded at: http://malexj.tk/kY