AWS Webcast - Location Based Search

Building Location-Based Search with

Amazon CloudSearch

Tom Hill April 3, 2013

Agenda

!   What is CloudSearch !   What is Location-Based search? !   Computing distance !   Location-Based Search in CloudSearch !   Sample App

Housekeeping

!   Type questions into GoToMeeting window •  We'll get to them at the end.

!   Recording will be on the AWS YouTube channel !   Slides will be on Slideshare.net

•  A link to slides will be mailed to all participants

!   You can maximize the shared screen's window

What is CloudSearch?

What is CloudSearch

!   Amazon CloudSearch is a fully-managed search service !   Easy to add search functionality to your application !   Fast and highly scalable !   Supports search features like

•  Faceting •  Synonyms, stopwords •  Ranking

How is CloudSearch Used

!   Create/Configure a "search domain" !   Post documents via HTTP !   Search via HTTP

•  results as XML or JSON

!   Scales automatically

http://search-‐DOMAIN-‐XYZ.REGION.cloudsearch.amazonaws.com/2011-‐02-‐01/search?q=cat

More Information

!   Webinar •  'Getting Started With Amazon CloudSearch"

!   April 17, 9:00am PT / 12:00pm ET !   Register at http://bit.ly/11WZRAZ

What is Location Based Search?

What is Location-Based Search?

!   Using location & distance as factors in search

Why Do You Care?

!   People care about things near them. •  Pizza, Classified Ads, etc. •  Find a Doctor, Lawyer,…

!   Mobile is a key driver

How do we use distance?

!   Limit to an Area •  Box •  Circle

!   Sort by Distance !   Include distance in ranking

•  combine text relevance and distance

Computing Distance

!   Many Formulas •  Rectangular distance •  Equi-rectangular projection •  Spherical Law of Cosines •  Haversine Formula

!   Speed Vs. Accuracy •  Speed: Rectangular distance •  Accuracy: Haversine Formula

Why So Many Ways to Compute Distance?

The earth isn't flat! It isn't a sphere either.

Is the Earth Flat?

!   If it's flat

!   If it's not flat

(x − x2)2 + (y− y2)2

2r× arcsin sin2 φ2 −φ12

'(+ cos(φ1)cos(φ2 )sin

2 λ2 −λ12

How Much Difference?

!   Distances between some cities •  Fort Lauderdale, FL to New Haven, CT

•  Haversine: 1813 •  Rectangular: 1872 •  Difference: 3%

•  Bangor, ME to Adak, AK •  Haversine: 7244 •  Rectangular: 12016 •  Difference: 66%

Difference in Detail

Haversine Cosines EuqiRect Rect - CosErr EquErr RecErr 1812.54997 1812.54997 1814.38516 1871.77660 - 0.000 0.001 0.033 Fort Lauderdale, FL to New Haven, CT7244.45661 7244.45661 8008.74698 12015.96646 - 0.000 0.106 0.659 Bangor, ME to Adak,AK

!   Four computations •  Rectangular Distance •  EquiRectangular Projection •  Spherical Law of Cosines •  Haversine

Same Close Not so close

How Much Accuracy?

!   "Pizza, 1 Mile" ! Haversine is more accurate

•  If you are a bird

!   Any distance is approximate

Which one is the Right Choice?

!   As usual: "It depends" !   Factors

•  Desired query speed •  Index size •  Accuracy needed

!   Start with Equirectangular Projection •  Test

Location-Based Search in CloudSearch

How to Compute Distance in CloudSearch?

!   Rank Expressions •  Computations run for each matching document

!   Can be used for •  Sorting •  Influencing Scoring

Two Types of Rank Expressions

!   Static Rank Expressions •  Computation based on values in index

!   Query Time Rank Expressions •  Allow including parameters at run time.

•  e.g. latitude, longitude.

Converting to Rank Expression

!   Rank expressions have JavaScript-like syntax !   Most math functions

•  log2, log10,sin, cos, atan, min, max, etc.

!   So this distance formula:

!   Becomes •  sqrt(pow(lat-userlat),2)+pow(lon-userlon),2))

(x − x2)2 + (y− y2)2

Query Time Rank Expressions

!   Define Rank Expression •  &rank-NAME=EXPRESSION •  &rank-geo=sqrt(pow(lat-userlat),2)+pow(lon-userlon),2))

!   Select Rank Expression •  &rank=NAME •  &rank=geo

http://searchendpoint?q=creek&rank=geo&rank-geo=sqrt(pow(la1-123),2)+pow(lo1-456),2))

Influencing Scoring

!   Include both text relevance and distance:

&rank-geo=(1000-text_relevance) + sqrt(pow(la1-ulat),2)+pow(lo1-ulon),2))

!   Relevance: Higher is better !   Distance: Lower is better

Combining Relevance and Distance

!   Which is more important distance or relevance? •  By how much?

!   Relative Weight •  N * text_relevance + M * distance

!   That's where the art comes in •  Will vary by your application •  Test & tune, and test again •  Rank expression comparator

Compare Rank Expressions

Storing Data

!   CloudSearch supports unsigned integers !   Have to convert latitude, longitude to positive ranges

•  latitude + 90 •  longitude + 180

!   Have to store as integers; need to scale •  latitude = (latitude + 90) * 1000 •  longitude= (longitude+ 180) * 1000

Performance

!   Don't query the whole world •  Can limit by literals or numeric fields. •  Literals are more efficient for limits.

!   Limit Options •  Literal

•  &bq=state:'CA' •  &bq=zip:'94402'

•  Numeric •  &bq=(and latitude:40..50 longitude:80..85)

Performance Measures

GeoMethod TextRel Limits Queries Seconds QTimeMS Threads CompletedQ AveHits NONE false 10 6.2255 622 1 10 8345450.00 CARTESIAN false 10 15.6064 1560 1 10 8345450.00 EQUI false 10 19.7106 1971 1 10 8345450.00 COSINES false 10 27.4968 2749 1 10 8345450.00 HAVERSINE false 10 31.2595 3125 1 10 8345450.00 NONE false Numeric 10 9.1758 917 1 10 3807.00 CARTESIAN false Numeric 10 9.0255 902 1 10 3807.00 EQUI false Numeric 10 9.1158 911 1 10 3807.00 COSINES false Numeric 10 9.8321 983 1 10 3807.00 HAVERSINE false Numeric 10 9.1272 912 1 10 3807.00 NONE false literal 10 0.8254 82 1 10 3781.00 CARTESIAN false literal 10 0.5936 59 1 10 3781.00 EQUI false literal 10 0.6173 61 1 10 3781.00 COSINES false literal 10 0.5916 59 1 10 3781.00 HAVERSINE false literal 10 0.6289 62 1 10 3781.00

Why you don't want query the whole world!

Geo-Spatial Demo Application

Demo Application Structure

HTML Page

Javascript Server (Tomcat)

CloudSearch Ajax

Demo Implementation

!   JavaScript •  Ajax •  JQuery •  Google Maps API

!   Tomcat Server •  Java •  Just for forwarding of requests

•  Because XSS, that's why.

Querying CloudSearch $.ajax({

url : "searchx", data : { 'q' : currentQuery, 'domain' : "geoname25", 'return-‐fields' : returnFields.join(), "rank" : "geo", "rank-‐geo" : "Math.sqrt(Math.pow(Math.abs(doc.latitude_90-‐ 12539),2) +

Math.pow(Math.abs(doc.longitude_180-‐ 5784),2))" }, dataType : "json", success : function(data) { var hits = data['hits']; displaySearchResults(hits['hit'], hits['found']); populateMap(hits['hit']); }

Getting Latitude & Longitude

!   What if you don't know the latitude & longitude? ! GeoCoding Services

•  Many services •  Some free tiers •  May have restrictions

•  Google, Bing, Yahoo, MapQuest, ArcGis, …

Wrap Up

!   Local search is a component of many applications !   CloudSearch supports local search

•  Using rank expressions

! GeoCoding services

Thanks for Coming!

!   Data •  http://www.geonames.org/export/

!  Slides •  Slideshare.net soon. We'll send you a link.

!   Sample Code •  Talk to me. (tomhill@amazon.com)

!   Computations •  http://www.movable-type.co.uk/scripts/latlong.html •  wikipedia

Any Questions?

Thanks for Coming!

Use Zip Code, City, State as a proxy

!   Multiple ways to select, map, zip, current location

Radians vs Degrees

!   Should note somewhere that all but rectangular distance require radians.

Querying Data !   Java/Javascript

•  Fields are latitude_90, longitude_180 •  user location is userlat, userlon

!   simple distance

rank = "Math.sqrt(Math.pow(Math.abs(latitude_90-‐(" + userlat + ")),2)+Math.pow(Math.abs(longitude_180-‐(" + userlon + ")),2))";

!   Spherical Law of Cosines

rank = "6371*Math.acos(Math.sin(" + userlat + ") * Math.sin(lat_rad/" + scale + ") + Math.cos(" + userlat + ") * Math.cos(lat_rad/" + scale + ") * Math.cos((lon_rad/" + scale + ") - " + userlon + ") )";

What CloudSearch Doesn't Do

Issues with assuming the earth is flat.

AWS Webcast - Location Based Search

Technology

AWS Webcast - Amazon EC2 Masterclass

AWS Webcast - Library Systems on the AWS Cloud

AWS Webcast - Sumo Logic

AWS Webcast - AWS OpsWorks Continuous Integration Demo

AWS Webcast - MS Exchange Implementation & Planning on AWS

AWS Webcast - Explore the AWS Cloud for Government

AWS Webcast - Neudesic Data Centermigrationtoaws

AWS Webcast - Running Windows Server on the AWS Cloud

Aws webcast - Scaling on AWS 13 08-20

AWS Webcast - Understanding the AWS Security Model

AWS Webcast - Managing Big Data in the AWS Cloud_20140924

AWS Webcast - AWS 101 - Journey to the AWS Cloud:...

AWS Webcast - Understanding database options

AWS Webcast - Introducing Amazon Redshift

AWS Webcast - Highly Available SQL Server on AWS

AWS Webcast - Explore the AWS Cloud