May 16, 2015
.
Problem Statement.Big Problem is.....
.
... that it is hard to come by realistic road maps for large-scale ITSsimulations.
.Small Problem is.....
.
... that even with Google Maps, quotas make it difficult to create large datasets
alone -- need social scaling (Google Maps quota: 2.5k reqs/day)
• EV battery replacement infrastructure simulations 03
• supply chain research -- again, battery replacement infrastructures
• autodrive simulation? -- in real life by around 2020?
• ... anyway who needs realistic road maps
03 myself+1 "...EV Use in Environments with V2H and Battery Replacement..." ITS Tokyo (2013)
M.Zhanikeev -- [email protected] -- Maps2Graphs: Social GIS Datasets Based on Google Maps API -- http://tinyurl.com/marat131204a --- 2/22...
2/22
.
Traditional GIS
M.Zhanikeev -- [email protected] -- Maps2Graphs: Social GIS Datasets Based on Google Maps API -- http://tinyurl.com/marat131204a --- 3/22...
3/22
.
GIS: The Shapefile• default in GIS datasets today
• a non-GPS = special format -- see ArcGIS manual 11• precision problems -- .SHP is all about relative shapes, not coordinates
11 Esri "ArcGIS Japan" http://www.esrij.com/products/arcgis (2013)
M.Zhanikeev -- [email protected] -- Maps2Graphs: Social GIS Datasets Based on Google Maps API -- http://tinyurl.com/marat131204a --- 4/22...
4/22
.
ArcGIS versus Google Maps• ArcGIS is
◦ paid◦ needsdatasets to work◦ address2location and placename2location are restricted by
quotas (credit)
◦ no APIs -- just plugins for PowerPoint and smartphone apps
• GoogleMaps is
◦ free with quotas, pay9$/m for nearly infinite quotas◦ all data is in one place
◦ many APIs -- Location, Direction, Renderer, ...
◦ address2location and placename2location are seamless -- no need tocare which you use
M.Zhanikeev -- [email protected] -- Maps2Graphs: Social GIS Datasets Based on Google Maps API -- http://tinyurl.com/marat131204a --- 5/22...
5/22
.
Google Maps : Route Datatype
M.Zhanikeev -- [email protected] -- Maps2Graphs: Social GIS Datasets Based on Google Maps API -- http://tinyurl.com/marat131204a --- 6/22...
6/22
.
Objectives
M.Zhanikeev -- [email protected] -- Maps2Graphs: Social GIS Datasets Based on Google Maps API -- http://tinyurl.com/marat131204a --- 7/22...
7/22
.
Objectives
• anwebapp which would create a graph -- all-to-all roadmap -- from alist of locations on top of Google Maps
• application has to be cloud based -- like Google Drive, Dropbox, etc... todistribute work socially
◦ terminology: serviceless app is a webapp that runs without a webserver
◦ otherwise need support from Google itself -- which is unlikely
• application has to scale socially -- full range from 1 to 10k collaborators
• rapid generation of datasets◦ example: need graph of 1k locations by tomorrow!
M.Zhanikeev -- [email protected] -- Maps2Graphs: Social GIS Datasets Based on Google Maps API -- http://tinyurl.com/marat131204a --- 8/22...
8/22
.
Definitions
• socially scalable: same design for any number of collaborators
• serverless webapp:1. a webapp that runs from a cloud drive2. uses cloud drive to store data3. can be shared by many collaborators
• dataset: a graph (nodes and links) of routes among many locations
1. easily merged from many pairwise routes
M.Zhanikeev -- [email protected] -- Maps2Graphs: Social GIS Datasets Based on Google Maps API -- http://tinyurl.com/marat131204a --- 9/22...
9/22
.
maps2graphs: design
M.Zhanikeev -- [email protected] -- Maps2Graphs: Social GIS Datasets Based on Google Maps API -- http://tinyurl.com/marat131204a --- 10/22...
10/22
.
Basic Idea
Full RESTful APIUser
URL-based API
Web
app
Dat
a
M.Zhanikeev -- [email protected] -- Maps2Graphs: Social GIS Datasets Based on Google Maps API -- http://tinyurl.com/marat131204a --- 11/22...
11/22
.
More Detail
Cloud Drive API
Collaborator
Owner
Serverless WebappUser-Generated Content
Content API
Mas
ter c
opy
Slav
e co
py
(syn
c)
M.Zhanikeev -- [email protected] -- Maps2Graphs: Social GIS Datasets Based on Google Maps API -- http://tinyurl.com/marat131204a --- 12/22...
12/22
.
maps2graphs: Datatypes
v11 v11 v11 v11 v11
v21 v21 v21 v21
v31 v31 v31
v41 v41
hA2A matrix of 5
Cloud Drive
becomes...0100?001001?01task.status.txt
task.list.json[“address 1”,“address 2”,“address 3”,….
]
base64( JSON as string)….
[collaborator].[time].[random tail]
…M.Zhanikeev -- [email protected] -- Maps2Graphs: Social GIS Datasets Based on Google Maps API -- http://tinyurl.com/marat131204a --- 13/22
...
13/22
.
maps2graphs: Batch
Collaborator Cloud Drive API Content API
Newbatch
Randomize
new file (chunk)
task.list.json
task.status.txt
vxy
write chunk(one line)
finalize file (set name)
read/fill/updatetask.status.txt
• specific forDropbox --Dropbox JS API in 01
• randomization is important tominimize overlaps with othercollaborators
• atomic: status string isupdated task.status.txt is updatedonly when the batch file issuccessfully closed
• continuous: batch bybatch
01 myself "Maps2Graphs Project Page" https://github.com/maratishe/maps2graphs (2013)
M.Zhanikeev -- [email protected] -- Maps2Graphs: Social GIS Datasets Based on Google Maps API -- http://tinyurl.com/marat131204a --- 14/22...
14/22
.
maps2graphs: social tricks
M.Zhanikeev -- [email protected] -- Maps2Graphs: Social GIS Datasets Based on Google Maps API -- http://tinyurl.com/marat131204a --- 15/22...
15/22
.
Social Features
• maps2graphs is a crowdsourcing effort
• much literature on crowdsourcing 04 05 06 07
• important feature is distribution: most people contribute very little, fewcontribute a lot
• translation for maps2graphs: most people will use the app for a fewminutes, few will run is for days -- just leave it running in the browser
04 G.Chatzimilioudis+3 "Crowdsourcing with Smartphones" IEEE J. Internet Computing (2012)
05 B.Yoo+2 "Travel Time Estimation Using Mobile Data" J. Transportation Studies (2005)
06 C.Wu+3 "Crowdsourcing Multimedia QoE Evaluation..." IEEE Trans. on Multimedia (2013)
07 D.Sui+2 "Crowdsourcing Geographic Knowledge: Volunteered Geographic Information (VGI)..." Springer Book (2013)
M.Zhanikeev -- [email protected] -- Maps2Graphs: Social GIS Datasets Based on Google Maps API -- http://tinyurl.com/marat131204a --- 16/22...
16/22
.
Actual Stats (later example)
0
70
140
210
Batc
h siz
e
Clie
nt 1
0
70
140
210
Batc
h siz
e
Clie
nt 2
0
70
140
210
Batc
h siz
e
Clie
nt 3
0 20 40 60 80 100 120 140Hour (relative to project start)
0
70
140
210
Batc
h siz
e
Clie
nt 4
.Aggressive Approach..
.
... high intensity until out of quota,
then hibernate until next 24H cycle
• high intensity: 1 req every 6 seconds
• quota: 2500 req/day per IP
• out of quota roughly in 3-5 hours
• stay slightly below quota for personal use
M.Zhanikeev -- [email protected] -- Maps2Graphs: Social GIS Datasets Based on Google Maps API -- http://tinyurl.com/marat131204a --- 17/22...
17/22
.
maps2graphs: implementation
M.Zhanikeev -- [email protected] -- Maps2Graphs: Social GIS Datasets Based on Google Maps API -- http://tinyurl.com/marat131204a --- 18/22...
18/22
.
Implementation
• maps2graphs: source code in 01
• working code at 02 --- you can run it right now!
◦ or just go tohttp://tinyurl.com/maps2graphs◦ 300 Family Marts in Northern Kyushu
01 myself "Maps2Graphs Project Page" https://github.com/maratishe/maps2graphs (2013)
02 myself "Maps2Graphs: Example Crowdsourcing Project" http://tinyurl.com/maps2graphs (2013)
M.Zhanikeev -- [email protected] -- Maps2Graphs: Social GIS Datasets Based on Google Maps API -- http://tinyurl.com/marat131204a --- 19/22...
19/22
.
Screenshot
• simple interface
• shows current volume (of current batchfile)
• shows details about current and pastroutes, quota status, etc.
• if you need to abort, press commit andgo
M.Zhanikeev -- [email protected] -- Maps2Graphs: Social GIS Datasets Based on Google Maps API -- http://tinyurl.com/marat131204a --- 20/22...
20/22
.
End Result
M.Zhanikeev -- [email protected] -- Maps2Graphs: Social GIS Datasets Based on Google Maps API -- http://tinyurl.com/marat131204a --- 21/22...
21/22
.
That’s all, thank you ...
M.Zhanikeev -- [email protected] -- Maps2Graphs: Social GIS Datasets Based on Google Maps API -- http://tinyurl.com/marat131204a --- 22/22...
22/22
.
[01] myself (2013)Maps2Graphs Project Pagehttps://github.com/maratishe/maps2graphs
[02] myself (2013)Maps2Graphs: Example Crowdsourcing Projecthttp://tinyurl.com/maps2graphs
[03] myself+1 (2013)...EV Use in Environments with V2H and Battery Replacement...ITS Tokyo
[04] G.Chatzimilioudis+3 (2012)Crowdsourcing with SmartphonesIEEE J. Internet Computing
[05] B.Yoo+2 (2005)Travel Time Estimation Using Mobile DataJ. Transportation Studies
M.Zhanikeev -- [email protected] -- Maps2Graphs: Social GIS Datasets Based on Google Maps API -- http://tinyurl.com/marat131204a --- 22/22...
22/22
.
[06] C.Wu+3 (2013)Crowdsourcing Multimedia QoE Evaluation...IEEE Trans. on Multimedia
[07] D.Sui+2 (2013)Crowdsourcing Geographic Knowledge: Volunteered Geographic Information (VGI)...Springer Book
[08] Google (2013)Dropbox Core APIhttps://www.dropbox.com/developers/core
[09] Google (2013)Google Maps APIhttps://developers.google.com/maps/
[10] (2013)JSON formatwww.json.org
M.Zhanikeev -- [email protected] -- Maps2Graphs: Social GIS Datasets Based on Google Maps API -- http://tinyurl.com/marat131204a --- 22/22...
22/22
.
[11] Esri (2013)ArcGIS Japanhttp://www.esrij.com/products/arcgis
[12] W3C (2013)HTML5W3C Candidate Recommendation at http://www.w3.org/TR/html5/
M.Zhanikeev -- [email protected] -- Maps2Graphs: Social GIS Datasets Based on Google Maps API -- http://tinyurl.com/marat131204a --- 22/22...
22/22