OpenStreetMap : Open Licensed GeoData

Harry Wood

OpenStreetMap : Open Licensed Geo DataBritish Computer Society : Monday 27th April 2009

Topics●OpenStreetMap purpose and premise

●Data structures: Nodes, Ways ,Tags etc

●Editor demo

●OpenStreetMap servers and architecture

●Rendering and map displays

●The license

●CloudMade products and services

●Imports and other mapping techniques

●Getting Involved

NOTE:Ran out of time for these topics on the day. The slides for these have also been removed from this deck. Could present them on another occasion!

Free as in Freedom

●Open license:–Creative Commons Attribution Share-alike

●“Open Content” like “Open Source”●Contributors retain ownership of copyright●People and commercial companies can use the maps for free under this license.

–Details of license requirements?... coming up

Getting an Open Licensed Map●Can't copy copyrighted maps

● Not allowed to import copyrighted data● Not allowed to copy from copyrighted maps● Not allowed to trace over copyrighted maps● Not allowed to “derive”

●Can copy some maps, but only...● Public domain. Unrestricted (incl. relicensing)● Get permission to release with an open license (big ask)

●Can create maps completely from scratch● crazy idea?

GPS traces

● How it started. Gadgets!● Cheap consumer GPS units or location-aware mobiles● Record a line of dots

Record many lines of dots....

Looks like a street map. Kind of

Recording dataNames of streets

Types of streets (trunk, residential, motorway)

One-way restrictions

Footpaths, tracks, pedestrian, rivers, railways

Parks, woodland, industrial areas, cemeteries

POI (pubs, cash machines, post offices, post boxes, bus stops, toilets, supermarkets, restaurants, kebab shops, monuments, hotels, picnic sites, barriers, light houses, piers, sports centres, petrol stations, playgrounds, cinemas, car parks, universities, bicycle parking, tourist information, etc etc etc

Mapping Techniques●Photo Mapping (geo-located photos)

●Audio mapping●Taking notes

●

●Ditch the GPS● Taking notes ● Local knowledge!● Yahoo Aerial Imagery

Mapping: A lot of effort

●Gather data ● GPS traces and other information

●Input data● using OSM “editor” software

●Requires a lot of effort● Requires a lot of people!

Community Contribution●Built by a large online community●Many hands make light work●Openly editable (and easy)●Poor quality contributions?

● Gradual refinement● Assume good faith ● Monitoring and correction

....Remarkably it works!

Sounds familiar?

Wikipedia●Large community coming together to build something great!●Wikipedia Principles● Openly editable● Open content license● Gradual refinement● Assume good faith ● “Soft Security” Monitoring and correction

looks like wikipedia

OpenStreetMap = The wikipedia of maps

Community

It's big. 100,000 registered users

Community

Increasing editing activity

CommunityVery Active

0 1000 2000 3000 4000 5000 6000 7000 8000 9000

User rank

Less Active

Lopsided

Long tail of less active users

Cambridge

Global Project!

Open Licensed Data

A copyrighted map

...is a justification for OpenStreetMap (It can't be used freely therefore OSM is better)

...cannot be a source for openstreetmap

Existing maps are very rarely free

Ordnance Survey●Wonderful data in the UK

●OS license use of maps (and charge ££££££)

●Never allow re-distributing with a different license

●Very strict about copying and their definition of “derived” work

Guardian 'Free Our Data' Campaign● Lobbying government

● Tax paid for data collection● Tax still pays indirectly● Economic benefits of free

● Slow progress● OS might release

● low quality data first● less-than-free license

● Or might be privatised!● Whine about it or take action?

OSM and Ordnance Survey

£many

£0

Low quality High quality

OS

OSM

● We can't can't use google maps● License their data from teleatlas● ...who license data from Navteq / Teleatlas● ...Ordnance Survey!● No access to underlying data● Google terms & conditions● Don't allow deriving data from their maps● Don't allow copying & re-distributing with a different license● Wonderful hi-res aerial imagery● T&Cs do not allow deriving maps (tracing)● Bought in (licensed) from multiple suppliers

Google Maps

Why not use Google Maps?Wonderful “free” (beer) mash-up API but...

●Errors and omissions●Car centric. Footpaths and other details●Cycle routes and Pistes●Colours / branding - Google maps fatigue●SVG export. Custom cartography●Underlying data access!● Details of OSM map access coming up

●Help OpenStreetMap!

Oxford University Website

OSM has better maps of Oxford

Encourages the OSM community

Other uses coming up...

Nodes, Ways, Relations

Node

Has latitude and longitude

Can stand alone, or form part of a way

Way

Joins together several nodes

Direction sometimes matters

Can form a 'closed way' (area)

Relation

For complex things such as routes

Tags

Applied to the Nodes, Ways, & Relations

Key value pairs

amenity=pubname=Hare & Hounds

highway=residentialname=Court Street

http://www.openstreetmap.org/?lat=53.548078&lon=-2.004771&zoom=18

Data Browser demo

Data Browser

Permalink

JOSM demo

JOSM demo

OpenStreetMap Servers

Hosted in UCL

Loads of bandwidth

~10 servers:

Where does the data go?

http://wiki.openstreetmap.org/wiki/Servers

OpenStreetMap Foundation

Custodian of servers and sysadmin access

Oversees funding and vehicle for fund raising

Protection from copyright and liability suits

Database ServerMotherboard Supermicro X7DWN+ motherboard with Intel 5400 (Seaburg) Chipset

CPU 2x Intel Xeon Processor E5420 Quad Core 2.5Ghz

Memory 32GB DDR2 667 ECC

Disk 2x 73GB (3.5) SAS 15K

10x 450GB (3.5) SAS 15K

donate.openstreetmap.org

Raised £10,000 in 2 days

API

● REST web service● HTTP GET & PUT● Get elements at URLs

http://www.openstreetmap.org/api/0.6/node/297556642

● No bloated request payloads

● Ruby on Rails

Ruby on Rails● It's easy. Web + REST● Fashionable. Developers like it

● Developers are our most limited resource.

● It's what SteveC used

● Problems?● Can't stream data from db● Memory hungry and leaks somewhere● Maybe use something else for core API

Nodeshttp://www.openstreetmap.org/api/0.6/node/297556642

<node id="297556642"lat="53.548223"lon="-2.0056012"version="2"changeset="648346"user="Guy"uid="10983"visible="true"timestamp="2008-09-16T20:42:44Z">

<tag k="name" v="Hare &amp; Hounds"/> <tag k="created_by" v="Potlatch 0.10b"/> <tag k="amenity" v="pub"/></node>

Wayshttp://www.openstreetmap.org/api/0.6/way/27120827 <way id="27120827" visible="true" timestamp="2008-09-19T13:19:53Z" version="2" changeset="664390" user="Guy" uid="10983"> <nd ref="298116100"/> <nd ref="297555192"/> <nd ref="297555193"/> <nd ref="297555194"/> <tag k="name" v="Court Street"/> <tag k="created_by" v="Potlatch 0.10b"/> <tag k="highway" v="residential"/></way>

Other API calls

GET a map

All elements within a bounding boxhttp://api.openstreetmap.org/api/0.6/map?bbox=11.54,48.14,11.543,48.145

PUT elements

Now requires “changeset open” request

Various other operations

History and changeset access

Get GPS points/tracks

Some database details

Switched from MySQL to Postgres last weekend!

Rails migrations in theory

In practice. C++ scripts running all weekend

Why the switch?

Lots of other planned restructuring

including new DB hardware

Good time to do it

MySQLGenerally fast and scalable enough

● Quadtile indexing extension

Several annoying flaws:● schema changes cause table copies● different features on different db engines● (transactions on InnoDB, spatial on MyISAM)● silently accepts invalid utf8● constraints can't be deferred● some non-standard SQL syntax

Postgres

● Addresses a lot of MySQL flaws:● Faster schema changes● Better support for transactions, utf8, etc

● Personal preference of our sysops

Full Revision History

Store a full history edits to elements

Essential wiki-like feature

Ideally provide simple roll-back

Access old versions of an element

Difficult to reconstruct old version of a map

Changesets

● Brand new feature● Every edit belongs in a change set● Every numbered version of every object

belongs in one particular changeset● Changesets have comments

● Great for monitoring

Changeset Displayshttp://www.openstreetmap.org/browse/changesets

Changeset revert?

● Reverting is still a difficult problem● Changesets are not atomic

● Changeset 1 User:Sam Node 12345 v1● Changeset 2 User:SallyNode 12345 v2● Changeset 3 User:Sid Node 12345 v3● Changeset 4 User:SallyNode 12345 v3● Changeset 1 User:Sam Node 12345 v4● Changeset 1 User:Sam Node 12345 v5

● Many interlinked elements

Conflicts

● Two users editing the same element– Rarely happens actually

● Version mismatch now reported– “Optimistic locking”

● Editors (should) do CVS style conflict resolution● Download reveals conflict● Upload not allowed until resolved

(Watch nice video)

OSM 2008: A Year of Edits

http://vimeo.com/2598878

planet.osm● Snapshot of the OpenStreetMap database● Entire planet. Every node, way, relation, tag● Only 'current' data. Not history● XML formatted .osm file● 5.2 GB with bzip2 compression● Uncompressed... 150 GB● Takes several hours to dump. Every Wednesday● Important part of Openness. Ensures longevity.

Osmosis

● Java toolkit for OpenStreetMap● Various data transformations● Minutely, Hourly, Daily diffs .osc.gz files● Created by Osmosis. Consumed by osmosis● Streamable changes

Open Tagging

● Mentioned tags briefly– amenity=pub highway=residential

● Free-form open tagging. Any tags you like!● Agree on standards● Main openstreetmap.org map rendering uses

one set of tags● Other map renderers, other tools, can use other

tagging schemes

'Map Features' wiki page

● BIG list of tags

Which tags go on this page?

● Wiki proposal process

● Wiki discussion and voting

● Wiki dabates (& blazing rows!)

– Different ways of tagging the same thing.

– Things which should not be tagged● Wiki documentation

Smoothness Debate● Vehement Objections

– Too subjective

– Verifiability

– Poor english

● Disruption– Disregarding vote

– Wiki fiddlers

vs Mappers

– Wiki edit wars

– New process?● Lock down?

The wrong way to think about tags

● Come up with lots of ideas for new tags● Submit proposals, organise votes, generally

fiddle with the OSM wiki a lot● Pester people to use tags in map renderings

...oh and maybe do a bit of actual mapping

The right way to think about tags● Do mapping!● Found something without a documented tag?

– Search thoroughly (in mailing list too)

– Use a less specific tag and qualify with type=

– Use a note= tag

– Just invent a tag

● Do more mapping!● Discuss politely. Improve existing docs.● maybe... possibly.... do a proposal● Focus on mapping. Don't worry about rendering

TagWatch

● Tag usage stats● Split by country● Tags used in

conjunction

Rendering● Topic follows on although...

tagging is not just about rendering● Go from geodata (nodes, ways, relations & tags)

to rasterized map images

Rendering

Which tags to render?

● Thousands of different tags in the DB● Can't show them all● Choose features to show at different

zoom levels– Cartography!

● What do you want to emphasise?

Rendering Toolchain

Slippy MapDisplay

Mapnik● Open Source rendering software● Fast!● C++● Requires PostGIS database

Mapnik Stylesheet

● XML format● 'styles', 'filters' and 'rules' ● >7000 lines long

● Pre-processing steps– Cascadenik

– and also...

osm2pgsql

● Step before using Mapnik (& stylesheet)● load OSM data into a Postgres database● Lossy conversion. Only take tags of interest● nodes and ways → linestrings and polygons

Slippy MapDisplay

● Open Source JavaScript library● Dynamic slippy map on your website● WMS layers● Tile based map layers● Transparent overlay layers● Markers, Boxes, Polygons, Click events

In the end we want a map display...

Tiles

● Small map images● Cacheable● Fast loading● Sized to optimize speed

– Too big. Unneeded map area

– Too small. Too many requests

– 255x255pixels

Tile Naming

● Slice the world into tiles at each zoom level● Tiles are always 255x255 pixels● Represent different sized area of the world at

different zoom levels

Tile Naming

Zoom level 0 has only one tile (whole world):

Tile Naming

Zoom level 1 has 2x2 tiles

Tile Naming

● Zoom level 2 has 4x4 tiles● Zoom level 3 has 8x8 tiles● Zoom level 4 has 16x16 tiles

...● Zoom level n has 2nx2n tiles

...● Zoom level 18 has

262144 x 262144 tiles

Tile Naming● Every tile has a URL

http://tile.openstreetmap.org/12/2047/1362.png

yZoom Level

(0-18) x

● Tile naming scheme followed by OpenLayers● Same used by google maps● Looks like filesystem URL

Tiles =High Performance Computing

262144 x 262144 = 68,719,476,736 tiles

inode problem!

5 kB each = 320 terabytes

But then there's zoom 17.... another 80 terabytes

etc...

Tiles =High Performance Computing

OpenStreetMap updates?● Apply diffs● Re-render tile images!● CPU problem!

Caching and mod_tile

● mod_tile– Apache module. Very fast

– Render-on-demand if necessary

– Clever caching

– Serves old cached images and labels as dirty

– Dirty tiles get re-rendered by render daemon

Slippy MapDisplay

Bandwidth

● Serving terrabytes of tile data. High bandwidth● UCL

OpenCycleMap.org

● Using OpenStreetMap– Presenting special interest map

– Same data. Different cartographic choices

● Toolchain running on another server– Updates fed in

– Passionate sub-set of the OSM community

Route relations, Cycle Parking, Bike Shops, Drink

Relief maps!

SRTM

● NASA - Shuttle Radar Tomography Mission● Public Domain● Problems

– Spot heights – not contours

– Course grid

– Voids and other anomalies

CycleMap tool chain● Downloads weekly planet dump● SRTM. More steps in the chain!

● Bandwidth problems. Now hosted by CloudMade

OpenPisteMap

Hiking Map

Whitewater Map

Bus map

Kosmos● .NET (windows only)● Desktop app● Can generate tiles● wiki based

style config

osmarender

● First good OSM renderer● Used to be the only way to get SVG

● Complex perl XSLT ● Generates SVG (XML vector graphics format)● Feed in .osm file and style config

● Can't be used to generate tiles.... or can it?

'osmarender' layer

tiles@home● Distributed tile rendering

– Instructions dished out from tiles@home server

– Many clients download via API and upload images

● 'osmarender' layer– Used to provide the fastest updates

● XSLT transforms & inkscape SVG rendering– Eats massive amounts of CPU

– Mapnik more sensible. ...no need to distribute

Other renderers?

● Plenty of scope to develop but..● high performance problem● Complex graphics problem● e.g. phprender

Needs a bit of

work!

We want people to be free to use our maps!

OSM License Requirements

● Free to bring maps into “collective” works– Must give “attribution”

● Free to create “derivative” works– must share-alike

● Awkward complications:– What exactly counts as “derivative work”?

– How do you give credit to the “authors”?

ODbL + ODC-Factual

● Open Data Commons● Open Database License● Factual Information License● Benefits:

– copyright, database right, and contract

– Expressly written for data

– More strict about underlying data (forcing sharing), but less strict about end products

Commercial use is allowed!● OSM destroys business models ...or does it?

– Destroys monopolies on geo data

● Allowed to charge for distribution– Can't disallow further distribution

– Monetary value tends towards zero

● Allowed to charge for services– Distribute different formats / renderings

● Solve difficult problems (+time dependant problems)● Hosting

– Consulting services

● Just use maps. Core business not in geo-data

Flickr

Nestoria

'Trails' iPhone app

Get Involved!

http://wiki.openstreetmap.org/http://lists.openstreetmap.org/

irc://irc.oftc.net #osmhttp://trac.openstreetmap.org

Harry Wood worked as an enterprise integration consultant for 8 years, but led a secret double-life as addicted contributer to wikipedia and other collaborative open content projects. He got involved in OpenStreetMap three years ago, as a mapper, wiki gardener, and developer. Since January this year (2009) he has worked for CloudMade, as a full time OpenStreetMap developer

CloudMade is a company providing products and services around OpenStreetMap.

More information at http://cloudmade.com

These slides are (of course) freely re-usable under the Creative Commons Attribution-ShareAlike 2.0 License