From publisher to platform: How the Guardian embraced the internet using content, search, and Open Source Stephen Dunn, Guardian News and Media [email protected], 25th May, 2011 Twitter: @cuica, @openplatform Thursday, 26 May 2011
Mar 15, 2016
From publisher to platform: How the Guardian embraced the internet using content, search, and Open Source
Stephen Dunn, Guardian News and [email protected], 25th May, 2011
Twitter: @cuica, @openplatform
Thursday, 26 May 2011
1
From publisher to platformHow the Guardian embraced the Internet using content, search, and Open SourceStephen Dunn, Guardian News and Media
2
Thursday, 26 May 2011
The publishing era
3
Thursday, 26 May 2011
We started a long time ago:
Thursday, 26 May 2011
Swine flu
Keyword page
Twitter updates
Content partnerships
Audio
Video Open platform API
Live blogs
Comment
Mobile siteApps
Newspapers
Thursday, 26 May 2011
To secure the financial and editorial independence of the Guardian in perpetuity. To promote freedom in the press and liberal journalism globally.
To become the world's leading liberal voice.
To secure the financial and editorial independence of the Guardian in perpetuity
To promote freedom in the press and liberal journalism globally
Thursday, 26 May 2011
7
Open Web Principles
Thursday, 26 May 2011
8
2009
Thursday, 26 May 2011
• “A cool URI is one that does not change” Tim Berners-Lee 1998• 1.5 million resources redirected to new scheme
9
1. Permanent
http://www.flickr.com/photos/fstorr/
Thursday, 26 May 2011
10
2. Addressable★ Resources are “about” something - ready for the
social web.
★ We live in “the age of point-at-things” (Coates 2005)
Thursday, 26 May 2011
11
★ Multiple routes to content
★ Tagging drives discovery
3. Discoverable
Thursday, 26 May 2011
12
4. Open
Thursday, 26 May 2011
/technology/internet
/technology/all
/environment/climatechange
Example: The Hackable Guardian
http://www.guardian.co.uk/....
/rss
/rss
+business/globaleconomy/rss
Thursday, 26 May 2011
Results...
14
Thursday, 26 May 2011
15
3,750,000
7,500,000
11,250,000
15,000,000
18,750,000
22,500,000
26,250,000
30,000,000
Sep 2005 Oct 2006 Nov 2007 Dec 2008
Uni
que
Use
rs Pre - project
First release
Final ReleaseSite traffic growthUnique Users
40M
Thursday, 26 May 2011
However...
16
Thursday, 26 May 2011
17
1 Billion+Internet Users!
Thursday, 26 May 2011
18
Thursday, 26 May 2011
19
Thursday, 26 May 2011
20
Thursday, 26 May 2011
21
...“How I stopped worrying about my website and learned to love the whole internet.”
Matt McAlister
Thursday, 26 May 2011
22
The Open Strategy
OPEN IN
Bring in data and apps from the Internet
OPEN OUT
Enable partners to build applications using Guardian content and services for other platforms
Thursday, 26 May 2011
23
Thursday, 26 May 2011
24
"Our most interesting experiments lie in combining what we know with the experience, opinions and expertise of the people who want to participate rather than passively receive.”
Thursday, 26 May 2011
25
Thursday, 26 May 2011
26
Thursday, 26 May 2011
27
Thursday, 26 May 2011
28
Thursday, 26 May 2011
29
Thursday, 26 May 2011
30
Thursday, 26 May 2011
31
Thursday, 26 May 2011
32
Thursday, 26 May 2011
33
Thursday, 26 May 2011
34
“The Guardian alongside Al Jazeera was the one news source that everybody on the streets in Tahrir - not just in Cairo but in surrounding cities and major centers of revolutionary activity - that people were talking about.”
Jack Shenker
Thursday, 26 May 2011
3522
The Open Strategy
OPEN IN
Bring in data and apps from the Internet
OPEN OUT
Enable partners to build applications using Guardian content and services for other platforms
Thursday, 26 May 2011
36
The Open Platform
Thursday, 26 May 2011
37
The suite of services enabling partners to build applications with
the Guardian
Thursday, 26 May 2011
3822
OPEN IN
Bring in data and apps from the Internet
OPEN OUT
Enable partners to build applications using Guardian content and services for other platforms
Thursday, 26 May 2011
CONTENT APIA service for selecting and
collecting content from the Guardian
for re-use
DATA STOREA directory of
useful data curated by Guardian editors
POLITICS APIOpen database of candidates, voting records, constituencies, election results,
live data on election day
Thursday, 26 May 2011
Mutualised news!
40
Thursday, 26 May 2011
Mutualised news!
41
Thursday, 26 May 2011
Mutualised news!
42
Thursday, 26 May 2011
43
Thursday, 26 May 2011
44
Thursday, 26 May 2011
45
Thursday, 26 May 2011
46
Thursday, 26 May 2011
DATA STOREA directory of
useful data curated by Guardian
editors
Thursday, 26 May 2011
POLITICS APIOpen database of candidates, voting
records, constituencies, election results, live data on election day
Thursday, 26 May 2011
POLITICS APIOpen database of candidates, voting
records, constituencies, election results, live data on election day
49
Thursday, 26 May 2011
<OBLIGATORY DOGFOOD SLIDE >
50
Thursday, 26 May 2011
51
Thursday, 26 May 2011
Thursday, 26 May 2011
Thursday, 26 May 2011
Thursday, 26 May 2011
Thursday, 26 May 2011
56
Open for Business
Thursday, 26 May 2011
57
3 Tiers of access3 Revenue models
Keyless: Take our headlines. You keep associated revenues.
Approved: Take our full article content, but with an advert. Guardian keeps ad revenue, you keep rest-of-page revenue.
Bespoke: Take, reformat, augment our content Revenue model to be negotiated. Combination of Media, Fees, Downloads.
Thursday, 26 May 2011
58
Thursday, 26 May 2011
59
What this means
Open Out: Developers can now access full content APIs on demand with keys post-approved
Platform is positioned as a place to do business
So rapid scalability, reliability and performance are now core requirements
Thursday, 26 May 2011
OPEN INBring in data and
apps from the internet
OPEN OUTAllow partners to build applications using Guardian
content and services for other
platforms
Thursday, 26 May 2011
61
A framework for integrating 3rd party
applications into guardian.co.uk
MICROAPPS Simple REST/HTTP framework allows lightweight development
Applications proxied for performance
Apps generally hosted in the cloud, allows hot deployment into production
Thursday, 26 May 2011
62
A framework for integrating 3rd party
applications into guardian.co.uk
MICROAPPS
Thursday, 26 May 2011
• What could I cook?
Thursday, 26 May 2011
64
Bringing it together
Thursday, 26 May 2011
65
Thursday, 26 May 2011
App showcase
66
Thursday, 26 May 2011
67
From publisher to platform
Seeking massive growth, but no longer only broadcasting content on the website
User/partner engagement & contribution onJournalismdatasoftwareapplicationsrevenue and ads
Support developers and partners with data and APIs,need scalability, reliability, speed
Thursday, 26 May 2011
68
Evolving the architecture
Thursday, 26 May 2011
App server App server App server
Web server Web server Web server
CMS
Oracle
Memcached (added later)
Thursday, 26 May 2011
App server App server App server
Web server Web server Web server
CMS
Oracle
Memcached
Why RDBMS?
5 years ago, fewer alternatives
Understand operations procedures
Can easily recruit DBAs / devs
Developer/ops tools
Business critical system: a safe choice
Thursday, 26 May 2011
71
3,750,000
7,500,000
11,250,000
15,000,000
18,750,000
22,500,000
26,250,000
30,000,000
Sep 2005 Sep 2006 Sep 2007 Sep 2008
Uni
que
Use
rs
Scaling trafficUnique Users
Thursday, 26 May 2011
72
Thursday, 26 May 2011
73
Thursday, 26 May 2011
74
Thursday, 26 May 2011
75
Thursday, 26 May 2011
76
Thursday, 26 May 2011
77
Thursday, 26 May 2011
We chose Solr/Lucene
78
Can perform complex queries, including full-text search
We can change the schema with no downtime
Most queries are of similar cost
Scales very well horizontally
“Just worked” in the cloud
No strange control processes/engines
Developers just loved working with it!
Thursday, 26 May 2011
79
Thursday, 26 May 2011
App server
Web servers
CMS
Memcached
RDBMS
80
Solr
Solr
Solr
Solr
Solr
Solr
Cloud, EC2
Api
Thursday, 26 May 2011
8122
OPEN IN
Bring in data and apps from the Internet
OPEN OUT
Enable partners to build applications using Guardian content and services for other platforms
What about Open In?
Thursday, 26 May 2011
App server
Web servers
CMS
Memcached
RDBMS
82
App
App
App
App
App
App
Apps
Proxy
external hostingapp engine etc
Thursday, 26 May 2011
App server
Web servers
CMS
Memcached
Solr
Core
Solr
Solr
Solr
Solr
Solr
Cloud, EC2
Out
App
App
App
App
App
App
In
Proxy
external hostingapp engine etc
rdbms
83
Thursday, 26 May 2011
84
Thursday, 26 May 2011
85
Thursday, 26 May 2011