Good Morning Percona Live Amsterdam 2015
Good MorningPercona Live Amsterdam 2015
Who Am Ifighting back against impostor syndrome
Nicola IarocciCo-founder and CTO at CIR 2000
Nicola IarocciMongoDB Master
Nicola IarocciOpen Source junkie
Eve • Cerberus • Events • Flask-Sentinel • Eve.NET • Etc.
Nicola IarocciConsultant
Mongo • RESTful Services • Python • My Open Source Projects
Nicola IarocciCoderDojo
Coding Clubs for Kids
MongoDB & REST APIsA Match Made in Heaven
Agenda
Agenda1. Our use case for a RESTful API
Agenda2. What is a RESTful API and why we need it
Agenda3. Why MongoDB is a good match for RESTful Services
Agenda4. Build and run a MongoDB RESTful Service from scratch, live on stage.
Agenda5. Stories from the field (if there’s any time left, which I doubt)
The Casefor RESTful Web APIs
Amica 10invoicing & accounting for italian small businesses
your old school desktop app
what we started with
Client
LAN/SQL
Database
DesktopApplication
GoalA remote service that client apps can leverage to stay in sync withc each other
What we need #1Must be accessible by any kind of client technology
What we need #2Abstract the data access layer
so we can update/replace the engine at any time with no impact on clients
What we need #3An appropriate data storage engine
What we need #4Easily (re)deployable and scalable multi-micro-service architecure
Where we want to go
Clients
“Cloud”
Database
RESTful Web API
APIiOS
Android
Website
Desktop Client
? ?
Constraints• minimum viable product first
• add features over time
• frequent database schema updates
• easily scalable
• avoid downtime as much as possible
• cater upfront for a microservices architecture
RESTSo What Is REST All About?
REST is not a standard
REST is not a protocol
REST is an architectural stylefor networked applications
Defines a set of simple principlesloosely followed by most API implementations
“resource”the source of a specific information
A web page is not a resourcerather the representation of a resource
“global permanent identifier”every resource is uniquely identified. Think a HTTP URI.
#3 standard interface
used to exchange representations of resources (think the HTTP protocol)
“a set of constraints”separation of concerns, stateless, cacheability, layered system, uniform interface, etc.
Web is built on RESTand it is meant to be consumed by humans
RESTful APIs are built on RESTand are meant to be consumed by machines
Representational State Transfer (REST)by Roy Thomas Fielding
http://www.ics.uci.edu/~fielding/pubs/dissertation/rest_arch_style.htm
Goals #1 and #2 are metREST layer allows all kinds of client technologies and abstracts the data away
MongoDB and RESTOr why we picked MongoDB for our REST API
JSON transportMost REST services and clients produce and consume use JSON
JSON-style data storeMongoDB stores data as Binary JSON
JSON & RESTful API
JSONaccepted media type
Client
JSON(BSON)
Mongo
GET
maybe we can push directly to client?
JSON & RESTful API
JSONaccepted media type
Client
JSON(BSON)
Mongo
JSONsubset of python dict
(kinda)
API
GET
almost.
JSON & RESTful API
JSONobjects
Client
JSON(BSON)
Mongo
JSON/dictmaps to python dict
(validation layer)
API
POST
also works when sending data the database
Similarity with RDBMSmakes NoSQL easy to grasp (even for a sql head like me)
TerminologyRDBMS Mongo
Database Database
Table Collection
Rows(s) JSON Document
Index Index
Join Embedding & Linking
What about Queries?Queries in MongoDB are represented as JSON-style objects
db.things.find({x: 3, y: "foo”});
Filtering and Sorting
nativeMongoquery syntax
Client
JSON(BSON)
Mongo
(very) thin parsing
& validation layer
API
Expose the native MongoDB syntax?
?where={“x”: 3, “y”: “foo”}
JSON all along the pipelinemapping to and from the database feels more natural
No need for ORMNo need to map objects to JSON and vice-versa (win!)
schema-lessdynamic documents allow for painless evolution
REST is statelessMongoDB lacks transactions
Ideal API SurfaceMongo collection maps to API resource endpoint
api.example.com/contacts
Maps to a Mongo collection
Ideal API SurfaceMongo document maps to a API document endpoint
api.example.com/contacts/4f46445fc88e201858000000
Maps to a collection ObjectID
Goal #3 is metAn appropriate data storage engine: MongoDB
Eve REST API for Humans™
Free and Open Source Powered by MongoDB and Good Intentions
eve
Philosopyeffortlessly build and deploy highly customizable, fully featured
RESTful Web Services
Quickstart
install$ pip install eve
run.pyfrom eve import Eveapp = Eve()
if __name__ == '__main__':app.run()
settings.py# just a couple API endpoints with no custom schema or rules.
DOMAIN = {‘people’: {}‘works’: {}
}
launch$ python run.py* Running on http://127.0.0.1:5000/
enjoy
HATEOAS AT WORK HERE
$ curl http://localhost:5000/people
{"_items": [],"_links": {"self": {"href": "people", "title": "people"},"parent": {“href": "/", "title": "home"},
},"_meta": {"max_results": 25,
"total": 0, "page": 1 }}
enjoy
CLIENTS CAN EXPLORE THE API PROGRAMMATICALLY
$ curl http://localhost:5000/people
{"_items": [],"_links": {"self": {"href": "people", "title": "people"},"parent": {“href": "/", "title": “home”}
},"_meta": {"max_results": 25,
"total": 0, "page": 1}}
$ curl http://localhost:5000/people
{"_items": [],"_links": {"self": {"href": "people", "title": "people"},"parent": {“href": "/", "title": "home"},
},"_meta": {"max_results": 25,
"total": 0, "page": 1 }}
enjoy
AND EVENTUALLY FILL THE UI
enjoy$ curl http://localhost:5000/people
{"_items": [],"_links": {"self": {"href": "people", "title": "people"},"parent": {“href": "/", "title": "home"},
},"_meta": {"max_results": 25,
"total": 0, "page": 1 }}
PAGINATION DATA
enjoy
EMTPY RESOURCE AS WE DID NOT
CONNECT A DATASOURCE
$ curl http://localhost:5000/people
{"_items": [],"_links": {"self": {"href": "people", "title": "people"},"parent": {“href": "/", "title": "home"},
},"_meta": {"max_results": 25,
"total": 0, "page": 1 }}
settings.py# let’s connect to a mongo instance
MONGO_HOST = 'localhost'MONGO_PORT = 27017MONGO_USERNAME = 'user'MONGO_PASSWORD = 'user'MONGO_DBNAME = 'percona'
settings.py# let’s also add a few validation rulesDOMAIN['people']['schema'] = { 'name': {'type': 'string’,'maxlength': 50,'unique': True},
'email': {'type': 'string','regex': '^\S+@\S+$'},
'location': {'type': 'dict','schema': {'address': {'type': 'string'},'city': {'type': 'string’}}},
'born': {'type': 'datetime'}}
UNIQUE STRING, MAX LENGTH 50
settings.py# let’s also add a few validation rulesDOMAIN['people']['schema'] = { 'name': {'type': 'string','maxlength': 50,'unique': True},
'email': {'type': 'string','regex': '^\S+@\S+$'},
'location': {'type': 'dict','schema': {'address': {'type': 'string'},'city': {'type': 'string’}}},
'born': {'type': 'datetime'}}
ONLY ACCEPT VALID EMAILS
settings.py# let’s also add a few validation rulesDOMAIN['people']['schema'] = { 'name': {'type': 'string’,'maxlength': 50,'unique': True},
'email': {'type': 'string','regex': '^\S+@\S+$'},
'location': {'type': 'dict','schema': {'address': {'type': 'string'},'city': {'type': 'string’}}},
'born': {'type': 'datetime'}}
THIS REGEX SUCKS!DON’T USE
IN PRODUCTION
settings.py# let’s also add a few validation rulesDOMAIN['people']['schema'] = { 'name': {'type': 'string','maxlength': 50,'unique': True},
'email': {'type': 'string','regex': '^\S+@\S+$'},
'location': {'type': 'dict','schema': {'address': {'type': 'string'},'city': {'type': 'string’}}},
'born': {'type': 'datetime'}}
SUBDOCUMENT WITH 2 STRING FIELDS
settings.py# let’s also add a few validation rulesDOMAIN['people']['schema'] = { 'name': {'type': 'string’,'maxlength': 50,'unique': True},
'email': {'type': 'string','regex': '^\S+@\S+$'},
'location': {'type': 'dict','schema': {'address': {'type': 'string'},'city': {'type': 'string’}}},
'born': {'type': 'datetime'}}
ONLY ACCEPT DATETIME VALUES
settings.py# allow write access to API endpoints# (default is [‘GET’] for both settings)
# /peopleRESOURCE_METHODS = ['GET','POST']
# /people/<id>ITEM_METHODS = ['GET','PATCH','PUT','DELETE']
ADD/CREATE ONE OR MORE ITEMS
settings.py# allow write access to API endpoints# (default is [‘GET’] for both settings)
# /peopleRESOURCE_METHODS = ['GET','POST']
# /people/<id>ITEM_METHODS = ['GET','PATCH','PUT','DELETE']
EDIT ITEM
settings.py# allow write access to API endpoints# (default is [‘GET’] for both settings)
# /peopleRESOURCE_METHODS = ['GET','POST']
# /people/<id>ITEM_METHODS = ['GET','PATCH','PUT','DELETE']
REPLACE ITEM
settings.py# allow write access to API endpoints# (default is [‘GET’] for both settings)
# /peopleRESOURCE_METHODS = ['GET','POST']
# /people/<id>ITEM_METHODS = ['GET','PATCH','PUT','DELETE']
YOU GUESSED IT
settings.py
CLIENT UI
# a few optional config options
DOMAIN['people'].update({'item_title': 'person','cache_control':'max-age=10,must-revalidate','cache_expires': 10,'additional_lookup': {'url’: 'regex("[\w]+")','field’: 'name'}
}}
)
settings.py
CLIENT CACHE OPTIONS
# a few optional config options
DOMAIN['people'].update({'item_title': 'person','cache_control':'max-age=10,must-revalidate','cache_expires': 10,'additional_lookup': {'url’: 'regex("[\w]+")','field’: 'name'}
}}
)
settings.py
ALTERNATE ENDPOINT
# a few optional config options
DOMAIN['people'].update({'item_title': 'person','cache_control':'max-age=10,must-revalidate','cache_expires': 10,'additional_lookup': {'url’: 'regex("[\w]+")','field’: 'name'}
}}
)
Features OverviewWe are going to focus on MongoDB power-ups
full range of CRUD operationsCreate/POST
Read/GET Update/PATCH and Replace/PUT
Delete/DELETE
filters, mongo style?where={“name”: “john”}
filters, the python way?where=name==john
sorting?sort=city,-name
SORT BY CITY, THEN NAME DESCENDING
projections?projection={"avatar": 0}
RETURN ALL FIELDS BUT ‘AVATAR’
projections?projection={"lastname": 1}
ONLY RETURN ‘LASTNAME’
pagination?max_results=20&page=2
MAX 20 RESULTS PER PAGE; PAGE 2
GeoJSONsupport and validation for all GeoJSON types
Point, LineString, Polygon, MultiPoint, MultiLineString, MultiPolygon, GeometricalCollection
document embeddingjoins
standard request$ curl example.com/works/<id>
{ "title": "Book Title", "description": "book description", "author": “52da465a5610320002660f94"
}RAW FOREIGN KEY
(DEFAULT)
request an embedded document$ curl example.com/works/<id>?embedded={“author”: 1}
{ "title": "Book Title", "description": "book description", "author": {
“firstname”: “Mark”, “lastname”: “Green”,
} }
REQUEST EMBEDDED AUTHOR
embedded document$ curl example.com/works/<id>?embedded={“author”: 1}
{ "title": "Book Title", "description": "book description", "author": {
“firstname”: “Mark”, “lastname”: “Green”,
} }
# embedding is configurable on per-field basis and# can be pre-set by API maintainer
EMBEDDED DOCUMENT
bulk insertsinsert multiple documents with a single request
request$ curl -d ‘
[{
"firstname": "barack", "lastname": “obama"
}, {
"firstname": "mitt", "lastname": “romney”
}]' -H 'Content-Type: application/json’ <url>
response[ { "_status": "OK", "_updated": "Thu, 22 Nov 2012 15:22:27 GMT", "_id": "50ae43339fa12500024def5b", "_etag": "749093d334ebd05cf7f2b7dbfb7868605578db2c" "_links": {"self": {"href": “<url>”, "title": "person"}} }, { "_status": "OK", "_updated": "Thu, 22 Nov 2012 15:22:27 GMT", "_id": "50ae43339fa12500024def5c", "_etag": "62d356f623c7d9dc864ffa5facc47dced4ba6907" "_links": {"self": {"href": “<url>", "title": "person"}} }] COHERENCE MODE OFF: ONLY META FIELDS ARE RETURNED
response[ { "_status": "OK", "_updated": "Thu, 22 Nov 2012 15:22:27 GMT", "_id": "50ae43339fa12500024def5b", "_etag": "749093d334ebd05cf7f2b7dbfb7868605578db2c" "_links": {"self": {"href": “<url>”, "title": “person”}}, "firstname": "barack", "lastname": "obama",
}, { "_status": "OK", "_updated": "Thu, 22 Nov 2012 15:22:27 GMT", "_id": "50ae43339fa12500024def5c", "_etag": "62d356f623c7d9dc864ffa5facc47dced4ba6907" "_links": {"self": {"href": “<url>", "title": "person"}} "firstname": "mitt", "lastname": "romney", }]
COHERENCE MODE ON: ALL FIELDS RETURNED
document versioning?version=3
?version=all?version=diffs
soft deletespreserve deleted documents and retrieve them with a simple
?show_deleted
file storagefiles are stored in GridFS by default; customizable for S3, file system, etc.
data validationpowerful and extensible data validation powered by Cerberus
rich validation grammarreferentrial integrity / unique values / defaults / regex / etc.
custom data typescreate your own data types to validate against
custom validation logicextended the validation system to cater for specific use cases
multi databaseserve endpoints and/or users from dedicated mongos
index maintenancedefine sets of resource indexes to be (re)created at launch
supports sparse, geo2d and background indexes
event hooksplug custom actions in the request/response cycle
run.pyfrom eve import Eveapp = Eve()
def percona(resource, response):documents = response['_items'] for document in documents: document['percona'] = 'is so cool!'
if __name__ == '__main__':app.on_fetched_resource += percona_liveapp.run()
CALLBACK FUNCTION
run.pyfrom eve import Eveapp = Eve()
def percona(resource, response):documents = response['_items'] for document in documents: document['percona'] = 'is so cool!'
if __name__ == '__main__':app.on_fetched_resource += percona_liveapp.run()
LOOP ON ALL DOCUMENTS
run.pyfrom eve import Eveapp = Eve()
def percona(resource, response):documents = response['_items'] for document in documents: document['percona'] = 'is so cool!'
if __name__ == '__main__':app.on_fetched_resource += percona_liveapp.run()
INJIECT FIELD
run.pyfrom eve import Eveapp = Eve()
def percona(resource, response):documents = response['_items'] for document in documents: document['percona'] = 'is so cool!'
if __name__ == '__main__':app.on_fetched_resource += percona_liveapp.run()
ATTACH CALLBACK TO EVENT HOOK
rate limitingpowered
settings.py# Rate limit on GET requests:
RATE_LIMIT_GET = (1, 60)
ONE REQUEST PER MINUTE (CLIENT)
$ curl -i <url>
HTTP/1.1 200 OKX-RateLimit-Limit: 1X-RateLimit-Remaining: 0X-RateLimit-Reset: 1390486659
rate limited request
CURRENT LIMIT
$ curl -i <url>
HTTP/1.1 200 OKX-RateLimit-Limit: 1X-RateLimit-Remaining: 0X-RateLimit-Reset: 1390486659
rate limited request
REMAINING
$ curl -i <url>
HTTP/1.1 200 OKX-RateLimit-Limit: 1X-RateLimit-Remaining: 0X-RateLimit-Reset: 1390486659
rate limited request
TIME TO RESET
$ curl -i <url>
HTTP/1.1 429 TOO MANY REQUESTS
rate limited request
OUCH!
operations loglog all operations and eventually expose a dedicated endpoint
HATEOASHypermedia As Engine Of The Application State
XML support$ curl -H ”Accept: application/xml” -i http://example.com/people
<resource href="people" title="people" ><link rel="parent" href="/" title="home" /><resource href="people/55fc149138345b0880f07e3d" title="person" >
<_created>Fri, 18 Sep 2015 13:41:37 GMT</_created> <_etag>5d057712ce792ebb4100b96aa98bfe9b6693c07b</_etag> <_id>55fc149138345b0880f07e3d</_id> <_updated>Fri, 18 Sep 2015 13:41:37 GMT</_updated>
<email>[email protected]</email> <name>john</name></resource>
</resource>
conditional requestsallow clients to only request non-cached content
If-Modified-SinceIf-Modified-Since: Wed, 05 Dec 2012 09:53:07 GMT
ONLY RETURN DOCUMENT IF MODIFIED SINCE
If-None-MatchIf-None-Match:1234567890123456789012345678901234567890
>
ONLY RETURN DOCUMENT ETAG CHANGED
data integrity and concurrencyno overwriting documents with obsolete versions
missing ETag# fails, as there is no ETag included with request
$ curl \-X PATCH \-i http://example.com/people/521d6840c437dc0002d1203c \-H "Content-Type: application/json" \-d '{"name": “ronald"}'
HTTP/1.1 403 FORBIDDEN NO ETAG
REJECTED
ETag mismatch# fails, as ETag does not match with server
$ curl \-X PATCH \-i http://example.com/people/521d6840c437dc0002d1203c \-H "If-Match: 1234567890123456789012345678901234567890" \-H "Content-Type: application/json” \-d '{"firstname": "ronald"}'
HTTP/1.1 412 PRECONDITION FAILED
ETAG MISMATCH
REJECTED
valid ETag# success at last! ETag matches with server
$ curl \-X PATCH \-i http://example.com/people/50adfa4038345b1049c88a37 \-H "If-Match: 80b81f314712932a4d4ea75ab0b76a4eea613012" \-H "Content-Type: application/json" \-d '{"firstname": "ronald"}'
HTTP/1.1 200 OK
# Like most of features, ETags can be disabled.
ETAG MATCH
ACCEPTED
custom data layersbuild your own data layer
authentication and authorizationbasic / token / hmac / BYO / OAuth2 / you name it
and (a lot) moreCORS, cache control, API versioning, JOSNP, Etc.
vibrant community90+ contributors / 350+ forks / 2500+ github stargazers
Eve-Docsautomatic documentation for Eve APIs in both HTML and JSON
CHARLES FLYNN
Eve-Docs
Eve-ElasticElasticsearch data layer for your Eve-powered API
PETR JASEK
Eve-SQLAlchemySQL data layer for Eve-powered APIs
PETR JASEK
Eve-Mongoengineenables mongoengines data models to be used as Eve schema
STANISLAV HELLER
Eve.NETHTTP and REST client for Eve-powered APIs
PETR JASEK
Eve-OAuth2leverage Flask-Sentinel to protect your API endpoints with OAuth2
THOMAS SILEO
REST Layer“golang REST API framework heavily inspired by Python Eve”
THOMAS SILEO
Goal # 4 achievedeasy to setup, launch and scale up; also a good fit for microservices infrastracture
A look back to initial draft
Clients
“Cloud”
Database
RESTful Web API
APIiOS
Android
Website
Desktop Client
? ?
Clients
MultipleMongoDBs
Database
Adameve instances
APIiOS
Android
Website
Desktop Client
what we have in production
stories from the trenches#1. when too much (magic) is too much #2. sometimes you don’t want to debug
#3. so how do I login into this thing?
Take Aways
Enhance MongoDBwith powerful features on top of native engine
validation, document embedding (joins), referential integrity, document versioning, transformations, rate limiting, etc.
Consider the REST layeras an ideal data access layer
the story of pymongo 3.0 breaking changes mongo or sql or elastic or …
Consider the REST layeras an ideal data access layer
the story of Adam dashboards
Consider Microservicesleverage Eve features create a network of isolated yet standardized services
each service has a dedicated role
runs as an eve instance, with its own configuration has its own database(s)
callbacks route traffic between services
Clients
User-reservedMongoDBs
eve-multidb
Data
Autheve-oauth2
(flask-sentinel)
API
iOS
Android
Website
Desktop Client
Adam 1
Adameve instance
Redisauth tokens,rate limiting
Auth/UsersMongoDB
Clients
service-reserved
MongoDBs
Data
Autheve-oauth2,flask-sentinel
API
iOS
Android
Website
Desktop Client
Adam 2
Adameve instance
Redisauth tokens,rate limiting
Serviceseve instances
Auth/UsersMongoDB
very simplified
thanksnicolaiarocci
python-eve.orgeve