Design Issues for Web APIs Paul Ashton. Overview Background (me, Tourplan) Design issues for web APIs Two general approaches: tunnelled and REST. Resources.

Design Issues for Web APIs

Paul Ashton

Overview

• Background (me, Tourplan)

• Design issues for web APIs

• Two general approaches: tunnelled and REST.

• Resources (esp apigee.com).

My background

• Lecturer at University of Canterbury (1987 to 2000). Interested in distributed systems. Communication mechanisms: sockets, Sun RPC, CORBA.

• Senior Developer, Tourplan (2000-2010, 2012-). XML web APIs for customer use. Also, internal APIs (some HTTP, some SOAP).

• Senior Developer, Ministry of Fisheries (2011). Small REST API.

Tourplan

The "classic" travel supply chain: customer, travel agent, outbound wholesaler, inbound wholesaler, supplier (accommodation, transport, attraction).

Tourplan:•A package used by travel wholesalers (mainly inbounders).•Developed circa 1986 (in dataflex) for a company wanting to run Haley's comet tours.•Current product is the second generation: SQL Server (2000+) / VB6 / .Net / Java.•Tourplan is Christchurch based. Offices in London, Johannesburg, Kuala Lumpur, Costa Rica.•Hundreds of clients in tens of countries. Installed at each client site.•Connectivity has always been critical. Initially fax, then email, and now web interfaces and web APIs.

Tourplan web APIs

• Tourplan terminology: agents, suppliers, (service) options (have rates and inventory), bookings, service lines.

• hostConnect web API. Product search and booking for agents. Some supplier functionality.

• productConnect web API. For use by Tourplan customers to access and update agent, supplier and service option information.

• We have written client code for a variety of supplier web APIs, and a number of payment gateway web APIs.

hostConnect

Tourplan's key web API. Java. Started in 2000 (mature)

Tens of client implementations (maybe over 100).

Used by:• Customers to write their own web front ends.• Clients of customers for system to system connections.• Customers to implement their own APIs for use by their

clients.• Tourplan to write our own web front ends (e.g. webConnect).• Tourplan to perform Tourplan to Tourplan connections.

Web API definition (Wikipedia)

"A web API (Application Programming Interface) is typically a defined set of HTTP request messages along with a definition of the structure of response messages, typically expressed in JSON or XML. While "web API" is sometimes considered a synonym for web service, the Web 2.0 applications typically have moved away from SOAP-based web services towards more direct REST-style communications.

Web APIs allow the combination of multiple services into new applications known as mashups."

Motivations

• component interfaces within a system

• facilitate system to system integration. Some hotels forcing dynamic rates on to wholesalers.

• open up access to software products (new third party web front ends, data import / export).

• APIs have long been a feature of packages; web APIs are much easier to consume than language / OS specific APIs (e.g. a COM DLL).

• My focus is on widely-used, long-lived APIs.

• APIs can be for internal use, external use (private or public), or both.

• Most developers will write client code.

• Some developers design and implement web APIs. They face a number of design choices.

• The client sees an HTTP interface (no matter what server-side support software is used to implement the interface).

• The developer using your API is your user; design for their convenience not yours

Request design

What operations do you want your API to provide?

• Some will read and return one item

• Some will search (return a collection of items, perhaps in summary form)

• Some will update (add / change / delete) one or more items.

Request Design Considerations

• For reads, different clients will want different collections of fields. • Balancing act: clients won’t want to make lots of requests, but

don’t want to receive lots of data they don’t need.

• Consider giving the client some control of what is returned (field level? groups of fields?).

• Netflix approach.

• For collections, are they paginated? Limited to some maximum number to prevent huge responses?

• For updates, the request is usually the transaction boundary.

Request implementation

• What HTTP verbs (GET, POST, PUT, DELETE, PATCH) do you use?• What is the structure of the URL path after the web app name. • URL query string parameters?• Data content (fields, nesting) for requests and replies.• Data representation. XML, JSON, form-url-encoded.• SOAP? Best avoided for external APIs. • ODATA???? Hmmm. Not sure. See apigee webcast.• ASP .NET's web-api can serialize .Net (transfer) objects to XML or

JSON based on the value of the Accept header (a common general approach).

• Use standard data formats (e.g. yyyy-MM-dd) and avoid localization.

Interface styles

Two common general approaches.

• "Tunnelled". hostConnect, productConnect.

• REST (Roy Fielding PhD). Pragmatists versus RESTafarians. Trademe, Xero public APIs.

An API should be distinct and separate from a web UI.

DO NOT DOCUMENT THE INTERFACE OF YOUR WEB APP AND CALL IT AN API.

Tunnelled

• Usually POST.

• Usually one (or a few) URLs (.../servlet/conn).

• Little use of HTTP capabilities outside of sending data and receiving data.

• Little guidance as to how to design operations. hostConnect has 34 requests, including OptionInfo, SupplierInfo, ListBookings, AddService, GetBooking, QuoteToBook.

<?xml version="1.0"?><!DOCTYPE Request SYSTEM "hostConnect_3_00_000.dtd"><Request> <OptionInfoRequest> <AgentID>BKGTST</AgentID> <Password></Password> <Opt>CHCACPJACHC?????</Opt> <Info>GS</Info> <DateFrom>2013-01-01</DateFrom> <SCUqty>1</SCUqty> <RoomConfigs><RoomConfig> <Adults>2</Adults> <RoomType>TW</RoomType> </RoomConfig></RoomConfigs></OptionInfoRequest> </Request>

<?xml version="1.0" encoding="ISO-8859-1"?><!DOCTYPE Reply SYSTEM

"http://dev04:8080/devtrunk/hostConnect_3_00_000.dtd"><Reply> <OptionInfoReply> <Option> <Opt>CHCACPJACHCGALA</Opt> <OptionNumber>7289</OptionNumber> <OptGeneral> ... </OptGeneral> <OptStayResults> ... </OptStayResults> </Option>

<Option> ...

(Pragmatic) REST

• Make use of capabilities offered by HTTP.

• Has conventions for structuring an API.

• Defining resources is central to this approach (agents, options, suppliers, bookings, service lines). It is OK for data to end up in multiple resources.

GET /suppliers?name=Smith*GET /suppliers/1234POST /suppliersPUT /suppliers/1234PATCH /suppliers/1234DELETE /suppliers/1234

• ASP.NET Web API (.Net 4 and above).

• REST is a natural fit for CRUD (partial updates cause debate); less so for some more complex operations (convert quote to booking, cancel booking).

Define data formats?

• XML Schemas (or old-school DTDs)

• JSON equivalents exist

• Defined data formats can form part of the API docs

• Can be validated against at run-time (for input AND output).

• hostConnect uses DTDs. Config settings as to whether validation is done (always done in dev systems).

Error handling

• Do you use HTTP status codes (200 OK, 400 Bad Request, 404 Not Found, 500 Internal server error, etc)?

• Recommended in REST, but some client libraries "swallow" some status codes.

• Reply content is error message (as text, XML, JSON, ...)

• Alternatively, always return 200 and define error fields in response formats (hostConnect).

• Whatever method, provide good error messages.

<?xml version="1.0" encoding="ISO-8859-1"?> <!DOCTYPE Reply SYSTEM

"http://dev04:8080/devtrunk/hostConnect_3_00_000.dtd"><Reply> <ErrorReply> <Error>1051 SCN XXXX</Error> </ErrorReply> </Reply>

Caching and Concurrency Control

• HTTP provides headers to control whether GET results can be cached (and for how long).

• HTTP If-Unmodified-Since can be used on GET to avoid retrieval if the resource hasn't changed.

• Etag header can be used for concurrency control, with If-Match header on PUT.

• REST interfaces can make use of the HTTP mechanisms.

• hostConnect does not support caching explicitly; some booking concurrency control incorporated in request design.

Preserving backwards compatibility

• Essential once there are production users (especially external ones)

• Don't tie your API structures to a database schema or business objects. Transfer objects are your friend.

• Non-breaking changes are OK: new requests, new data in existing requests (so long as structures are not changed).

• Aside: if writing clients, DO NOT assume field B will always follow field A (at some time field C might come between).

Breaking change options

• Break the interface (and deal with fall out from clients!).

• Make the new behaviour optional, with the old behaviour being the default.

• Add new requests (e.g. by version number in the URL). Phase out old requests after a suitable notice period (could be years)

• Start a new API

Other considerations

• Keep the API stateless. Avoid: login request, various "do stuff" requests, logout request. Avoid session state.

• Ids should be stable• Docs and sample requests. Web-based test harness? SDKs?• Test systems• Authentication: one of the web mechanisms? OAUTH? Roll your

own?• hostConnect: rolled our own (id / password). Roles: standard,

extensions. Could do with a super-user role.• SSL? Needed for some authentication mechanisms to be secure.• Signup processes (fully on-line; or with manual intervention)• Management: error logs, performance measures, throttling.

API Standards

Example: OTA in the travel industry.

• XML / tunnelled. Lots of schemas. • Two versions per year (A and B)• OTA is huge (kitchen sink approach)!• Everyone implements their own subset. • Plug and play between two OTA-using parties is unlikely, but they

start much closer together than they would without OTA.• hostConnect and productConnect are not OTA-based (have looked

at OTA for ideas); Tourplan has one API that is OTA-based.

If there are standard APIs in your domain, you might want to base what you do on them (or use them as the source of ideas).

Choosing an interface style

• Pragmatic REST has received a lot of recent attention (JSON, XML).

• Tunnelled interfaces are also common (often XML; can be JSON, form-url-encoded, etc). Most travel interfaces we have come across are tunnelled / XML.

• Consider consistency with other APIs in the same area.

• API standards may be a consideration.

• Aside: in a system to system connection, semantic issues are usually the hardest: different underlying concepts, terminology, data fields used, max field lengths.

Resources

Gain experience from APIs you use.

Look at other APIs (there are lots documented on the web).

Book: APIs: A Strategy Guide - Jacobson, Brail and Woods.

apigee.com (API Best Practices page)

• eBooks. Especially "Web API Design" (describes pragmatic REST)

• webcasts, blogs

• The api-craft google group (warning: tends to be dominated by REST religious debates).

• And...

Free stickers!

Summary

• Currently, two general styles to work within

• Many design decisions to make

• Lots of prior work to learn from

• Design for your API clients

• Backwards compatibility

• APIs have a habit of "escaping" once developed.