Handling Changes to Your Server-Side Data Model

Evan Coyne Maloney Principal Engineer

Mobile

Gracefully handling changes to your server-side data model

Mobile

Things change.

…sometimes more often than we’d like.

Mobile

Handling change on the Web

• No app review process

• “Go live” with the push of a button• Immediate deployment to production• All users get changes right away

• No need to test “on device”• Won’t spend hours in provisioning hell• Easy to perform automated testing on the hardware you’ll

be using in production

Handling change Web vs. iOS

Mobile

Handling change on iOS

• Mandatory app review• Can take a week or more

• Must test “on device”• If you only test in the simulator, you don’t really know

what your code is doing• How does it perform?• It is exhibiting device-only crashing?


Mobile

Web vs. iOS


Web iOS

Fully automated testing Partially manual testing

Push a button and go live Going live requires Apple’s permission, which in turn requires waiting

Can fix stuff right away

Being able to fix stuff right away requires careful planning and lots of work

(…and even then you can’t fix everything)

Mobile

These differences have consequences

Your iOS apps can become an anchoron your server-side environment


Mobile

An Example

Mobile

Chirper

Handling change An Example

A hypothetical social network

• Unidirectional relationships between users (following)• Encourages very short messages; optimized for mobile input• Allows followers to tag a message to categorize it

Mobile

Chirper


A hypothetical social network

• Launched by an early-stage startup with limited resources • Built a simple system to launch quickly• Having trouble scaling• Need to bring hosting and bandwidth costs down

Mobile

Chirper’s Message Feed


• The feed contains all messages that:• Are new to the user• Were delivered previously, but have newly-applied tags

[{"message_id": 10271972,"message": "Is there going to be an encore? I really hope they play Penelope!","author_id": 5375,"tags": ["Pinback", "concert"]

}]

• When the Chirper app launches, it retrieves the updated message feed from the server

• Messages in the feed arrive within a JSON array:

Mobile

Not the most efficient


• Any time anyone tags a message, the entire message will be re-sent in the feed update, even though previously-received messages are already stored in the client app

• Tags themselves can add a lot of redundant data, especially in heavily active feeds with long tag names

• Chirper has decided to refactor their feeds to be more efficient

Mobile

An example scenario


1. Someone adds a new tag to an old message

2. Two new messages arrive

Mobile

How it looks in the current feed


[{ "message_id": 10271972, "content": "Is there going to be an encore? I really hope they play Penelope!", "author_id": 5375, "tags": ["Pinback", "concert", "live music"], "is_tag_update": true},{ "message_id": 10272003, "content": "I like the live Penelope so much better than the album version.", "author_id": 6022, "tags": ["Pinback", "concert", "live music"]},{ "message_id": 10272012, "content": "They played that last night. Expect June or Grey Machine tonight.", "author_id": 4216, "tags": ["Pinback", "concert", "live music"]}]

503 bytes (compact JSON)

Mobile

A refactored feed


{ "tags": ["Pinback", "concert", "live music"], "message_tags": { "10271972": [ 0, 1, 2 ], "10272003": [ 0, 1, 2 ], "10272012": [ 0, 1, 2 ] }, "messages": { "10272003": { "content": "I like the live Penelope so much better than the album version.", "author_id": 6022 }, "10272012": { "content": "They played that last night. Expect June or Grey Machine tonight.", "author_id": 4216 } }}

344 bytes (compact JSON)

Mobile

Success?


• The response size is now about ⅔ of what it was• Savings will be even better under many scenarios

Great, but…These savings won’t lower Chirper’s bandwidth costs until:

1. They build and submit a new app2. Apple approves it3. A critical mass of users update to the latest version

Mobile

The root of the problem

Handling change Designing for change

• The client app and the server must always agree on the structure of the data

• On the client side, this knowledge has to be compiled in to the application binary

• Once the app has been submitted, it can’t be modified

• But, we can design a way around this!

Mobile

How we build server-driven apps today


1. The server maintains knowledge of the data necessary to drive the app

2. The app requests data, and receives a data structure in response

3. The app will “ask questions of the data” to do its work

4. This requires having knowledge of the data structure

Mobile

How we build server-driven apps today



2. The app requests data, and receives a data structure in response

3. The app will “ask questions of the data” to do its work

4. This requires having knowledge of the data structure

Mobile

How we could build server-driven apps



2. The app requests data, and receives in response:• The data itself• A set of instructions for how to extract values from the data structure

3. The app will still “ask questions of the data” to do its work, but it will use the server-provided instructions to do it — the client no longer needs advance knowledge of the server’s data model!

Mobile

1. The exact structure of the data

2. What questions it needs answered from the data

3. How to translate those questions into procedures for extracting answers from the data

The Client needs to know: The Server needs to know:1. The exact structure of the

data

Rethink where we require knowledge


Mobile

Rethink where we require knowledge


1. What questions it needs answered from the data

1. The exact structure of the data

2. The set of questions each app version needs answered from the data

3. How to translate those questions into procedures for extracting answers from the data

The Client needs to know: The Server needs to know:

Mobile

How can we do this?


Mobile

Introducing Mockingbird Expressions

Mobile

• Text strings that specify values within an iOS application’s runtime environment • Objective-C and Swift variables• Server-side data models

• Device information• Screen size, system clock

• OS-level information• iOS version, system permissions, user settings

• Provided by the Mockingbird Data Environment library

Mockingbird Expressions An Introduction

Mockingbird ExpressionsWhat are they?

Mobile


Mockingbird ExpressionsWhat do they look like?

Variables FunctionsThe expression: The expression:

$foo ^currentTime()

Returns the object bound to the Mockingbird variable name “foo”.

Returns an NSDate object containing the current time.

$dict[key].propVal ^formatLongDate(^currentTime())

Selects the object associated with key from the NSDictionary bound to the name “dict”, and returns the value of its propVal property.

Returns an NSString containing the current date in the format “August 11, 2014”.

$UIDevice.model ^filter($users|$item.firstName == Jill)

Equivalent to the Objective-C code:

[[UIDevice currentDevice] model]

Returns a subset of the collection $users containing only the items whose firstName property equals “Jill”.

Mobile

We can use Mockingbird Expressions to describe how to extract

answers from data


Mobile

Chirper + Mockingbird

Mobile

1. What are the IDs of the new messages?

2. What are the IDs of the messages with newly-applied tags?

3. What is the content of a given message?

4. Who is the author of a given message?

5. Which tags are associated with a given message?

Mockingbird Expressions A Use Case

Questions Chirper asks its data

Mobile




1. What are the IDs of the new messages?

^list(^filter($feed|!$item.is_tag_update)|$item.message_id)Old feed

New feed ^list($feed.messages|$key)

Mobile

2. What are the IDs of the messages with newly-applied tags?




^list(^filter($feed|$item.is_tag_update)|$item.message_id)

^filter(^list($feed.message_tags|$key)|!$feed.messages[$item])

Old feed

New feed

Mobile

3. What is the content of a given message?




âssociate($feed|$item.message_id|$item.content)[$id]

$feed.messages[$id].content

Old feed

New feed

Mobile

4. Who is the author of a given message?




âssociate($feed|$item.message_id|$item.author_id)[$id]

$feed.messages[$id].author_id

Old feed

New feed

Mobile

5. Which tags are associated with a given message?




âssociate($feed|$item.message_id|$item.tags)[$id]

^list($feed.message_tags[$id]|$feed.tags[$item])

Old feed

New feed

Mobile

{ "new_message_ids": "^list(^filter($feed|!$item.is_tag_update)|$item.message_id)", "newly_tagged_message_ids": "^list(^filter($feed|$item.is_tag_update)|$item.message_id)", "content_for_message_with_id": "âssociate($feed|$item.message_id|$item.content)[$id]", "author_id_for_message_with_id": "âssociate($feed|$item.message_id|$item.author_id)[$id]", "tags_for_message_with_id": "âssociate($feed|$item.message_id|$item.tags)[$id]"}


Old feed

We can now separate out the expressions needed to query the data

{ "new_message_ids": "^list($feed.messages|$key)", "newly_tagged_message_ids": "^filter(^list($feed.message_tags|$key)|!$feed.messages[$item])", "content_for_message_with_id": "$feed.messages[$id].content", "author_id_for_message_with_id": "$feed.messages[$id].author_id", "tags_for_message_with_id": "^list($feed.message_tags[$id]|$feed.tags[$item])"}

New feed

Mobile

This architecture now relies on“model/query decoupling”

wherein


The ability to query a set of data is separated from

knowledge of that data’s structure

Mobile

How would this work in practice?

Mobile

1. The server maintains a schema version for the data model it uses• Whenever a non-backwards-compatible schema change is introduced, the

schema version increments

2. Any time the schema version changes, the server publishes a corresponding set of query expressions for that schema version

3. When an app fetches new data, it checks the schema version returned by the server. If the app doesn’t have the query expressions for that version, it can get them from the server• Apps will cache query expressions• Apps can include a compiled-in set of query expressions to avoid having to

download them the first time an app launches

Mockingbird Expressions Code Samples

Model/Query Decoupling using Mockingbird Expressions

Mobile

1. When we retrieve feed data from the server, we expose it to the variable space:


Populating the Mockingbird Data Environment

NSData* json = ... // Unicode text containing JSON structure from server

id feedData = [NSJSONSerialization JSONObjectWithData:json options:0 error:nil];

[[MBVariableSpace instance] setVariable:@"feed" value:feedData];

The expression $feed now refers to the value of feedData

2. We load the query expressions associated with the current schema into a dictionary and expose it to the variable space:

NSDictionary* queries = ... // query names -> expressions

[[MBVariableSpace instance] setVariable:@"queries" value:queries];

The expression $queries now refers to the NSDictionary queries

Mobile

3. Query for new messages:


Querying the data

NSString* query = [MBExpression asString:@"$queries.new_message_ids"];NSArray* newMessageIDs = [MBExpression asObject:query];

The array newMessageIDs contains two elements:

( 10272003, 10272012 )

4. Query for messages with updated tags:NSString* query = [MBExpression asString:@"$queries.newly_tagged_message_ids"];NSArray* newlyTaggedMessageIDs = [MBExpression asObject:query];

The array newlyTaggedMessageIDs contains one element:

( 10271972 )

5. Get the content of a message given a message ID:[[MBVariableSpace instance] setVariable:@"id" value:@(10272003)];NSString* query = [MBExpression asString:@"$queries.content_for_message_with_id"];NSString* content = [MBExpression asNumber:query];

The string content is “I like the live Penelope so much better than the album version.”

Mobile


Querying the data

7. Get the tags associated with a given message:[[MBVariableSpace instance] setVariable:@"id" value:@(10272012)];NSString* query = [MBExpression asString:@"$queries.tags_for_message_with_id"];NSArray* tags = [MBExpression asObject:query];

The array tags contains three elements:

( "Pinback", "concert", "live music" )

6. Get the author ID of a message given a message ID:[[MBVariableSpace instance] setVariable:@"id" value:@(10271972)];NSString* query = [MBExpression asString:@"$queries.author_id_for_message_with_id"];NSNumber* authorID = [MBExpression asNumber:query];

The NSNumber authorID contains the integer value 5375.

Mobile

What can you use this for?

Mobile

1. No App Store update required to introduce support for new schemas • New sets of query expressions can be downloaded to add immediate support for new

schemas

2. Maintain legacy support while evolving your services • You won’t need to run multiple versions of your backend just to maintain legacy support

3. Simplify your backend codebase • Avoid spaghetti code — no need for version-check conditionals littered throughout your

server-side codebase

4. A/B testing made easy • Want to support multiple schemas simultaneously? No problem! Just switch between sets of

query expressions. This makes server-driven A/B testing easy.

Mockingbird Expressions Use Cases

Benefits

Mobile

When can you use this?

Mobile

Mockingbird Availability

Mockingbird Data Environment availabilityThe Mockingbird Data Environment is part of the

Mockingbird Library open-source project from Gilt Groupe.

Coming Fall 2014

Mobile

Mockingbird Data Environment availability

Available today!

Coming Fall 2014


The Mockingbird Data Environment is part of the Mockingbird Library open-source project from Gilt Groupe.

Mobile

Mockingbird Toolbox open source project

https://github.com/gilt/mockingbird-toolbox


available at:

Evan Coyne Maloney Principal Engineer

Mobile

Gracefully handling changes to your server-side data model

Handling Changes to Your Server-Side Data Model

Technology

mobile handling

mobile chirper handling

ios handling

mobile web

production handling

ecient handling

mobile input

mobile things