Top Banner
Data transferring faster, stronger, better and not harder
50

Data transfering: faster, stronger, better and not harder. UA Mobile 2016.

Apr 07, 2017

Download

Software

UA Mobile
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
  • Data transferringfaster, stronger, better and not harder

  • DisclaimerData transferring optimizations requires both, client side and server side development, could be hard to implement and it could break your system (and may be life)

    ...But it worth to be done

  • Is it really a problem?

  • Is it really a problem?

  • General advice GZIP Cache (!!!) Partial requests/responses Minimize data structure Use streams where possible Use Binary formats

  • General advice GZIP Cache (!!!) Partial requests/responses Minimize data structure Use streams where possible Use Binary formats

  • GZIP Network cost vs CPU Measure it

  • General advice GZIP Cache (!!!) Partial requests/responses Minimize data structure Use streams where possible Use Binary formats

  • Cache Client DB Network client level cache Server side support (ETag, Cache-Control, etc.) Use push notifications for cache invalidating

  • General advice GZIP Cache (!!!) Partial requests/responses Minimize data structure Use streams where possible Use Binary formats

  • Partial requests/responses

  • Partial requests/responses Transfer only what you really need Control field count (Data Diet) Pagination is our friend

  • Partial requests/responses

    https://www.googleapis.com/demo/v1?key=YOUR-API-KEY

  • Partial requests/responses

    https://www.googleapis.com/demo/v1?key=YOUR-API-KEY

    { "kind": "demo",

    ...

    "items": [

    {

    "title": "First title",

    "comment": "First comment.",

    "characteristics": {

    "length": "short",

    "accuracy": "high",

    "followers": ["Jo", "Will"],

    }, ... ]

    }

  • Partial requests/responses

    https://www.googleapis.com/demo/v1?key=YOUR-API-KEY&fields=kind,items(title,characteristics/length)

  • Partial requests/responses

    https://www.googleapis.com/demo/v1?key=YOUR-API-KEY&fields=kind,items(title,characteristics/length)

    { "kind": "demo",

    "items": [

    {

    "title": "First title",

    "characteristics": {

    "length": "short"

    }

    }, ... ]

    }

  • Partial requests/responses

    { "kind": "demo",

    ...

    "items": [

    {

    "title": "First title",

    "comment": "First comment.",

    "characteristics": {

    "length": "short",

    "accuracy": "high",

    "followers": ["Jo", "Will"],

    }, ... ]

    }

    { "kind": "demo",

    "items": [

    {

    "title": "First title",

    "characteristics": {

    "length": "short"

    }

    }, ... ]

    }

  • Partial requests/responses

    { "kind": "demo",

    ...

    "items": [

    {

    "title": "First title", "comment": "First comment.", "characteristics": { "length": "short", "accuracy": "high", "followers": ["Jo", "Will"], }, ... ]

    }

    { "kind": "demo",

    "items": [

    {

    "title": "First title", "characteristics": { "length": "short" }

    }, ... ]

    }

  • Fields parameter syntax summary Use a comma-separated list to select multiple fields. Use a/b to select a field b that is nested within field a; use a/b/c to select a

    field c nested within b. Exception: For API responses that use "data" wrappers, where the response

    is nested within a data object that looks like data: { ... }, do not include "data" in the fields specification. Including the data object with a fields specification like data/a/b causes an error. Instead, just use a fields specification like a/b.

  • Fields parameter syntax summary Use a sub-selector to request a set of specific sub-fields of arrays or objects

    by placing expressions in parentheses "( )".

    For example: fields=items(id,author/email) returns only the item ID and author's email for each element in the items array. You can also specify a single sub-field, where fields=items(id) is equivalent to fields=items/id.

  • Fields parameter syntax summary Use wildcards in field selections, if needed.

    For example: fields=items/pagemap/* selects all objects in a pagemap.

  • Use PATCH (yes, there is a HTTP Verb PATCH) https://tools.ietf.org/html/rfc5789 If youve changed one field - send one field Retrofit support included ;) Could be used with Partial Responses

  • General advice GZIP Cache (!!!) Minimize data Minimize data structure Use streams where possible Use Binary formats

  • Minimize data structure

    Remove spaces Remove long field names Remove braces Remove data (see previous slides)

  • Make you data homogeneousWhat is an Homogenous Collection?

    Before...

    [

    {"a":"A","b":"B"},

    {"a":"C","b":"D"},

    {"a":"E","b":"F"}

    ]

    After

    [2,"a","b","A","B","C","D","E","F"]

  • Tools CJSON Algorithm JSON.hpack Algorithm Brain (use it carefully)

  • General advice GZIP Cache (!!!) Partial requests/responses Minimize data structure Use streams where possible Use Binary formats

  • Why streams? Start parsing from the first byte Reduce memory usage Most of the tools support it by default

  • General advice GZIP Cache (!!!) Partial requests/responses Minimize data structure Use streams where possible Use Binary formats

  • Binary formats Fast Small Optimized for every platform Static typed

  • Binary formats Protobuf Thrift (Never used it :( ) Capn proto

  • Protobufmessage Person {

    required string name = 1;

    required int32 id = 2;

    optional string email = 3;

    }

  • ProtobufPerson john = Person.newBuilder()

    .setId(1234)

    .setName("John Doe")

    .setEmail("jdoe@example.com")

    .build();

    output = new FileOutputStream(args[0]);

    john.writeTo(output);

  • Protobuf Advanced android support (Retrofit, Wire, etc.) Obj-c/Swift/JS are also supported Easy to implement on server side

  • Capn proto Faster than Protobuf Same author No Obj-C/Swift support (use C++ and suffer) No Android specific realization (Java only) BETA

  • Why it worth to check Incremental reads Random access Mmap support Tiny generated code Tiny runtime library

  • Why you shouldnt use it in production Still BETA Has not yet had a security review The Capn Proto programming interface may still change in ways that break

    existing code Performance (Currently it only beats Protobufs in realistic-ish end-to-end

    benchmarks by around 2x-5x. We can do better)

  • Capn proto: Schemastruct Person { name @0 :Text = Bob; birthdate @3 :Date;

    email @1 :Text = default@value.com; phones @2 :List(PhoneNumber);

    struct PhoneNumber { number @0 :Text; type @1 :Type;

    enum Type { mobile @0; home @1; work @2; } }}

  • Capn proto: Schema Types come after names @N annotations # Comments also are supported (should appear after the declaration)

  • Capn proto: Built-in Types Void: Void Boolean: Bool Integers: Int8, Int16, Int32, Int64 Unsigned integers: UInt8, UInt16, UInt32, UInt64

    Floating-point: Float32, Float64

    Blobs: Text, Data

    Lists: List(T)

  • Capn proto: Unionsstruct Shape { area @0 :Float64;

    union { circle @1 :Float64; # radius square @2 :Float64; # width }}

  • Capn proto: Unions Unions members are numbered in the same number space as fields Void is useful if no value is needed By default, when a struct is initialized, the lowest-numbered field in the union

    is set

  • Capn proto: Groupsstruct Person { # ...

    # Note: This is a terrible way to use groups, and meant # only to demonstrate the syntax. address :group { houseNumber @8 :UInt32; street @9 :Text; city @10 :Text; country @11 :Text; }}

  • Capn proto: Groups Behaves as one field of other structure (address :Address) Group is not separated object! Essentially, a group is just a namespace Useful with Unions

  • Capn proto: Groupsstruct Shape { area @0 :Float64;

    union { circle :group { radius @1 :Float64; } rectangle :group { width @2 :Float64; height @3 :Float64; } }}

  • Capn proto: Other features Dynamically-typed Fields Enums Interfaces (RPC) Generic Types Generic Methods (RPC) Constants Nesting, Scope, and Aliases Imports Annotations

  • Capn proto: Other features

  • Summary Dont transfer data at all If you should transfer something - get only part that you need ...then minimize data structure of that part ...then compress it ...then optimize data parsing ...then cache result it and see p.1 Use binary data!

  • Thanks!URLs:

    https://capnproto.org/https://developers.google.com/web/fundamentals/performance/optimizing-content-efficiency/http-caching?hl=enhttps://developers.google.com/site-verification/v1/performancehttps://developers.google.com/protocol-buffers/

    Contact me:

    Twitter: @AntonMinashkinEmail: anton.minashkin@outlook.comFB: https://www.facebook.com/anton.minashkin.1