Introducing the App Engine datastore

Thursday, May 26, 2011

2

Hands on with the App Engine Datastore

Ikai LanMay 9th, 2011


About the speaker

• Ikai Lan - Developer Programs Engineer, Developer Relations• Twitter: @ikai• Google Profile: http://profiles.google.com/ikai.lan

3


http://profiles.google.com/ikai.lan

http://profiles.google.com/ikai.lan

Lab prerequisites

• JDK 1.5+• Apache Ant• Codelab package: http://code.google.com/p/2011-datastore-

bootcamp-codelab/downloads/detail?name=2011-datastore-bootcamp-codelab.zip

Shortlink: http://tinyurl.com/datastore-bootcamp

4


http://code.google.com/p/2011-datastore-bootcamp-codelab/downloads/detail?name=2011-datastore-bootcamp-codelab.zip






Goals of this talk

• Understand a bit of how the datastore works underneath the hood

• Have a conceptual background for the persistence codelab

5


Understanding the datastore

• The underlying Bigtable• Indexing and queries• Complex queries• Entity groups• Underlying infrastructure

6


Datastore layers

7

Complex queries

Entity Group Transactions

Queries on properties

Key range scan

Get and set by key

Datastore ✓ ✓ ✓ ✓ ✓Megastore ✓ ✓ ✓ ✓Bigtable ✓ ✓


Datastore layers

8

Complex queries



Get and set by key, key range scans

Datastore ✓ ✓ ✓ ✓Megastore ✓ ✓ ✓Bigtable ✓

Complex queries



Key range scan

Get and set by key



What does a Bigtable row look like?

9

Source: http://static.googleusercontent.com/external_content/untrusted_dlcp/labs.google.com/en/us/papers/bigtable-osdi06.pdf


Bigtable API

• “Give me the column ‘name’ at key 123”• “Set the column ‘name’ at key 123 to ‘ikai’”• “Give me all columns where the key is greater than 100 and less

than 200”

10


Datastore layers

11

Complex queries





Complex queries



Key range scan

Get and set by key



Megastore API

• “Give me all rows where the column ‘name’ equals ‘ikai’”• “Transactionally write an update to this group of entities”• “Do a cross datacenter write of this data such that reads will be

strongly consistent” (High Replication Datastore)• Megastore paper: http://www.cidrdb.org/cidr2011/Papers/

CIDR11_Paper32.pdf

12


http://www.cidrdb.org/cidr2011/Papers/CIDR11_Paper32.pdf




Datastore layers

13

Complex queries





Complex queries



Key range scan

Get and set by key



App Engine Datastore API

• “Give me all Users for my app where the name equals ‘ikai’, company equals ‘Google’, and sort them by the ‘awesome’ column, descending”

14



Queries


Let’s save an Entity with the low-level Java API DatastoreService datastore = DatastoreServiceFactory

.getDatastoreService();

Entity ikai = new Entity("User", "[email protected]");

ikai.setProperty("firstName", "ikai"); ikai.setProperty("company", "google");

ikai.setUnindexedProperty("biography", "Ikai is a great man, a great, great man."); datastore.put(ikai);

16


mailto:[email protected]


Get an instance of the DatastoreServiceDatastoreService datastore = DatastoreServiceFactory





17

Fetch a client instance




Instantiate a new EntityDatastoreService datastore = DatastoreServiceFactory





18

Set the Entity Kind




Instantiate a new EntityDatastoreService datastore = DatastoreServiceFactory





19

Set a unique key




Set indexed propertiesDatastoreService datastore = DatastoreServiceFactory





20

First argument is the property name

Second argument is the property value




Set unindexed propertiesDatastoreService datastore = DatastoreServiceFactory





21

This property will be saved, but we will not run queries against it




Commit the entity to the datastoreDatastoreService datastore = DatastoreServiceFactory





22

Save the thing!




What happens when we save?

23

Write the entity

Write the indexes

Make the write RPC Success!


What actually gets written?

24

Bigtable key Value

AppId:User:[email protected] ( Protobuf serialized entity - includes firstName, company and biography values )

Read more: http://code.google.com/appengine/articles/storage_breakdown.html

Bigtable key Value

AppId:User:firstName:ikai:[email protected] ( Empty )

AppId:User:company:google:[email protected] ( Empty )

Entities table

Indexes table




http://code.google.com/appengine/articles/storage_breakdown.html






Now let’s run a query

• If we have the key, we can fetch it right away by key• What if we don’t? We need indexes.

25


Let’s run a queryDatastoreService datastore = DatastoreServiceFactory


Query queryByName = new Query("User");

queryByName.addFilter("firstName", FilterOperator.EQUAL, "ikai");

List<Entity> results = datastore.prepare( queryByName).asList( FetchOptions.Builder.withDefaults());

// Roughly equivalent to: // SELECT * from User WHERE firstname = ‘ikai’;

26


Step 1: Query the indexes table

27

Bigtable key Value



Bigtable key Value



Entities table

Indexes table

Scan the indexes table for values >= AppId:User:firstName:










Step 2: Start extracting keys

28

Bigtable key Value



Bigtable key Value



Entities table

Indexes table

That gets us this row - extract the key [email protected]












Step 3: Batch get the entities themselves

29

Bigtable key Value



Bigtable key Value



Entities table

Indexes tableNow let’s go back to the entities table and fetch that key. Success!










Key takeaways

• This isn’t a relational database– There are no full table scans– Indexes MUST exist for every property we want to query– Natively, we can only query on matches or startsWith queries– Don’t index what we never need to query on

• Get by key = one step. Query on property value = 2 steps

30


Let’s run a more complex query!DatastoreService datastore = DatastoreServiceFactory


Query queryByName = new Query("User");

queryByName.addFilter("firstName", FilterOperator.EQUAL, "ikai");

queryByName.addFilter("company", FilterOperator.EQUAL, "google");

List<Entity> results = datastore.prepare( queryByName).asList( FetchOptions.Builder.withDefaults());

// Roughly equivalent to: // SELECT * from User WHERE firstname = ‘ikai’ // AND company = ‘google’;

31


Query resolution strategies

• This query can be resolved using built in indexes– Zig zag merge join - we’ll cover this example

• Can be optimized using composite indexes

32


Zig zag across multiple indexes

33


Bigtable key

AppId:User:firstName:alfred:[email protected]

AppId:User:firstName:ikai:[email protected]



AppId:User:firstName:zed:[email protected]

Bigtable key

AppId:User:company:acme:[email protected]

AppId:User:company:google:[email protected]



AppId:User:company:megacorp:[email protected]

Begin by scanning indexes >= AppId:User:company:google

























34


Bigtable key






Bigtable key






There’s at least a partial match, so we “jump” to the next index























35


Bigtable key






Bigtable key





AppId:User:company:megacorp:[email protected] to the next index. Start a scan for keys >= AppId:User:firstName:ikai:[email protected]























36


Bigtable key






Bigtable key





AppId:User:company:megacorp:[email protected], so that’s a twist. The first value that matches has key [email protected]! Does this value exist in the first index?
























37


Bigtable key






Bigtable key






Let’s advance the original cursor to >= AppId:User:company:google:[email protected]

























38


Bigtable key






Bigtable key






Alright! We found a match. Let’s add the key to our in memory list and go back to the first index

























39


Bigtable key






Bigtable key






Bigtable key






Bigtable key






Let’s move on to see if there are any more matches. Let’s start at [email protected]

























Bigtable key






Bigtable key







40


Are there any keys >= AppId:User:firstName:ikai:[email protected]?

























41


Bigtable key






Bigtable key






Bigtable key






No. We’re at the end of our index scans. Let’s do a batch key of our list of keys: [ ‘[email protected]’ ]

Bigtable key





























Batch get the entities themselves

42

Bigtable key Value



Entities table

Now let’s go back to the entities table and fetch that key. Success!






Let’s change the shape of the data

• Zig zag performance is HIGHLY dependent on the shape of the data

• Let’s go ahead and muck with the data a bit

43


Same query, sparsely distributed matches

44


Bigtable key



AppId:User:firstName:igor:[email protected]



Bigtable key






























45


Bigtable key






Bigtable key






Begin by scanning indexes >= AppId:User:company:google

























46


Bigtable key






Bigtable key






Move to the next index. Start a scan for keys >= AppId:User:firstName:ikai:[email protected]


























47


Bigtable key






Bigtable key






Oh ... no matches. Let’s move back to the first index and move the cursor down

























48


Bigtable key






Bigtable key






Okay, we’ve got another Googler
























49


Bigtable key






Bigtable key






Move to the next index. Start a scan for keys >= AppId:User:firstName:ikai:[email protected]

























50


Bigtable key






Bigtable key






Oh ... no matches here either. Let’s go back to the first index.

























51


Bigtable key






Bigtable key






Oh ... no matches here either. Let’s go back to the first index.

... if these indexes were huge, we could be here for a while!
























What happens in this case?

• If we traverse too many indexes, the datastore throws a NeedIndexException

• We’ll want to build a composite index

52


Composite index

53


Bigtable key

AppId:User:company:acme:firstName:alfred:[email protected]

AppId:User:company:google:firstName:david:[email protected]

AppId:User:company:google:firstName:ikai:[email protected]

AppId:User:company:google:firstName:max:[email protected]

AppId:User:company:megacorp:firstName:zed:[email protected]














Composite index

54


Bigtable key






Search for all keys >= AppId:User:company:google:firstName:ikai














Composite index

55


Bigtable key






Well, that was much faster, wasn’t it?














Composite index tradeoffs

• Created at entity save time - incurs additional datastore CPU and storage quota

• You can only create 200 composite index• You need to know the possible queries ahead of time!

56


Complex Queries takeaways

• This isn’t a relational database – There are no full table scans– Indexes MUST exist for every property we want to query

• Performance depends on the shape of the data• Worse case scenario: if your query matches are highly sparse• Build composite indexes when you need them

57



Entity Groups


Why entity groups?

• We can perform transactions within this group - but not outside• Data locality - data are stored “near” each other• Strongly consistent queries when using High Replication

datastore within this entity group

59


Entity groups and transactions

• A hierarchical structuring of your data into Megastore’s unit of atomicity

• Allows for transactional behavior - but only within a single entity group

• Key unit of consistency when using High Replication datastore

60


Example: Data for a blog hosting service

61

Comment

Blog

Entry

User

Has manyHas many

Has many


Example: Data for a blog hosting service

62

Comment

Blog

Entry

User

Has manyHas many

Has many

This can be structured as an entity group (tree structure)!


Structure this data as an entity group

63

Blog

Entry

User

Blog

Entry Entry

CommentCommentComment

Entity group root


How are entity groups stored?

64

Bigtable key ValueAppId:User:[email protected] ( Protobuf serialized User )

AppId:User:[email protected]/Blog:123 ( Protobuf serialized Blog )

AppId:User:[email protected]/Blog:123/Entry:456 ( Protobuf serialized Entry )


AppId:User:[email protected]/Blog:123/Entry:456/Comment:111

( Protobuf serialized Comment )





Read more: http://code.google.com/appengine/docs/python/datastore/entities.html

Entities table
















http://code.google.com/appengine/docs/python/datastore/entities.html



65












Entities table Entity groups have a single root entity



















66












Entities table

Child entities embed the entire ancestry in their keys


















Let’s write an entity group transactionallyDatastoreService datastore = DatastoreServiceFactory


Entity ikai = new Entity("User", "[email protected]"); Entity blog = new Entity("Blog", "ikaisays.com", ikai.getKey()); Entity entry = new Entity("Entry", "datastore-intro", blog.getKey()); // Auto assign an ID Entity comment = new Entity("Comment", entry.getKey()); Transaction tx = datastore.beginTransaction(); // Helper function for clarity datastore.put(Arrays.asList(ikai, blog,entry, comment)); tx.commit();

67







68

Create the root entity







69

This is the first child entity - notice the third argument, which specifies the parent entity key







70

The next deeper entity sets the blog as the parent







71

We can also opt to not provide a key name and just use a parent key for a new entity







72

Start a new transaction







73

Put the entities in parallel







74

Actually commit the changes




Step 1: Commit

75

Commit Changes to entities visible

Changes to entities and indexes visible

Roll the timestamp forward on the root entity


Step 2: Entity visible

76



On read, check for the most recent timestamp on the root entity

This is the version we want since it represents a complete write


Step 3: Indexes updated

77



Indexes are written - now we can query for this entity with the new properties


Entity group and transactions takeaways

• Structure data into hierarchical trees– Large enough to be useful, small enough to maximize

transactional throughput

• Transactions need an entity group root - roughly 1 transaction/second– If you write N entities that are all part of 1 entity group, it counts as

1 write

• Optimistic locking used - can be expensive with a lot of contention

78


General datastore tips

• Denormalize as much as possible– As much as possible, treat datastore as a key-value store

(Dictionary or Map like structure)– Move large reporting to offline processing. This lets you avoid

unnecessary indexes

• Use entity groups for your data• Build composite indexes where you need them - “need” depends

on shape of your data

79



Questions?



Introducing the App Engine datastore

Technology

helper function

auto assign

roughly equivalent

protobuf serialized

protobuf serialized

protobuf serialized

scanning indexes

protobuf serialized