This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
dynamodb-pm@
Event Driven Computing Enabled by DynamoDB Streams
Launch update on Cross-region replication and Database Triggers with AWS Lambda Integration
Worker worker = new Worker(recordProcessorFactory, workerConfig, adapterClient, dynamoDBClient, ..);
Thread t = new Thread(worker);t.start();
Full code available online dynamodb-pm@
Processing streams (KCL)
public class StreamsRecordProcessor implements IRecordProcessor { .. @Override public void processRecords(List<Record> records,.. ) { for(Record record : records) {
if (record instanceof RecordAdapter) {
Record ddbStreamRecord = ((RecordAdapter) record).getInternalObject();
switch(ddbStreamRecord.getEventName()) { case "INSERT" : case "MODIFY" : DemoHelper.putItem(dynamoDBClient, tableName,
• Allow each person to vote only once• No changing votes• Real-time aggregation• Voter analytics, demographics
Real-time voting architecture
AggregateVotesTable
Voters
RawVotes Table
Voting App
Partition 11000 WCUs
Partition K1000 WCUs
Partition M1000 WCUs
Partition N1000 WCUs
Votes Table
Candidate A Candidate B
Scaling bottlenecks
50,000/sec
70,000/sec
Voters
Provision 200,000 WCUs
Write sharing
Candidate A_2
Candidate B_1
Candidate B_2
Candidate B_3
Candidate B_5
Candidate B_4
Candidate B_7
Candidate B_6
Candidate A_1
Candidate A_3
Candidate A_4Candidate A_7 Candidate B_8
Candidate A_6 Candidate A_8
Candidate A_5
Voter
Votes Table
Write sharding
Candidate A_2
Candidate B_1
Candidate B_2
Candidate B_3
Candidate B_5
Candidate B_4
Candidate B_7
Candidate B_6
Candidate A_1
Candidate A_3
Candidate A_4Candidate A_7 Candidate B_8
UpdateItem: “CandidateA_” + rand(0, 10)ADD 1 to Votes
Candidate A_6 Candidate A_8
Candidate A_5
Voter
Votes Table
Votes Table
Shared aggregation
Candidate A_2
Candidate B_1
Candidate B_2
Candidate B_3
Candidate B_5
Candidate B_4
Candidate B_7
Candidate B_6
Candidate A_1
Candidate A_3
Candidate A_4
Candidate A_5
Candidate A_6 Candidate A_8
Candidate A_7 Candidate B_8
Periodic Process
Candidate ATotal: 2.5M
1. Sum2. Store Voter
Correctness in voting
UserId Candidate DateAlice A 2013-10-02
Bob B 2013-10-02
Eve B 2013-10-02
Chuck A 2013-10-02
RawVotes Table
Segment VotesA_1 23
B_2 12
B_1 14
A_2 25
AggregateVotes Table
Voter1. Record vote and de-dupe; retry 2. Increment candidate counter
Correctness in aggregation?
UserId Candidate DateAlice A 2013-10-02
Bob B 2013-10-02
Eve B 2013-10-02
Chuck A 2013-10-02
RawVotes Table
Segment VotesA_1 23
B_2 12
B_1 14
A_2 25
AggregateVotes Table
Voter
Real-time voting architecture (improved)
AggregateVotesTable
Amazon Redshift Amazon EMR
Your Amazon Kinesis–
Enabled App
Voters RawVotes TableVoting App RawVotesDynamoDB
Stream
Real-time voting architecture
AggregateVotesTable
Amazon Redshift Amazon EMR
Your Amazon Kinesis-
Enabled App
Voters RawVotes TableVoting App RawVotesDynamoDB
Stream
Handle any scale of election
Real-time voting architecture
AggregateVotesTable
Amazon Redshift Amazon EMR
Your Amazon
Kinesis-Enabled app
Voters RawVotes TableVoting App RawVotesDynamoDB
Stream
Vote only once, no changing votes
Real-time voting architecture
AggregateVotesTable
Amazon Redshift Amazon EMR
Your Amazon
Kinesis–Enabled App
Voters RawVotes TableVoting app RawVotesDynamoDB
Stream
Real-time, fault-tolerant, scalable aggregation
Real-time voting architecture
AggregateVotesTable
Amazon Redshift Amazon EMR
Your Amazon
Kinesis–Enabled App
Voters RawVotes TableVoting app RawVotesDynamoDB
Stream
Voter analytics, statistics
Analytics with DynamoDB Streams
• Collect and de-dupe data in DynamoDB• Aggregate data in-memory and flush periodically• Important when: Performing real-time aggregation and analytics
Op: PUTJohnTokyo
Op: UPDATEJohnPluto
Op: UPDATEJohnMars
DynamoDB writes and Streams
Operation #
DynamoDB Operation
Data inDynamoDB
Data in Streams
1 PUT : {John:Tokyo}
{John:Tokyo} PUT John Tokyo
2 UPDATE:{John:Mars}
{John:Mars} UPDATE John Mars
3 UPDATE:{John:Pluto}
{John:Pluto} UPDATE John Pluto
Data: {Name:Destination}
dynamodb-pm@
View Type Destination
Old Image – Before update Name = John, Destination = Mars
New Image – After update Name = John, Destination = Pluto
Old and New Images Name = John, Destination = MarsName = John, Destination = Pluto
Keys Only Name = John
View types
dynamodb-pm@
Features of DynamoDB Streams
dynamodb-pm@
Streams Characteristics
• Each item update appears exactly once • Records are strictly ordered by time• Streams are Asynchronous
dynamodb-pm@
Durability & high availabilityHigh throughput consensus protocolReplicated across multiple AZs
dynamodb-pm@
Managed StreamsSimply enable streams
dynamodb-pm@
ElasticityAdjusts to table throughput
dynamodb-pm@
PerformanceDesigned for sub-second latency
| |
Sub-second latency
dynamodb-pm@
DurabilityRecords available for 24 hours
dynamodb-pm@
How much does it cost?
• Free to turn it on• First 2.5 million reads per month are Free• $0.20 per million reads after that
DynamoDB StreamsDynamoDBdynamodb-pm@
dynamodb-pm@
What Customers Are Saying
Mapbox
ProblemMaking mapping data highly available, even faster.
DynamoDB Streams use-caseCross-region Replication
In their own words“DynamoDB Streams unlocks cross-region replication - a critical feature that enabled us to fully migrate to DynamoDB. Cross-region replication allows us to distribute data across the world for redundancy and speed.” - Jake Pruitt, Software Developer, Mapbox
TOKYU HANDS ProblemAugmenting Point of Sale system to react in real time to inventory and customer data
In their own words“TOKYU HANDS is running in-store Point Of Sales system backed by DynamoDB and various AWS services. We really like full-managed services such as DynamoDB. I believe DynamoDB Streams would help us making the system more sophisticated and more automated.” - Yamazaki-san, Cloud Architect, TOKYU HANDS.
The local version of DynamoDB
• Desktop Installable• Development & Testing• Publicly available at DynamoDB.com