Top Banner
5 Signs You’ve Outgrown DynamoDB © 2018 Aerospike, Inc. All rights reserved. Aerospike and the Aerospike logo Are trademarks or registered trademarks of Aerospike. All other names and trademarks are for identification purposes and are the property of their respective owners. 2525 E. Charleston Road, Suite 201 Mountain View, CA 94043 Tel: 408 462 AERO (2376) www.aerospike.com Executive Summary Companies choose databases with both core business objectives and technical objectives in mind. Often they select a database that seems to be the best choice at first glance, as well as the path of least resistance, and then are subsequently surprised by cost overruns and technology limitations that quickly hinder productivity and put the business at risk. This seems to be the case with many enterprises that chose Amazon Web Service’s (AWS) DynamoDB. DynamoDB is a solid product that appears to be a sound choice for companies that already leverage the AWS platform. However, it’s not a one-size-fits-all-for-all-uses product, and there are areas of concern that could clearly indicate you’ve outgrown this technology. Unfortunately, you can’t change your business requirements to fit the limitations of the technology, and the technology is unlikely to change to meet your expanding needs. What are the signs that you’ve outgrown DynamoDB? Here are 5 that we cover in this paper: Sign 1 Your business is growing but on-demand pricing is now pricing you out Sign 2 You can no longer tolerate variable performance and latency Sign 3 Your business requires a duality of cloud and non-cloud platforms Sign 4 You are paying more and more for DevOps and testing Sign 5 You are stuck with one vendor and have no room to maneuver Ask yourself these questions: Are you concerned about the limitations and issues of DynamoDB? Are you looking for a viewpoint based upon broader experience? In this paper, we’ll remove the noise from the world of AWS DynamoDB. We’ll look specifically at what this technology does, and, more importantly, limitations you should consider to help determine if you’ve outgrown DynamoDB. Or, issues you should consider before you select this technology in the first place. Throughout this paper, we will provide you with an objective view of the information that’s currently available, and help you make the right choices, in terms of both the business and the technology you’re looking to deploy.
14

5 Signs You’ve Outgrown DynamoDB - Amazon S3€¦ · 5 Signs You’ve Outgrown DynamoDB 3 Cost of Capacity: Higher read performance costs more. A lot more. Looking at Figure 1,

May 20, 2020

Download

Documents

dariahiddleston
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: 5 Signs You’ve Outgrown DynamoDB - Amazon S3€¦ · 5 Signs You’ve Outgrown DynamoDB 3 Cost of Capacity: Higher read performance costs more. A lot more. Looking at Figure 1,

5 Signs You’ve Outgrown DynamoDB

©2018Aerospike,Inc.Allrightsreserved.AerospikeandtheAerospikelogoAretrademarksorregisteredtrademarksofAerospike.Allothernamesandtrademarksareforidentificationpurposesandarethepropertyoftheirrespectiveowners.

2525 E. Charleston Road, Suite 201 Mountain View, CA 94043 Tel: 408 462 AERO (2376) www.aerospike.com

ExecutiveSummary

Companies choose databases with both core business objectives and technical objectives in mind. Often they select a database that seems to be the best choice at first glance, as well as the path of least resistance, and then are subsequently surprised by cost overruns and technology limitations that quickly hinder productivity and put the business at risk.

This seems to be the case with many enterprises that chose Amazon Web Service’s (AWS) DynamoDB. DynamoDB is a solid product that appears to be a sound choice for companies that already leverage the AWS platform. However, it’s not a one-size-fits-all-for-all-uses product, and there are areas of concern that could clearly indicate you’ve outgrown this technology. Unfortunately, you can’t change your business requirements to fit the limitations of the technology, and the technology is unlikely to change to meet your expanding needs.

What are the signs that you’ve outgrown DynamoDB? Here are 5 that we cover in this paper:

Sign 1 Your business is growing but on-demand pricing is now pricing you out

Sign 2 You can no longer tolerate variable performance and latency

Sign 3 Your business requires a duality of cloud and non-cloud platforms

Sign 4 You are paying more and more for DevOps and testing

Sign 5 You are stuck with one vendor and have no room to maneuver

Ask yourself these questions: Are you concerned about the limitations and issues of DynamoDB? Are you looking for a viewpoint based upon broader experience?

In this paper, we’ll remove the noise from the world of AWS DynamoDB. We’ll look specifically at what this technology does, and, more importantly, limitations you should consider to help determine if you’ve outgrown DynamoDB. Or, issues you should consider before you select this technology in the first place.

Throughout this paper, we will provide you with an objective view of the information that’s currently available, and help you make the right choices, in terms of both the business and the technology you’re looking to deploy.

Page 2: 5 Signs You’ve Outgrown DynamoDB - Amazon S3€¦ · 5 Signs You’ve Outgrown DynamoDB 3 Cost of Capacity: Higher read performance costs more. A lot more. Looking at Figure 1,

5 Signs You’ve Outgrown DynamoDB

2

Sign1:Yourbusinessisgrowingbuton-demandpricingisnowpricingyouout

While this was a question of CapEx versus OpEx at the beginning of the cloud computing discussion, many enterprises find that on-demand pricing is not as advantageous as they once hoped. Some enterprises are getting cloud bills that are almost double and triple what they originally thought they would be.1,2 While there is no hardware to own, the price/performance for a database as a service is often many times that of on-premises systems.

There are a couple of key attributes that drive higher pricing when it comes to Amazon DynamoDB. One is the migration of data when onboarding and the other is the pricing basis that tends to rely heavily upon usage capacity. Many of these factors are not considered when initially comparing one database to another. For DynamoDB, true costs are uncovered a few months after deployment when the usage and cost patterns are better known.

CostofStrategicProvisioning:MigratingdatafromanotherdatasourceintoDynamoDB When a few existing database customers moved from Aerospike3 to DynamoDB (and then later returned – ergo our uncovering this issue), these customers found that they needed a high number of capacity units to migrate from Aerospike in a timely manner. This ended up costing them 10 times the amount they expected for migration.

After the migration, they then decreased the number of capacity units to a steady state value to reduce costs, but they could not decrease the number of partitions needed for DynamoDB. It’s safe to say that if you increase your capacity requirement for the sake of faster data load, DynamoDB creates a lot of internal partitions. Indeed, there is a cap of 1000 write units per partition that you need to consider.

Since the pricing is based upon RCU and WCU (Read Capacity Units and Write Capacity Units) - customers pay a premium to migrate data into the platforms at a faster rate. If you increase your capacity requirement for the sake of faster data load, AWS creates many internal partitions, but does not allow you to reduce the number of partitions for steady-state usage.

So, in a way, customers are paying for excess capacity they never really use. Further, if you have a read/write pattern which has affinity to a partition, you will be limited by the per-partition limit, i.e. 1000, even though you have more overall capacity.

Many DynamoDB customers fall into this trap. They went with high capacity to migrate data from Aerospike or other databases, and now are stuck with excess capacity. Migration costs can increase as much as 3 to 5 times that of leveraging a database such as Aerospike. Moreover, it’s difficult to anticipate the additional costs associated with overprovisioning for migration.

What options remain? Either customers continue to pay tens of thousands of dollars a month, or they reduce the capacity and thus reduce throughput. The result is that customers pay for more capacity than what they actually use when leveraging DynamoDB. This is one of the primary considerations when cost-optimizing DynamoDB usage, with a focus on lean, strategic provisioning and migration.

1 https://medium.com/google-cloud/the-hidden-costs-of-cloud-ddb702495e93 2 https://assets.rightscale.com/uploads/pdfs/Optimize-Cloud-Costs-RightScale-White-Paper-by-EMA.pdf 3 https://hackernoon.com/beware-of-dilution-of-dynamodb-throughput-due-to-excessive-scaling-4df51063edae

Page 3: 5 Signs You’ve Outgrown DynamoDB - Amazon S3€¦ · 5 Signs You’ve Outgrown DynamoDB 3 Cost of Capacity: Higher read performance costs more. A lot more. Looking at Figure 1,

5 Signs You’ve Outgrown DynamoDB

3

CostofCapacity:Higherreadperformancecostsmore.Alotmore. Looking at Figure 1, let’s examine the test case of one billion records with a record size of 4 KB and a 150,000 reads-per-second with 150,000 writes-per-second and a replication factor of three. It is easy to compare the yearly costs of capacity between hourly rate for a year, hourly rate for a 1-year upfront purchase, and hourly rate for a 3-year upfront purchase. All things are never equal, but it’s helpful to understand the basic metrics of cost evaluation for each database. This data set is representative of what you’ll typically see, in terms of costs differences on AWS between DynamoDB and Aerospike.

Figure 1: As you can see by this cost data, what you pay for each database very much depends on the pattern of use. However, as you can also see, the cost of DynamoDB with DAX (needed to match Aerospike performance) is typically much higher.

Note: There is a 3-year upfront pricing option for DynamoDB that is typically a one-time deal with a heavy (70-80%) discount that reverts to 1-year pricing thereafter. If customers want more flexible hourly/annual rate, they can get that too - just at a considerably higher rate. Many enterprises fail to factor these issues in.

When using DynamoDB, you pay a flat, hourly rate based upon how much capacity you have provisioned - Read Capacity Units (RCU) and Write Capacity Units (WCU). Pricing also varies by the region. Capacity planning is obviously a huge aspect of ensuring cost efficiency for DynamoDB.

$-

$1,000,000

$2,000,000

$3,000,000

$4,000,000

$5,000,000

$6,000,000

$7,000,000

DynamoDB DynamoDBw/DAX AerospikeCrossAZDataCosts

AerospikeInstanceCost

1YearOperationalChargesonAWS:DynamoDBvs.Aerospike1Brecords,4krecordsize,150kTPSReads,150kTPSWrites,RF3

Hourlyrate 1yearupfrontrate 3yearupfrontrate

Page 4: 5 Signs You’ve Outgrown DynamoDB - Amazon S3€¦ · 5 Signs You’ve Outgrown DynamoDB 3 Cost of Capacity: Higher read performance costs more. A lot more. Looking at Figure 1,

5 Signs You’ve Outgrown DynamoDB

4

Recently DynamoDB introduced DAX – an accelerator for super low latency queries. DAX is a caching mechanism and is priced independently as Memory/hr. For example, if you want half your records on DAX for faster queries, you need to understand that costs will increase accordingly.

Table 1 (below) exhibits a business that has a mix of reads/writes at very high TPS. The operational charges from Amazon for DynamoDB/DAX will cost you approximately 85% more than a comparable Aerospike database.

Table 1: When looking at usage patterns, you need to understand the cost over time, as well as growth. While the on-demand model is often attractive, it could be costing you several times that of other database options.

These operational costs do not include any scaling nor growth you may encounter in subsequent years. Moreover, to achieve similar latency requirements as those from Aerospike (e.g., <1ms), DAX will be required – which incurs significant costs (as indicated) – even with 1- and 3-year upfront discounting (modeled; not typically available). If you don’t care about latency (which can be 10x that of Aerospike), then you can use Dynamo without DAX and save some money. However, as your business requirements change, you will be forced to revisit this decision.

(For DynamoDB and DAX performance dynamics see Sign 2.)

Page 5: 5 Signs You’ve Outgrown DynamoDB - Amazon S3€¦ · 5 Signs You’ve Outgrown DynamoDB 3 Cost of Capacity: Higher read performance costs more. A lot more. Looking at Figure 1,

5 Signs You’ve Outgrown DynamoDB

5

CostofConsistency:Ifyoudon’tlikestale,payup

DynamoDB has two modes of reads – Eventually Consistent and Strongly Consistent. It does allow more “Eventually Consistent Reads” per hour than “Strongly Consistent Reads”.

In Eventually Consistent Reads, the response might not reflect the results of a recently completed write operation. Therefore, the response might include some stale data. On the other hand, Strongly Consistent Reads will return the most up-to-date copy of the data. Determining whether the needs will be sufficiently served by one or the other depends upon the business.

It is important to note that strong consistency has a price. Strongly Consistent Reads cost twice as much, hence customers should choose wisely.

Conclusion: The published numbers do not make DynamoDB more compelling when looking at growth over time. Upfront costs for migration, costs for high performance and costs beyond three years will result in costs greater than expected for DynamoDB.

Sign2:Youcannolongertoleratevariableperformanceandlatency.

Running workloads on public clouds has many advantages, but latency is not one of them. You need to consider the true cost to leverage an on-demand platform considering that network latency and noisy neighbor issues could be systemic.

NoisyNeighborswilldriveinconsistentperformance Noisy neighbors, also known as stolen CPUs, implies some negative intent from your virtual neighbors, but intent has nothing to do with how your performance can be adversely affected. In reality, it is a relative measure of the cycles a CPU should deliver to your database but could not due to other tenants on that infrastructure diverting cycles away. Databases are especially sensitive to noisy neighbors because of their CPU-intensive tasks. This results in burst-y behavior when there are drops and rises in performance based upon how the public cloud-based systems allocate CPU time to your specific process - in this case, a database.

Some of these “stolen” cycles are from the hypervisor enforcing a quota based upon the type of subscription you have in place which would represent an upper limit that would be hit by the spikes in CPU utilization. In other cases, such as the one shown below (Figure 2), the amount of diverted CPU cycles varies over time, presumably due to the same physical hardware that also requests CPU cycles from the underlying hardware. In short, you’re competing for the same hardware resources, and you won’t win all of the time.

Figure 2 shows a graph of CPU usage on a host, with ‘stolen’ CPUs in yellow. Light blue denotes ‘idle’ or available cycles, purple denotes ‘user’ or cycles spent executing application code, and dark blue denotes ‘system’ or cycles spent executing kernel code. In this case we can see that the amount of ‘stolen’ CPUs (yellow) is significant and in a couple of points in time pushes CPU utilization to 100% threatening your processes (purple) headroom (light blue).

Page 6: 5 Signs You’ve Outgrown DynamoDB - Amazon S3€¦ · 5 Signs You’ve Outgrown DynamoDB 3 Cost of Capacity: Higher read performance costs more. A lot more. Looking at Figure 1,

5 Signs You’ve Outgrown DynamoDB

6

Figure 2 In this graphic you can clearly see the stolen CPUs which are in yellow. This is from a typical AWS performance monitoring graphic (Source: Alexis Lê-Quôc)

While many databases run on the AWS public cloud as well, including Aerospike, this really is a consideration of the platform options and a downside of leveraging public cloud platforms that use all types of software systems. However, keep in mind that database technology such as Aerospike can run either on-premises or within the public AWS cloud as well as on other cloud providers’ platforms.

Since public clouds such as AWS are multitenant systems, they have to allocate resources as needed to tenant workloads that need them. Of course, while you can estimate needed capacity as to the pool of resources needed. There are a few ways to deal with this issue:

1. You need to leverage a database technology that can utilize the cloud-native platform features in ways that remove multitenant-induced latency due to either CPU or storage I/O cycles that are taken by other tenants. Since Aerospike is not an AWS cloud service, but a database that runs within AWS as a tenant, these latencies will be reduced due to the process AWS uses to allocate and prioritize resources for customer processes above those for their own native platform services.

2. You need to leverage database technology that runs on a public cloud provider as well as on premises. (See Sign 3.) While it seems like a trendy thing to leverage cloud-only databases, such as DynamoDB, it’s still important to have the option of running in both places, or one or the other. This choice provides you with ways to work around latency and cost issues that can’t be easily and inexpensively resolved if you’re locked into public cloud-only databases, such as DynamoDB.

3. You should look for a database that utilizes the CPU more efficiently to further mitigate the noisy neighbor conundrum.

Benchmark results for DynamoDB and DynamoDB with DAX expose their variability inperformancewhencomparedtoAerospike

In our benchmark analysis (see Figure 3 below), we prove that Aerospike performs best with the lowest latency and highest query throughput when compared to DynamoDB under scenarios of very heavy Read workloads and a 50-50 Read/Write workload. Keep in mind that performance is not just about completing database tasks the fastest. It’s also about productivity, which can cost much more than the cost of the database technology, depending upon how you’re set up within your IT organization.

Page 7: 5 Signs You’ve Outgrown DynamoDB - Amazon S3€¦ · 5 Signs You’ve Outgrown DynamoDB 3 Cost of Capacity: Higher read performance costs more. A lot more. Looking at Figure 1,

5 Signs You’ve Outgrown DynamoDB

7

Our tests involved the following:

- 1 billion records @4KB data size

- Heavy Read workloads

- Heavy Write workloads

- 3 DAX instances per cluster

- a cache hit ratio of 52%

Summaryfindings:

Figure 3 Benchmark data, Aerospike vs. DynamoDB using different configurations and usage patterns. Note that both DynamoDB and DynamoDB with DAX has latency numbers are much higher for average, 95% and 99.9%.

Aerospike is guaranteed to be under 1ms 99% of time under very high TPS of 200K-500K (see Appendix). This is indicative of workloads a firm that utilizes transactional analytics will see. Though Dynamodb with DAX (accelerator) is intended to for lower latencies, the costs of DAX can be prohibitively high.

Latency can only be improved by upgrading the AWS instance type, increasing the number of clusters and sharding, creating a hot key set that allows for a subset of data to fit onto a DAX cluster or upping the cache hit ratio for DAX. For example, with a cache hit ratio of 52%, we observed average latencies of 2.5ms to 3.5ms (see Appendix). For sub-ms (<1ms) performance, it was observed that a cache hit ratio of 90% would be required.

Worse, there is also throttling between the DAX servers and the DynamoDB servers. So trying to warm cache can be a long and tedious process. The process of warming cache may take as long as 5 or 6 hours, depending upon the cache hit ratio that you are trying to achieve. A 50% cache hit ratio will most likely be 2 to 3 hours.

2.98

240 113

4.54

60.15 99

02468

101214161820

Aerospike,100%Reads

DynamoDB,100%Reads

Dynamo/DAX,100%Reads

Aerospike,50/50R/W

DynamoDB,50/50R/W

Dynamo/DAX,50/50R/W

LatencyBenchmark(ms)

AvgLatency 95%Latency 99.9%Latency

Page 8: 5 Signs You’ve Outgrown DynamoDB - Amazon S3€¦ · 5 Signs You’ve Outgrown DynamoDB 3 Cost of Capacity: Higher read performance costs more. A lot more. Looking at Figure 1,

5 Signs You’ve Outgrown DynamoDB

8

Trying to improve DynamoDB latency with DAX is far from a cure-all:

• Applications need additional code to use DAX: applications need to know which keys are in which DAX cluster.

• When using DAX, write latency actually goes up at the cost of having good reads; you essentially have to “write-through”.

• You're essentially playing a game of hot/warm/cold data by placing some in DAX, some in Dynamo. If your data is in cache, you can expect good read performance with DAX. If not, you’ll see a sizable performance difference/variance.

• Aerospike’s range of high performance is much broader. For instance, with Aerospike you can get up to 700K TPS without any cost increment while at the same latency levels (see Appendix). By contrast, DynamoDB is literally priced by TPS – you get nothing for free.

Conclusion: While latency issues within multitenant public clouds are a well-documented technical fact, you need to consider the impact of this type of latency on your organization as well. The trends indicate most choices are to either run within a public cloud, such as AWS, or run on-premises. Keep in mind that most database requirements, including those that deal with performance issues, will change over time. The more adaptable the database technology, such as the ability to offer both a public cloud and on premises version, the more likely the organization is to be successful with that database technology.

Sign3:Yourbusinessrequiresadualityofcloudandnon-cloudplatforms

As we stated in the discussion above, Aerospike can run on both cloud and non-cloud platforms. This capability provides lock-in avoidance value. Another value is Aerospike’s ability to support both platforms’ operations, cloud and non-cloud. DynamoDB supports only a public IaaS cloud, AWS.

Many enterprises looking to build databases systems need the operational flexibility of running both cloud and non-cloud platforms to deal with real world business issues. Examples include laws that require the data to remain on premises, or latency issues that can only be addressed with the data being co-located with the applications, either cloud or non-cloud.

A theme that often comes up is Business Continuity. Enterprises tend to adopt a Hybrid model when it comes to cloud. Some data is deemed to be ok in public cloud but some aren't and are kept in their own data centers. If an enterprise needs a similar high performance database in its public cloud as well as in its own private cloud, then a cloud-only DynamoDB choice limits their options. With Aerospike, the enterprise can run it in the on-premises datacenter as well as a selection of public cloud choices be it AWS, Azure or GCP.

Let’s consider an enterprise that needs to support on-premises data due to compliance issues. However, they like the agility and cost advantages of the public cloud. With DynamoDB, that enterprise is locked-in to the use of a cloud, and cloud only. Any requirements to keep data local for data governance compliance wont’ be compatible with the on-demand only options that AWS offers.

One need only look to the regulations that govern data in the EU as well as any organization that does business in the EU.

Page 9: 5 Signs You’ve Outgrown DynamoDB - Amazon S3€¦ · 5 Signs You’ve Outgrown DynamoDB 3 Cost of Capacity: Higher read performance costs more. A lot more. Looking at Figure 1,

5 Signs You’ve Outgrown DynamoDB

9

The General Data Protection Regulation (GDPR) (Regulation (EU) 2016/679) is a regulation by which the European Parliament, the Council of the European Union and the European Commission intend to unify data protection for all in the European Union (EU). Moreover, they also address the export of personal data outside the EU. GDPR includes aspects that address what data is being housed and where, how it is secured, and the flow of how users access and use it. As enterprises embrace public, private, and hybrid cloud strategies, and thus store data in diverse environments, they need to understand the regulatory and legal hold considerations of each location – including cross-border and security sensitivities.

In addition, the Payment Services Directive 2 (PSD2) is an EU Directive, administered by the European Commission to regulate payment services and payment service providers throughout the European Union (EU) and European Economic Area (EEA). PSD2 sets the stage for open banking by providing standardized access to customer data, enhancing payment security and lowering the barriers to entry for TPP and FinTech. The value nexus between financial institutions and payment providers lies in the specific ability to transact that customer data. Simply put, customer data needs to be exposed to many players across many services at a much faster rate, and all of it done securely.

These may manifest into two core issues:

First, PII (personally identifiable information) may only be stored within the company where the EU citizen resides.

Second, the ability to transport data back and forth between their own data center and the public cloud becomes a matter of compliance. Indeed, many public cloud providers do not guarantee that the data will remain in-country, since they leverage backup services that my replicate the data to storage systems out of the country4.

While most consider the cloud to be the ultimate platform, there are no absolutes when it comes to what platform to leverage. Flexibility of data hosting is key. Without the choice of both cloud and non-cloud, enterprises could find themselves migrating off of a cloud-based database, such as DynamoDB, at great risk and expense.

Conclusion: This section is about leveraging both cloud and non-cloud technology to deal with changing operational needs. As privacy, protection and compliance issues continue to arise, the ability to leverage both cloud and non-cloud, at the same time, is becoming not only an advantage but critical to success.

Sign4:YouarepayingmoreandmoreforDevOpsandtesting

Moving to DevOps? You’ll need dev and test instances as part of your database. If you pay on demand pricing for those instances, you could find that your database is cost prohibitive. Sticker shock around the cost of outfitting dev and test with required database instances drives many enterprises to the use of DevOps back on premises.5 This is a typical outcome of leveraging DynamoDB for a few core reasons:

1. While on-demand costs appear to provide better cost metrics for dev and test use of databases, the resources needed when considering dev, test, staging, and deployment are often far greater than those budgeted.

4 https://enterprisersproject.com/article/2017/1/three-things-companies-must-know-about-data-sovereignty-when-moving-cloud 5 https://www.infoworld.com/article/3195082/devops/the-3-big-speed-bumps-to-devops-in-the-cloud.html

Page 10: 5 Signs You’ve Outgrown DynamoDB - Amazon S3€¦ · 5 Signs You’ve Outgrown DynamoDB 3 Cost of Capacity: Higher read performance costs more. A lot more. Looking at Figure 1,

5 Signs You’ve Outgrown DynamoDB

10

2. Latency issues from leveraging a database which is remotely hosted are not typically factored in. This means that just minor adjustments for performance can run into thousands of lost dollars a day when considering the hit on dev test productivity.

3. Security testing becomes more complex considering that more security approaches and enabling technology need to come into play to handle native security on a public cloud, as well as native security within the on-premises systems.

If an enterprise’s DevOps included a lot of validation with dynamic data sets, the enterprise will pay a premium every time it must test its product/solution. Customers try to mitigate this by reserve instance, or paying upfront for cloud service, or spot instances (be there at the right time), but this quickly becomes a manageability issue.

Dedicated resources in the DevOps team that hunt for the cheapest way to get it done must constantly monitor, track and analyze service usage and “look for a deal” to reduce costs. This quickly causes a distraction, and reduces the effectiveness of DevOps processes and teams. This can be tracked if DevOps organizations are able to gather productivity metrics, which we recommend that you do to spot costs that are typically not tracked. For instance, in a typical DevOps organization with a hundred people doing Dev and Test at just a 5 percent loss in productivity can cost the business over $5,000 per day, or $1.3 million dollars per year.

It’s helpful to do a comparison of Dev and Test when it comes to database costs for a typical DevOps process or chain. In looking at Figure 4 below, it’s helpful to note how Dev and Test compare when using each database technology. While the on-demand aspect of AWS’s DynamoDB does assist in some costs, the use within a DevOps process and organization not only creates a cost disadvantage, but can be a disruption that carries many hidden costs.

Figure 4 When running a DevOps organization, you need to consider the cost of the database across Dev, Test, and Staging (Source: Cloud Technology Partners)

$-

$50,000.00

$100,000.00

$150,000.00

$200,000.00

$250,000.00

$300,000.00

CostofDev CostofTest CostofStaging

ImplementingDevOps:DynamoDBvs.Aerospike

DynamoDB Aerospike

Page 11: 5 Signs You’ve Outgrown DynamoDB - Amazon S3€¦ · 5 Signs You’ve Outgrown DynamoDB 3 Cost of Capacity: Higher read performance costs more. A lot more. Looking at Figure 1,

5 Signs You’ve Outgrown DynamoDB

11

Moreover, we’re also looking at the cost of staging, which many DevOps organizations leverage. This is often different than test, considering that we’re placing a database, and once again paying on demand pricing, that cost a great deal more than typically anticipated, especially considering that we’re not yet into production.

It’s a clear trend that leveraging on demand databases are far too expensive when it comes to DevOps, and DynamoDB shares that limitation. An emerging best practice is to consider cost models that are friendlier to the DevOps way of working, which means dealing with a lot of non-production use. If we begin paying for that all of the time, the value of DevOps is greatly diminished.

Conclusion: On demand database pricing may be fine for production. However, when you consider the cost of Dev and Test, and the fact that you’ll need a database instance for each, as well as staging, the costs are typically prohibitive when compared to on premises databases such as Aerospike.

You should also know that you have the ability to leverage Aerospike within the AWS cloud as well, where DynamoDB only supports on-demand. You need to carefully consider visible costs, such as the cost of database use itself. Moreover, consider hidden costs such as lost productivity.

Sign5:Youarestuckwithonevendorandhavenoroomtomaneuver

We all fear lock-in. Why? The idea that you’ll be held hostage by a technology is scary, and it adds costs and risk to your database solutions. While there is rarely a technology that can’t be abandoned, the core metric here is the cost of doing so, as well as the risk and disruption to the business.

As we’ve covered thus far, it’s an advantage to leverage technology that runs in more than one place, including on public clouds. Keep in mind that while DynamoDB is an AWS IaaS cloud-only product, Aerospike can run on both cloud-based and traditional on premises platforms. This capability provides you with the flexibility to pick either cloud or non-cloud platforms. As your needs change over time, you can move between platforms (covered next). Lock-in is non-existent when compared to DynamoDB, where it’s built into the database platform.

When it comes to cloud computing and lock-in, a study by Kleiner Perkins showed a sharp increase in the number of buyers who are citing the possibility of lock-in as one of their top three concerns in moving to the cloud6. According to a 2017 Stratoscale report7, eight out of ten enterprises fear vendor lock-in in the public cloud.

Grumbling about lock-in has become more pronounced over the last few years as AWS consolidates its market share and continues to crank out compelling new features that require just enough customization work to embrace the requirements. Indeed, AWS and its cloud services, including DynamoDB, are considered “sticky” technology with migration away from the technology at some point in the future often cost prohibitive.

The advice we’re getting around dealing with cloud lock-in includes8:

• Take maximum advantage of the new capabilities of the cloud to help the business.

6 https://www.geekwire.com/2017/kleiner-perkins-meeker-concerns-cloud-vendor-lock-soaring/ 7 https://www.stratoscale.com/solutions/hybrid-cloud/whitepaper-thanks-hybrid-cloud-survey/ 8 https://www.forbes.com/sites/danwoods/2017/06/20/five-ways-to-avoid-cloud-lock-in/ - 2857dda04bf6

Page 12: 5 Signs You’ve Outgrown DynamoDB - Amazon S3€¦ · 5 Signs You’ve Outgrown DynamoDB 3 Cost of Capacity: Higher read performance costs more. A lot more. Looking at Figure 1,

5 Signs You’ve Outgrown DynamoDB

12

• Do so in a way that if and when the rules or prices change, you have a strong negotiating position, which, in practical terms, means making it as easy as possible to migrate to another cloud or data center.

Note that there are no saviors here in terms of dealing with the cost and risk of lock-in. You’re largely on your own.

Figure 5 : More enterprises are concerned about vendor lock-in, according to a recent study by Stratoscale

While enterprises often fear lock-in, they don’t often consider it when selecting database and other technology products. The trend seems to be to follow the crowd. As a result, many enterprises find themselves stuck with technology where the cost of change will far exceed that of continuing with technology that’s no longer a fit.

Conclusion: This is a matter of choice, not one platform or technology over the other. Ideally you need to pick a technology that will not lock you into a platform, such as a public cloud, and you may have business or technical needs where the platform is no longer a fit. Hedging your bets seems to be the best path to success, including picking technology that runs on the public cloud and on traditional systems.

Page 13: 5 Signs You’ve Outgrown DynamoDB - Amazon S3€¦ · 5 Signs You’ve Outgrown DynamoDB 3 Cost of Capacity: Higher read performance costs more. A lot more. Looking at Figure 1,

5 Signs You’ve Outgrown DynamoDB

13

CalltoAction

In this paper we removed the noise from the world of AWS DynamoDB. Looking specifically at what this technology does, and, more importantly, the limitations you should consider to determine if you’ve outgrown this technology. Or, issues you should consider before you select this technology.

The call to action here is that you consider your own requirements that include function, costs, and performance with an open mind about how both technologies will work with your enterprise. Look at the situation not only now, but years from now, and consider all aspects of use cases that you’re like to leverage, including DevOps, production, non-production, etc.

Enterprises often make decisions based upon the understanding of only a single dimension. Do this, and you’re likely to miss the bigger picture, especially the tactical details you should consider such as those presented in this paper.

For more information: http://www.aerospike.com/

AboutAerospike Aerospike is an enterprise-class, hybrid memory architecture database solution that enables digital transformation by powering real-time, mission-critical applications and analytics across industries, including AdTech, ecommerce, gaming, telecommunications, and financial services. Aerospike delivers predictable performance at scale, superior uptime, and high availability at the lowest total cost of ownership.

Page 14: 5 Signs You’ve Outgrown DynamoDB - Amazon S3€¦ · 5 Signs You’ve Outgrown DynamoDB 3 Cost of Capacity: Higher read performance costs more. A lot more. Looking at Figure 1,

5 Signs You’ve Outgrown DynamoDB

14

Appendix

Complete DynamoDB vs. Aerospike benchmark results used for this paper (click to enlarge) with 1 billion objects, at 4KB record size, with a mix of 100% Reads and 50-50 Read-Writes.