How to auto scale Amazon DynamoDB throughput?

Question

Amazon DynamoDB doesn’t provide inbuild capabilities to auto tune throughput based on Dynamic Load. It provide API to increase or Decrease throughput. Customers are being c

Answer 1

AWS added native auto scaling support for DynamoDB in June 2017. See the announcement here.

You can configure this using code (Java SDK example), but if you have just a few tables, you can use the Management Console. Click in your table configuration and select the Capacity tab. The following image shows what are your options:

Answer 2

I just discovered this project that will autoscale up and down your Dynamodb and looks better than Dynamic Dynamo, because it uses Lambda functions rather than EC2 instances:

https://github.com/channl/dynamodb-lambda-autoscale

• 5 minute setup process
• Serverless design
• Flexible code over configuration style
• Autoscale table and global secondary indexes
• Autoscale multiple tables
• Autoscale by fixed settings
• Autoscale by provisioned capacity utilisation
• Autoscale by throttled event metrics
• Optimised for large spikes in usage and hotkey issues by incorporating throttled event metrics
• Optimised performance using concurrent queries
• RateLimitedDecrement as imposed by AWS
• Statistics via 'measured'
• AWS credential configuration via 'dotenv'
• Optimised lambda package via 'webpack'
• ES7 code
• 100% Flow static type checking coverage

Answer 3

Now that AWS has announced scheduled execution of lambda services, these seem a great fit to do time-based auto scaling. I wrote up an example of how to use this on medium. Example code is on github.

Answer 4

The answer from Chris is an accurate answer. Just to add a few points from prior experience using DynamoDB …

The situation with DynamoDB is different from EC2. The elastic compute service has an API supported directly as a web service by Amazon to allow you to program how to scale up or down according some logic such as how much demand exists. You program this by defining a monitoring threshold and automatically triggering creation or deletion of instances in a group.

Data servers do not work in the same way with triggers to adjust their capacity. But the capacity of DynamoDB is very flexible and may be controlled as Chris has pointed out. The API to provide this is good enough to make one off changes. Or equivalent manual changes from the console.

The different language bindings to program create and update actions with DynamoDB is here …

http://docs.aws.amazon.com/cli/latest/reference/dynamodb/index.html

The important operation to modify capacity is here …

http://docs.aws.amazon.com/cli/latest/reference/dynamodb/update-table.html

So this gives you the ability to make an increase or decrease in the ReadCapacityUnits or WriteCapacityUnits of ProvisionedThroughput.

Which is fine for a predicted or one-off change. But that is not the same thing as a flexibility tool to allow you to trigger the change automatically.

Programmatically, what you are most likely to want to do is to adjust capacity in response to change in utilization in the preceding time interval. In particular you may need to scale up rapidly in response to a surge in demand by defining an appropriate time slot and a lower and upper threshold to trigger.

A more complete solution to achieve this is described here …

https://aws.amazon.com/blogs/aws/auto-scale-dynamodb-with-dynamic-dynamodb/

The solution is maintained by Sebastian Dahlgren and may be found with all instructions at …

https://github.com/sebdah/dynamic-dynamodb

I see that the current release is 1.18.5 which is more recent than when I last used it.

Judging from earlier releases it is simple to configure by means of a dynamodb.conf properties style file …

Having provided credentials and region, the most crucial settings are

• check-interval — to test throughput in seconds
• min-provisioned-reads, max-provisioned-reads; reads-upper-threshold, reads-lower-threshold; increase-reads-with, decrease-reads-with — These are all percentages
• min-provisioned-writes, max-provisioned-writes; writes-upper-threshold, writes-lower-threshold; increase-writes-with, decrease-writes-with — These are all percentages

Is this information up to date?

Well if you look at http://aws.amazon.com/new/ you will see just one additional recent change affecting DynamoDB which affects the documents stored. The entry for Dynamic DynamoDB is the last published entry dealing with scaling actions. So this is the best maintained DynamoDB automatic scaling capability at this time.

Answer 5

I added new features to Rockeee Dynamic DynamoDB Lambda. You can see this project:

https://github.com/touchvie/dynamic-dynamodb-lambda

• Global Secondary Index support
• Enable/Disable read/write autoscaling in config json file
• Throttle Events in CloudWatch support
• Enable/Disable throttle-read/throttle-write checking in config json file
• Added test to lambda

I hope that it can help you.

Answer 6

You can manage throughput programmatically through the updateTable API or manually through the console.

There's also tools like Dynamic DynamoDB, though you could roll your own version as well: you'd use the updateTable API and have some background process running to detect those circumstances and call updateTable as necessary.

Some things to watch out for when changing the scale of DynamoDB:

1. You get billed for allocated throughput, whether you're actually using it or not.
2. Wen you scale up, Dynamo may allocate new partitions for you - but it won't remove them when it scales down. This can result in unexpected hot hash key problem where you have a lot of partitions but very low throughput on each of them.