How to implement pagination when using amazon Dynamo DB in rails

Question

I want to use amazon Dynamo DB with rails.But I have not found a way to implement pagination.

I will use AWS::Record::HashModel as ORM.

This ORM

Answer 1

You issue queries using LIMIT. If the subset returned does not contain the full table, a LastEvaluatedKey value is returned. You use this value as the ExclusiveStartKey in the next query. And so on...

From the DynamoDB Developer Guide.

Answer 2

You can provide 'page-size' in you query to set the result set size. The response of DynamoDB contains 'LastEvaluatedKey' which will indicate the last key as per the page size. If response does't contain 'LastEvaluatedKey' it means there are no results left to fetch. Use the 'LastEvaluatedKey' as 'ExclusiveStartKey' while fetching next time.

I hope this helps.

DynamoDB Pagination

Answer 3

Here's a simple copy-paste-run proof of concept (Node.js) for stateless forward/reverse navigation with dynamodb. In summary; each response includes the navigation history, allowing user to explicitly and consistently request either the next or previous page (while next/prev params exist):

GET /accounts                -> first page
GET /accounts?next=A3r0ijKJ8 -> next page
GET /accounts?prev=R4tY69kUI -> previous page

Considerations:

1. If your ids are large and/or users might do a lot of navigation, then the potential size of the next/prev params might become too large.
2. Yes you do have to store the entire reverse path - if you only store the previous page marker (per some other answers) you will only be able to go back one page.
3. It won't handle changing pageSize midway, consider baking pageSize into the next/prev value.
4. base64 encode the next/prev values, and you could also encrypt.
5. Scans are inefficient, while this suited my current requirement it won't suit all!

// demo.js
const mockTable = [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20]
const getPagedItems = (pageSize = 5, cursor = {}) => {

  // Parse cursor
  const keys = cursor.next || cursor.prev || [] // fwd first
  let key = keys[keys.length-1] || null // eg ddb's PK

  // Mock query (mimic dynamodb response)
  const Items = mockTable.slice(parseInt(key) || 0, pageSize+key)
  const LastEvaluatedKey = Items[Items.length-1] < mockTable.length 
                            ? Items[Items.length-1] : null

  // Build response
  const res = {items:Items}
  if (keys.length > 0) // add reverse nav keys (if any)
    res.prev = keys.slice(0, keys.length-1)
  if (LastEvaluatedKey) // add forward nav keys (if any)
    res.next = [...keys, LastEvaluatedKey]
  
  return res
}

// Run test ------------------------------------
const runTest = () => {
  const PAGE_SIZE = 6
  let x = {}, i = 0

  // Page to end
  while (i == 0 || x.next) {
    x = getPagedItems(PAGE_SIZE, {next:x.next})
    console.log(`Page ${++i}: `, x.items)
  }

  // Page back to start
  while (x.prev) {
    x = getPagedItems(PAGE_SIZE, {prev:x.prev})
    console.log(`Page ${--i}: `, x.items)
  }
}
runTest()

Answer 4

I faced a similar problem.

The generic pagination approach is, use "start index" or "start page" and the "page length". 

The "ExclusiveStartKey" and "LastEvaluatedKey" based approach is very DynamoDB specific.

I feel this DynamoDB specific implementation of pagination should be hidden from the API client/UI.

Also in case, the application is serverless, using service like Lambda, it will be not be possible to maintain the state on the server. The other side is the client implementation will become very complex.

I came with a different approach, which I think is generic ( and not specific to DynamoDB)

When the API client specifies the start index, fetch all the keys from the table and store it into an array.

Find out the key for the start index from the array, which is specified by the client.

Make use of the ExclusiveStartKey and fetch the number of records, as specified in the page length.

If the start index parameter is not present, the above steps are not needed, we don't need to specify the ExclusiveStartKey in the scan operation.

This solution has some drawbacks -

We will need to fetch all the keys when the user needs pagination with start index.

We will need additional memory to store the Ids and the indexes. Additional database scan operations ( one or multiple to fetch the keys )

But I feel this will be very easy approach for the clients, which are using our APIs. The backward scan will work seamlessly. If the user wants to see "nth" page, this will be possible.