looking for ideas/alternatives to providing a page/item count/navigation of items matching a GAE datastore query

Question

I like the datastore simplicity, scalability and ease of use; and the enhancements found in the new ndb library are fabulous.

As I understand datastore best practices, o

Answer 1

Since the question was "looking for ideas/alternatives to providing a page", maybe the very simple alternative of fetching 10 pages worth of key_only items, then handling navigation through within this set is worth considering.

I have elaborated on this in answering a similar question, you will find sample code there :

Backward pagination with cursor is working but missing an item

The sample code would be more appropriate for this question. Here is a piece of it:

def session_list():
    page = request.args.get('page', 0, type=int)

    sessions_keys = Session.query().order(-Session.time_opened).fetch(100, keys_only=True)
    sessions_keys, paging = generic_list_paging(sessions_keys, page)
    # generic_list_paging will select the proper sublist.
    sessions = [ sk.get() for sk in sessions_keys ]

    return render_template('generic_list.html', objects=sessions, paging=paging)

See the referenced question for more code.

Of course, if the result set is potentially huge, some limit to the fetch must still be given, the hard limit being 1000 items I think. Obviously, it the result is more than some 10 pages long, the user will be asked to refine by adding criteria.

Dealing with paging within a few hundreds of keys_only items is really so much simpler, that it's definitely worth considering. It makes it quite easy to provide direct page navigation as mentionned in the question. The actual entity items are only fetched for the actual current page, the rest is only keys so it's not so costly. And you may consider keeping the keys_only result set in memcache for a few minutes so that a user quickly browsing through pages will not require the same query to be performed again.