Query with paging by cursor causes error because of limitations for “IN filter” in cursor() method… What should be the alternative?

Question

I am developing a twitter like microblogging system by using the following models:

class Member(db.Model):    
    user = db.UserProperty(required=True)
             


        

Answer 1

The reason for this restriction is that IN and != queries are executed by splitting the query into multiple underlying queries, which are executed individually by the datastore, then merged together in sorted order.

If you want to do a query like this in a paginated fashion, you would have to execute the queries yourself, and do the merge yourself. To fetch a cursor, you'd need to fetch cursors from the individual sub-queries and concatenate them together. Further, you'd need to keep track of how many fetched but un-consumed results you had, so you can pick up exactly from where you left off.

As you can see, this is complicated and leads to excessively long cursor values, which is why it isn't implemented by the SDK currently. Unfortunately, it's the only practical way to do it, unless you can find a way to avoid the use of an IN clause, or drop your requirement for ordering by another one (in which case you could just execute the queries serially, paginating each).