Query response size limit on appengine?

Question

Appengine docs mention a 1Mb limit on both entity size and batch get requests (db.get()): http://code.google.com/appengine/docs/python/datastore/overview.html

Is there also a limit on the total size of all entities returned by a query for a single fetch() call?

Example query:

db.Model.all().fetch(1000)

Update: As of 1.4.0 batch get limits have been removed!

• Size and quantity limits on datastore batch get/put/delete operations have been removed. Individual entities are still limited to 1 MB, but your app may batch as many entities together for get/put/delete calls as the overall datastore deadline will allow for.

Answer 1

Theres no longer a limit on the number of entities that can be returned by a query, but the same entity size limit applies when you are actually retrieving / iterating over the entities. This will only be on a single entity at a time though; it is not a limit on the total size of all entities returned by the query.

Bottom line: as long as you don't have a single entity that is > 1Mb you should be OK with queries.

Answer 2

I tried it out on production and you can indeed exceed 1 Mb total for a query. I stopped testing at around 20 Mb total response size.

from app import models

# generate 1Mb string
a = 'a'
while len(a) < 1000000:
    a += 'a'

# text is a db.TextProperty()
c = models.Comment(text=a)
c.put()


for c in models.Comment.all().fetch(100):
    print c

Output:

<app.models.Comment object at 0xa98f8a68a482e9f8>
<app.models.Comment object at 0xa98f8a68a482e9b8>
<app.models.Comment object at 0xa98f8a68a482ea78>
<app.models.Comment object at 0xa98f8a68a482ea38>
....

Answer 3

Yes there is a size limit; the quotas and limits section explicitly states there is a 1 megabyte limit to db API calls.

You will not be able to db.get(list_of_keys) if the total size of the entities in the batch is over 1 megabyte. Likewise, you will not be able to put a batch if the total size of the entities in the batch is over 1 megabyte.

The 1,000 entity limit has been removed, but (at present) you will need to ensure the total size of your batches is less than 1 megabyte yourself.