GAE python NDB projection query working in development but not in production

Question

I've been hitting my head against the wall because my Google App Engine python project has a very simple NDB projection query which works fine on my local machine, but mysteriously fails when deployed to production.

Adding to the mystery... as a test I added an identical projection on another property, and it works in both dev and production! Could anyone help please?! Here are more details:

I have the following entity that represents an expense:

class Entry(ndb.Model):
    datetime = ndb.DateTimeProperty(indexed=True, required=True)
    amount = ndb.IntegerProperty(indexed=False, required=True)
    payee = ndb.StringProperty(indexed=True, required=True) 
    comment = ndb.StringProperty(indexed=False)
    # ...

Later on in the code I am doing a projection on Entry.payee (to get a list of all payees). As a test I also added a projection on Entry.datetime:

log_msg = '' # For passing debug info to the browser

payeeObjects = Entry.query(ancestor=exp_traq_key(exp_traq_name), projection=[Entry.payee]).fetch()
payees = []
for obj in payeeObjects:
    payees.append(obj.payee)
log_msg += '%d payees: %s' % (len(payees), str(payees))

log_msg += ' ------------------- ' # a visual separator

dtObjects = Entry.query(ancestor=exp_traq_key(exp_traq_name), projection=[Entry.datetime]).fetch()
dts = []
for obj in dtObjects:
    dts.append(obj.datetime)
log_msg += '%d datetimes: %s' % (len(dts), str(dts))

#...other code, including passing log_msg down to the client

Here's the output in dev environment (notice a list of payees and a list of datetimes are displayed in console):

And here's the output when deployed to app engine. I can't get it to return a list of payees. It keeps returning an empty list even though in dev it returns the list fine:

I've ensured that I have the indexes properly set up on GAE:

Please help!

---

2018-12-05 Update: I added a couple more entries in production and they got picked up! See screenshot. But the older entries are still not being returned.

My immediate reaction is that the datastore index needs to be "refreshed" somehow so it can "see" old entries. BUT the thing is I removed and recreated the index yesterday, which means it should have old entries... So still need help resolving this mystery!

Answer 1

I figured it out. Darn it wasn't intuitive at all. I wish GAE documentation was better on this point...

My datastore in production contains a lot of previously created entries. As part of my latest code where I'm trying to do the projection on Entry.payee, I had to change the definition of Entry.payee from unindexed to indexed, like so:

payee = ndb.StringProperty(indexed=True, required=True) # Originally was indexed=False

So now all those entries sitting in the datastore are being ignored by the projection query because the index on payee ignores those entries.

So what I need to do now is somehow migrate all those old entities to be indexed=True.

---

Update - here's how I did this migration. Turned out simpler than expected.

def runPayeeTypeMigration(exp_traq_name):
  Entry.query(ancestor=exp_traq_key(exp_traq_name)).fetch()
  for entry in entries:
    entry.put()

This works by reading all entries into the updated datastructure (the one where Entry.payee is indexed=True) and writes it back to the datastore, so that the entity will now be indexed.