db.ReferenceProperty() vs ndb.KeyProperty in App Engine

Question

ReferenceProperty was very helpful in handling references between two modules. Fox example:

class UserProf(db.Model):
    name = db.StringProperty(required=True)

class Team(db.Model):
    manager_name = db.ReferenceProperty(UserProf, collection_name='teams')
    name = db.StringProperty(required=True)

• To get 'manager_name' with team instance, we use team_ins.manager_name.
• To get 'teams' which are managed by particular user instance, we use user_instance.teams and iterate over.

Doesn't it look easy and understandable?

In doing same thing using NDB, we have to modify

db.ReferenceProperty(UserProf, collection_name='teams') --> ndb.KeyProperty(kind=UserProf)

• team_ins.manager_name.get() would give you manager name

• To get all team which are manger by particular user, we have to do

  for team in Team.query(Team.manager_name == user_ins.key): 
      print "team  name:", team.name
  

As you can see handling these kind of scenarios looks easier and readable in db than ndb.

• What is the reason for removing ReferenceProperty in ndb?
• Even db's query user_instance.teams would have doing the same thing as it is done in ndb's for loop. But in ndb, we are explicitly mentioning using for loop.
• What is happening behind the scenes when we do user_instance.teams?

Thanks in advance..

Answer 1

Tim explained it well. We found that a common anti-pattern was using reference properties and loading them one at a time, because the notation "entity.property1.property2" doesn't make it clear that the first dot causes a database "get" operation. So we made it more obvious by forcing you to write "entity.property1.get().property2", and we made it easier to do batch prefetching (without the complex solution from Nick's blog) by simply saying "entity.property1.get_async()" for a bunch of entities -- this queues a single batch get operation without blocking for the result, and when you next reference any of these properties using "entity.property1.get().property2" this won't start another get operation but just waits for that batch get to complete (and the second time you do this, the batch get is already complete). Also this way in-process and memcache integration comes for free.

Answer 2

I don't know the answer as to why Guido didn't implement reference property.

However I found a spent a lot of time using pre_fetch_refprops http://blog.notdot.net/2010/01/ReferenceProperty-prefetching-in-App-Engine (pre fetches all of the reference properties by grabbing all the keys with get_value_for_datastore,) and then it does a get_multi on the keys.

This was vastly more efficient.

Also if the object referenced doesn't exist you would get an error when trying to fetch the object.

If you pickled an object which had references you ended up pickling a lot more than you probably planned too.

So I found except for the one case, where you have single entity and you wanted to grab the referenced object with .name type accessor you had to jump through all sorts of hoops to prevent the referenced entity from being fetched.