Custom properties not saved correctly for Expando models in repeated StructuredProperty

Question

I am trying to use an Expando model as a repeated StructuredProperty in another model. Namely, I would like to add an indefinite number of Accounts to my User model. As Accounts can have different properties depending on their types (Accounts are references to social network accounts, and for example Twitter requires more information than Facebook for its OAuth process), I have designed my Account model as an Expando. I've added all basic information in the model definition, but I plan to add custom properties for specific social networks (e.g., a specific access_token_secret property for Twitter).

1/ Can you confirm the following design (Expando in repeated StructuredProperty) should work?

class Account(ndb.Expando):
    account_type = ndb.StringProperty(required=True, choices=['fb', 'tw', 'li'])
    account_id = ndb.StringProperty()
    state = ndb.StringProperty()
    access_token = ndb.StringProperty()

class HUser(User):
    email = ndb.StringProperty(required=True, validator=validate_email)
    created = ndb.DateTimeProperty(auto_now_add=True)
    accounts = ndb.StructuredProperty(Account, repeated=True)

2/ Now the problem I am facing is: when I add a Facebook account to my HUser instance, everything works fine ; however the problem rises when I append a Twitter account to that same instance, and add a new property not declared in the model, like that:

for account in huser.accounts:
    if account.state == "state_we_re_looking_for" and account.account_type == 'tw':
        # we found the appropriate Twitter account reference
        account.access_token_secret = "..." # store the access token secret fetched from Twitter API
        huser.put() # save to the Datastore
        break

This operation is supposed to save the access token secret in the Twitter Account instance of my User, but in fact it saves it in the Facebook Account instance (at index 0)!

What am I doing wrong?

Thanks.

Answer 1

This is a fundamental problem with how ndb stores the StructuredProperty. Datastore does not currently have a way to store this, so ndb basically explodes your properties.

For example, consider the entity:

HUser(email='test@example.com'.
      accounts=(Account(type='fb',
                        account_id='1',
                        state='1',
                        access_token='1'),
                Account(type='tw',
                        account_id='2',
                        state='2',
                        access_token='2',
                        access_token_secret='2')))

This will actually get stored in an entity that looks like:

{
 email : 'test@example.com',
 accounts.type : ['fb', 'tw'],
 accounts.account_id : ['1', '2'],
 accounts.state : ['1', '2'],
 accounts.access_token : ['1', '2'],
 accounts.access_token_secret : ['2']
}

Because you are using an ndb.Expando, ndb doesn't know that it should populate the access_token_secret field with a None for the facebook account. When ndb repopulates your entities, it will fill in the access_token_secret for the first account it sees, which is the facebook account.

Restructuring your data sounds like the right way to go about this, but you may want to make your HUser an ancestor of the Account for that HUser so that you query for a user's accounts using strong consistency.

Answer 2

From what I understand, it seems like App Engine NDB does not support Expando entities containing Expando entities themselves.

One thing that I didn't realize at first is that my HUser model inherits from Google's User class, which is precisely an Expando model!

So without even knowing it, I was trying to put a repeated StructuredProperty of Expando objects inside another Expando, which seemingly is not supported (I didn't find anything clearly written on this limitation, however).

The solution is to design the data model in a different way. I put my Account objects in a separate entity kind (and this time, they are truly Expando objects!), and I've added a KeyProperty to reference the HUser entity. This involves more read/write ops, but the code is actually much simpler to read now...

I'll mark my own question as answered, unless someone has another interesting input regarding the limitation found here.