google-cloud-datastore

I have run into an interesting limit with the App Engine datastore. I am creating a handler to help us analyze some usage data on one of our production servers. To perform the analysis I need to query and summarize 10,000+ entities pulled from the datastore. The calculation isn't hard, it is just a histogram of items that pass a specific filter of the usage samples. The problem I hit is that I can't get the data back from the datastore fast enough to do any processing before hitting the query deadline. I have tried everything I can think of to chunk the query into parallel RPC calls to improve

I am working with NodeJS on Google App Engine with the Datastore database. Due to the fact that Datastore does not have support the OR operator , I need to run multiple queries and combine the results. I am planning to run multiple queries and then combine the results into a single array of entity objects. I have a single query working already. Question: What is a reasonably efficient way to combine two (or more) sets of entities returned by Datastore including de-duplication? I believe this would be a "union" operation in terms of set theory. Here is the basic query outline that will be run

I need to assign a unique identifier to each entity in datastore. I've found AllocateIDs but the issue is that it generates integers and I use strings for the keys. Is it safe to convert the integer to a string or there is a risk of collision (i.e. datastore to return me the same integer that I'm currently using as a string key). ? Let's clear some things first: The identifier part of an entity's key can either be a key name string or an integer numeric ID But not both. So when you save an entity, its key either has a string id called name OR (exclusive OR) an int64 id called intID . The 2

I am trying to execute a dataflow pipeline job which would execute one function on N entries at a time from datastore. In my case this function is sending batch of 100 entries to some REST service as payload. This means that I want to go through all entries from one datastore entity, and send 100 batched entries at once to some outside REST service. My current solution Read input from datastore Create as many keys as there are workers specified in pipeline options (1 worker = 1 key). Group by key, so that we get iterator as output (iterator input in step no.4) Programatically batch users in

There are many properties in my model that I currently don't need indexed but can imagine I might want indexed at some unknown point in the future. If I explicitly set indexed=False for a property now but change my mind down the road, will Datastore rebuild the entire indices automatically at that point, including for previously written data? Are there any other repercussions for taking this approach? Patrick Costello No, changing indexed=True to indexed=False (and vice-versa) will only affect entities written after that point to the datastore. Here is the documentation that talks about it and