app-engine-ndb

Processing a large (>32mb) xml file over appengine

匆匆过客 提交于 2019-12-06 11:07:08
I'm trying to process large (~50mb) sized xml files to store in the datastore. I've tried using backends, sockets (to pull the file via urlfetch), and even straight up uploading the file within my source code, but again keep running into limits (i.e. the 32 mb limit). So, I'm really confused (and a little angry/frustrated). Does appengine really have no real way to process a large file? There does seem to be one potential work around, which would involve remote_apis, amazon (or google compute I guess) and a security/setup nightmare... Http ranges was another thing I considered, but it'll be

App Engine NDB query with multiple inequalities?

☆樱花仙子☆ 提交于 2019-12-06 10:03:26
问题 The only two answers on here involve essentially restructuring the database to accommodate this limitation, but I am unsure how to do that in my case. I have a list of thousands of contacts, each with many many properties. I'm making a page that has an ability to filter on multiple properties at once. For example: Age < 15, Date Added > 15 days ago, Location == Santa Cruz, etc. Potentially a ton of inequality filters required. How does one achieve this in GAE? 回答1: According to the docs (for

validator for repeated ndb.StructuredProperty fails to fire

梦想的初衷 提交于 2019-12-06 06:09:16
Here is my ndb Model from google.appengine.ext import ndb from mainsite.rainbow.models.CFCSocialUser import CFCSocialUser class CFCSocialGroup(ndb.Model): def remove_duplicate(self, value): raise Exception("Duplicate user detected") name = ndb.StringProperty(required=True) created_on = ndb.DateTimeProperty(auto_now_add=True) updated_on = ndb.DateTimeProperty(auto_now=True) created_by = ndb.StructuredProperty(CFCSocialUser) members = ndb.StructuredProperty(CFCSocialUser, repeated=True, validator=remove_duplicate) @staticmethod def create_group(name): """Create a new group""" group =

multiple filters vs OR , ndb query

不羁的心 提交于 2019-12-06 03:07:26
What is the difference between these queries: With consequent filters: qry1 = Account.query() # Retrieve all Account entitites qry2 = qry1.filter(Account.userid >= 40) # Filter on userid >= 40 qry3 = qry2.filter(Account.userid < 50) # Filter on userid < 50 as well Using ndb.OR: qry = Article.query(ndb.OR(Account.userid >= 40, Account.userid < 50)) Using ndb.AND: qry = Article.query(ndb.AND(Account.userid >= 40, Account.userid < 50)) The first query does an AND. Thus, only the entities that match both inequalities will be returned by the query. The second query does an OR. Thus, entities that

Why doesn't appengine auto-convert datetime to UTC when calling put()

眉间皱痕 提交于 2019-12-06 02:54:07
问题 Here's what I'm trying to do: the user submits a time in pacific, once submitted I use .replace to set the timezone to Pacific. Pacific = time.USTimeZone(-8, "Pacific", "PST", "PDT") addEvent.date = addEvent.date.replace(tzinfo=Pacific) Once i've set the tzinfo, I'm doing a put. According to the python documentation of google appengine it says: "If the datetime value has a tzinfo attribute, it will be converted to the UTC time zone for storage. Values come back from the datastore as UTC, with

How to update an NDB Model's Schema [closed]

Deadly 提交于 2019-12-06 02:25:31
Closed . This question needs details or clarity . It is not currently accepting answers. Want to improve this question? Add details and clarify the problem by editing this post . Closed 4 years ago . I have seen the solution to this question using App Engine's older DB Datastore API , but cannot find a solution while using the newer NDB API. What is the best way to add migration support, so that I am able to migrate from an old version of a schema to a new version. Would it be best to write a migration script , and how would this work? Something like migrating a schema like this in (Note that

App engine NDB: how to access verbose_name of a property

有些话、适合烂在心里 提交于 2019-12-06 02:13:20
问题 suppose I have this code: class A(ndb.Model): prop = ndb.StringProperty(verbose_name="Something") m = A() m.prop = "a string value" Now of course if I print m.prop, it will output "a string value" while in fact it's a StringProperty instance. So verbose_name can't be accessed the "normal" way, i.e m.prop._verbose_name . I read the code and found a way to access it: m._properties["prop"]._verbose_name , it works, but it looks hacky o_o. So tell me, is there another way to do it? Note: I'm

Is it safe to use images.get_serving_url_async() inside a ndb tasklet?

大憨熊 提交于 2019-12-06 01:31:31
问题 NDB tasklets and yield are a great way to do async/parallel code. However, it is not quite clear from the documentation if this mecanism can be safely used with non-ndb async functions such as images.get_serving_url_async() . The NDB Asynchronous Operation documentation page has a very tiny section about using the NDB context's own version of urlfetch_async() , where it is stated (emphasis mine): The URL Fetch service has its own asynchronous request API. It's fine, but not always easy to use

Are ndb cached read ops still counted as datastore read ops for billing purposes?

余生颓废 提交于 2019-12-05 23:35:19
From NDB Caching : NDB manages caches for you. There are two caching levels: an in-context cache and a gateway to App Engine's standard caching service, memcache. Both caches are enabled by default for all entity types, but can be configured to suit advanced needs. My app doesn't make any ndb caching configuration change, so it must be using the defaults - both caching levels enabled. I'm running some tests on my staging environment (a separate, dedicated GAE project) where I can totally isolate sequences of activity from any spurious external requests. Each sequence of activity consists of a

NDB querying results that start with a string

帅比萌擦擦* 提交于 2019-12-05 22:57:26
Working with Google App Engine's NDB, I'm looking to query for all items that start with a user-inputted string. Example: abc_123 abcdefg 123abc Querying for "abc" should return abc_123, abcdefg (however, not 123abc as it doesn't start with "abc") I previously used the code below for a similar but different purpose: q = q.filter(order._properties[kw].IN(values_list)) which filtered for all values in values_list that were in kw, I am now looking to filter for all values that start with a string that are in kw. Try: Kind.query(ndb.AND(Kind.property >= "abc", Kind.property <= "abcz")) 来源: https:/