bulkloader

Bulkloader raises the following error when importing a CSV file with large cells: [ERROR ] Error in data source thread: field larger than field limit (131072) This is a common problem for the csv module, which can be fixed with: csv.field_size_limit(sys.maxint) How can I make bulkloader execute this? Try this: In bulkloader.yaml add: python_preamble: - import: csv_fix ... # the rest of your imports In csv_fix.py add: import csv, sys csv.field_size_limit(sys.maxint) 来源： https://stackoverflow.com/questions/5973363/bulkloader-csv-size-error

I want to use bulkloader to download all entities in a model with some self-defined Property . If I define a model like this, class MyType: def __init__(self, arg): self.name = arg['name'] self.id = arg['id'] class MyProperty(db.Property): def get_value_for_datastore(self, instance): val = super(MyProperty, self).get_value_for_datastore(instance) if type(val) == dict: val = MyType(val) return pickle.dumps(val) def make_value_from_datastore(self, val): return None if val is None else pickle.loads(str(val)) class MyModel(db.Model): info = MyProperty() then how can I download MyModel using the

My application uses Django non-rel . I don't have access to model. I have my bulkloader.yaml file autogenerated by appcfg.py create_bulkloader_config . Problem is entities numeric ID's are being imported as string key names. So if I export entity with int ID of, for example, '62' , it gets imported as entity with string key name of '61' which screws up Django. Revelant bulkloader.yaml Fragment: property_map: - property: __key__ external_name: key export_transform: transform.key_id_or_name_as_string I'm trying to setup download/upload od data using bulkloader, and I want to have data as easy to

I'm currently using the --kind parameter to specify which kind to download and the --filename to specify the name of the csv file to produce. The --rps_limit , --bandwidth_limit and --batch_size are used to speed up the download. For example, to download my Game kind I'm using: appcfg.py download_data --config_file=bulkloader.yaml --kind=Game --filename=game.csv --application=MyAppId --url=http://MyAppId.appspot.com/remote_api --rps_limit=500 --bandwidth_limit=2500000 --batch_size=100 So is there an appcfg.py command to download the entire datastore without being kind specific to one csv file?

I'm using the following to download all instances of one of my kinds: appcfg.py download_data --config_file=bulkloader.yaml --kind=ModelName --filename=ModelName.csv --url=http://appid.appspot.com/remote_api If the kind has more instances than the batch size, then I get this warning: No descending index on __key__, performing serial download I don't have any custom indexes, or any properties with indexes disabled. Do I 'need' to do something to resolve this warning, or is it just a warning I can safely ignore? Does it effect the speed of the download? This article on the bulkloader includes