pickle | 易学教程

Reading multiple Python pickled data at once, buffering and newlines?

阅读更多关于 Reading multiple Python pickled data at once, buffering and newlines?

问题 to give you context: I have a large file f , several Gigs in size. It contains consecutive pickles of different object that were generated by running for obj in objs: cPickle.dump(obj, f) I want to take advantage of buffering when reading this file. What I want, is to read several picked objects into a buffer at a time. What is the best way of doing this? I want an analogue of readlines(buffsize) for pickled data. In fact if the picked data is indeed newline delimited one could use readlines,

What's the fastest way to pickle a pandas DataFrame?

阅读更多关于 What's the fastest way to pickle a pandas DataFrame?

问题 Which is better, using Pandas built-in method or pickle.dump ? The standard pickle method looks like this: pickle.dump(my_dataframe, open('test_pickle.p', 'wb')) The Pandas built-in method looks like this: my_dataframe.to_pickle('test_pickle.p') 回答1: Thanks to @qwwqwwq I discovered that pandas has a built-in to_pickle method for dataframes. I did a quick time test: In [1]: %timeit pickle.dump(df, open('test_pickle.p', 'wb')) 10 loops, best of 3: 91.8 ms per loop In [2]: %timeit df.to_pickle(

How to pickle a scapy packet?

阅读更多关于 How to pickle a scapy packet?

问题 I need to pickle a scapy packet. Most of the time this works, but sometimes the pickler complains about a function object. As a rule of thumb: ARP packets pickle fine. Some UDP packets are problematic. 回答1: My solution (as inspired by the scapy mailing list) is as follows: class PicklablePacket: """A container for scapy packets that can be pickled (in contrast to scapy packets themselves).""" def __init__(self, pkt): self.contents = bytes(pkt) self.time = pkt.time def __call__(self): """Get

EOFError Ran out of input Python

阅读更多关于 EOFError Ran out of input Python

I am trying to use pickle to create a save file for my game, but When I try to submit my dictionary, and then take that information back, but it isn't working. import pickle data = {'health':100, 'gold':1560, 'name': 'mariano'} with open('s.txt','wb') as f: pickle.dump(data, f, protocol = 2) with open('s.txt','rb') as f: data = pickle.load(f) then when I run that code it gives me this error EOFError: Ran out of input I figured out what went wrong, or at least how I fixed the situation. You were trying to use a file that was already pickled, and since it ran with an error it broke it, all you

Reduce memory usage of this Pandas code reading JSON file and pickling

阅读更多关于 Reduce memory usage of this Pandas code reading JSON file and pickling

问题 I can't figure out a way to reduce memory usage for this program further. Basically, I'm reading from JSON log files into a pandas dataframe, but: the list append function is what is causing the issue. It creates two different objects in memory, causing huge memory usage. .to_pickle method of pandas is also a huge memory hog, because the biggest spike in memory is when writing to the pickle. Here is my most efficient implementation to date: columns = ['eventName', 'sessionId', "eventTime",

Pickling a django model with a binary field in cacheops

阅读更多关于 Pickling a django model with a binary field in cacheops

问题 I have a simple Django model with a binary field that I would like to pickle. class MyModel(models.Model): bin_data = models.BinaryField() From the context of my unittests, I do the following: import pickle tmp_obj = MyModel.objects.create(bin_data="12345") obj = MyModel.objects.get(pk=tmp_obj.pk) # load from DB data = pickle.dumps(obj) obj2 = pickle.loads(data) However the pickle.dumps() fails with: TypeError: can't pickle buffer objects When I use the following command to pickle: data =

How does Python 3 know how to pickle extension types, especially Numpy arrays?

阅读更多关于 How does Python 3 know how to pickle extension types, especially Numpy arrays?

问题 Numpy arrays, being extension types (aka defined using in extensions the C API), declare additional fields outside the scope of the Python interpreter (for example the data attribute, which is a Buffer Structure , as documented in Numpy's array interface. To be able to serialize it, Python 2 used to use the __reduce__ function as part of the pickle protocol, as stated in the doc, and explained here. But, even if __reduce__ still exists in Python 3, the Pickle protocol section (and Pickling

pathos multiprocessing cannot pickle

阅读更多关于 pathos multiprocessing cannot pickle

I am having a similar issue to this person . I am unable to run a simple multiprocessing routine in the pathos module and receive a pickling error. Below is the code and error. from pathos.multiprocessing import ProcessingPool import dill class ProcClass(object): def __init__(self): pass def f(self,x): return x*x pc = ProcClass() pl = ProcessingPool(3) print pl.map(pc.f, range(10)) The returned error: Exception in thread Thread-2: Traceback (most recent call last): File "/Library/Frameworks/EPD64.framework/Versions/7.3/lib/python2.7/threading.py", line 551, in __bootstrap_inner self.run() File

unpickle OrderedDict from python3 in python2

阅读更多关于 unpickle OrderedDict from python3 in python2

I'm trying to unpickle objects pickled in python3. This works in python3 but not in python2. The issue can be reproduced down to pickle protocol 0. Example code: import pickle import collections o = collections.OrderedDict([(1,1),(2,2),(3,3),(4,4)]) f = open("test.pkl", "wb") pickle.dump(o, f, 0) f.close() This results in the following pkl file: python2: ccollections OrderedDict p0 ((lp1 (lp2 I1 aI1 aa(lp3 I2 aI2 aa(lp4 I3 aI3 aa(lp5 I4 aI4 aatp6 Rp7 python3: cUserString OrderedDict p0 (tRp1 L1L L1L sL2L L2L sL3L L3L sL4L L4L s. When I try to load the pickle file created in python3 from

Pickling decorated callable class wrapper

阅读更多关于 Pickling decorated callable class wrapper

问题 I'm struggling to pickle a wrapped function when I use a custom callable class as a wrapper. I have a callable class "Dependee" that keeps track of dependencies for a wrapped function with a member variable "depends_on". I'd like to use a decorator to wrap functions and also be able to pickle the resulting wrapped function. So I define my dependee class. Note the use of functools.update_wrapper. >>> class Dependee: ... ... def __init__(self, func, depends_on=None): ... self.func = func ...