Python pickling error: TypeError: object pickle not returning list. Issue with numpy?

Question

I wrote a script that does a bit of data processing, then pickles the results before doing some more processing so I can fiddle with final parameters for plotting. I am pickling lists of two classes that I've created. Pickling is fine for each list, but unpickling one list returns the following error:

Traceback (most recent call last):
  File "script.py", line 980, in <module>
    main()
  File "script.py", line 956, in promoter_main
    open(os.path.join(pickle_dir, 'doublep.pickle'), 'rb'))
  File "/usr/lib/python2.6/pickle.py", line 1370, in load
    return Unpickler(file).load()
  File "/usr/lib/python2.6/pickle.py", line 858, in load
    dispatch[key](self)
  File "/usr/lib/python2.6/pickle.py", line 1217, in load_build
    setstate(state)
  File "/home/jmerkin/software/numpy/numpy/ma/core.py", line 5468, in __setstate__
    ndarray.__setstate__(self, (shp, typ, isf, raw))
TypeError: object pickle not returning list

As I said, one goes fine (list of instances of SingleP), the other gives me this error when unpickling (instances of DoubleP). I hesitate to post the entire code for the class because it's long, but basically, it has a couple methods that populate some attributes. These attributes consist of either booleans, lists of tuples, strings, ints, or lists of numpy.ma.arrays. Some of these arrays have masked numpy.nan.

This seems to be a potentially related issue: http://projects.scipy.org/numpy/ticket/897. In that case, the problem is arrays with an np.object as the array's dtype. However, I've been pickling and unpickling numpy.ma.arrays, including arrays that have masked numpy.nan, without problems for quite a while.

Am I missing something silly with pickling? I am not changing the object between when it was pickled and when I try to unpickle.

---

i rewrote bits of the code so that rather than than having an array of n x m x l dimensions as some attributes, i have a list with n entries, each of which is an m x l array and now it's happy and pickles and unpickles. i'm very confused but it runs. i'm not going to close this, because my questions persists, why did i get that error?

Answer 1

Had a very similar issue: I rewrote a script so that the np.nan values were masked as well (I wish numpy, scipy and matplotlib could handle arrays consistently) but this caused pickle.load() to baulk. I got rid of the masked arrays and everything worked again. The solution seems to be to add and remove masks to please pickle. Not much of a solution, I know, but it seems to work.