Pickled file won't load on Mac/Linux

Question

I have an application that imports data from a pickled file. It works just fine in Windows but Mac and Linux behaviour is odd.

In OS X, the pickled file (file extens

Answer 1

As mentioned by Adam, the problem is likely to be the newline format of the pickle file.

Unfortunately, the real problem is actually caused on save rather than load. This may be recoverable if you're using text mode pickles, rather than binary. Try opening the file in universal newline mode, which will cause python to guess what the right line-endings are ie:

char_file=open('filename.char','rU')

However, if you're using a binary format (cPickle.dump(file, 1)) you may have an unrecoverably corrupted pickle (even when loading in Windows) - if you're lucky and no \r\n characters show up then it may work, but as soon as this occurs you could end up with corrupted data, as there's no way to distinguish between a "real" \r\n code and one windows has inserted on seeing just \n.

The best way to handle things to be loaded in multiple platforms is to always save in binary mode. On your windows machine, when saving the pickle use:

char_file = open('filename.char','wb')
cPickle.dumps(data, char_file)