Pickle Queue objects in python

Question

I have a class that uses a list of Queue objects. I need to pickle this class including the information saved in the queue objects. For example:

import Queue
import pickle

class QueueTest(object):
    def __init__(self):
        self.queueList = []
    def addQueue(self):
        q = Queue.Queue()
        q.put('test')
        self.queueList.append(q)


obj = QueueTest()
obj.addQueue()

with open('pickelTest.dat','w') as outf:
    pickle.dump(obj,outf)

returns the error

raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: can't pickle lock objects

Is there a work around to pickle Queue objects?

Answer 1

I suggest replacing your uses of Queue.Queue with collections.deque. The Queue class is intended to be used for synchronized communication between threads, so it will have some unnecessary overhead when used as a regular data structure. collections.deque is a faster alternative. (The name "deque" is pronounced "deck" and means "double-ended queue".)

The deque class does have a different API than the Queue type, but it should be pretty easy to translate between them. Use deque.append in place of Queue.put and deque.popleft in place of q.get() (or appendleft and pop, if you feel like going the other direction). Rather than calling Queue.empty, just use a deque instance as a Boolean value (like you do to test for an empty list).

deque instances are picklable:

>>> import collections, pickle
>>> q = collections.deque(["test"])
>>> pickle.dumps(q)
b'\x80\x03ccollections\ndeque\nq\x00]q\x01X\x04\x00\x00\x00testq\x02a\x85q\x03Rq\x04.'

Answer 2

As you have commented to @Blckknght, you don't need the synchronization features of Queue.Queue. So just use collections.deque that the Queue.Queue class uses itself as the underlying queue data structure. You will have to use .appendleft to emulate the FIFO Queue.put and .pop to emulate Queue.get