C++ pthread blocking queue deadlock (I think)

Question

I am having a problem with pthreads where i think i am getting a deadlock. I have created a blocking queue which I thought was working, but after doing some more testing I h

Answer 1

I'm not exactly familiar with pthread_cancel() - I prefer cooperative termination.

Wouldn't a pthread_cancel() leave your mutex locked? I suppose you need to cleanup with a cancellation handler.

Answer 2

pthread_cancel() is best avoided.

You can unblock all your threads blocked on Blocking_Queue::pull() by throwing an exception from there.

One weak spot in the queue is that T t = _queue.front(); invokes the copy constructor of T that may throw an exception, rendering you queue mutex locked forever. Better use C++ scoped locks.

Here is an example of graceful thread termination:

$ cat test.cc
#include <boost/thread/mutex.hpp>
#include <boost/thread/thread.hpp>
#include <boost/thread/condition_variable.hpp>
#include <exception>
#include <list>
#include <stdio.h>

struct BlockingQueueTerminate
    : std::exception
{};

template<class T>
class BlockingQueue
{
private:
    boost::mutex mtx_;
    boost::condition_variable cnd_;
    std::list<T> q_;
    unsigned blocked_;
    bool stop_;

public:
    BlockingQueue()
        : blocked_()
        , stop_()
    {}

    ~BlockingQueue()
    {
        this->stop(true);
    }

    void stop(bool wait)
    {
        // tell threads blocked on BlockingQueue::pull() to leave
        boost::mutex::scoped_lock lock(mtx_);
        stop_ = true;
        cnd_.notify_all();

        if(wait) // wait till all threads blocked on the queue leave BlockingQueue::pull()
            while(blocked_)
                cnd_.wait(lock);
    }

    void put(T t)
    {
        boost::mutex::scoped_lock lock(mtx_);
        q_.push_back(t);
        cnd_.notify_one();
    }

    T pull()
    {
        boost::mutex::scoped_lock lock(mtx_);

        ++blocked_;
        while(!stop_ && q_.empty())
            cnd_.wait(lock);
        --blocked_;

        if(stop_) {
            cnd_.notify_all(); // tell stop() this thread has left
            throw BlockingQueueTerminate();
        }

        T front = q_.front();
        q_.pop_front();
        return front;
    }
};

void sleep_ms(unsigned ms)
{
    // i am using old boost
    boost::thread::sleep(boost::get_system_time() + boost::posix_time::milliseconds(ms));
    // with latest one you can do this
    //boost::thread::sleep(boost::posix_time::milliseconds(10));
}

void thread(int n, BlockingQueue<int>* q)
try
{
    for(;;) {
        int m = q->pull();
        printf("thread %u: pulled %d\n", n, m);
        sleep_ms(10);
    }
}
catch(BlockingQueueTerminate&)
{
    printf("thread %u: finished\n", n);
}

int main()
{
    BlockingQueue<int> q;

    // create two threads
    boost::thread_group tg;
    tg.create_thread(boost::bind(thread, 1, &q));
    tg.create_thread(boost::bind(thread, 2, &q));
    for(int i = 1; i < 10; ++i)
        q.put(i);
    sleep_ms(100); // let the threads do something
    q.stop(false); // tell the threads to stop
    tg.join_all(); // wait till they stop
}

$ g++ -pthread -Wall -Wextra -o test -lboost_thread-mt test.cc

$ ./test
thread 2: pulled 1
thread 1: pulled 2
thread 1: pulled 3
thread 2: pulled 4
thread 1: pulled 5
thread 2: pulled 6
thread 1: pulled 7
thread 2: pulled 8
thread 1: pulled 9
thread 2: finished
thread 1: finished

Answer 3

I've had similar experience with pthread_cond_wait() / pthread_cancel(). I had issues with a lock still being held after the thread returned for some reason, and it was impossible to unlock it, since you have to unlock in the same thread as you locked. I noticed these errors when doing pthread_mutex_destroy() since I had a single producer, single consumer situation so the deadlock didn't occur.

pthread_cond_wait() is supposed to lock the mutex when returning, and this could have happened, but the final unlock didn't go through since we forcefully canceled the thread. For safety I generally try to avoid using pthread_cancel() altogether since some platforms don't even support this. You could use a volatile bool or atomics and check if thread should be shut down. That way, the mutexes will be handled cleanly as well.