race-condition in pthread_once()?

Question

I have a std::future in one thread which is waiting on a std::promise being set in another thread.

EDIT: Updated the question with an exemplar app which will block forever:

UPDATE: If I use a pthread_barrier instead, the below code does not block.

I have created a test-app which illustrates this:

Very basically class foo creates a thread which sets a promise in its run function, and waits in the constructor for that promise to be set. Once set, it increments an atomic count

I then create a bunch of these foo objects, tear them down, and then check my count.

#include <iostream>
#include <thread>
#include <atomic>
#include <future>
#include <list>
#include <unistd.h>

struct foo
{
    foo(std::atomic<int>& count)
        : _stop(false)
    {
        std::promise<void> p;
        std::future <void> f = p.get_future();

        _thread = std::move(std::thread(std::bind(&foo::run, this, std::ref(p))));

        // block caller until my thread has started 
        f.wait();

        ++count; // my thread has started, increment the count
    }
    void run(std::promise<void>& p)
    {
        p.set_value(); // thread has started, wake up the future

        while (!_stop)
            sleep(1);
    }
    std::thread _thread;
    bool _stop;
};

int main(int argc, char* argv[])
{
    if (argc != 2)
    {
        std::cerr << "usage: " << argv[0] << " num_threads" << std::endl;
        return 1;
    }
    int num_threads = atoi(argv[1]);
    std::list<foo*> threads;
    std::atomic<int> count(0); // count will be inc'd once per thread

    std::cout << "creating threads" << std::endl;
    for (int i = 0; i < num_threads; ++i)
        threads.push_back(new foo(count));

    std::cout << "stopping threads" << std::endl;
    for (auto f : threads)
        f->_stop = true;

    std::cout << "joining threads" << std::endl;
    for (auto f : threads)
    {
        if (f->_thread.joinable())
            f->_thread.join();
    }

    std::cout << "count=" << count << (num_threads == count ? " pass" : " fail!") << std::endl;
    return (num_threads == count);
}

If I run this in a loop with 1000 threads, it only has to execute it a few times until a race occurs and one of the futures is never woken up, and therefore the app gets stuck forever.

# this loop never completes
$ for i in {1..1000}; do ./a.out 1000; done

If I now SIGABRT the app, the resulting stack trace shows it's stuck on the future::wait The stack trace is below:

// main thread
    pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
    __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at libstdc++-v3/include/x86_64-unknown-linux-gnu/bits/gthr-default.h:846
    std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../libstdc++-v3/src/condition_variable.cc:56
    std::condition_variable::wait<std::__future_base::_State_base::wait()::{lambda()#1}>(std::unique_lock<std::mutex>&, std::__future_base::_State_base::wait()::{lambda()#1}) (this=0x93a050, __lock=..., __p=...) at include/c++/4.7.0/condition_variable:93
    std::__future_base::_State_base::wait (this=0x93a018) at include/c++/4.7.0/future:331
    std::__basic_future<void>::wait (this=0x7fff32587870) at include/c++/4.7.0/future:576
    foo::foo (this=0x938320, count=...) at main.cpp:18
    main (argc=2, argv=0x7fff32587aa8) at main.cpp:52


// foo thread
    pthread_once () from /lib64/libpthread.so.0
    __gthread_once (__once=0x93a084, __func=0x4378a0 <__once_proxy@plt>) at gthr-default.h:718
    std::call_once<void (std::__future_base::_State_base::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>&, bool&), std::__future_base::_State_base* const, std::reference_wrapper<std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()> >, std::reference_wrapper<bool> >(std::once_flag&, void (std::__future_base::_State_base::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, ...) at include/c++/4.7.0/mutex:819
    std::promise<void>::set_value (this=0x7fff32587880) at include/c++/4.7.0/future:1206
    foo::run (this=0x938320, p=...) at main.cpp:26

I'm pretty sure that I'm not doing anything wrong in my code, right?

Is this an issue with the pthread implementation, or the std::future/std::promise implementation?

My library versions are:

libstdc++.so.6
libc.so.6 (GNU C Library stable release version 2.11.1 (20100118))
libpthread.so.0 (Native POSIX Threads Library by Ulrich Drepper et al Copyright (C) 2006)

Answer 1

Indeed, there is a race condition between the destructor of the local promise object (at the end of the constructor and the call to set_value() from the thread. That is, set_value() wakes the main tread, that just next destroys the promise object, but the set_value() function has not yet finished, and dead-locks.

Reading the C++11 standard, I'm not sure if your use is allowed:

void promise<void>::set_value();

Effects: atomically stores the value r in the shared state and makes that state ready.

But somewhere else:

The set_value, set_exception, set_value_at_thread_exit, and set_exception_at_thread_exit member functions behave as though they acquire a single mutex associated with the promise object while updating the promise object.

Are set_value() calls supposed to be atomic with regards to other functions, such as the destructor?

IMHO, I'd say no. The effects would be comparable to destroying a mutex while other thread is still locking it. The result is undefined.

The solution would be to make p outlive the thread. Two solutions that I can think of:

1. Make p a member of the class, just as Michael Burr suggested in the other answer.

2. Move the promise into the thread.

In the constructor:

std::promise<void> p;
std::future <void> f = p.get_future();
_thread = std::thread(&foo::run, this, std::move(p));

BTW, you don't need the call to bind, (the thread constructor is already overloaded), or call to std::move to move the thread (the right value is already an r-value). The call to std::move into the promise is mandatory, though.

And the thread function does not receive a reference, but the moved promise:

void run(std::promise<void> p)
{
    p.set_value();
}

I think that this is precisely why C++11 defines two different classes: promise and future: you move the promise into the thread, but you keep the future to recover the result.

Answer 2

Try moving the std::promise<void> p; so that instead of being local to the constructor it will be a member of struct foo:

struct foo
{
    foo(std::atomic<int>& count)
        : _stop(false)
    {
        // std::promise<void> p;    // <-- moved to be a member
        std::future <void> f = p.get_future();

        // ...same as before...
    }
    void run(std::promise<void>& p)
    {
        // ... same ...
    }

    std::promise<void> p;   // <---
    std::thread _thread;
    bool _stop;
};

I beleive that what may be happening is that you get into a race where p is destroyed in the constructor while p.set_value() is acting on the reference to that promise. Something occurs inside set_value() while it's finishing up/cleaning up; acting on the reference to the already destroyed std::promise is corrupting some state in the pthread library.

This is just speculation - I don't have ready access to a system that reproduces the problem at the moment. But making p a member will ensure its lifetime extends well past the completion of the set_value() call.