What does “release sequence” mean?

Question

I don\'t understand, why will be problems without release sequence, if we have 2 threads in the example below. We have only 2 operations on the atomic variable

Answer 1

i stumbled over the exact same question like you did. i thought i got the understanding right and then he comes in with this example and only uses std::memory_order_aquire. it was difficult to find any good information on this, but finally i found some helpful sources. the main information i was not aware of was the simple fact, that read-modify-write operations ALWAYS work on the newest/latest value, no matter what memory order given (even std::memory_order_relaxed). this ensures, that you wont have the same index two times in the example. still the ordering of operations can mix up (so you dont know which fetch_sub will happens before the other).

this is an answer of anthony williams himself stating that read-modify-write operations always work on the latest value: Concurrency: Atomic and volatile in C++11 memory model

additionally, someone asked about the fetch_sub in combination with the shared_ptr ref count. here anthony williams responded too and brings clarity into the situation with the reordering of the fetch_sub: https://groups.google.com/a/isocpp.org/forum/#!topic/std-discussion/OHv-oNSuJuk

Answer 2

What does he mean? That both threads should see the value of count is 20? But in my output count is sequently decremented in threads.

No he doesn't. All modification to count are atomic, so both reader threads would always see different values for it in the given code.

He's talking about the implications of the release sequence rule, namely that when a given thread performs a release store, other multiple threads that then perform acquire loads of the same location form a release sequence, in which each subsequent acquire load has a happens-before relationship with the storing thread (i.e. the completion of the store happens-before the load). This means that the load operation in the reader thread is a synchronisation point with the writer thread, and all memory operations in the writer prior to the store must complete and be visible in the reader when its corresponding load completes.

He's saying that without this rule, only the first thread would be thus synchronised to the writer. The second thread would therefore have a data race in accessing queue (note: not count, which is protected anyway by atomic access). Theoretically, memory operations on data occurring before the store on count could be seen by reader thread number 2 only after its own load operation on count. The release sequence rule assures that this will not happen.

In summary: the release sequence rules assures multiple threads can synchronise their loads on a single store. The synchronisation in question is that of memory accesses to data other than the actual atomic variable being synchronised on (which is guaranteed to be synchronised anyway due to being atomic).

Note to add here: for the most part these kind of issues are only of concern on CPU architectures that are relaxed about reordering their memory operations. The Intel architecture is not one of them: it is strongly-ordered and has only a few very specific circumstances in which memory operations can ever be reordered. These kind of nuances are mostly only relevant when talking about other architectures, such as ARM and PowerPC.

Answer 3

it means that the initial store is synchronized-with the final load even if the value read by the final load isn't directly the same value stored at beginning, but it is the value modified by one of the atomic instruction which could race into. A simpler example, assuming there are three threads racing which executes these instruction (assume x initialized to 0 before the race)

// Thread 1:
A;
x.store(2, memory_order_release);

// Thread 2:
B;
int n = x.fetch_add(1, memory_order_relaxed);
C;

// Thread 3:
int m = x.load(memory_order_acquire);
D;

What are the possible values read for n and m according to possible results of the race? And what are the guarantees that we have on the ordering of instructions A, B, C, and D based on what we read on m and n? For n we have two cases, either 0 or 2. For m we could read 0, 1, 2, and 3. There are six valid combinations of the two. Let's see each case:

• m = 0, n = 0. We don't have any synchronizes-with relationship, thus we can't infer any happens-before relationship except for the obvious B happens-before C

• m = 0, n = 2. Even though the fetch_add operation read the value written by the store, since the fetch_add has a relaxed memory ordering there is no synchronizes-with relationship between the two instruction. We can't say that A happens-before C

• m = 1, n = 0. Similarly as before, since fetch_add don't have a release semantic we can't infer a synchronizes-with relationship between the fetch_add and the load operation, hence we don't know whether B happens-before D

• m = 2, n = 0. The value we read with the acquire semantic load has been written with a release semantic store. We are guaranteed that the store synchronizes-with the load, hence A happens-before D

• m = 2, n = 2. Same as above, the store synchronizes-with the load, hence A happens-before D. As usual, the fact that the value read from fetch_add is the same as the one stored from thread 1 do not imply any synchronization relationship.

• m = 3, n = 2. In this case the data read by the load has been written by the fetch_add, and the data read by the fetch_add has been written by the store. However because fetch_add has relaxed semantic, no synchronization can be assumed between store and fetch_add and between fetch_add and load. Apparently, in this case no synchronization can be assumed, same as the case m = 0, n = 0. Here is where the release sequence concept comes in handy: the release semantic store in thread 1 will synchronize-with the acquire semantic load in thread 3 as long as the value that is being read has been written in the release sequence, which includes

  1. all the stores performed later in the same thread as the release operation
  2. all the atomic read-modify-write operation which read a value from the same release sequence.

  In this case since fetch_add is an atomic read-modify-write operation we know that the store in thread 1 synchronizes-with the load in thread 3, and thus A happens-before D. We still can't say anything about the ordering of B and C though.

In your case you have this pseoudocode, assuming number_of_items = 2:

// Thread 1
Item[0] = ...;
Item[1] = ...;
count.store(2,memory_order_release);

// Thread 2
int i2 = 0;
while (i2 = count.fetch_sub(1,memory_order_acquire) <= 0 ) sleep();
auto x2 = Item[i2-1];
process(x2);

// Thread 3
int i3 = 0;
while (i3 = count.fetch_sub(1,memory_order_acquire) <= 0 ) sleep();
auto x3 = Item[i3-1];
process(x3);

Let's assume that the first positive value read into i2 is 2, and thus the first positive value read into i3 is 1. Since the value read from Thread 2 has been written from the store in Thread 1, the store synchronizes-with the load, and we know that Item[1] = ...; from Thread 1 happens-before auto x2 = Item[1]; in Thread 2. However the value 1 read from Thread 3 has been written by Thread 2, with fetch_sub which has no release semantic. The fetch_sub from Thread 2 thus does not synchronizes-with the fetch_sub from Thread 3, however since the fetch_sub from Thread 2 is part of the release chain of the store in Thread 1, the store in Thread 1 also synchronizes-with the fetch_sub in Thread 3, from which we know that Item[0] = ...; happens-before auto x3 = Item[0];

Answer 4

fetch_sub is read-modify-write action. it atomically reads the value from the memory address, decrement it by the argument provided and then writes it back to the memory address . it all happens atomically.

now, every atomic action reads and writes directly to the memory address. the CPU does not rely on a value cachced in the registers or the cache-lines for performance gain. it reads and writes to the memory address directly and prevent othr CPU's to do so in that time.

what "plain" (==relaxed) atomicity does not provide is reordering. both the compiler and the CPU scramble reads and writes in order to speed up the execution of the program.

look at the example below:

atomic integer i
regular integer j

Thread A:
i <- 5
//do something else
i -> j
//make some decisions regarding j value.

Thread B:
i++

if no memory order is supllied the compiler and the CPU are allowed to transform the code to

Thread A:
i -> j
i <- 5
//do something else
//make some decisions regarding j value.

Thread B:
i++

Which is of-course not what we wanted. the decision making is wrong.

what we need is memory reordering.

memory order acquire: don't scramble memory accesses before
memory order release: don't scramble memory accesses after

going back to the question:

fetch_sub is both reading a value and writing a value. by specifying a memory order acquire we say "I only care about the order of actions the happened before the reading"
by specifying memory order release we say "I only care about the order of actions happened after the writing.

But you do care about memory access before and after!

if you only have one consumer thread, than the sub_fetch does not effect anyone, because the producer anyway uses plain store and the affects of fetch_sub are only visible to the thread which invoked fetch_sub. and in this case, you only care about the reading - the reading gives you the current and updated index. what happens after you store the updated index (lets say x-1) is not that important.

but since there are two threads which read and write to counter it is important that thread A will be aware that thread B wrote a new value to the counter and Thread B is aware that Thread A is about the read the value of counter. also vice versa- Thread B must be aware that Thread A wrote a new value to counter and Thread A must be aware that Thread B is about to read a value from counter

you need both guarantees - every thread states that it is about to both read and write to the shared counter. the memory order you need is std::memory_order_acquire_release.

But the example is tricky. the producer thread simply stores a new value in counter regardless of the value which was there before. if the producer thread was to incremenet the counter each time it pushes new item - you had to use std::memory_order_acquire_release in both the producer and the consumer threads even if you had one consumer