Shared semaphore between user and kernel spaces

Question

Short version

Is it possible to share a semaphore (or any other synchronization lock) between user space and kernel space? Named POSIX semaphores have kernel persi

Answer 1

Multiple solutions exist in Linux/GLIBC but none permit to share explicitly a semaphore between user and kernel spaces. The kernel provides solutions to suspend threads/processes and the most efficient is the futex. Here are some details about the state of the art of the current implementations to synchronize user space applications.

SYSV services

The Linux System V (SysV) semaphores are a legacy of the eponymous Unix OS. They are based on system calls to lock/unlock semaphores. The corresponding services are:

• semget() to get an identifier
• semop() to make operations on the semaphores (e.g. incrementation/decrementation)
• semctl() to make some control operations on the semaphores (e.g. destruction)

The GLIBC (e.g. 2.31 version) does not provide any added value on top of those services. The library service directly calls the eponymous system call. For example, semop() (in sysdeps/unix/sysv/linux/semtimedop.c) directly invokes the corresponding system call:

int
__semtimedop (int semid, struct sembuf *sops, size_t nsops,
          const struct timespec *timeout)
{
  /* semtimedop wire-up syscall is not exported for 32-bit ABIs (they have
     semtimedop_time64 instead with uses a 64-bit time_t).  */
#if defined __ASSUME_DIRECT_SYSVIPC_SYSCALLS && defined __NR_semtimedop
  return INLINE_SYSCALL_CALL (semtimedop, semid, sops, nsops, timeout);
#else
  return INLINE_SYSCALL_CALL (ipc, IPCOP_semtimedop, semid,
                  SEMTIMEDOP_IPC_ARGS (nsops, sops, timeout));
#endif
}
weak_alias (__semtimedop, semtimedop)

Nowadays, SysV semaphores (as well as other SysV IPC like shared memory and message queues) are considered deprecated because as they need a system call for each operation, they slow down the calling processes with systematic context switches. New applications should use POSIX compliant services available through the GLIBC.

POSIX services

POSIX semaphores are based on Fast User Mutexes (FUTEX). The principle consists to increment/decrement the semaphore counter in user space with atomic operations as long as there is no contention. But when there is contention (multiple threads/processes want to "lock" the semaphore at the same time), a futex() system call is done to either wake up waiting threads/processes when the semaphore is "unlocked" or suspend threads/processes waiting for the semaphore to be released. From performance point of view, this makes a big difference compared to the above SysV services which systematically required a system call for any operation. The POSIX services are implemented in GLIBC for the user space part of the operations (atomic operations) with a switch into kernel space only when there is contention.

For example, in GLIBC 2.31, the service to lock a semaphore is located in nptl/sem_waitcommon.c. It checks the value of the semaphore to decrement it with an atomic operation (in __new_sem_wait_fast()) and invokes the futex() system call (in __new_sem_wait_slow()) to suspend the calling thread only if the semaphore was equal to 0 before the attempt to decrement it.

static int
__new_sem_wait_fast (struct new_sem *sem, int definitive_result)
{
[...]
  uint64_t d = atomic_load_relaxed (&sem->data);
  do
    {
      if ((d & SEM_VALUE_MASK) == 0)
    break;
      if (atomic_compare_exchange_weak_acquire (&sem->data, &d, d - 1))
    return 0;
    }
  while (definitive_result);
  return -1;
[...]
}
[...]
static int
__attribute__ ((noinline))
__new_sem_wait_slow (struct new_sem *sem, clockid_t clockid,
             const struct timespec *abstime)
{
  int err = 0;

[...]
  uint64_t d = atomic_fetch_add_relaxed (&sem->data,
      (uint64_t) 1 << SEM_NWAITERS_SHIFT);

  pthread_cleanup_push (__sem_wait_cleanup, sem);

  /* Wait for a token to be available.  Retry until we can grab one.  */
  for (;;)
    {
      /* If there is no token available, sleep until there is.  */
      if ((d & SEM_VALUE_MASK) == 0)
    {
      err = do_futex_wait (sem, clockid, abstime);
[...]

The POSIX services based on the futex are for examples:

• sem_init() to create a semaphore
• sem_wait() to lock a semaphore
• sem_post() to unlock a semaphore
• sem_destroy() to destroy a semaphore

To manage mutex (i.e. binary semaphores), it is possible to use the pthread services. They are also based on the futex. For examples:

• pthread_mutex_init() to create/initialize a mutex
• pthread_mutex_lock/unlock() to lock/unlock a mutex
• pthread_mutex_destroy() to destroy a mutex