Segmentation fault when using C callback user data to store a boxed Rust closure

问题

I am creating a Rust wrapper around a C API. One function in this C API sets a callback and accepts a void pointer which will be passed to the callback. It stores a reference to the callback and user data for later, so I am using the last code section from this answer.

Here is my code. The Test::trigger_callback(...) function is meant to emulate the C library calling the callback.

extern crate libc;

use libc::c_void;
use std::mem::transmute;

struct Test {
    callback: extern "C" fn(data: i32, user: *mut c_void) -> (),
    userdata: *mut c_void,
}

extern "C" fn c_callback(data: i32, user: *mut libc::c_void) {
    unsafe {
        println!("Line {}. Ptr: {}", line!(), user as u64);
        let func: &mut Box<FnMut(i32) -> ()> = transmute(user);
        println!("Line {}. Data: {:?}", line!(), data);
        (*func)(data);
        println!("Line {}", line!());
    }
}

impl Test {
    fn new<F>(func: F) -> Test
    where
        F: FnMut(i32) -> (),
        F: 'static,
    {
        let func = Box::into_raw(Box::new(Box::new(func)));
        println!("Line: {}, Ptr: {}", line!(), func as u64);

        Test {
            callback: c_callback,
            userdata: func as *mut c_void,
        }
    }

    fn trigger_callback(&self, data: i32) {
        (self.callback)(data, self.userdata);
    }
}

fn main() {
    let test = Test::new(|data: i32| {
        println!("Inside callback! Data: {}", data);
    });

    test.trigger_callback(12345);
}

As mentioned in the linked answer, Boxing the closure is meant to store it on the heap so that a pointer to it will be valid for an arbitrarily long time, and then Boxing that Box is because it's a fat pointer, but needs to be converted to a regular pointer so that it can be cast to a void pointer.

When run, this code prints out:

Line: 29, Ptr: 140589704282120
Line 13. Ptr: 140589704282120
Line 15. Data: 12345
Segmentation fault (core dumped)

It segfaults when trying to call the closure inside of the extern "C" function.

Why? As far as I understand it, putting the closure in a Box and then using Box::into_raw(...) should store it on the heap and 'leak' the memory, so the pointer should be valid for as long as the program is running. What part of this is wrong?

回答1:

Box::into_raw(Box::new(Box::new(func)));

This does not produce the type you think it does:

   = note: expected type `()`
              found type `*mut std::boxed::Box<F>`

You assume it's a trait object:

let func: &mut Box<FnMut(i32) -> ()> = transmute(user);

Instead, make the input value into a trait object when you box it. I advocate for explicit lines with comments explaining each step:

// Trait object with a stable address
let func = Box::new(func) as Box<FnMut(i32)>;
// Thin pointer
let func = Box::new(func);
// Raw pointer
let func = Box::into_raw(func);

Box<FnMut(i32) -> ()>

The return type of () is redundant; use Box<FnMut(i32)>

let func: &mut Box<FnMut(i32) -> ()> = transmute(user);

Try really hard to avoid transmute. There are usually smaller tools to use:

extern "C" fn c_callback(data: i32, user: *mut libc::c_void) {
    let user = user as *mut Box<FnMut(i32)>;
    unsafe {
        (*user)(data);
    }
}

Avoid restating the same type all over. Introduce a type alias:

type CallbackFn = Box<FnMut(i32)>;

let user = user as *mut CallbackFn;

let func = Box::new(func) as CallbackFn;