Allocation in execv, execvp, and execve

[hniksic]

The exec family of functions replace the current process with the new one, thus programs that wish to spawn a child normally call exec*() soon after a fork(). Unfortunately, fork interacts badly with threads, because of which POSIX restricts the functions that are allowed to be called after fork() to a small set of async-signal-safe functions. The part that affects Rust is that, in particular, calls to malloc() are not allowed, which means any heap allocation after fork() is forbidden. See std::process documentation and source for additional details.

The nix interface to execv, execve, and execvp calls to_exec_array to allocate the kind of array that the corresponding C functions expect. It makes sense that this is done inside exec; the argv and envp pointers are laid out in a way completely alien to Rust, and in violation to its safety rules. However, since exec* functions are typically invoked after a fork, it would be very nice to make the allocation optional, while still providing a fully safe wrapper.

One way to avoid allocation after fork and retain safety is allocate the needed buffers before forking. For example, a type that encapsulates an owning vector of C strings could be instantiated before fork(), and passed to actual exec. Since it would always store both the vector of CString objects and a vector of pointers to their content, its as_c_vec() method could produce the layout needed by exec* without further allocation:

struct CVec {
    strings: Vec<CString>,

    // nullptr-terminated vector of pointers to data inside self.strings.
    ptrs: Vec<*const libc::c_char>,
}

impl CVec {
    fn new<S>(slice: &[S]) -> Result<CVec>
        where S: AsRef<OsStr>
    {
        // ...
    }

    pub fn as_c_vec(&self) -> *const *const libc::c_char {
        self.ptrs.as_ptr()
    }
}

A safe variant of unistd::execvp() would only accept CVec objects:

pub fn safe_execv(path: &CString, argv: &CVec) -> Result<Void> {
    unsafe {
        libc::execv(path.as_ptr(), argv.as_c_vec())
    };
    Err(Error::Sys(Errno::last()))
}

Another option, which is more of a hassle to implement, but perhaps more convenient to use, is to completely encapsulate the allocation done inside the exec wrapper. It would effectively support staged execution, where first stage would prepare the needed data and return a closure that carries out the actual call to libc::exec, to be called after forking:

let do_exec = unistd::stage_execvp("cmd", &["arg0", "arg1", "arg2"])?;
if unistd::fork()?.is_child() {
    do_exec();  // handle errors
    _exit(127);
}

String arguments are supported by accepting AsRef<OsStr>, again converted to C strings before fork(). Here is an example of a library that implements this approach to wrap exec* and implement the equivalent of execvpe. (While developing this library, the code exhibited real hangs before its own exec wrapper was switched to the approach described above.)

[kamalmarhubi]

Thanks for raising this. I need to read up a bit more on the restrictions post-fork.

[kamalmarhubi]

I was writing a shell last weekend, and came up against this. The shell was single-threaded, but I found it quite uncomfortable to be doing allocation after fork.

Do you know if unsafety can result from allocating after fork, or is it just deadlock?

[hniksic]

If the process has only one thread, then you have nothing to worry about. The problem arises if and only if fork() is executed while another thread holds a lock that will also be needed in the child. One thing you need to consider, though, is that a library or a crate you are using might be spawning threads behind your back.

The problem is this issue only concerns deadlock, no Rust-level unsafety is present. Again, third-party C libraries can do what they like - for example, libdispatch, which can spawn its own threads, is known to use pthread_atfork() to poison its data structures in the child process to force an instant crash if someone tries to call it there.

[Kixunil]

Note that exec could require &CStr in arguments. &CString is not needed.