Box<[T]>::into_raw is useless

[vinipsmaker]

Sorry about the horrible title, but I really don't know how to else "summarize" this problem.

I tried to get an answer for this several times on StackOverflow (after documentation proved useless) and now I'm coming here. Maybe it's just a matter of updating documentation to have guarantees of a behaviour that will continue to work on future versions of Rust (the main purpose of documentation in the first place) or maybe the error is me who failed to spot the useful part.

I need to expose a Rust library to C applications and I need to expose an array of size only know at runtime to a function which will own this data and later will call a Rust function to deallocate it. I have a Vec<u8> and I need to get this (*mut u8, usize) tuple out of it. Later I need to get receive this same tuple and deallocate its data.

One option is to use mem::forget + Vec::from_raw_parts, but this is far from ideal as shrink_to_fit isn't guaranteed to work and I'd also need to store capacity (the C library really shouldn't need to worry about that).

The perfect option would be to use Vec::into_boxed_slice. However, it crashed when I tried to use the pointer for the first element (slice::as_mut_ptr) to Box::from_raw, as it should. And I cannot pass the pointer return from Box::into_raw to the C function, as the memory layout is unspecified and I don't have any guarantee it'll be a pointer to the first element.

How do we fix this? Update documentation? Add some extra function to Box<[T]> like Box<[T]>::from_raw_parts(ptr: *mut T, len: usize)?

[nagisa]

I have a Vec and I need to get this (*mut u8, usize) tuple out of it. Later I need to get receive this same tuple and deallocate its data.

vec.shrink_to_fit();
let ptr = vec.as_mut_ptr();
let len = vec.len();
::std::mem::forget(vec);
// <snip>
drop(Vec::from_raw_parts(ptr, len, len));

this is far from ideal as shrink_to_fit isn't guaranteed to work… The perfect option would be to use Vec::into_boxed_slice. However, it crashed when I tried to use the pointer for the first element (slice::as_mut_ptr) to Box::from_raw,

This is how you construct a fat slice box out of pointer and capacity:

let slice = slice::from_raw_parts_mut(ptr, capacity);
let boxed_slice: Box<[T]> = Box::from_raw(slice);

shrink_to_fit is exactly what Vec::into_boxed_slice does before making a boxed slice, so there’s really absolutely no difference between using that or Vec::into_boxed_slice.

And I cannot pass the pointer return from Box::into_raw to the C function, as the memory layout is unspecified and I don't have any guarantee it'll be a pointer to the first element.

You should be able to pass [T]::as_mut_ptr() to C code just fine.

[ollie27]

@nagisa shrink_to_fit doesn't guarantee that the length will equal the capacity so I think into_boxed_slice is the correct thing to use.

[nagisa]

@ollie27 then we have a larger problem of invalid implementation of Vec::into_boxed_slice.

[ollie27]

See #29847.

[ustulation]

I think the problem is with documentation. shrink_to_fit from the documentation isn't guaranteed to make capacity == len. So if you loose information about capacity, from_raw_parts is an undefined behaviour (if you assume len == capacity and pass it as such). I assume the only way into_vec() on Boxed-slice will be well behaved is if it guarantees to make len == capacity but documentation says it's exactly as if we did shrink_to_fit. This seems contradictory.

[vinipsmaker]

I think boxed slice is more appropriate here. We don't need to give more guarantees to Vec. We could just "fix" boxed slice.

[vinipsmaker]

Hey guys, any update on this? Our FFI interfaces still feel weird taking a capacity member everywhere.

[ollie27]

I'm not sure what update you're waiting on. The documentation certainly needs to be improved but as far as I'm aware you can do the following now with no problems:

pub fn vec_to_ptr_len(v: Vec<u8>) -> (*mut u8, usize) {
    let mut b = v.into_boxed_slice();
    let ptr = b.as_mut_ptr();
    let len = b.len();
    std::mem::forget(b);
    (ptr, len)
}

pub unsafe fn ptr_len_to_vec(ptr: *mut u8, len: usize) -> Vec<u8> {
    Box::from_raw(std::slice::from_raw_parts_mut(ptr, len)).into_vec()
}

[steveklabnik]

I would like to close out this ticket. Here's how I understand it: What's left is basically @ustulation's comment above, that is, shrink_to_fit says it does not guarantee capacity but into_boxed_slice seems like it must, yet says it's the same as shrink_to_fit. Is this accurate?

@rust-lang/libs, what do you want done here?

[bluss]

I think that there is lacking clarity and it needs one hard look at usable size again (if that is ok)? Rust could gain some memory utilization there, removing clown shoes, without breaking any current Vec API. It requires looking at jemalloc's size bins and working out the relevant details. The theory is that all capacities in the same bin are equivalent, and usable size gives you the largest one in the allocation's current bin.

With that in place, docs related to this can be clarified with confidence.

[alexcrichton]

Taking advantage of extra space given to us by the allocator seems like it falls under #29931, in which case it sounds to me like @steveklabnik is right, we should just remove the clause from into_boxed_slice about it being equivalent to shrink_to_fit.

The fact that into_boxed_slice calls shrink_to_fit is just an implementation detail we need not worry about. If we change the implementation of Vec to take advantage of excess capacity then we'll fix that implementation as well.

[bluss]

That sounds good!

[QuietMisdreavus]

In case anyone is willing to tackle this, the docs team wants to take this approach:

...we should just remove the clause from into_boxed_slice about it being equivalent to shrink_to_fit.

[seanprashad]

Hi there! I'm a student at Seneca College whose enrolled in an Open Source course taught by my professor and Mozillian, David Humphrey. I would love to tackle this bug for my course if it's still available and unassigned.

[steveklabnik]

That'd be great! Is @QuietMisdreavus 's post on what to do clear enough?

[seanprashad]

@steveklabnik I would love to say yes and get on my way but I'm still a little unclear. Would I be removing documentation for into_boxed_slice being equivalent to shrink_to_fit? Any and all clarification is welcomed!!

[steveklabnik]

Would I be removing documentation for into_boxed_slice being equivalent to shrink_to_fit?

Yup! To elaborate a little further, that means modifying these lines:

rust/src/liballoc/vec.rs

Lines 510 to 516 in 94a82ad

	/// Note that this will drop any excess capacity. Calling this and
	/// converting back to a vector with [`into_vec`] is equivalent to calling
	/// [`shrink_to_fit`].
	///
	/// [owned slice]: ../../std/boxed/struct.Box.html
	/// [`into_vec`]: ../../std/primitive.slice.html#method.into_vec
	/// [`shrink_to_fit`]: #method.shrink_to_fit

The sentence about excess capacity should stay, but the second one and the link should be removed.

[seanprashad]

Thank you so much @steveklabnik! I will have to prepare my local environment on my laptop and get to work. Once I'm done and ready to submit a PR, I will let you know!

[vinipsmaker]

Thanks.