Tracking issue for RFC 1861: Extern types #43467
Comments
aturon
added
B-RFC-approved
aturon
changed the title
|
This is not explicitly mentioned in the RFC, but I'm assuming different instances of extern {
type A;
type B;
}
fn convert_ref(r: &A) -> &B { r } |
|
@jethrogb That's certainly the intention, yes. |
|
Relatedly, is deciding whether we want to call it EDIT: rust-lang/rfcs#2071 is also relevant here w.r.t. the connotations of |
|
Can we get a bullet for the panic vs DynSized debate? |
Mark-Simulacrum
added
the
C-tracking-issue
|
I've started working on this, and I have a working simple initial version (no generics, no DynSized). I've however noticed a slight usability issue. In FFI code, it's frequent for raw pointers to be initialized to null using Despite being unsized, extern types are used through thin pointers, so it should be possible to use It is still possible to cast an integer to an extern type pointer, but this is not as nice as just using the function designed for this. Also this can never be done in a generic context. extern {
type foo;
}
fn null_foo() -> *const foo {
0usize as *const foo
}Really we'd want is a new trait to distinguish types which use thin pointers. It would be implemented automatically for all sized types and extern types. Then the cast above would succeed whenever the type is bounded by this trait. Eg, the function fn null<T: ?Sized + Thin>() -> *const T {
0usize as *const T
}However there's a risk of more and more such traits creeping up, such as |
|
I think we can add extern types now and live with |
|
@SimonSapin yeah, it's definitely a minor concern for now. I do think this problem of not having a trait bound to express "this type may be unsized be must have a thin pointer" might crop up in other places though. |
|
Oh yeah, I agree we should solve that eventually too. I’m only saying we might not need to solve all of it before we ship any of it. |
|
I've pushed an initial implementation in #44295 |
Implement RFC 1861: Extern types A few notes : - Type parameters are not supported. This was an unresolved question from the RFC. It is not clear how useful this feature is, and how variance should be treated. This can be added in a future PR. - `size_of_val` / `align_of_val` can be called with extern types, and respectively return 0 and 1. This differs from the RFC, which specified that they should panic, but after discussion with @eddyb on IRC this seems like a better solution. If/when a `DynSized` trait is added, this will be disallowed statically. - Auto traits are not implemented by default, since the contents of extern types is unknown. This means extern types are `!Sync`, `!Send` and `!Freeze`. This seems like the correct behaviour to me. Manual `unsafe impl Sync for Foo` is still possible. - This PR allows extern type to be used as the tail of a struct, as described by the RFC : ```rust extern { type OpaqueTail; } #[repr(C)] struct FfiStruct { data: u8, more_data: u32, tail: OpaqueTail, } ``` However this is undesirable, as the alignment of `tail` is unknown (the current PR assumes an alignment of 1). Unfortunately we can't prevent it in the general case as the tail could be a type parameter : ```rust #[repr(C)] struct FfiStruct<T: ?Sized> { data: u8, more_data: u32, tail: T, } ``` Adding a `DynSized` trait would solve this as well, by requiring tail fields to be bound by it. - Despite being unsized, pointers to extern types are thin and can be casted from/to integers. However it is not possible to write a `null<T>() -> *const T` function which works with extern types, as I've explained here : #43467 (comment) - Trait objects cannot be built from extern types. I intend to support it eventually, although how this interacts with `DynSized`/`size_of_val` is still unclear. - The definition of `c_void` is unmodified
While it is possible for Sync, Send, UnwindSafe and RefUnwindSafe, doing Should Or is it possible to declare an extern type is safe-by-default, which opt-out instead of opt-in? extern {
#[unsafe_impl_all_auto_traits_by_default]
type Foo;
}
impl !Send for Foo {} |
|
@kennytm What's the usecase? The semantics of |
|
@eddyb Use case: Trying to see if it's possible to make CStr a thin DST. I don't see anything related to a cell in #44295? It is reported to LLVM as an |
|
@kennytm So with |
|
@Skepfyr That design is one of the common ways to implement external types in stable, and it's mentioned in the RFC. The main downside is that code with access to It's possible that the subset of |
Rust's requirements for extern types would be to not require alignment since we don't know it. Same for data validity. There's really no problem here. Undefined Behavior arises when the code violates the assumptions made by the compiler, but here the compiler cannot make any assumptions, and therefore the code cannot violate them. This is, in fact, already implemented correctly both in codegen and in Miri.
I tend to agree, what the compiler does here is a hack.
That's impossible in the design proposed by @Skepfyr as that struct is private to the module. |
I'm not trying to be snarky when I write this response, so please read literally: if It seems to me that the main blocker for
mod hide_the_internals {
struct ExternType;
pub struct OwnedExternType(*mut ExternType);
pub struct ExternTypeRefMut<'a>(*mut ExternType, PhantomData<&'a mut ExternType>);
pub struct ExternTypeRef<'a>(*const ExternType, PhantomData<&'a ExternType>);
fn do_something(x: &ExternType) { ... }
}I know it's not the most critical failure mode, but considering that solving that specific drawback is the motivation for the RFC that this tracking issue exists for, I think it's worth at least keeping in mind. |
Because that's the most correct least dangerous semantics. I was only talking about the runtime semantics. The type system clearly needs work. I never claimed the feature is ready. We were talking about the question of alignment requirements of references, why are you now deflecting by talking about size_of_val? Your claim, as I understood it, was specifically that there's a problem defining the validity invariant for references (which typically involves alignment) as we don't know the alignment of these types. Two people now told you that this is in fact not a problem as we can just make the validity invariant not require any particular alignment for extern types.
I don't know what you are referring to here. Stabilization is blocked on the issue that the Rust type system does not understand the concept of a type that does not have a dynamically computable size.
If you change other people's working code then it may not work any more. Not sure which point you are trying to make here. Usually when we evaluate the correctness of other people's code we only consider what can be done by calling that code as-if it was in another crate; we can't just access private fields or private types. It's trivial to break basically everything if you allow yourself to access private fields and private types. But if we have e.g. a macro that generates the |
My point is that A practical way to solve that problem is to prevent external types from being used as references, because if the only thing they're good for is their bit pattern then they're not actually references in the Rust sense.
I'm referring to #115709, in which a
I thought I was clear in my point, but in case not, the RFC for this tracking issue exactly describes the use of a ZST for extern types in the "motivation" section, and also describes why that is not the ideal solution. Scrapping the |
I have a use case that looks similar to this, that utilizes references to opaque types: #![feature(extern_types)]
#[repr(C)]
#[derive(Debug)]
struct TextureSize {
width: u32,
height: u32,
}
extern "C" {
type FfiSprite;
fn ffi_sprite_create() -> *mut FfiSprite;
fn ffi_sprite_get_texture(sprite: *const FfiSprite) -> *const Texture;
type Texture;
fn ffi_texture_get_size(texture: *const Texture) -> TextureSize;
}
struct Sprite<'texture> {
handle: *mut FfiSprite,
// The handle to the texture is on the FFI side
_texture: core::marker::PhantomData<&'texture Texture>,
}
impl<'texture> Sprite<'texture> {
fn new() -> Self { Self {
handle: unsafe { ffi_sprite_create() },
_texture: std::marker::PhantomData,
}}
fn texture(&self) -> &Texture {
unsafe { &*ffi_sprite_get_texture(self.handle) }
}
}
// Directly implement the opaque type
impl Texture {
// Method by reference
fn size(&self) -> TextureSize {
unsafe { ffi_texture_get_size(self) }
}
}
fn main() {
let sprite = Sprite::new();
let texture = sprite.texture();
dbg!(texture.size());
}Is this/should this be undefined behavior because you can't have a reference to an opaque type? Here is a real life example in rust-sfml (using pseudo-opaque types similar to what bindgen generates, but the principle is the same): https://github.com/jeremyletang/rust-sfml/blob/3e275e06f408343c63133108e65009d1f4e6295c/src/graphics/texture.rs Is this undefined behavior? In all my testing it worked fine. |
|
I think there are usability problems with what I've described, reborrowing the Ref(Mut)? versions is painful, and it means that no-one else can name a I agree we should consider getting the opaque tail stuff working a separate issue, it's incredibly painful due to alignment, and I haven't seen a working proposal yet. @jmillikin |
Yes. As I said, that's a type system issue. They crash in a safe way (a non-unwinding panic) so this is not unsound.
This does not follow, not at all. UB is a very big hammer we use to enable the compiler to optimize code better. "size_of_val would panic" is most definitely not an excuse for introducing UB. The consequences of accidental UB are way too severe to gratuitously use it here. Ideally we will use the type system to prevent size_of_val on references or pointers to extern types. But if that doesn't work we can use a runtime check like we do right now, with a panic. That's still miles better than Undefined Behavior. There's no need to risk arbitrary memory corruption just because someone created a reference to an extern type.
That issue isn't about |
|
IIUC #43467 (comment) the proposed One big issue of
But given that Rust can never safely create a
I don't see how banning
it can already prevent I'm not sure what "references in the Rust sense" encompasses. IMO at least calling methods given a Footnotes
|
|
@kennytm The problem with the proposed Given the following sets of goals for an
... the proposal to forbid their use as references solves all four, and allowing references but forbidding generics would leave two unsolved. |
the types |
|
@jmillikin Forbidding Meanwhile forbidding reference but not generic meant the following code is still possible: // crate A
pub struct Tailed<T: ?Sized> {
a: usize,
b: T,
}
// crate B
extern { type Texture; }
type TailedTexture = crate_a::Tailed<Texture>;The |
|
@kennytm I don't think I don't understand why the code you provided would be valid for |
|
You can't forbid references without forbidding extern types in generics. Otherwise it's always possible to use generics to bypass the restriction. |
I think it should be valid for extern {
#[repr(align(4))]
type ExternTypeWithKnownAlignment;
} |
What really matters is the intrinsic they are being forwarded to (min_align_of_val) are actually used by the language to compute alignment of a struct tail field.
The
While it sounds good in theory I doubt in practice if there is any actual scenario where the FFI type specified the exact alignment yet the size and all fields are implementation details. Usually you either have none or all of these 3 details. Like for the #[repr(C)]
struct FILE {
_flags: c_int,
_IO_read_ptr: *mut c_char,
...
}I see no case that you specifically only want to know The only exception might be dynamic-sized array, like an array tail like |
I agree they are not the same (and my blog post said that). However, my point was that I do not think that understanding this distinction is as important as you do, at least at first, and I think that by the time it is, people will be able to handle it. In contrast, the Let me explain this a different way. Early on in their Learning Rust journey, I think the main way that people will interact with Maybe they then go to author an impl that needs All that said, I think we would really benefit from doing some experimentation here. It's often hard to judge how something will feel after using it for a while. My hunch is that |
|
(Has anybody written lints to detect when a type parameter could have been |
|
I have a concern that drop glue for extern types should not be instantiated, meanwhile it currently resolves with empty P.S. It should probably become some sort of a post-monomorphization (or linker) error, since this feature usually relies on the linker finding symbols anyway. I could try implement that. Perhaps this could also be done to |
|
@zetanumbers Could you give a code example of what your talking about? I can't imagine a scenario where drop glue would be emitted for an extern types, because you can't ever have an owned extern type in Rust. Similarly, I'd expect |
As you have said, so that
You cannot implement I understand you don't see the difference for yourself, I am just looking which behavior would people prefer if they care. |
Extern types do not have any implicit drop glue, so there's no good reason for Maybe we want to allow |
|
I am worried if such behavior may affect support for unforgettable types. Extern types seem like very much raw feature for unsafe code only, which is not inherently bad, but I wonder if usage from safe code is planned. Also to note if unforgettable types are indeed only meaningful when they have a lifetime, as I believe it would be without accounting other hypothetical features, then extern type should probably not be allowed to have lifetime arguments. |
|
I don't see how extern types are particularly interesting here. In terms of |
|
Since no one seems to have asked this question yet: The current "idiom" for opaque types is having a #[repr(C)]
pub struct Shader<'texture> {
_opaque: [u8; 0],
// Keeps track of the lifetime of the texture the Shader borrows,
// but of which Rust has no idea otherwise, since
// the field that "borrows" the texture is on the C side
_texture: PhantomData<&'texture Texture>,
}How would I go about this using extern types? |
I would assume by doing something like this: extern {
type ShaderExtTy;
}
#[repr(transparent)]
pub struct Shader<'texture> {
inner: ShaderExtTy,
_phantom: PhantomData<&'texture Texture>,
} |
|
@programmerjake #[repr(transparent)]
pub struct Shader<'texture> {
_phantom: std::marker::PhantomData<&'texture Texture>,
inner: ShaderExtTy,
}Since only the last field can be unsized. |
and others
This is a tracking issue for RFC 1861 "Extern types".
Steps:
Unresolved questions:
Rust does not support types that don't have dynamically computed alignment -- we need the alignment to compute the field offset in structs.
extern typeviolates this basic assumption, causing pain, suffering, and ICEs all over the compiler. What is the principled fix for this?Should we allow generic lifetime and type parameters on extern types?
If so, how do they effect the type in terms of variance?
In std's source, it is mentioned that LLVM expects
i8*for C'svoid*.We'd need to continue to hack this for the two
c_voids in std and libc.But perhaps this should be done across-the-board for all extern types?
Somebody should check what Clang does. Also see "extern type" should use
opaquetype in LLVM #59095.How should this interact with unsized arguments? Currently it ICEs: unsized_fn_params should not accept types that don't have a dynamically fixed size (such as
externtypes) #115709The text was updated successfully, but these errors were encountered: