extern types #1861

canndrew · 2017-01-21T11:25:01Z

Add extern type declarations for declaring types from external libraries which have an unknown size/layout.

glaebhoerl · 2017-01-21T12:12:25Z

Given that this is declaring a new type and not an alias to an existing type (presumably you should be allowed to write trait impls for it, and so on), I think it should be extern struct rather than extern type.

nagisa · 2017-01-21T13:34:46Z

text/0000-extern-types.md

+
+Add a new kind of type declaration, an `extern type`:
+
+    extern type Foo;


This diverges somewhat from extern fn where its a definition and not declaration. For consistency IMHO declarations should stay constrained to extern blocks, like below.

Right, of course. They're two different things.

- niether -> neither - Use actual em dash

petrochenkov · 2017-01-21T13:55:19Z

What do size_of/align_of return for extern types?

Ericson2314 · 2017-01-21T14:01:40Z

@petrochenkov it's a DST, you mean {size,align}_of_val? Those shouldn't work either, see #1524 (comment).

petrochenkov · 2017-01-21T14:13:35Z

Oh, right, size_of_val.
What do you mean "shouldn't work", report an error during monomorphization?
size_of_val is usable unconditionally, so it can be used with type parameters that can become extern types later.

Ericson2314 · 2017-01-21T14:17:00Z

@petrochenkov I think will need to change #1524 (comment). I had a bunch of things to add to this RFC so I'm making a PR-on-PR; I propose making it panic for stop-gap.

arielb1 · 2017-01-21T14:28:19Z

@glaebhoerl

It's sort of like a "truely" abstract type, so I think extern type is OK.

glaebhoerl · 2017-01-21T14:30:12Z

But it's a nominal type that's knowably disjoint from all other types, isn't it?

Ericson2314 · 2017-01-21T14:37:44Z

I think the fact that it is, after all, called a "struct" in C tips me over the fence into @glaebhoerl's camp. Yes, this is used as a hack around not having real abstract types in C [and there should be a way to make the size of a concrete struct with private fields abstract in Rust!], but I'm OK with skeuomorphisms for FFI.

petrochenkov · 2017-01-21T14:46:42Z

I may be a struct in C, or it may be a union, or it may be just a malloc-ated storage.

presumably you should be allowed to write trait impls for it, and so on

I wonder why would you need to do such a thing.
Extern type refers to some type on C side, you don't know anything about it, what traits can you possibly implement?

A lot of misc changes to extern types

canndrew · 2017-01-21T17:53:45Z

Thanks @Ericson2314 for some much-needed extra material.

canndrew · 2017-01-21T18:09:47Z

Given that this is declaring a new type and not an alias to an existing type (presumably you should be allowed to write trait impls for it, and so on), I think it should be extern struct rather than extern type.

The fact they can only appear in extern blocks (where things already look wierd), and that they don't have = Something after them should help prevent them from being confused with type aliases. Besides, in some sense they are a type alias - they're an alias to some type in an external library.

I'd feel a little weird calling them structs when the type might not actually be a struct. That feels like a bit of a C-anachronism.

canndrew · 2017-01-21T18:14:08Z

Extern type refers to some type on C side, you don't know anything about it, what traits can you possibly implement?

There's a lot of object-oriented C APIs that work with pointers to some opaque type. You could wrap these APIs and expose safe versions of the C functions as methods on the extern type.

glaebhoerl · 2017-01-21T19:41:17Z

I'd feel a little weird calling them structs when the type might not actually be a struct. That feels like a bit of a C-anachronism.

I think it'd be fine to just allow struct, enum, and union interchangeably fwiw.

I feel that the difference between whether it's an alias or a fresh type is much more significant than what variety of fresh type it is (especially given that the representation is hidden so that the difference is academic). type is pretty well entrenched as denoting an alias in Rust. Yes, this is an "alias to an external type" in some sense, but it's not [presumably going to be] an alias from the perspective of Rust's actual in-language semantics which is what really matters, given that it's what the Rust programmer will be dealing with.

Stebalien · 2017-01-21T20:58:41Z

I really want a feature like this but have a couple of concerns about the semantics of this proposal:

Personally, I wouldn't call these DSTs (dynamically sized). I'd call them unsized types. That's why we have to be careful about size_of_val and friends.
They're not really extern. Unlike every other extern item, the name doesn't matter: they don't necessarily refer to some existing external item.
They're useful beyond externally defined structures. For example, CStr should be defined as an unsized type (&CStr should be a thin pointer) but is currently a DST.

Stebalien · 2017-01-21T21:00:43Z

An alternative is to drop the extern part and allow type MyType;:

type A = B; means there exists a type A that is equivalent to type B.
type A; means there is a type A that isn't equivalent to anything. It's not a struct, enum, union, etc. Just a type, no defined structure.

Posted separately to allow separate reactions (I like this alternative more then the proposal but I'm not entirely sure about it).

briansmith · 2017-01-22T08:54:07Z

The main issue I see with type A; is that it doesn't indicate explicitly that it is a FFI-compatible type or not. #[repr(C)] type A; or extern { type A; } makes it clear that it is FFI-compatible. I guess #[repr(C)] type A; is more consistent with #[repr(C)] struct A { ... } so I suggest going with that.

type A; makes the most sense of all alternatives proposed above. Using struct A; to refer to a type defined as a non-struct type in C would be confusing to newcomers, and distracting to the pedantic (e.g. me). Allowing all of enum, union, and struct means that we'd potentially need to add more such options for non-C languages (class for C++, for example, or a Haskell tuple type) or be at risk of the same kind of confusion when interfacing with non-C languages.

The RFC should be updated to specifically mention that references to such types are allowed. It should also mention that moves from these types are not allowed.

briansmith · 2017-01-22T08:56:20Z

text/0000-extern-types.md

+struct FfiStruct {
+    data: u8,
+    more_data: u32,
+    tail: OpaqueTail,


It's not clear whether you are proposing that this should be accepted by the compiler or not. I think that this is not as essential as the rest of the proposal and I'd suggest that you remove this example and specify that such types can only be used as pointer and reference types.

I'm saying it should be accepted. This isn't really hard to implement - just make extern types effect the struct pointer the same way that slices and trait objects do. It's also consistent with allowing other DSTs at the end of structs. I think it should be accepted because it's a pattern I've seen a lot in C code.

This structure seems useless to me. Consider 3 cases where FfiStruct could be used:

fn by_return() -> FfiStruct; fn by_sret(retptr: *mut FfiStruct); fn by_argument(arg: FfiStruct);

Now, all of these are invalid or can’t be made work:

For by_return it may under covers be either a return by value or by sret pointer (analysed in next point); If its returned by value its all right. However you cannot really know if its by value or by sret without knowing the full defn' of structure;

For by_sret compiler simply cannot how much of stack space to allocate for the retptr slot;

For by_argument compiler cannot properly how to correctly pass such a structure to the C side (i.e. if the FfiStruct was supposed to be passed to C via registers, how many registers does the C side expect to be used?).

EDIT: only case where this could work is

fn double_indirection(retptr: *mut *mut FfiStruct)

What about:

fn return_reference(&T) -> &FfiStruct {} fn consume_reference(&FfiStruct) {}

One of the main uses of this type would be to define opaque types with headers of known layouts such that they can be handled and have lifetimes managed as though they are Rust types, despite being defined in C++ land.

As an example, in the gecko codebase there is a type nsAString, which represents an abstract string. This type is an abstract base class which has (approximately) the following layout:

struct nsAString { const uint16_t* data; uint32_t length; uint32_t flags; };

And has multiple subclasses, which may or may not add extra data after the above data, such as nsFixedString which has a layout like:

struct nsFixedString : public nsAString { uint32_t capacity; uint16_t* buffer; };

In rust we can then define nsAString as:

#[repr(C)] struct nsAString { data: *const u16, length: u32, flags: u32, _rest: OpaqueTail }

And then we could take *mut nsAString, *const nsAString, *const nsFixedString etc. and cast them (through unsafe code) into &'a nsAString, working with them as though they were a rust object, able to directly access members from rust like length and flags, without having to worry about accidentally moving the data inside and breaking C++-defined invariants.

Currently in our rust bindings we're working around this limitation by defining #[repr(C)] struct nsAString([u8;0]); and doing casts to extract the fields from the header. You can see this here: http://searchfox.org/mozilla-central/rev/d3307f19d5dac31d7d36fc206b00b686de82eee4/xpcom/rust/nsstring/src/lib.rs#160-163

fn return_reference(&T) -> &FfiStruct {}

This can work, but you’ll need to either know the full type of T or get it on Rust side with *mut *mut T in the first place, basically moving the responsibility from FfiStruct directly to T.

Where’s the point of having a reference to

#[repr(C)] struct nsAString { data: *const u16, length: u32, flags: u32, _rest: OpaqueTail }

over a reference to

#[repr(C)] struct nsAString { data: *const u16, length: u32, flags: u32, }

So still, I’m not seeing the point of allowing such a thing, given in how many cases this cannot reasonably work in a FFI context.

Yeah the current hacks look weird, and poorly approximate the true semantics to boot as @mystor shows.

@nagisa have you looked at the custom DST RFC? IMO thinking about then together helps if this seems too narrow on its own.

@nagisa
by_sret is intended to work. extern type pointers are "fat" pointers in the sense that they point to a DST but they're still pointer-sized (the extra metadata on the pointer is just a ()). So *mut FfiStruct is pointer-sized and ffi-safe.

@canndrew How? Namely, please describe how much memory would the caller need to allocate on the stack so it could produce a valid pointer to pass to the function?

@Ericson2314 yes, I’ve seen it.

@nagisa: The entire point of this RFC is that no Rust code can possibly create such a value on the stack - it can only obtain pointers to such values, and only from foreign (or unsafe pointer-casting) code.

It is meant almost exactly to mimic the C/C++ notion of an "incomplete type" - which cannot be allocated on the stack in C/C++ either, but can be pointed/referred to.

@nagisa 32 bits on a 32 bit system, 64 bits on a 64 bit system? I'm not sure I understand the question.

Edit: Oh I get it. Yes, what @eternaleye said.

rfcbot · 2017-07-22T05:18:05Z

The final comment period is now complete.

liigo · 2017-07-22T09:58:42Z

In Rust, pointers to DSTs carry metadata about the object being pointed to. For strings and slices this is the length of the buffer, for trait objects this is the object's vtable. For extern types the metadata is simply (). This means that a pointer to an extern type is identical to a raw pointer.

Saying that it is not 'fat pointer' is more clear for me.

canndrew · 2017-07-24T10:43:49Z

@aturon So we're sweet to merge this baby yeah?

retep998 · 2017-07-24T14:55:46Z

text/0000-extern-types.md

@@ -64,7 +64,7 @@ These types are FFI-safe. They are also DSTs, meaning that they do not implement
 In Rust, pointers to DSTs carry metadata about the object being pointed to.
 For strings and slices this is the length of the buffer, for trait objects this is the object's vtable.
 For extern types the metadata is simply `()`.
-This means that a pointer to an extern type is identical to a raw pointer.
+This means that a pointer to an extern type is identical to a raw pointer (ie. it is not a "fat pointer").


Probably better to just say it is not a fat pointer and not even bother bringing up raw pointers because raw pointers can be fat too! *const [T] is fat for example despite being a raw pointer.

aturon · 2017-07-25T04:13:38Z

Huzzah! This RFC has been merged! Tracking issue.

sdroege · 2018-12-07T14:54:30Z

text/1861-extern-types.md

+One is, we define the type as an uninhabited type. eg.
+
+```rust
+enum MyFfiType {}


This is still in the RFC but wrong as it causes UB. See https://doc.rust-lang.org/nomicon/ffi.html#representing-opaque-structs and https://www.reddit.com/r/rust/comments/a3bf2p/patterns_of_refactoring_c_to_rust_the_case_of/eb5wxtw

This should probably also be fixed in the RFC text for people who look here because they want extern type but can't use it yet because it's not stable and need a workaround.

Where is the UB in *mut MyFfiType?

Ever doing *ptr is UB. This includes &*ptr as *const MyFfiType.

See https://doc.rust-lang.org/nightly/nomicon/ffi.html#representing-opaque-structs for what is currently recommended.

Those two new language traits are at the intersection of rust-lang#1861, rust-lang#2580, rust-lang#2594, and this RFC's raison d'être is solely to try to get some progress for those 3 RFCs all at once by specifying their common core part.

Add extern types RFC

6085dc4

nagisa reviewed Jan 21, 2017

View reviewed changes

One sentance per line so future diffs are readable. 2 and typo fixes too

b2d8042

- niether -> neither - Use actual em dash

Expand teaching section

e6f7f93

Ericson2314 added 3 commits January 21, 2017 09:02

Shout-out to roadmap with it's focus on FFI in the motivation section

083adad

Add ```rust in a few more places to get syntax highlighting

2f77316

Mention size_of_val/align_of_val in detailed design

949c292

Ericson2314 added 2 commits January 21, 2017 09:20

OOps

9f6d098

"Sized" is the trait, not "?Sized"

cf183e9

Talk about c_void, and associated representation concerns

434b472

canndrew added 2 commits January 22, 2017 01:47

Merge pull request #1 from Ericson2314/extern-types

b3d5ba3

A lot of misc changes to extern types

Grammatic changes

bce8a7b

Remove indentation and invalid syntax

156ed4b

briansmith reviewed Jan 22, 2017

View reviewed changes

rfcbot removed the proposed-final-comment-period Currently awaiting signoff of all team members in order to enter the final comment period. label Jul 12, 2017

Clarify that extern type pointers are thin pointers

263b7c9

retep998 reviewed Jul 24, 2017

View reviewed changes

re-word last commit

e884a0b

aturon mentioned this pull request Jul 25, 2017

Tracking issue for RFC 1861: Extern types rust-lang/rust#43467

Open

3 tasks

RFC 1861 is Extern types

4b971bd

aturon merged commit 4a27a85 into rust-lang:master Jul 25, 2017

jimblandy mentioned this pull request Jul 26, 2017

Poor error message for type definitions in extern blocks rust-lang/rust#43495

Closed

sdroege mentioned this pull request Jul 29, 2017

Unions: Remove feature-gate for stable rustc gtk-rs/gir#415

Merged

jplatte mentioned this pull request Aug 31, 2017

Named existentials and impl Trait variable declarations #2071

Merged

joshtriplett mentioned this pull request Sep 7, 2017

Support defining C-compatible variadic functions in Rust #2137

Merged

scottmcm mentioned this pull request Jan 19, 2018

Empty structs should not need #[repr(C)] #750

Closed

petrochenkov mentioned this pull request Jan 28, 2018

RFC: truly unsized types #813

Closed

Centril added A-typesystem Type system related proposals & ideas A-repr #[repr(...)] related proposals & ideas A-dst Proposals re. DSTs labels Nov 23, 2018

sdroege reviewed Dec 7, 2018

View reviewed changes

kennytm mentioned this pull request May 5, 2020

RFC: Generic member access for dyn Error trait objects #2895

Open

nox mentioned this pull request Sep 10, 2020

Introduce Pointee and DynSized #2984

Open

burdges mentioned this pull request Nov 11, 2020

Storing an object as &Header, but reading the data past the end of the header rust-lang/unsafe-code-guidelines#256

Open

eternaleye mentioned this pull request Nov 29, 2021

Should Rust permit conversions between pointers to data and pointers to functions? rust-lang/unsafe-code-guidelines#309

Closed

cuviper mentioned this pull request Nov 11, 2022

RFC: core::mem::replace_with for temporarily moving out of ownership. #1736

Closed

Skepfyr mentioned this pull request Feb 26, 2023

Extern types v2 #3396

Open

Amanieu mentioned this pull request May 23, 2023

RFC: #[export] (dynamically linked crates) #3435

Open

JarvisCraft mentioned this pull request Aug 27, 2023

Allow running MIRI flipperzero-rs/flipperzero#100

Open


		Add a new kind of type declaration, an `extern type`:

		extern type Foo;

extern types #1861

extern types #1861

Conversation

canndrew commented Jan 21, 2017 • edited by traviscross Loading

glaebhoerl commented Jan 21, 2017

nagisa Jan 21, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

petrochenkov commented Jan 21, 2017 • edited Loading

Ericson2314 commented Jan 21, 2017

petrochenkov commented Jan 21, 2017

Ericson2314 commented Jan 21, 2017 • edited Loading

arielb1 commented Jan 21, 2017

glaebhoerl commented Jan 21, 2017

Ericson2314 commented Jan 21, 2017 • edited Loading

petrochenkov commented Jan 21, 2017

canndrew commented Jan 21, 2017

canndrew commented Jan 21, 2017

canndrew commented Jan 21, 2017

glaebhoerl commented Jan 21, 2017 • edited Loading

Stebalien commented Jan 21, 2017

Stebalien commented Jan 21, 2017

briansmith commented Jan 22, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nagisa Feb 14, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eternaleye Feb 16, 2017 • edited Loading

Choose a reason for hiding this comment

canndrew Feb 16, 2017 • edited Loading

Choose a reason for hiding this comment

rfcbot commented Jul 22, 2017

liigo commented Jul 22, 2017

canndrew commented Jul 24, 2017

Choose a reason for hiding this comment

aturon commented Jul 25, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RalfJung Dec 7, 2018 • edited Loading

Choose a reason for hiding this comment

canndrew commented Jan 21, 2017 •

edited by traviscross

Loading

nagisa Jan 21, 2017 •

edited

Loading

petrochenkov commented Jan 21, 2017 •

edited

Loading

Ericson2314 commented Jan 21, 2017 •

edited

Loading

Ericson2314 commented Jan 21, 2017 •

edited

Loading

glaebhoerl commented Jan 21, 2017 •

edited

Loading

nagisa Feb 14, 2017 •

edited

Loading

eternaleye Feb 16, 2017 •

edited

Loading

canndrew Feb 16, 2017 •

edited

Loading

RalfJung Dec 7, 2018 •

edited

Loading