New custom type system

[bendk]

PR notes: I split this out out from #2087 so that we can discuss the custom type changes separate from the remote type changes. I think the new syntax is pretty uncontroversial, but this does require a breaking change so I wanted to make room to discuss that separate from the main conversation. On the topic of breaking changes: I tried but failed on avoiding them. Maybe it's possible with some extra work, but I don't think it's worth it at this point. How does that sound to others?

---

Implemented a new custom type system described in the new docs. This system is easier to use and also allows us to do some extra tricks behind the scene. For example, I hope to add a flag that will toggle between implementing FfiConverter for the local tag vs a blanket impl. It's not possible to do that with the current system and adding support would be awkward.

I wanted to keep backwards compatibility for a bit, but couldn't figure out a good way to do it. The main issue is that for the old system, if a custom type is declared in the UDL then we generate the FfiConverter implementation, while the new system expects the user to call custom_type! to create the impl. I tried to get things working by creating a blanket impl of FfiConverter for types that implemented UniffiCustomTypeConverter, but ran into issues -- the first blocker I found was that there's no way to generate TYPE_ID_META since we don't know the name of the custom type.

Removed the nested-module-import fixture. The custom type code will no longer test it once we remove the old code, since it's not using UniffiCustomTypeConverter. I couldn't think of a way to make it work again.

[bendk]

@mhammond I just added an upgrading guide for users. I'm not sure how to hook it in to the new doc system though. Does this mean updating the gh-pages branch after we merge this? Maybe you could walk me through it sometime.

[mhammond]

oops, I had these pending but I think they are still worthwhile

[mhammond]

This is the entry-point to custom type docs - so maybe add a para or 2 intro here - eg, describe our Guid/Handle types, how they wrap their types, what bindings see, before getting into the detail below. I'd even think about moving the custom_newtype example above this.

[bendk]

Great idea. I added an overview/motivation for custom types here and thought it looked good so I didn't move the custom_newtype example. WDYT?

[mhammond]

I'm not sure how to hook it in to the new doc system though. Does this mean updating the gh-pages branch after we merge this? Maybe you could walk me through it sometime.

I'm not quite sure what you are asking - you need to list it in https://github.com/mozilla/uniffi-rs/blob/main/mkdocs.yml, but what's less clear is how this new file (which I think could maybe be renamed to just "upgrading.md"?) will function. Off-hand, I'd say our release process should be tweaked to say (a) remove old entries from this file and (b) have a quick check over the changelog for new items that should be added.

Or did you have something else in mind?

[bendk]

I'm not sure how to hook it in to the new doc system though. Does this mean updating the gh-pages branch after we merge this? Maybe you could walk me through it sometime.

I'm not quite sure what you are asking - you need to list it in https://github.com/mozilla/uniffi-rs/blob/main/mkdocs.yml, ...

I was just looking for that YAML file, thanks!

[jeddai]

I really like this adjustment — I feel like it works better with the switch to macros for the bindings for other types

[mhammond]

I've spent more time trying to understand the bigger picture here than looking at the implementation details of this - which got me back to looking more at #2087 so see how all these pieces fit together. I think I'm getting my head around it and I like it!

Here's a possibly controversial take on the naming though:

I never really liked the old names very much (I think I might have introduced them),
and sadly I think I like the new ones even less - but I see an opportunity:

So here's the old world:

impl UniffiCustomTypeConverter for MyType {
    type Builtin = UniFFIBuiltinType;

    fn into_custom(val: Self::Builtin) -> uniffi::Result<Self> {
        ...
    }

    fn from_custom(obj: Self) -> Self::Builtin {
        ...
    }
}

What I like about this is the liberal use of the term "builtin" and the fact that the types are explicit.
It's clear what Self refers to and it's very clear what's the "builtin" vs the "custom".

Here's the new world:

// Expose `http::HeaderMap` to Uniffi.
uniffi::custom_type!(HeaderMap, Vec<HttpHeader>, {
    from_custom: |obj| { ... }
    try_into_custom: |val| { ... }
}

What I don't like about this is that the term "builtin" is no longer used and there are no types in the signatures.
Instead, just the term "custom" appears, and without much experience with UniFFI it might not be obvious what is meant by the term (eg, it might be easy to convince yourself that the term "custom" actually is referring to the builtin if that's the only term actually used)

When I read the other docs and the implementation though, I see:

// Defining `From<Handle> for i64` also gives us `Into<i64> for Handle`
impl From<Handle> for i64 { ... }
impl TryFrom<i64> for Handle { ... }

Which makes me think - why not change the new model to:

// Expose `http::HeaderMap` to Uniffi.
uniffi::custom_type!(HeaderMap, Vec<HttpHeader>, {
    into_builtin: |obj| { ... }
    try_from_builtin: |val| { ... }
}

The improvements here IMO are:

• The types become that little more obvious - what the "builtin" means seems more obvious.
• It better matches how Rust names the equivalent functionality, so is more likely to be familiar to the reader.
• It better matches the actual implementation and makes the docs that bit easier to read.

eg, the docs can say something like "into_builtin defaults to Into<Builtin> and "try_from_builtin" defaults to TryFrom<Builtin>.
Indeed, they can even encourage you to implement those builtin traits instead of manually specifying the "overrides" (which they kinda do in your patch, but having this consistency makes that story stronger)

It could be argued it makes the breakage that little bit more difficult to mechanically fix,
but it seems like it is actually a much better fit for how it actually works and seems far more intuitive for new UniFFI users.
It's an opportunity we will probably not get again.

(I'll also admit this came to me just last night, so I haven't thought it through in great detail, but though it worth throwing out there anyway!)

[bendk]

Instead, just the term "custom" appears, and without much experience with UniFFI it might not be obvious what is meant by the term (eg, it might be easy to convince yourself that the term "custom" actually is referring to the builtin if that's the only term actually used)

This is a very good point. I'd love to get a very specific and easy to understand name here.

The reason I didn't go with "builtin" is that type doesn't actually need to be a builtin type. You could convert the custom type into a user-defined record/enum/interface. Would this actually happen in practice? I wasn't really sure.

Maybe this would be a good opportunity to introduce a term here. There were a lot of times when I was writing docs that I wanted a word for a type that had the FFI traits implemented, but couldn't find a good one. Sometimes I wanted to call it a "UniFFI type", but it's not so clear what that means from the words alone. I usually ended up saying something like "a type that can be used in the exported UniFFI API", which was pretty awkward. I think it could be good to pick a term and add a section in the manual that defined it. "UniFFI type" or "UniFFI-enabled type" are the best names that I could come up with. WDYT?

..the docs can say something like "into_builtin defaults to Into and "try_from_builtin" defaults to TryFrom

True, but isn't this also true for the current code? You could say "from_custom defaults to From<Custom>". I think the main issue is that "custom" isn't a great name.

[mhammond]

The reason I didn't go with "builtin" is that type doesn't actually need to be a builtin type. You could convert the custom type into a user-defined record/enum/interface. Would this actually happen in practice? I wasn't really sure.

This is the http-header example right? The "custom type" is a remote type and the "builtin" type is a record?

I agree "builtin" isn't a great name, but I think it's important the concept have a name.

I think the main issue is that "custom" isn't a great name.

My take is that:

• It's ideal that the concept clearly includes 2 different terms - one for the new type being exposed to the object model (what we call the "custom type") and the other being the "uniffi compatible type used to represent that custom type" (ie, what we call the builtin).
• The old model wasn't great, but the fact it used 2 distinct terms meant less confusion was likely. Even if we came up with a better name for "custom", I think the fact only one name is used is a sticking point - users may be confused about which of the 2 types involved it refers to.

[mhammond]

or to put my last comment another way - I think the term "builtin" is more problematic than "custom", because it implies records etc can not be used - but having 2 distinct "not ideal" terms is somewhat better than just using a single "ideal" term.

[bendk]

Using two terms for the two different types makes a lot of sense. What if we leaned into that more and used a syntax like this?

uniffi::custom_type!(
   custom: HeaderMap,
   builtin: Vec<HttpHeader>,
   into_builtin: |obj| { ... },
   try_from_builtin: |val| { ... },
);

• Maybe replacing "builtin" with some other term that we come up with.
  • We could call it uniffi or uniffi_type, but those names don't feel right to me. "UniFFI" feels too overloaded to be meaningful to me.
  • What about converted? That would be a horrible name by itself, but if you think about "custom" and "converted" together it makes some sense. The docs could say explain it as you're converting your custom type into a UniFFI compatible type.
• Maybe brackets instead of parens? I think Rust lets you use either interchangeably so it's just a question of which one we use in our docs/examples.

[mhammond]

uniffi::custom_type!(
custom: HeaderMap,
builtin: Vec,
into_builtin: |obj| { ... },
try_from_builtin: |val| { ... },
);

What I do like about this is what I liked about my idea above - ie, I think this is basically as good as simply

uniffi::custom_type!(HeaderMap, Vec<HttpHeader>
    into_builtin: |obj| { ... },
    try_from_builtin: |val| { ... },
);

because it does use the 2 terms. In both cases though, the sticking point remains more "builtin" than "custom". Another bad option might be "existing"?

[bendk]

I've been thinking about this one off and on for a bit and "existing" seems like the best option to me, should we just go with that one?

[mhammond]

should we just go with that one?

That sgtm, at least as the next straw-man!

[mhammond]

This implementation and motivation is great, thanks for your patience while we catch up with your thinking.

I'm now a bit worried that my naming suggestions of before have actually made things worse. Maybe some small tweaks to the docs are all that's necessary, but I do think it's still worth thinking about before landing - that naming is almost the only part of this which will be visible by users so it's worth getting right, and I'm really sorry if I'm just making it worse or more confusing. Once I get over that I think this is ready to go.

[mhammond]

gah - a bit of a side issue, but these links will be 404s until the next release, which doesn't seem ideal. 2 options we have seem to be:

• link to /next instead. This works in the short term, but is likely to go stale over the longer term. I'm not sure going stale matters in practice though in terms of old changelog entries.
• Our docs process is mildly tweaked so the presumed next version (0.29 in this case) is always created as soon as the last version is released.

The problem with that second option is that it assumes 0.29 will be the next version - while I see no reason to believe that's not true, you never know. Eg, if we manage to squeeze a number of other breaking improvments in, we might decide to skip 0.29 and move directly to 0.30

Regardless, this is less about this PR and more about our docs in general, so this doesn't need to block anything.

[bendk]

Let's use next. It works for now and I agree that going stale over the long term is not the biggest deal.

[mhammond]

Reading the docs below, it looks like we might also use Into and TryFrom depending on the direction/context. If so, could you please tweak this? In particular, this sentence almost seems like the reverse of what's sensible (ie, that we have a try_from method, which uses From if omitted, and we have an into method which, if omitted, uses TryInto appears wrong and inconsistent).

(If I'm not wrong about that, the changes make here could make the text quite vague, which I think is OK if the alternative makes things appear more restrictive than it is)

[bendk]

Thinking about this some more, I think we should spell out the exact trait and always use the into version. So this should say "For example, the try_from_existing and into_existing can be omitted in many cases, and will use the TryInto<Custom> and Into<Existing> traits by default." The reason for this is that Into is auto-derived from From, but not the other way around.

Maybe we should rename try_from_existing to try_into_custom to match this?

[mhammond]

really nit-picking here, but I wonder if this line should say, eg, try_from_existing: |val| { Ok(...) }, to highlight the fact this is a Result<>?

[mhammond]

tuple structs are supported by procmacros.

[mhammond]

My brain hurts - following on from the last comment - have we got these names correct? We have into_existing which doesn't return a result, but here talking about "TryInto" (which does) seems confusing and backwards.

I'm worried my last suggestion which gave us these new names actually made things more complicated?

[bendk]

I think I just got this backwards and I'll reverse them.

[bendk]

Please nit-pick, I agree that the names are very important here. I just pushed some fixes/clarifications to the docs.

I'm currently 50/50 on renaming try_from_existing to try_into_custom. What's your feeling on this?

[mhammond]

Think of these comments more as brainstorming than a concrete proposal :)

[gruberb]

Just a few thoughts. I think it's a bit confusing reading existing, custom_newtype and custom_type.

I would like to define overall naming patterns somewhere.

We have:

• Built-in types from the Rust world
• User created types via struct and enum
• UniFFI types
• Types from external crates

How about:

• primitive_type
• custom_type
• bridge_type
• external_type

There is a lot of context that I am properly missing. But might be worth defining the terms we use first, and then adjust the conventions in the codebase.

[bendk]

It's great to hear a take coming from someone who hasn't been so deep in the weeds on this one. I'm going to try to rephrase it slightly, please tell me if I'm missing anything:

1. There should be an overview of the UniFFI type system that describes all of the kinds of types in broad strokes.

2. One obvious split is types that are defined by UniFFI (u32, String, HashMap, etc.) vs ones defined outside of UniFFI via UDL/proc-macros. This split should inform the terms we use. My preferred terms are:

• builtin types for the first category. I like this better than primitive, since HashMap doesn't feel like a primitive type to me.
• user-defined types for the second category.
• Stop using the term custom types since custom type could easily be interpreted as user-defined type.

3. Use a term like bridge type for the feature we're describing here to more clearly distinguish it from other user-defined types. bridge type would work for me, but I like adapter type or converter type a bit better. Are there other terms like this that we can consider?

4. custom_newtype is confusing. You've convinced me on this and made we want to step back and reconsider this feature altogether. It seems to me that what would be really nice is first-class support for newtypes. You could have Rust code like this:

#[derive(uniffi::newtype)]
pub struct Guid(String);

On the Rust side, this would work exactly like a custom_newtype today. On the foreign side, we could define a Guid newtype rather than passing a raw string around. One potential drawback is that users can currently define their own custom type converters for custom newtypes on the foreign side, for example maybe they could convert them into some standard Kotlin Guid type. I think this is tractable though.

At this point, I think we're talking about more than one PR worth of work. Maybe we could split this up into multiple ones:

• This PR is where we pick the names/syntax for bridge types/adapter types/whatever we end up calling them. It will implement the custom_newtype macro, but we'll plan to rework this before the next release.
• A PR to add the type overview manual page
• A PR that reworks newtypes
• A PR that reworks converting types in the foreign bindings.

[jplatte]

Use a term like bridge type for the feature we're describing here to more clearly distinguish it from other user-defined types. bridge type would work for me, but I like adapter type or converter type a bit better. Are there other terms like this that we can consider?

Idk whether this question was directed at @gruberb or everybody, but personally I really like bridge(d) type! :)

[bendk]

If others like bridge type, that works for me.

[bendk]

We discussed this today, got some more clarity and came up with some decisions. I misunderstood part of Bastian's point. He's fine with "custom type" as a name, the issue was with the other type involved, which he wants to name "bridge type".

The new plan is to:

• Keep "custom types" as the name for this feature
• Call the types involved "custom type" and "bridge type"
• Refer to "bridge type" in the macro. Use into_bridge_type and try_from_bridge_type as the attribute names.

[bendk]

Updated the language as describe above. I think this is maybe finally ready to land, tell me what you think.

[mhammond]

This is great, thanks so much for your patience shepherding this through! I'd still love to get more eyes on this, so maybe give it a week or so for them to magically materialize :)

[jhugman]

I'm really happy to see this work happen. Thank you so much Ben!

Having read the discussion about names, I totally agree with you about custom and builtin, and see the difficulties in picking a new color of bikeshed more descriptive name.

My suggestion would be to keep the custom type naming, but lean into the lifting and lowering metaphors, specifically use a lowered type instead of builtin:

uniffi::custom_type!(HeaderMap, Vec<HttpHeader>
    into_lowered: |obj| { ... },
    try_from_lowered: |val| { ... },
);

alternates, using verbs:

uniffi::custom_type!(HeaderMap, Vec<HttpHeader>
    lower: |obj| { ... },
    try_lift: |val| { ... },
);

My hunch would be that if you're at the stage of adding new types to cross the FFI, then you've already found the lifting and lowering concepts already prominent in the docs.

[bendk]

My suggestion would be to keep the custom type naming, but lean into the lifting and lowering metaphors, specifically use a lowered type instead of builtin:

This is a pretty interesting idea. Conceptually, it means that values would get lowered/lifted several times (Guid is lowered to String is lowered to RustBuffer). That adds some complexity to the concept of lifting/lowering, but then you don't need to introduce a new concept of "converting into/from the existing type". I remember you suggesting something like this way at the beginning.

My initial feeling is that I like it, what do others think?

[bendk]

I like the suggestion of going with lower and lift because IMO it lets is paint a better picture in the description. I updated the documentation to lean into this, tell me what you think.

[bendk]

We discussed this yesterday and there's general agreement to lean into the lower/lift terminology to describe this. I updated the PR to use lower/try_lift instead of into_bridge_type/try_from_bridge_type.

I'd love to get this merged. I think @mhammond has some suggestions for the new doc page, does anyone else have suggestions/comments?