Separate functional from ergonomic API (add cbindgen)

[sffc]

No description provided.

[sffc]

I think the best first step here is to try to get an ICU4X component like PluralRules to compile with cbindgen.

It's not hard to get started. I annotated the try_new function with #[no_mangle] and extern as follows:

    #[no_mangle]
    pub extern fn try_new<'d, D: DataProvider<'d>>(
        langid: LanguageIdentifier,
        data_provider: &D,
        type_: PluralRuleType,
    ) -> Result<Self, PluralRulesError>

And the PluralRules struct with #[repr(C)]:

#[repr(C)]
pub struct PluralRules

Then I can run cbindgen --crate icu_plurals. I get:

$ cbindgen --crate icu_plurals
WARN: Can't find LanguageIdentifier. This usually means that this type was incompatible or not found.
WARN: Can't find LanguageIdentifier. This usually means that this type was incompatible or not found.
WARN: Can't find D. This usually means that this type was incompatible or not found.
#include <cstdarg>
#include <cstdint>
#include <cstdlib>
#include <ostream>
#include <new>

/// A type of a plural rule which can be associated with the [`PluralRules`] struct.
///
struct PluralRuleType;

/// A list of possible error outcomes for the [`PluralRules`](crate::PluralRules) struct.
///
struct PluralRulesError;

template<typename T = void, typename E = void>
struct Result;

/// An enum storing models of
/// handling plural rules selection.
struct RulesSelector;

/// `PluralRules` is a struct which provides an ability to retrieve an appropriate
/// [`Plural Category`] for a given number.
///
/// # Examples
///
/// ```
/// use icu_locid_macros::langid;
/// use icu_plurals::{PluralRules, PluralRuleType, PluralCategory};
/// use icu_provider::InvariantDataProvider;
///
/// let lid = langid!("en");
///
/// let dp = InvariantDataProvider;
///
/// let pr = PluralRules::try_new(lid, &dp, PluralRuleType::Cardinal)
///     .expect("Failed to construct a PluralRules struct.");
///
/// assert_eq!(pr.select(5_usize), PluralCategory::Other);
/// ```
///
/// [`ICU4X`]: ../icu/index.html
/// [`Plural Type`]: PluralRuleType
/// [`Plural Category`]: PluralCategory
struct PluralRules {
  LanguageIdentifier _langid;
  RulesSelector selector;
};

extern "C" {

/// Constructs a new `PluralRules` for a given locale, [`type`] and [`data provider`].
///
/// This constructor will fail if the [`Data Provider`] does not have the data.
///
/// # Examples
///
/// ```
/// use icu_locid_macros::langid;
/// use icu_plurals::{PluralRules, PluralRuleType};
/// use icu_provider::InvariantDataProvider;
///
/// let lid = langid!("en");
///
/// let dp = InvariantDataProvider;
///
/// let _ = PluralRules::try_new(lid, &dp, PluralRuleType::Cardinal);
/// ```
///
/// [`type`]: PluralRuleType
/// [`data provider`]: icu_provider::DataProvider
Result<PluralRules, PluralRulesError> try_new(LanguageIdentifier langid,
                                              const D *data_provider,
                                              PluralRuleType type_);

} // extern "C"

I guess that's a start! I'm not really sure where to go next. In particular, I don't know how to fix the errors on top:

WARN: Can't find LanguageIdentifier. This usually means that this type was incompatible or not found.
WARN: Can't find LanguageIdentifier. This usually means that this type was incompatible or not found.
WARN: Can't find D. This usually means that this type was incompatible or not found.

I annotated LanguageIdentifier with #[repr(C)], but that didn't seem to do anything.

Once we can get some sort of PluralRules compiling to FFI in ICU4X, the next step will be to work on how the FFI relates to the data provider. I think, at least, we should add a new API that builds a PluralRules directly from data structs and other elementary components, rather than abstract traits/interfaces.

CC @Manishearth @nciric

[zbraniecki]

Do we want cbindgen or cxx?

[Manishearth]

I annotated LanguageIdentifier with #[repr(C)], but that didn't seem to do anything.

I think it needs to be all the way down. What we need to do is one of:

• Declare this function as taking an &LanguageIdentifier so it can operate on opaque objects (we need a way to construct these that has the right ownership model
• Declare a separate FFI-compat LanguageIdentifier type that can be appropriately converted in Rust
• Make everything repr(C), including TinyStr. We could potentially have a cfg that does this so that normal rust consumers aren't affected by the loss of layout optimizations.

Do we want cbindgen or cxx?

cxx could work, but it has a different model. I figured cbindgen would be useful since you need to do a lot of that work for wasm anyway. Worth trying cxx!

[Manishearth]

WARN: Can't find D. This usually means that this type was incompatible or not found.

For generics we'd have to do some kind of dynamic dispatch instead, sadly.

[filmil]

FWIW.

We can not expect that we can just run either cbindgen or cxx on arbitrary Rust code, and that we will get sensible output. #[repr(C)] is necessary, but not sufficient. And frankly I'm not sure that "repr(C) all the way" is what we want anyways.

The API for the code that you expect to be usable over FFI needs to be designed with FFI in mind. IIUC, cbindgen doesn't help us at all with that, while cxx might -- though it's not there yet.

Some types such as Result<_,_> don't really have a repr(C), so that needs to be taken into account too. It would be cool to have a library that handles the representation differences, and it looks like this is something that could be generated automatically. cxx goes some of the way.

[sffc]

Clearly the FFI layer needs to be written in a way to make it C- and WASM-compatible, with simpler types. In wrapper_layer.md, I proposed making the Rust API wrap over the FFI API. I'm wondering whether this is the right approach, or whether the FFI API should be a "standalone" wrapper over the Rust API, more like in ICU4C. I would prefer making the Rust API wrap FFI, but I don't want to do that if it incurs a significant cost (in safety, performance, maintenance, etc).

[Manishearth]

So the typical way to do this is to have intermediate FFI structs that contain the same data and are cheaply convertible to the Rust format.

"Rust API wrap FFI" is rarely a good idea since you lose out on all of the benefits of Rust enums/etc.

[sffc]

We should schedule a deep-dive session where we figure this out, similar to how we did with WASM.

[Manishearth]

I might also do some experiments with pluralrules

[Manishearth]

FWIW, i've filed mozilla/cbindgen#646, which will eventually let us directly pass TinyStr over FFI, until then we'll have to make a shim type.

[Manishearth]

cbindgen (master) is now compatible, opened zbraniecki/tinystr#30 for making tinystr work

[Manishearth]

Got some experiments up at #453

[eerhardt]

Any idea when this will be started/implemented?
As I stated in the discussion I opened yesterday, we are investigating using ICU4X in the .NET runtime. However, in order to invoke it, we will need to call it through a "C" API.

[Manishearth]

We have a rough plan for this and I hope to start implementing it soonish. It should still be a few months out though.

[Manishearth]

Currently blocked on mozilla/cbindgen#665

[Manishearth]

I think the design work here was mostly resolved in #552 (and this issue can be closed, or kept around as an FFI tracking issue)