[vm/ffi] int, long, size_t, uintptr_t, wchar_t, etc #36140

dcharkes · 2019-03-07T13:58:42Z

Update 2022-01-20: This will be available in dart:ffi from Dart 2.17 (and from now on tip of tree). In the mean time this is also available in package:ffi for Dart 2.16 (dev release).

Update 2021-01-07: We will implement this by implementing #42563 and probably adding the common types to package:ffi.

Currently dart:ffi only supports:

int8_t
int16_t
int32_t
int64_t
uint8_t
uint16_t
uint32_t
uint64_t
intptr_t
float
double

However, many C APIs use C types such as int, long, size_t, uintptr_t, and wchar_t.

We could support these types. The question is if we support these types, then what other types should we support.

An alternative could be to not support these types but provide a way for users to specify types which differ in size per platform.

@jonasfj @sjindel-google

The text was updated successfully, but these errors were encountered:

dcharkes · 2020-01-21T09:59:30Z

sizeof(int) on iOS arm64 returns 4. Thus, we cannot safely substitute int with intptr_t.

sizeof(int) on iOS arm64 in Flutter test projects returns 4. Issue: #36140 Issue: #39637 Change-Id: I7b471fa1da653e9bee169c34d1bd36a96fa6a704 Cq-Include-Trybots: luci.dart.try:vm-ffi-android-debug-arm-try,vm-ffi-android-debug-arm64-try,app-kernel-linux-debug-x64-try,vm-kernel-linux-debug-ia32-try,vm-kernel-win-debug-x64-try,vm-kernel-win-debug-ia32-try,vm-kernel-precomp-linux-debug-x64-try,vm-dartkb-linux-release-x64-abi-try,vm-kernel-precomp-android-release-arm64-try,vm-kernel-asan-linux-release-x64-try,vm-kernel-linux-release-simarm-try,vm-kernel-linux-release-simarm64-try,vm-kernel-precomp-android-release-arm_x64-try,vm-kernel-precomp-obfuscate-linux-release-x64-try,dart-sdk-linux-try,analyzer-analysis-server-linux-try,analyzer-linux-release-try,front-end-linux-release-x64-try,vm-kernel-precomp-win-release-x64-try Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/132607 Reviewed-by: Martin Kustermann <kustermann@google.com> Commit-Queue: Daco Harkes <dacoharkes@google.com>

artob · 2020-05-15T05:09:58Z

From a user perspective, the lack of these standard variable-sized types is probably the single largest remaining deficiency in Dart's FFI support. It does rather complicate making FFI bindings for Dart.

As someone routinely working on a number of FFI bindings for a multitude of different languages (most recently, OpenXR bindings), the specific list of missing, absolutely essential variable-sized types in Dart is:

int and unsigned int
long and unsigned long
size_t (stddef.h)

And some other, somewhat rarer variable-sized types to consider supporting would be:

off_t (sys/types.h)
ssize_t (sys/types.h)
wchar_t (stddef.h)

All the aforementioned types listed in priority order, in terms of how badly and frequently I have needed them.

What's the plan for supporting at least int, long, and size_t?

dcharkes · 2020-05-20T16:39:06Z

These types are not part of the ABI, and the C standard has a very flexible specification (see this quick explanation). So, a compiler can choose what size/alignment these types have.

dart:ffi needs to be able to know what the size/alignment of types is of the compiled shared object loaded into Dart. The DartVM knows what OS (Linux,Windows,MacOS,iOS,Android) and hardware (x64,ia32,arm32,arm64) it is running on, so it can rely on the ABI and use the types specified in the ABI.

However, dart:ffi does not know with what compiler (and what compiler options) the shared object that is being loaded is compiled. So it can not know how to treat these C types. (Adding a possibly endless list of compilers/compiler options to DartVM is probably not a good approach, because it would require us to change the DartVM every time someone tries to use a new combination.)

It might be the case that for some of these types, all the compilers targeting the various ABIs (OS/Hardware combinations) agree on a size/alignment. Then we could incorporate these types into the ABI-logic. The downside is that if ever some new compiler comes up that does not respect the status quo, stuff breaks.

Does anyone have experience with whether the most common compilers agree on these types are handled in specific ABIs? Do we have any prior art in FFIs in other languages?

Another direction we could go to is to pass in the specs for these types into a bindings generator. The downside here is that one needs to re-run the bindings generator for every compiler/compiler options/hardware/OS combination when releasing an app.

@artob, what are you currently doing to work around this limitation?

artob · 2020-05-21T05:30:36Z

dart:ffi needs to be able to know what the size/alignment of types is of the compiled shared object loaded into Dart. The DartVM knows what OS (Linux,Windows,MacOS,iOS,Android) and hardware (x64,ia32,arm32,arm64) it is running on, so it can rely on the ABI and use the types specified in the ABI.

However, dart:ffi does not know with what compiler (and what compiler options) the shared object that is being loaded is compiled. So it can not know how to treat these C types. (Adding a possibly endless list of compilers/compiler options to DartVM is probably not a good approach, because it would require us to change the DartVM every time someone tries to use a new combination.)

@dcharkes Respectfully, you might be overthinking this. Just about any FFI for any programming language has the same theoretical problem, yet they all provide these FFI types.

It might be the case that for some of these types, all the compilers targeting the various ABIs (OS/Hardware combinations) agree on a size/alignment. Then we could incorporate these types into the ABI-logic. The downside is that if ever some new compiler comes up that does not respect the status quo, stuff breaks.

Does anyone have experience with whether the most common compilers agree on these types are handled in specific ABIs? Do we have any prior art in FFIs in other languages?

Every single FFI in every single programming language I've worked with--with the sole exception of Dart--supports these basic types: at the very least, int, long, and size_t are universal. Off the top of my head, here's a partial list of FFIs I have prior experience with, and how they map the types I mentioned:

Common Lisp CFFI: :int, :long, :size-t
Haskell FFI: CInt, CLong, CSize, CSsize, CWchar
Java JNA: int, NativeLong, Structure.FFIType.size_t, platform.linux.XAttr.ssize_t, char
Julia: Cint, Clong, Csize_t, Cssize_t, Cwchar_t
Node.js FFI: int, long, size_t
Python ctypes: c_int, c_long, c_size_t, c_ssize_t, c_wchar
OCaml ctypes: int, long, size_t, ssize_t
Racket FFI: _int, _long, _size, _ssize, _wchar
Ruby FFI: :int, :long, :size_t, :ssize_t, :wchar_t
Rust FFI: c_int, c_long, size_t, ssize_t

I omitted Go, LuaJIT, Nim, PHP, and Zig from the list, since their FFI implementations all parse C header syntax directly (and implicitly support these types).

@artob, what are you currently doing to work around this limitation?

I'm not. This, plus #35763, is blocking my efforts for nontrivial FFI bindings for Dart and Flutter. It would be a royal pain in the arse to work around both of these blockers, so I'm focusing my efforts elsewhere for now and hoping to come back to all this after a future Dart release announcement.

artob · 2020-05-21T06:13:17Z

@artob, what are you currently doing to work around this limitation?

Oh, and I might as well mention this here as an aside given that I haven't found an existing issue for it: the lack of general type aliases--for example, defining CLong as a typedef for either Int32 or Int64 at compile time--also hampers creating FFI bindings.

dcharkes · 2020-05-21T07:40:42Z

Oh, and I might as well mention this here as an aside given that I haven't found an existing issue for it: the lack of general type aliases--for example, defining CLong as a typedef for either Int32 or Int64 at compile time--also hampers creating FFI bindings.

See dart-lang/language#65 for that.

dcharkes · 2020-06-23T11:03:00Z

Assuming that the sizes are stable per ABI (so checking a single compiler gcc/clang/msvc per platform) and only for the subset of OS x hardware platforms that DartVM runs on:

Types we already support:

sizeof(int) == 4 -> use Int32.
sizeof(unsigned int) == 4 -> use Uint32.
sizeof(ssize_t) == sizeof(intptr_t) -> use IntPtr.

Types that require dart:ffi support:

sizeof(size_t) == sizeof(intptr_t) -> Introduce UintPtr (range in Dart 0 - +2^32 on 32 bit, range -2^63 - +2^63 on 64 bit, because Dart does not have unsigned 64 bit ints).
sizeof(long) == sizeof(intptr_t) except for Windows where sizeof(long) == 4. -> Introduce Long.
sizeof(unsigned long) == sizeof(intptr_t) except for Windows where sizeof(long) == 4. -> Introduce UnsignedLong (when 8 bytes, will be signed in Dart for the same reason).
sizeof(off_t) == sizeof(long) -> Introduce Long.
sizeof(wchar_t) == 4 except for Windows where sizeof(wchar_t) == 2. -> Introduce WChar.

Sources:

https://godbolt.org/ (linux/msvc and cross compilation to arm/arm64).
MS documentation about long.
Manual compilation on macos gist.

TODO:

Check iOS behavior.
Check Android arm/arm64 with Linux arm/arm64 behavior.

artob · 2020-07-05T13:23:45Z

@dcharkes Would you be willing to give any estimate which future Dart release those new types (Long, WChar, etc) might be expected to land in?

dcharkes · 2020-07-06T08:11:08Z

No, we generally don't give estimates. We're working on it, but we don't want to rush a suboptimal solution (idea for a general solution to ABI-specific types: #42563).

ndusart · 2021-02-09T15:13:54Z

As the types names do not match between dart:ffi and C. It would be helpful to provide a mapping table somewhere in dart:ffi documentation (at least for C types which have a correct sized type available in dart:ffi).

I had to search quite a lot to find how to safely map int to finally fall on this issue which tells that, yeah, given the current set of platforms supported by DartVM, dart:fii.Int32 is okay.

But still, I wouldn't say that int or unsigned int have a proper type already available. It means that we can go for Int32 without much consideration. But in a few months or years, DartVM will run on a platform that uses another size, and you'll be stuck or will finally provide a proper type for it but lot of codes will fail to run correctly because they won't update their code.

Why not introducing aliases for these types as well just now ?

dcharkes · 2021-02-09T15:17:50Z

I'll introduce them after #42563 to package:ffi.

alexmercerind · 2021-08-24T12:47:55Z

Wait, I've been working with wchar_t all this time, isn't it Utf16 in ffi?

dcharkes · 2022-01-20T15:13:06Z

This will be available in dart:ffi from Dart 2.17 (and from now on tip of tree).

In the mean time this is also available in package:ffi for Dart 2.16 (dev release).

Wait, I've been working with wchar_t all this time, isn't it Utf16 in ffi?

On Windows they're both 16-bit integers yes. Not on other platforms.

dcharkes · 2022-01-20T15:16:01Z

I'll introduce them after #42563 to package:ffi.

Please note that we've decided to put them into dart:ffi instead, so that they can be used with FfiNative (which needs to be usable without package imports).

Because they did not make it into Dart 2.16, but AbiSpecificInteger did, they are available in package:ffi for Dart 2.16 and will be deprecated from there with Dart 2.17 arrives.

This reverts commit 85a87ca. Reason for revert: * Adding `Char` breaks `package:win32` which is used in Flutter so it breaks the Flutter build: https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8824468064587445729/+/u/Android_Views_Integration_Tests/stdout For reference: full list of Flutter failues: https://github.com/flutter/flutter/runs/4890844911 Original change's description: > [vm/ffi] Add common C types > > We're adding these types to `dart:ffi` rather than `package:ffi` so that > they can be used with `FfiNative`s. > > Adds `NativeType`s for the following C types: > > * char > * unsigned char > * signed char > * short > * unsigned short > * int > * unsigned int > * long > * unsigned long > * long long > * unsigned long long > * uintptr_t > * size_t > * wchar_t > > Because the C standard only defines minimum sizes for many of these > types, future platforms might diverge from the typical size even if all > platforms currently agree on a size. To avoid having to reification > later, we define all types as AbiSpecificIntegers rather than typedefs, > even if all current target platforms agree on the size. > > Closes: #36140 > > TEST=tests/ffi/c_types_test.dart > > Change-Id: Ie97d253856d787386529231e8060f879069be886 > Cq-Include-Trybots: luci.dart.try:dart-sdk-linux-try,dart-sdk-mac-try,dart-sdk-win-try,vm-ffi-android-debug-arm64c-try,vm-ffi-android-debug-arm-try,vm-canary-linux-debug-try,vm-fuchsia-release-x64-try,vm-kernel-gcc-linux-try,vm-kernel-asan-linux-release-x64-try,vm-kernel-linux-debug-x64-try,vm-kernel-linux-debug-ia32-try,vm-kernel-mac-debug-x64-try,vm-kernel-mac-release-arm64-try,vm-kernel-nnbd-win-release-ia32-try,vm-kernel-nnbd-win-release-x64-try,vm-precomp-ffi-qemu-linux-release-arm-try,vm-kernel-win-release-x64-try > Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/228541 > Reviewed-by: Martin Kustermann <kustermann@google.com> # Not skipping CQ checks because original CL landed > 1 day ago. Change-Id: Ic56df88c653b1395ed5e5a71af5e571b1adc3671 Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/229152 Reviewed-by: Daco Harkes <dacoharkes@google.com> Commit-Queue: Daco Harkes <dacoharkes@google.com>

dcharkes · 2022-01-22T10:52:24Z

package:ffigen (starting version 5.0.0-dev.0) now recognizes the following C types:

char

unsigned char

signed char

short

unsigned short

int

unsigned int

long

unsigned long

long long

unsigned long long

uintptr_t

size_t

wchar_t

dcharkes added the area-core-library SDK core library issues (core, async, ...); use area-vm or area-web for platform specific libraries. label Mar 7, 2019

dcharkes mentioned this issue Mar 7, 2019

dart:ffi resolve outstanding design decisions #35764

Closed

12 tasks

lrhn added the library-ffi label Mar 7, 2019

dcharkes added the type-design label Jun 3, 2019

dcharkes mentioned this issue Nov 21, 2019

How to define a type like size_t?? #39372

Closed

artob mentioned this issue May 21, 2020

[dart:ffi] Inline arrays in Structs #35763

Closed

artob mentioned this issue May 21, 2020

Type aliases: Typedefs for non-function types dart-lang/language#65

Closed

dcharkes mentioned this issue Jun 19, 2020

Support size_t, int, and other common C types // platform independent bindings dart-lang/native#530

Closed

artob mentioned this issue Jul 23, 2020

[ffi] Add support for abi-specific integer sizes #42563

Closed

dcharkes changed the title ~~dart:ffi int, long, etc~~ [vm/ffi] int, long, size_t, uintptr_t, wchar_t, etc Jan 7, 2021

dcharkes mentioned this issue Mar 4, 2021

Use package:ffigen to generate bindings google/webcrypto.dart#15

Closed

dcharkes mentioned this issue Jan 4, 2022

ABI-specific integers dart-archive/ffi#128

Merged

copybara-service bot closed this as completed in 85a87ca Jan 20, 2022

dcharkes reopened this Jan 22, 2022

rustui mentioned this issue Jan 25, 2022

Support a way to access low level hidapi methods rustui/hid#7

Open

copybara-service bot closed this as completed in 045b9c1 Jan 27, 2022

sagudev mentioned this issue Apr 29, 2022

[Bug] u8,u16,i8,i16 not working in struct fzyzcjy/flutter_rust_bridge#429

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[vm/ffi] int, long, size_t, uintptr_t, wchar_t, etc #36140

[vm/ffi] int, long, size_t, uintptr_t, wchar_t, etc #36140

dcharkes commented Mar 7, 2019 •

edited

Loading

dcharkes commented Jan 21, 2020

artob commented May 15, 2020

dcharkes commented May 20, 2020

artob commented May 21, 2020

artob commented May 21, 2020

dcharkes commented May 21, 2020

dcharkes commented Jun 23, 2020 •

edited

Loading

artob commented Jul 5, 2020

dcharkes commented Jul 6, 2020

ndusart commented Feb 9, 2021 •

edited

Loading

dcharkes commented Feb 9, 2021

alexmercerind commented Aug 24, 2021

dcharkes commented Jan 20, 2022

dcharkes commented Jan 20, 2022

dcharkes commented Jan 22, 2022

[vm/ffi] int, long, size_t, uintptr_t, wchar_t, etc #36140

[vm/ffi] int, long, size_t, uintptr_t, wchar_t, etc #36140

Comments

dcharkes commented Mar 7, 2019 • edited Loading

dcharkes commented Jan 21, 2020

artob commented May 15, 2020

dcharkes commented May 20, 2020

artob commented May 21, 2020

artob commented May 21, 2020

dcharkes commented May 21, 2020

dcharkes commented Jun 23, 2020 • edited Loading

artob commented Jul 5, 2020

dcharkes commented Jul 6, 2020

ndusart commented Feb 9, 2021 • edited Loading

dcharkes commented Feb 9, 2021

alexmercerind commented Aug 24, 2021

dcharkes commented Jan 20, 2022

dcharkes commented Jan 20, 2022

dcharkes commented Jan 22, 2022

dcharkes commented Mar 7, 2019 •

edited

Loading

dcharkes commented Jun 23, 2020 •

edited

Loading

ndusart commented Feb 9, 2021 •

edited

Loading