Support flexible array members in low-level API

[edsko]

See https://en.wikipedia.org/wiki/Flexible_array_member .

[phadej]

The question is: how? Structs with flexible array member are "incomplete", they cannot be Storable. The size of that member is application specific.

As far as I can tell, the only thing we can reliable do is to ignore that member. We could somehow provide the offsetof information to the Haskell user though, custom typeclass? rust-bindgen does something I'm not completely understand, but it looks like that:

struct foo {
        int len;
        char data[];
};

/* automatically generated by rust-bindgen 0.70.1 */

#[repr(C)]
#[derive(Default)]
pub struct __IncompleteArrayField<T>(::std::marker::PhantomData<T>, [T; 0]);
impl<T> __IncompleteArrayField<T> {
    #[inline]
    pub const fn new() -> Self {
        __IncompleteArrayField(::std::marker::PhantomData, [])
    }
    #[inline]
    pub fn as_ptr(&self) -> *const T {
        self as *const _ as *const T
    }
    #[inline]
    pub fn as_mut_ptr(&mut self) -> *mut T {
        self as *mut _ as *mut T
    }
    #[inline]
    pub unsafe fn as_slice(&self, len: usize) -> &[T] {
        ::std::slice::from_raw_parts(self.as_ptr(), len)
    }
    #[inline]
    pub unsafe fn as_mut_slice(&mut self, len: usize) -> &mut [T] {
        ::std::slice::from_raw_parts_mut(self.as_mut_ptr(), len)
    }
}
impl<T> ::std::fmt::Debug for __IncompleteArrayField<T> {
    fn fmt(&self, fmt: &mut ::std::fmt::Formatter<'_>) -> ::std::fmt::Result {
        fmt.write_str("__IncompleteArrayField")
    }
}
#[repr(C)]
#[derive(Debug)]
pub struct foo {
    pub len: ::std::os::raw::c_int,
    pub data: __IncompleteArrayField<::std::os::raw::c_char>,
}
#[allow(clippy::unnecessary_operation, clippy::identity_op)]
const _: () = {
    ["Size of foo"][::std::mem::size_of::<foo>() - 4usize];
    ["Alignment of foo"][::std::mem::align_of::<foo>() - 4usize];
    ["Offset of field: foo::len"][::std::mem::offset_of!(foo, len) - 0usize];
    ["Offset of field: foo::data"][::std::mem::offset_of!(foo, data) - 4usize];
};

[edsko]

Discussed this with the client. We propose something along the following lines:

Given

struct foo {
        int len;
        char data[];
};

(which potentially has many more fields than just these two, of course), generate something like

data PartialFoo = PartialFoo {
      len :: Int
     -- plus other fields, if any, but without the flexible array member
    }

-- generated instance, as usual
instance Storable PartialFoo where

In addition, we should also generate an instance of this class:

class SupportsFlexibleArray a where
  arrayMemberOffset :: Proxy a -> Int

Moreover, the user must provide an instance of this class:

class SupportsFlexibleArray a => HasFlexibleArrayLength a where
  arrayMemberLength :: a -> Int

Finally, we provide the following infrastructure in hs-bindgen-patterns:

newtype WithFlexbileArrayMember a = WithFlexbileArrayMember a

-- Single instance, defined once and for all 
instance (Storable a, HasFlexibleArrayLength a) => Storable (WithFlexibleArrayMember a) where 
  ...  

[phadej]

instance (Storable a, HasFlexibleArrayLength a) => Storable (WithFlexibleArrayMember a) where

cannot be defined. Even we have sizeOf :: a -> Int; the a there is really meant to be Proxy a (but this is pre-Proxy) interface.

One way to understand this is that Storable types are of fixed size; consider usage in Storable.Vector.

We can have Storable-like class though, with peek and poke like functions.

[edsko]

Sorry, I should have answered here. Good point regarding Storable implying a fixed size. Yes, let's introduce a custom class, and perhaps we should then derive instances for that custom class for all types, and Storable instances additionally for some types?