Split crate in smaller crates

[jorgecarleitao]

Hi,

I would like to gauge interest in splitting this crate in 3 smaller crates,

• arrow2-core
• arrow2-compute
• arrow2-io

so that we can:

1. apply a different versioning to them. E.g. the core part hasn't had a backward incompatible change for some time now, but all other parts had some recently.
2. add features more dedicated to each of the parts. E.g. compute currently drags a lot of stuff, even if people would just like part of the compute
3. Reduce the compilation times

arrow2-core would contain:

alloc
array
bitmap
buffer
datatypes
ffi
types
util
error

arrow2-compute would contain the compute, and arrow2-io would contain io.

[jorgecarleitao]

any thoughts, @ritchie46 @sundy-li @elferherrera ?

[ritchie46]

add features more dedicated to each of the parts. E.g. compute currently drags a lot of stuff, even if people would just like part of the compute

hear hear. 😄

I think it is a good idea to have a minimal core crate. I am mostly for cherry picking IO and compute utils, but that can be achieved with feature gates. I think this is a good step to improve compile times. No extra thoughts.

[elferherrera]

I like the idea as well, specially for the compilation times.
However, what would be the plan for future versioning if Arrow2 becomes part of apache arrow with multiple crates?
Or are you really considering leaving arrow2 outside apache? (I dont mind either, just wondering how it will work in the future)

[yjhmelody]

Good idea, in my project, I only use arrow2 computing related APIs.