Updating arrow2 branch

[dbr]

The arrow2 branch is currently using v0.10.0 - I'd like to update it to work with v0.12.0. Is there anything I should know before starting this?

I ask as last time I made a PR for the branch, there was an impending rework of that branch, and I'm not sure what the state of things are currently!

[andygrove]

Would it make sense to consider having a separate, optional, datafusion-arrow2 crate containing a physical planner + operators that use arrow2? It seems to me that this would be easier to maintain. Maybe @houqp has an opinion on this?

[alamb]

cc @Igosuki as well who may have some thoughts

[Igosuki]

Hello, sorry for my absence. Maintaining full compatibility requires constant updates, and updating the arrow2 branch is often in the hundred files because of the API surface conflict. So adding a new crate where only arrow2 itself needs to be updated is a good idea in my opinion.

[houqp]

+1 on @andygrove 's idea for isolating changes to a self-contained crate. @dbr if you are interested in that, please feel free to start that project at the datafusion contrib github org. If you are only interested in updating arrow2 version in the current arrow2 branch, then you can just send a PR to the arrow2 branch :)

[andygrove]

Should we have a discussion/vote about whether this should be in datafusion-contrib rather than under Arrow governance? There are the options of creating a new repo such as arrow-datafusion-arrow2 or a new optional crate within the main repo. The work on this so far has been under Arrow governance so we should make sure that all contributors are in agreement on moving it out and also be aware that moving it back in is a lot of work. In hindsight, I wish we had not moved the DataFusion Python bindings out of Arrow governance after all the work that was done to get them donated in the first place.

[andygrove]

@alamb @jorgecarleitao Do you have opinions on this?

[alamb]

I do not have any preference, but I also don't really spend any effort on arrow2 or the arrow2 branch in datafusion -- perhaps @Igosuki or others who have expended significant effort would be better to weigh on

[jorgecarleitao]

@andygrove, thanks for the ping! I do not have a preference - I am very happy to support anyone that wishes to use arrow2/parquet2, and I am very thankful for everyone that has been maintaining the arrow2 branch. I unfortunately do not have the time left to spearhead it.

@dbr I would be very happy to support you in the migration - let me know how I can help and feel free to assign tasks directly. Arrow2's core API is essentially stable and that we we usually have small breaking changes in IO, so most of the changes should be on the datafusion's side.

For full transparency: imo the primary question here is one of control - although we demonstrated that arrow2 had superior support, safety, performance, documentation, and UX, I sensed that the main requirement was that arrow2 was donated to Apache, which for me was unacceptable. This demotivated me from contributing here.

I understand that companies that have a strong dependency on DataFusion need control over its core dependency. For me it demos that we are a bit past the idea that people in Apache are individual contributors.

The core development of Polars and Databend have been more pragmatic and I have thus been more aware of the requests there. However, if there is anything needed, just drop an issue and I will gladly take it, irrespectively of where the development happens (here or in a separate repo / org).

I continue to believe that DataFusion would benefit immensely from arrow2's current and future capabilities - arrow2 is the backend of likely the fastest in-node dataframe APIs out there and just this week we exposed a new API that enable compute to be 2x-5x faster ;)

[alamb]

although we demonstrated that arrow2 had superior support, safety, performance, documentation, and UX

I believe there are differing opinions on some of these items

I do think there is broad agreement that the governance model of arrow2 (benign dictator) is not suitable for all users for a variety of reasons.

[Igosuki]

The main issue a few months ago was that the datafusion codebase was not split like it is now, so it created conflicts in the various places where it wasn't possible to confine differences between arrow and arrow2::io to extension traits. So my question is, can a contrib crate exist since arrow2 is essentially required at the core ?

[andygrove]

I do think there is broad agreement that the governance model of arrow2 (benign dictator) is not suitable for all users for a variety of reasons.

I agree. Although there is some great work happening in arrow2, I am not motivated (or even permitted) to contribute there. I am heavily invested in working with the Apache Arrow community to continue to improve things here together. The Apache governance model is not perfect but it is working well enough from my point of view. I am a big believer in community over code.

[andygrove]

The main issue a few months ago was that the datafusion codebase was not split like it is now, so it created conflicts in the various places where it wasn't possible to confine differences between arrow and arrow2::io to extension traits. So my question is, can a contrib crate exist since arrow2 is essentially required at the core ?

Regardless of where the code lives, I would view us as having two main options:

1. Continue to maintain a copy of DataFusion "core" with modifications to allow building with arrow or arrow2. This will be an ongoing maintenance burden and IMO only worth the effort if we are working towards having this merged into DataFusion master at some point and I am not sure how much appetite there is for this.

2. Fork datafusion-core to create a new query engine based on arrow2 that can leverage the new DataFusion crates for logical plan building, SQL query planning, logical plan optimizations etc. Over time maybe more functionality can be moved out of core and re-used by other query engines. This would decouple the development work and allow those interested in arrow2 to move faster and does not put any burden on the core DataFusiuon development. That would be my choice if I were working on this.

[alamb]

One approach that @jimexist started with, that I thought was promising was to use traits to collect the differences in the two apis -- so that DataFusion used the traits and then there were impls for both arrow and arrow2

For example, from_slice in #1588 and #1589

I would be quite supportive of such a strategy and would be a nice way to let people use which every library they wanted

[andygrove]

I would be quite supportive of such a strategy and would be a nice way to let people use which every library they wanted

Bonus points if it could one day support a GPU implementation of Arrow as well 😉