[FEA] Upgrade to Arrow 14

[bdice]

This issue tracks the migration of Arrow to major version 14.

The conda-forge packaging of libarrow has been split up (conda-forge/arrow-cpp-feedstock#1201), resulting in this list of packages split from libarrow:

• libarrow-all is a metapackage which depends on all the following packages
• libarrow
• libarrow-acero
• libarrow-dataset
• libarrow-flight
• libarrow-flight-sql
• libarrow-gandiva
• libarrow-substrait
• libparquet

The pyarrow package has run dependencies on all of the above packages (excluding libarrow-all).

As a first attempt at migration, I think we can use libarrow-all in place of libarrow in our conda builds, and if that works, we can try just using the components that we need. My best guess is that this includes only libarrow and libarrow-dataset, and maybe libparquet.

cc: @galipremsagar

[bdice]

This update is important because if an environment already contains a package that depends on (or solves with) pyarrow or libarrow version 14, it's not possible for the conda/mamba solver to downgrade from libarrow 14 to libarrow 13 due to the differences in package structure. We're starting to see this issue pop up across RAPIDS. Pinning this to Arrow 13 everywhere this problem shows up is not a good strategy because this should be solved within cudf's pinnings.

[bdice]

#14371 updated cudf to use Arrow 14. The next step I would take before closing this is to identify which subpackages we rely on and only install those subpackages where needed, rather than libarrow-all.

Based on this list:

cudf/cpp/cmake/thirdparty/get_arrow.cmake

Lines 181 to 182 in 723f0e4

	GLOBAL_TARGETS arrow_shared parquet_shared arrow_acero_shared arrow_dataset_shared arrow_static
	parquet_static arrow_acero_static arrow_dataset_static

I think we want libarrow, libparquet, libarrow-acero, libarrow-dataset. We probably won't need libarrow-flight, libarrow-flight-sql, libarrow-gandiva, or libarrow-substrait.

[galipremsagar]

We probably won't need libarrow-flight, libarrow-flight-sql, libarrow-gandiva, or libarrow-substrait.

Just a note, pyarrow=14 will still depend on these and fetch them. The change we make would be beneficial to someone who is depending only on libcudf and not on cudf.

[h-vetinari]

Just a note, pyarrow=14 will still depend on these and fetch them.

This is something we want to change, and which may well still happen in the 14 series. The first iteration only split the libraries, but getting the installation size of pyarrow down (where the whole enchilada isn't needed) has been something arrow wanted to do for a while, and a bunch of the necessary work already landed in 14.

[jakirkham]

Glad to hear that! Thanks Axel 🙏