[16.x] make pyarrow{,-all} noarch

[h-vetinari]

Both pyarrow and pyarrow-all are wrapper packages with no actual content. They could can just as well be noarch, and the rest should work (i.e. the constraints on pyarrow-core will ensure the right python version etc. gets used).

[conda-forge-webservices]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

[h-vetinari]

Any help from the noarch experts here? CC @jaimergp @beckermr @jakirkham?

The way the package is layered, there are two outputs that are effectively empty meta-packages which could easily be noarch:

pyarrow-core    # depends on libarrow*;     per python version resp. CUDA/non-CUDA
pyarrow         # depends on pyarrow-core;  could be noarch
pyarrow-all     # depends on pyarrow;       could be noarch
pyarrow-tests   # depends on pyarrow-all;   per python version

I've tried doing that here now, and it works fine in native builds, but fails with IMO non-sensical errors on aarch/ppc:

TEST END: /home/conda/feedstock_root/build_artifacts/linux-aarch64/pyarrow-core-16.0.0-py39ha95f412_1_cpu.conda
TEST START: /home/conda/feedstock_root/build_artifacts/noarch/pyarrow-16.0.0-hd8ed1ab_1.conda
WARNING: Multiple meta files found. The meta.yaml file in the base directory (/tmp/tmpnher07k4/info/recipe) will be used.
Reloading output folder (local): ...working... done
Solving environment (_test_env): ...working... failed
WARNING: failed to get package records, retrying.  exception was: Unsatisfiable dependencies for platform linux-64: {MatchSpec("16.0.0=*_1_"), MatchSpec("pyarrow==16.0.0=hd8ed1ab_1"), MatchSpec("pyarrow-core=16.0.0[build=*_1_*]")}
Encountered problems while solving:
  - nothing provides requested pyarrow-core 16.0.0.* *_1_*
  - nothing provides pyarrow-core 16.0.0 *_1_* needed by pyarrow-16.0.0-hd8ed1ab_1

When I say non-sensical, I mean that that the TEST END: at the top just confirmed the local existence of an artefact that should match the requested pattern.

Note that all jobs here want to build:

noarch/pyarrow-16.0.0-hd8ed1ab_1.conda
noarch/pyarrow-all-16.0.0-hd8ed1ab_1.conda

which is the goal - we need to build it in each job in order to successfully build the per-python pyarrow-tests on top, but only one build will be uploaded in the end across all jobs, due to the (intentional!) hash collision.

[h-vetinari]

Ah, I see the problem now:

TEST END: /home/conda/feedstock_root/build_artifacts/linux-aarch64/pyarrow-core-16.0.0-py39ha95f412_1_cpu.conda
[...]                                                ^^^^^^^^^^^^^
Unsatisfiable dependencies for platform linux-64: ...
                                        ~~~~~~~~

so it uses the wrong platform (probably due to the noarch specification?)! I had suspected something like that before and had tried

diff --git a/conda-forge.yml b/conda-forge.yml
index 076b676..a1f93b6 100644
--- a/conda-forge.yml
+++ b/conda-forge.yml
@@ -13,4 +13,13 @@ github:
 provider:
   linux_aarch64: azure
   linux_ppc64le: azure
+# to be able to build pyarrow{,-all} as noarch, while building
+# the python-version-dependent pyarrow-tests on top (per job)
+noarch_platforms:
+  - linux_64
+  - linux_aarch64
+  - linux_ppc64le
+  - osx_64
+  - osx_arm64
+  - win_64
 test: native_and_emulated

but that doesn't generate any noticeable difference in the rerendered files.

It still sounds like a conda-build issue IMO if the it tries to change architecture mid-build due to noarch:. CC @mbargull

[conda-forge-webservices]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe/meta.yaml) and found it was in an excellent condition.