PyArrow broke the build on ARM platform [Regression]

[jaksmid]

pip install apache-superset==0.35.1 on Raspbian (Raspberry pi 4, ARM) ends with

ERROR: Could not find a version that satisfies the requirement pyarrow<0.15.0,>=0.14.1 (from apache-superset) (from versions: 0.9.0, 0.10.0, 0.11.0, 0.11.1, 0.12.0, 0.12.1, 0.13.0, 0.14.0, 0.15.1)
ERROR: No matching distribution found for pyarrow<0.15.0,>=0.14.1 (from apache-superset)

Version `0.34.1` installed correctly.

Expected results

Apache superset installs correctly.

Actual results

The installation ends with an error.

Screenshots

If applicable, add screenshots to help explain your problem.

How to reproduce the bug

1. Use system with ARM architecture.
2. Run pip install apache-superset==0.35.1

Environment

(please complete the following information):

• superset version: 0.35.1
• python version: 3.7.3
• OS: Linux raspberrypi 4.19.75-v7l+ GPL licence for select2.sortable.js? #1270 SMP Tue Sep 24 18:51:41 BST 2019 armv7l GNU/Linux

Checklist

Make sure these boxes are checked before submitting your issue - thank you!

• I have checked the superset logs for python stacktraces and included it here as text if there are any.
• I have reproduced the issue with at least the latest released version of superset.
• I have checked the issue tracker for the same issue and I haven't found one similar.

Additional context

I tried to installed the latest version of pyarrow (as I saw the recent not released bump to 0.15.1).

This fails with: ERROR: Could not build wheels for pyarrow which use PEP 517 and cannot be installed directly

[issue-label-bot]

Issue-Label Bot is automatically applying the label #bug to this issue, with a confidence of 0.94. Please mark this comment with 👍 or 👎 to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

[dpgaspar]

Thank you for reporting this.

Strange... any idea why it does not consider 0.14.1 has a valid version on RPI?

[jaksmid]

I guess that you can add tags to the build regarding the supported architecture. Maybe 0.14.1 is missing the proper tag.

However, even other versions fail during the cmake. I have a set of common build tools installed (everything else builds fine) but the 0.15.1 and 0.14.0 fail. So far did not found a solution on PyArrow issue tracker.

[jaksmid]

This discussion about ARM support for arrow seems to be related:
https://issues.apache.org/jira/browse/ARROW-7042

[stone-bits]

Have the same error on Wondows. Python 3.6.8, 3.7.5

[dpgaspar]

@igor-tk Windows is not supported by superset since we don't have tests that cover this platform. I've marked #8583 to be included on superset next release. You can keep using 0.34.1. It would be great if you could test #8583 commit tag and report back.

[Guires]

#Anaconda prompt
Step1:Install - "pip install virtualenv"
Step2: Create VM- "py -m venv your_env_name" or "python -m venv your_env_name" or just create environment using conda
Step3:Activate -"your_env_name\Scripts\activate"
Step4:Downlaod- python-geohash wheel with compitable version from the link:https://www.lfd.uci.edu/~gohlke/pythonlibs/#python-geohash
Step5:Install the python-geohash library "pip install python-geohash directory"
step6:anaconda prompt -"pip install superset"
step7:# Create an admin user (you will be prompted to set username, first and last name before setting a password) by typing this- "fabmanager create-admin --app superset"
step8: type -"cd your_env_name/Scripts"
Step8:python superset db upgrade
Step9:python superset load_examples
step10:python superset init
step11:python superset run -p 8088 --with-threads --reload --debugger

After installation, you should be able to point your browser to the right hostname:port http://localhost:8088

Output screenshot

[jaksmid]

Hello @dpgaspar, thanks for looking into this.

I have tested the #8583 and I get the different error but also related to pyarrow.

Here are the details:

1. created a fresh virtual environment (Python 3.7.3)
2. tried to install the merge commit of #8583 - fc12e53ff58e32756bd7fcf5f0013f424b2fc901 by running pip install git+git://github.com/apache/incubator-superset.git@fc12e53ff58e32756bd7fcf5f0013f424b2fc901

It sucessfully downloaded pyarrow==0.15.1. Got error during

Building wheels for collected packages: pyarrow
Building wheel for pyarrow (PEP 517) ... error

Error:

ERROR: Command errored out with exit status 1:
   command: /home/pi/.virtualenvs/superset-arrow/bin/python3 /home/pi/.virtualenvs/superset-arrow/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py build_wheel /tmp/tmpz55jr2rb
       cwd: /tmp/pip-install-mfsz8sux/pyarrow
Complete output (428 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-armv7l-3.7
  creating build/lib.linux-armv7l-3.7/pyarrow
...
Lots of copy files lines
...
  running build_ext
  creating build/temp.linux-armv7l-3.7
  -- Running cmake for pyarrow
  cmake -DPYTHON_EXECUTABLE=/home/pi/.virtualenvs/superset-arrow/bin/python3  -DPYARROW_BOOST_USE_SHARED=on -DCMAKE_BUILD_TYPE=release /tmp/pip-install-mfsz8sux/pyarrow
  error: command 'cmake' failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for pyarrow
 Running setup.py clean for pyarrow
Failed to build pyarrow
Building wheels for collected packages: apache-superset, msgpack
  Building wheel for apache-superset (setup.py) ... done
  Created wheel for apache-superset: filename=apache_superset-0.999.0.dev0-cp37-none-any.whl size=19405754 sha256=70769042bd106ea807ab21a0a2aee23e186339a749baf91323613cdb5f019bda
  Stored in directory: /tmp/pip-ephem-wheel-cache-9vxgy7gv/wheels/79/01/4b/64bb882d74d8f02a24d1df1b4521a46c0cf61cbc03fd76b970
  Building wheel for msgpack (setup.py) ... done
  Created wheel for msgpack: filename=msgpack-0.6.2-cp37-cp37m-linux_armv7l.whl size=264049 sha256=433873d162ac01e013ffe2e2f0d8eda2bc96c67bfdff9c2e37ac826dc79feaf5
  Stored in directory: /home/pi/.cache/pip/wheels/27/e6/c7/adc38b19993e22e1f7bea6ff388c7fcda941a2a17258d6e8f3
Successfully built apache-superset msgpack
ERROR: Could not build wheels for pyarrow which use PEP 517 and cannot be installed directly

Seems like pyarrow does not like 32 bit ARM builds.

[robdiciuccio]

The PyArrow installation docs "strongly" recommends using a 64-bit system, but it's a bit odd that older versions have worked for you.

Have you tried the --no-binary install option discussed here?
https://discuss.python.org/t/pep-517-and-projects-that-cant-install-via-wheels/791

[jaksmid]

@robdiciuccio The old version simply worked for me because there was no pyarrow. It was added only in the most recent version. Hence the regression. I am aware of the strong arrow recommendation to use 64-bit system.

However, the official rapsbian os for Rapsberry is for now released for 32-bit only, with the 64-bit version under testing.

I have tried the no binary option (pip install pyarrow --no-binary pyarrow) with the same result.

[jingw222]

pip install pyarrow also failed to build on Ubuntu Server (19.10, 64-bit) on Raspberry Pi 3b+ with the same error

'cmake' failed with exit status 1
ERROR: Could not build wheels for pyarrow which use PEP 517 and cannot be installed directly

System info:

$ uname -a
Linux ubuntu 5.3.0-1014-raspi2 #16-Ubuntu SMP Tue Nov 26 11:18:23 UTC 2019 aarch64 aarch64 aarch64 GNU/Linux

[jaksmid]

I got the same error on the beta version of 64-bit Raspbian.

After I installed cmake by sudo apt-get install cmake, I got the same error as mentioned in #8843 ModuleNotFoundError: No module named 'numpy' even though thenumpy is installed.

@jingw222 could you please try to install the cmake and rerun the command to see if the Ubuntu has the same behavior?

[dpgaspar]

@jaksmid thank you for testing #8583, sorry that it did not work.

RPI and ARM are not tested on superset and they are not a "normal" OS to deploy Superset on.

Having said that I found this https://gist.github.com/heavyinfo/04e1326bb9bed9cecb19c2d603c8d521 on ARROW-7042, maybe it works.

[jingw222]

@jaksmid Yes exactly. That was not expected since numpy had just been compiled, built and installed in my local env, and I ran it in python without an issue.

  CMake Error at cmake_modules/FindNumPy.cmake:62 (message):
    NumPy import failure:

    Traceback (most recent call last):

      File "<string>", line 1, in <module>

    ModuleNotFoundError: No module named 'numpy'

  Call Stack (most recent call first):
    CMakeLists.txt:189 (find_package)


  -- Configuring incomplete, errors occurred!
  See also "/tmp/pip-install-7728kxma/pyarrow/build/temp.linux-aarch64-3.7/CMakeFiles/CMakeOutput.log".
  See also "/tmp/pip-install-7728kxma/pyarrow/build/temp.linux-aarch64-3.7/CMakeFiles/CMakeError.log".
  error: command 'cmake' failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for pyarrow
  Running setup.py clean for pyarrow
Failed to build pyarrow
ERROR: Could not build wheels for pyarrow which use PEP 517 and cannot be installed directly

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. For admin, please label this issue .pinned to prevent stale bot from closing the issue.

[exwhyz]

I have the exact same issue on Windows 10 64-bit, Python 3.8 with Arrow 0.16.0 - Requirement already satisfied: pyarrow==0.16.0 in c:\python38\lib\site-packages (0.16.0). When installing apache-superset==0.35.2, it fails with the error - ModuleNotFoundError: No module named 'numpy' even though Numpy is installed fine and there are no environment conflicts - Requirement already satisfied: numpy in c:\python38\lib\site-packages (1.18.1).

Update: I suspect this is tied to the following issue (apache/arrow#5987) and since apache-superset expects pyarrow==0.15.1 which I am unable to downgrade to (from 0.16.0 because of the same issue reported above i.e. Numpy import failure) this issue manifests in the build of apache superset.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. For admin, please label this issue .pinned to prevent stale bot from closing the issue.