gradlew :metadata-ingestion:installDev subprocess-exited-with-error #4715

iosMental · 2022-04-21T13:35:51Z

Installing collected packages: numpy
Running setup.py install for numpy: finished with status 'error'
error: subprocess-exited-with-error

× Running setup.py install for numpy did not run successfully.
│ exit code: 1
╰─> [1327 lines of output]
Running from numpy source directory.

  Note: if you need reliable uninstall behavior, then install
  with pip instead of using `setup.py install`:
  
    - `pip install .`       (from a git repo or downloaded source
                             release)
    - `pip install numpy`   (last NumPy release on PyPi)

clang: error: no such file or directory: 'build/temp.macosx-10.14-arm64-3.8/numpy/core/src/multiarray/dragon4.o'
error

numpy/core/src/multiarray/dragon4.c:3141:1: error: implicit declaration of function 'Dragon4_PrintFloat_Intel_extended128' is invalid in C99 [-Werror,-Wimplicit-function-declaration]

  ^
  /private/var/folders/3k/pjpmrxl91gsfwl734jv4lf4w0000gp/T/pip-install-f8mmyw5c/numpy_efa59bb6a8334616987bbb43e729d281/numpy/distutils/checks/cpu_neon_vfpv4.c:10:5: error: use of undeclared identifier 'float32x4_t'
      float32x4_t v3 = vdupq_n_f32(3.0f);
      ^
  /private/var/folders/3k/pjpmrxl91gsfwl734jv4lf4w0000gp/T/pip-install-f8mmyw5c/numpy_efa59bb6a8334616987bbb43e729d281/numpy/distutils/checks/cpu_neon_vfpv4.c:11:20: error: implicit declaration of function 'vgetq_lane_f32' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
      int ret = (int)vgetq_lane_f32(vfmaq_f32(v1, v2, v3), 0);
                     ^
  /private/var/folders/3k/pjpmrxl91gsfwl734jv4lf4w0000gp/T/pip-install-f8mmyw5c/numpy_efa59bb6a8334616987bbb43e729d281/numpy/distutils/checks/cpu_neon_vfpv4.c:11:35: error: implicit declaration of function 'vfmaq_f32' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
      int ret = (int)vgetq_lane_f32(vfmaq_f32(v1, v2, v3), 0);
                                    ^
  /private/var/folders/3k/pjpmrxl91gsfwl734jv4lf4w0000gp/T/pip-install-f8mmyw5c/numpy_efa59bb6a8334616987bbb43e729d281/numpy/distutils/checks/cpu_neon_vfpv4.c:11:45: error: use of undeclared identifier 'v1'
      int ret = (int)vgetq_lane_f32(vfmaq_f32(v1, v2, v3), 0);
                                              ^

The text was updated successfully, but these errors were encountered:

anshbansal · 2022-04-21T14:12:39Z

What operating system is this happening on?

iosMental · 2022-04-22T01:15:04Z

What operating system is this happening on?

mac m1

iosMental · 2022-04-22T06:11:57Z

mac m1

anshbansal · 2022-04-22T08:02:55Z

What command are you running? Please share with me full command.

I use the below on Mac M1

./gradlew :metadata-ingestion:installDev
# This fails once 

cd metadata-ingestion
source venv/bin/activate
pip install --upgrade pip wheel setuptools
CPATH="$(brew --prefix librdkafka)/include"
export CPATH
C_INCLUDE_PATH="$(brew --prefix librdkafka)/include"
export C_INCLUDE_PATH
LIBRARY_PATH="$(brew --prefix librdkafka)/lib"
export LIBRARY_PATH
pip install --no-cache confluent-kafka

./gradlew :metadata-ingestion:installDev
#passes this time

iosMental · 2022-04-22T08:26:06Z

i use this on Mac m1
cd datahub
./gradlew :metadata-ingestion:installDev

#Then start execution
<===========--> 91% EXECUTING [56s]

IDLE
IDLE
:metadata-ingestion:runPreFlightScript
Task :metadata-ingestion:runPreFlightScript FAILED

FAILURE: Build failed with an exception.

anshbansal · 2022-04-22T08:31:36Z

Can you try the commands I shared?

iosMental · 2022-04-22T08:32:24Z

What command are you running? Please share with me full command.

I use the below on Mac M1

./gradlew :metadata-ingestion:installDev
# This fails once 

cd metadata-ingestion
source venv/bin/activate
pip install --upgrade pip wheel setuptools
CPATH="$(brew --prefix librdkafka)/include"
export CPATH
C_INCLUDE_PATH="$(brew --prefix librdkafka)/include"
export C_INCLUDE_PATH
LIBRARY_PATH="$(brew --prefix librdkafka)/lib"
export LIBRARY_PATH
pip install --no-cache confluent-kafka

./gradlew :metadata-ingestion:installDev
#passes this time

try to execute in the datahub ingestion directory
cd metadata-ingestion
...
...
..
Final execution ./gradlew :metadata-ingestion:installDev
Remind me ./gradlew :metadata-ingestion:installDev
zsh: no such file or directory: ./gradlew

anshbansal · 2022-04-22T08:53:22Z

The final one will be in datahub folder. I missed that

iosMental · 2022-04-22T09:09:50Z

The final one will be in datahub folder. I missed that

i finally execute ./gradlew :metadata-ingestion:installDev in datahub folder Still prompt error

-lopenblas -lopenblas -o build/lib.macosx-10.14-arm64-3.8/numpy/core/_multiarray_umath.cpython-38-darwin.so" failed with exit status 1
INFO:
########### EXT COMPILER OPTIMIZATION ###########
INFO: Platform :
Architecture: aarch64
Compiler : clang

  CPU baseline  :
    Requested   : 'min'
    Enabled     : none
    Flags       : none
    Extra checks: none
  
  CPU dispatch  :
    Requested   : 'max -xop -fma4'
    Enabled     : none
    Generated   : none
  INFO: CCompilerOpt.cache_flush[817] : write cache to path -> /private/var/folders/3k/pjpmrxl91gsfwl734jv4lf4w0000gp/T/pip-install-pe7i81xb/numpy_732bfe74a3ee4e68b8c8e12bd450ca3b/build/temp.macosx-10.14-arm64-3.8/ccompiler_opt_cache_ext.py
  INFO:
  ########### CLIB COMPILER OPTIMIZATION ###########
  INFO: Platform      :
    Architecture: aarch64
    Compiler    : clang
  
  CPU baseline  :
    Requested   : 'min'
    Enabled     : none
    Flags       : none
    Extra checks: none
  
  CPU dispatch  :
    Requested   : 'max -xop -fma4'
    Enabled     : none
    Generated   : none
  INFO: CCompilerOpt.cache_flush[817] : write cache to path -> /private/var/folders/3k/pjpmrxl91gsfwl734jv4lf4w0000gp/T/pip-install-pe7i81xb/numpy_732bfe74a3ee4e68b8c8e12bd450ca3b/build/temp.macosx-10.14-arm64-3.8/ccompiler_opt_cache_clib.py
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> numpy

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

Task :metadata-ingestion:runPreFlightScript FAILED

FAILURE: Build failed with an exception.

What went wrong:
Execution failed for task ':metadata-ingestion:runPreFlightScript'.

Process 'command 'scripts/datahub_preflight.sh'' finished with non-zero exit value 1

Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
Get more help at https://help.gradle.org

Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.9.2/userguide/command_line_interface.html#sec:command_line_warnings

BUILD FAILED in 1m 0s
23 actionable tasks: 8 executed, 15 up-to-date

iosMental · 2022-04-22T09:12:49Z

In addition, I'd like to ask if this is the way ( ./gradlew :metadata-ingestion:installDev )
I want to debug metadata ingression locally

iosMental · 2022-04-22T09:21:33Z

The final one will be in datahub folder. I missed that

i finally execute ./gradlew :metadata-ingestion:installDev in datahub folder Still prompt error

-lopenblas -lopenblas -o build/lib.macosx-10.14-arm64-3.8/numpy/core/_multiarray_umath.cpython-38-darwin.so" failed with exit status 1 INFO: ########### EXT COMPILER OPTIMIZATION ########### INFO: Platform : Architecture: aarch64 Compiler : clang
  CPU baseline  :
    Requested   : 'min'
    Enabled     : none
    Flags       : none
    Extra checks: none
  
  CPU dispatch  :
    Requested   : 'max -xop -fma4'
    Enabled     : none
    Generated   : none
  INFO: CCompilerOpt.cache_flush[817] : write cache to path -> /private/var/folders/3k/pjpmrxl91gsfwl734jv4lf4w0000gp/T/pip-install-pe7i81xb/numpy_732bfe74a3ee4e68b8c8e12bd450ca3b/build/temp.macosx-10.14-arm64-3.8/ccompiler_opt_cache_ext.py
  INFO:
  ########### CLIB COMPILER OPTIMIZATION ###########
  INFO: Platform      :
    Architecture: aarch64
    Compiler    : clang
  
  CPU baseline  :
    Requested   : 'min'
    Enabled     : none
    Flags       : none
    Extra checks: none
  
  CPU dispatch  :
    Requested   : 'max -xop -fma4'
    Enabled     : none
    Generated   : none
  INFO: CCompilerOpt.cache_flush[817] : write cache to path -> /private/var/folders/3k/pjpmrxl91gsfwl734jv4lf4w0000gp/T/pip-install-pe7i81xb/numpy_732bfe74a3ee4e68b8c8e12bd450ca3b/build/temp.macosx-10.14-arm64-3.8/ccompiler_opt_cache_clib.py
  [end of output]
note: This error originates from a subprocess, and is likely not a problem with pip. error: legacy-install-failure

× Encountered error while trying to install package. ╰─> numpy

note: This is an issue with the package mentioned above, not pip. hint: See above for output from the failure.

Task :metadata-ingestion:runPreFlightScript FAILED

FAILURE: Build failed with an exception.

What went wrong:
Execution failed for task ':metadata-ingestion:runPreFlightScript'.

Process 'command 'scripts/datahub_preflight.sh'' finished with non-zero exit value 1

Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

Get more help at https://help.gradle.org

Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.9.2/userguide/command_line_interface.html#sec:command_line_warnings

BUILD FAILED in 1m 0s 23 actionable tasks: 8 executed, 15 up-to-date

and my jdk is zulu-8.jdk

iosMental · 2022-04-23T16:39:53Z

× Running setup.py install for numpy did not run successfully.
│ exit code: 1
╰─> [1352 lines of output]
Running from numpy source directory.

i try to modify datahub_preflight.sh and run success,

#pip3 install --no-binary :all: --no-use-pep517 numpy
pip3 install numpy --no-use-pep517
pip3 install pythran
#pip3 install --no-binary :all: --no-use-pep517 scipy
pip3 install scipy --no-use-pep517

iosMental added the bug Bug report label Apr 21, 2022

maggiehays assigned anshbansal Apr 21, 2022

iosMental closed this as completed Apr 23, 2022

anshbansal mentioned this issue Apr 28, 2022

fix(m1): tweak m1 preflight #4771

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gradlew :metadata-ingestion:installDev subprocess-exited-with-error #4715

gradlew :metadata-ingestion:installDev subprocess-exited-with-error #4715

iosMental commented Apr 21, 2022

anshbansal commented Apr 21, 2022

iosMental commented Apr 22, 2022

iosMental commented Apr 22, 2022

anshbansal commented Apr 22, 2022

iosMental commented Apr 22, 2022

anshbansal commented Apr 22, 2022

iosMental commented Apr 22, 2022

anshbansal commented Apr 22, 2022

iosMental commented Apr 22, 2022

iosMental commented Apr 22, 2022

iosMental commented Apr 22, 2022

iosMental commented Apr 23, 2022 •

edited

Loading

gradlew :metadata-ingestion:installDev subprocess-exited-with-error #4715

gradlew :metadata-ingestion:installDev subprocess-exited-with-error #4715

Comments

iosMental commented Apr 21, 2022

anshbansal commented Apr 21, 2022

iosMental commented Apr 22, 2022

iosMental commented Apr 22, 2022

anshbansal commented Apr 22, 2022

iosMental commented Apr 22, 2022

anshbansal commented Apr 22, 2022

iosMental commented Apr 22, 2022

anshbansal commented Apr 22, 2022

iosMental commented Apr 22, 2022

iosMental commented Apr 22, 2022

iosMental commented Apr 22, 2022

iosMental commented Apr 23, 2022 • edited Loading

iosMental commented Apr 23, 2022 •

edited

Loading