Add pyproject.toml specifying numpy and cython as build deps #730

aaronayres35 · 2021-05-05T17:59:34Z

closes #372

It adds a pyproject.toml specifying cython and numpy as build deps.

Note I tried building an sdist in my bootstrap env (has effectively everything installed, generally with editable installs for ets things), then created a fresh env and tried pip installing the sdist.

Previously I had seen:

$ pip install chaco-4.8.1.dev187.tar.gz
Processing ./chaco-4.8.1.dev187.tar.gz
    ERROR: Command errored out with exit status 1:
     command: /Users/aayres/.edm/envs/dumb/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/j9/ps8s26f91_1_lgf7mtd4smyr0000gn/T/pip-req-build-4u0ttll3/setup.py'"'"'; __file__='"'"'/private/var/folders/j9/ps8s26f91_1_lgf7mtd4smyr0000gn/T/pip-req-build-4u0ttll3/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/j9/ps8s26f91_1_lgf7mtd4smyr0000gn/T/pip-req-build-4u0ttll3/pip-egg-info
         cwd: /private/var/folders/j9/ps8s26f91_1_lgf7mtd4smyr0000gn/T/pip-req-build-4u0ttll3/
    Complete output (5 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/j9/ps8s26f91_1_lgf7mtd4smyr0000gn/T/pip-req-build-4u0ttll3/setup.py", line 7, in <module>
        from numpy import get_include
    ModuleNotFoundError: No module named 'numpy'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

and then after installing numpy:

$ pip install chaco-4.8.1.dev187.tar.gz
Processing ./chaco-4.8.1.dev187.tar.gz
    ERROR: Command errored out with exit status 1:
     command: /Users/aayres/.edm/envs/dumb/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/j9/ps8s26f91_1_lgf7mtd4smyr0000gn/T/pip-req-build-1uijyzf6/setup.py'"'"'; __file__='"'"'/private/var/folders/j9/ps8s26f91_1_lgf7mtd4smyr0000gn/T/pip-req-build-1uijyzf6/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/j9/ps8s26f91_1_lgf7mtd4smyr0000gn/T/pip-req-build-1uijyzf6/pip-egg-info
         cwd: /private/var/folders/j9/ps8s26f91_1_lgf7mtd4smyr0000gn/T/pip-req-build-1uijyzf6/
    Complete output (5 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/j9/ps8s26f91_1_lgf7mtd4smyr0000gn/T/pip-req-build-1uijyzf6/setup.py", line 9, in <module>
        from Cython.Build import cythonize
    ModuleNotFoundError: No module named 'Cython'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Now, I see:

$ pip install chaco-4.8.1.dev188.tar.gz
Processing ./chaco-4.8.1.dev188.tar.gz
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  ERROR: Command errored out with exit status 1:
   command: /Users/aayres/.edm/envs/dumb/bin/python3 /Users/aayres/.edm/envs/dumb/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /var/folders/j9/ps8s26f91_1_lgf7mtd4smyr0000gn/T/tmp0e0ib9ww
       cwd: /private/var/folders/j9/ps8s26f91_1_lgf7mtd4smyr0000gn/T/pip-req-build-szg8gt4a
  Complete output (16 lines):
  Traceback (most recent call last):
    File "/Users/aayres/.edm/envs/dumb/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py", line 257, in <module>
      main()
    File "/Users/aayres/.edm/envs/dumb/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py", line 240, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/Users/aayres/.edm/envs/dumb/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py", line 91, in get_requires_for_build_wheel
      return hook(config_settings)
    File "/private/var/folders/j9/ps8s26f91_1_lgf7mtd4smyr0000gn/T/pip-build-env-k7vxu8mp/overlay/lib/python3.6/site-packages/setuptools/build_meta.py", line 155, in get_requires_for_build_wheel
      config_settings, requirements=['wheel'])
    File "/private/var/folders/j9/ps8s26f91_1_lgf7mtd4smyr0000gn/T/pip-build-env-k7vxu8mp/overlay/lib/python3.6/site-packages/setuptools/build_meta.py", line 135, in _get_build_requires
      self.run_setup()
    File "/private/var/folders/j9/ps8s26f91_1_lgf7mtd4smyr0000gn/T/pip-build-env-k7vxu8mp/overlay/lib/python3.6/site-packages/setuptools/build_meta.py", line 150, in run_setup
      exec(compile(code, __file__, 'exec'), locals())
    File "setup.py", line 132, in <module>
      from chaco import __requires__, __version__
  ModuleNotFoundError: No module named 'chaco'
  ----------------------------------------
ERROR: Command errored out with exit status 1: /Users/aayres/.edm/envs/dumb/bin/python3 /Users/aayres/.edm/envs/dumb/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /var/folders/j9/ps8s26f91_1_lgf7mtd4smyr0000gn/T/tmp0e0ib9ww Check the logs for full command output.

which feels very strange, and I admittedly don't understand the problem right now / I haven't investigated. Perhaps this is a bug, perhaps I am doing something wrong / dumb. In any case, I believe the changes in this PR are changes we want so I am opening the PR now.

EDIT: this PR now closes #534

rahulporuri · 2021-05-10T05:39:19Z

Like we discussed offline, i think we need to update the setup.py in chaco to not import __version__ and __requires__ from chaco.__init__. Once that is fixed, I suspect that the weird failures that you saw will disappear.

rahulporuri · 2021-05-11T05:19:14Z

setup.py

+# If this file is part of a Git export (for example created with "git archive",
+# or downloaded from GitHub), ARCHIVE_COMMIT_HASH gives the full hash of the
+# commit that was exported.
+ARCHIVE_COMMIT_HASH = "$Format:%H$"


you'll have to add a .gitattributes file to the root of the git repository if you want this to work correctly. ref https://github.com/enthought/traitsui/blob/master/.gitattributes

rahulporuri · 2021-05-11T05:38:23Z

I tested this PR as follows -

I created a new chaco-build environment, ran python -m pip install --upgrade pip setuptools wheel and created the sdist using python setup.py sdist
Then from a completely new chaco-install environment, ran python -m pip install --upgrade pip setuptools wheel and tried installing from the sdist using cd dist and python -m pip install chaco-4.8.1.dev189.tar.gz.

This then gave rise to the error

> python -m pip install chaco-4.8.1.dev189.tar.gz
Processing c:\users\rporuri\work\github\ets\chaco\dist\chaco-4.8.1.dev189.tar.gz
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting pyface
  Using cached pyface-7.3.0-py3-none-any.whl
Collecting traitsui
  Using cached traitsui-7.1.1-py3-none-any.whl
Collecting traits
  Using cached traits-6.2.0-cp36-cp36m-win_amd64.whl (5.0 MB)
Collecting numpy
  Using cached numpy-1.19.5-cp36-cp36m-win_amd64.whl (13.2 MB)
Collecting enable
  Using cached enable-5.1.0.tar.gz (3.0 MB)
    ERROR: Command errored out with exit status 1:
     command: 'C:\Users\rporuri\.edm\envs\temp-install\python.exe' -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\rporuri\\AppData\\Local\\Temp\\pip-install-uqt5se6v\\enable_1a8aa81e0927462ba8f773a3822a49e0\\setup.py'"'"'; __file__='"'"'C:\\Users\\rporuri\\AppData\\Local\\Temp\\pip-install-uqt5se6v\\enable_1a8aa81e0927462ba8f773a3822a49e0\\setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base 'C:\Users\rporuri\AppData\Local\Temp\pip-pip-egg-info-2dk2esl1'
         cwd: C:\Users\rporuri\AppData\Local\Temp\pip-install-uqt5se6v\enable_1a8aa81e0927462ba8f773a3822a49e0\
    Complete output (5 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:\Users\rporuri\AppData\Local\Temp\pip-install-uqt5se6v\enable_1a8aa81e0927462ba8f773a3822a49e0\setup.py", line 18, in <module>
        from Cython.Distutils import build_ext
    ModuleNotFoundError: No module named 'Cython'
    ----------------------------------------

From the looks of it, this should be fixed when the same changes in enable are available in a release.

aaronayres35 · 2021-05-13T19:34:37Z

In the latest commit I require enable master not a released version of enable (undo this commit before merging).
Previously, in a fresh env running pip install -e . yielded:

$ pip install -e .
Obtaining file:///Users/aayres/Desktop/chaco
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting traits
  Using cached traits-6.2.0-cp36-cp36m-macosx_10_14_x86_64.whl (5.0 MB)
Collecting numpy
  Using cached numpy-1.19.5-cp36-cp36m-macosx_10_9_x86_64.whl (15.6 MB)
Collecting traitsui
  Downloading traitsui-7.2.1.tar.gz (6.7 MB)
     |████████████████████████████████| 6.7 MB 4.3 MB/s 
Collecting enable
  Using cached enable-5.1.0.tar.gz (3.0 MB)
    ERROR: Command errored out with exit status 1:
     command: /Users/aayres/.edm/envs/dumb/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/j9/ps8s26f91_1_lgf7mtd4smyr0000gn/T/pip-install-0q0n0df4/enable/setup.py'"'"'; __file__='"'"'/private/var/folders/j9/ps8s26f91_1_lgf7mtd4smyr0000gn/T/pip-install-0q0n0df4/enable/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/j9/ps8s26f91_1_lgf7mtd4smyr0000gn/T/pip-install-0q0n0df4/enable/pip-egg-info
         cwd: /private/var/folders/j9/ps8s26f91_1_lgf7mtd4smyr0000gn/T/pip-install-0q0n0df4/enable/
    Complete output (5 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/j9/ps8s26f91_1_lgf7mtd4smyr0000gn/T/pip-install-0q0n0df4/enable/setup.py", line 18, in <module>
        from Cython.Distutils import build_ext
    ModuleNotFoundError: No module named 'Cython'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

and now:

$ pip install -e .
Obtaining file:///Users/aayres/Desktop/chaco
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting traitsui
  Using cached traitsui-7.2.1.tar.gz (6.7 MB)
Collecting numpy
  Using cached numpy-1.19.5-cp36-cp36m-macosx_10_9_x86_64.whl (15.6 MB)
Processing /Users/aayres/Library/Caches/pip/wheels/c9/24/5a/1231d8847ac0ff839cc5b7810612b851318a6d2c66a891c031/pyface-7.3.0-py3-none-any.whl
Collecting enable@ git+http://github.com/enthought/enable.git#egg=enable
  Cloning http://github.com/enthought/enable.git to /private/var/folders/j9/ps8s26f91_1_lgf7mtd4smyr0000gn/T/pip-install-ovasowiu/enable
  Running command git clone -q http://github.com/enthought/enable.git /private/var/folders/j9/ps8s26f91_1_lgf7mtd4smyr0000gn/T/pip-install-ovasowiu/enable
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  ERROR: Command errored out with exit status 1:
   command: /Users/aayres/.edm/envs/dumb/bin/python3 /Users/aayres/.edm/envs/dumb/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /var/folders/j9/ps8s26f91_1_lgf7mtd4smyr0000gn/T/tmp3g92bo64
       cwd: /private/var/folders/j9/ps8s26f91_1_lgf7mtd4smyr0000gn/T/pip-install-ovasowiu/enable
  Complete output (29 lines):
  Traceback (most recent call last):
    File "setup.py", line 441, in verify_swig_version
      with subprocess.Popen(cmd, **opts) as proc:
    File "/Users/aayres/.edm/envs/dumb/lib/python3.6/subprocess.py", line 729, in __init__
      restore_signals, start_new_session)
    File "/Users/aayres/.edm/envs/dumb/lib/python3.6/subprocess.py", line 1364, in _execute_child
      raise child_exception_type(errno_num, err_msg, err_filename)
  FileNotFoundError: [Errno 2] No such file or directory: 'swig': 'swig'
  
  During handling of the above exception, another exception occurred:
  
  Traceback (most recent call last):
    File "/Users/aayres/.edm/envs/dumb/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py", line 257, in <module>
      main()
    File "/Users/aayres/.edm/envs/dumb/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py", line 240, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/Users/aayres/.edm/envs/dumb/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py", line 91, in get_requires_for_build_wheel
      return hook(config_settings)
    File "/private/var/folders/j9/ps8s26f91_1_lgf7mtd4smyr0000gn/T/pip-build-env-pynytf0r/overlay/lib/python3.6/site-packages/setuptools/build_meta.py", line 155, in get_requires_for_build_wheel
      config_settings, requirements=['wheel'])
    File "/private/var/folders/j9/ps8s26f91_1_lgf7mtd4smyr0000gn/T/pip-build-env-pynytf0r/overlay/lib/python3.6/site-packages/setuptools/build_meta.py", line 135, in _get_build_requires
      self.run_setup()
    File "/private/var/folders/j9/ps8s26f91_1_lgf7mtd4smyr0000gn/T/pip-build-env-pynytf0r/overlay/lib/python3.6/site-packages/setuptools/build_meta.py", line 150, in run_setup
      exec(compile(code, __file__, 'exec'), locals())
    File "setup.py", line 454, in <module>
      verify_swig_version()
    File "setup.py", line 450, in verify_swig_version
      raise Exception(msg)
  Exception: SWIG is a required build dependency of Enable. Furthermore, there is currently a known issue with SWIG 4.0, see enthought/enable#360. Please install SWIG 3.0.x (see http://www.swig.org/).
  ----------------------------------------
ERROR: Command errored out with exit status 1: /Users/aayres/.edm/envs/dumb/bin/python3 /Users/aayres/.edm/envs/dumb/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /var/folders/j9/ps8s26f91_1_lgf7mtd4smyr0000gn/T/tmp3g92bo64 Check the logs for full command output.

If I then install swig (e.g edm install swig which gets version 3.0.11, but generally this is assuming people using pip not edm, so something like brew install swig@3 would be equivalent) and run pip install -e . now I get:

$ pip install -e .
Obtaining file:///Users/aayres/Desktop/chaco
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting traitsui
  Using cached traitsui-7.2.1.tar.gz (6.7 MB)
Collecting enable@ git+http://github.com/enthought/enable.git#egg=enable
  Cloning http://github.com/enthought/enable.git to /private/var/folders/j9/ps8s26f91_1_lgf7mtd4smyr0000gn/T/pip-install-qeoapmny/enable
  Running command git clone -q http://github.com/enthought/enable.git /private/var/folders/j9/ps8s26f91_1_lgf7mtd4smyr0000gn/T/pip-install-qeoapmny/enable
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting numpy
  Using cached numpy-1.19.5-cp36-cp36m-macosx_10_9_x86_64.whl (15.6 MB)
Collecting traits
  Using cached traits-6.2.0-cp36-cp36m-macosx_10_14_x86_64.whl (5.0 MB)
Processing /Users/aayres/Library/Caches/pip/wheels/c9/24/5a/1231d8847ac0ff839cc5b7810612b851318a6d2c66a891c031/pyface-7.3.0-py3-none-any.whl
Collecting pillow
  Using cached Pillow-8.2.0-cp36-cp36m-macosx_10_10_x86_64.whl (2.8 MB)
Collecting fonttools
  Downloading fonttools-4.23.0-py3-none-any.whl (852 kB)
     |████████████████████████████████| 852 kB 2.9 MB/s 
Collecting importlib-resources>=1.1.0
  Downloading importlib_resources-5.1.3-py3-none-any.whl (26 kB)
Collecting importlib-metadata
  Using cached importlib_metadata-4.0.1-py3-none-any.whl (16 kB)
Collecting zipp>=0.4; python_version < "3.8"
  Using cached zipp-3.4.1-py3-none-any.whl (5.2 kB)
Collecting typing-extensions>=3.6.4; python_version < "3.8"
  Using cached typing_extensions-3.10.0.0-py3-none-any.whl (26 kB)
Building wheels for collected packages: enable
  Building wheel for enable (PEP 517) ... done
  Created wheel for enable: filename=enable-5.1.0.dev286-cp36-cp36m-macosx_10_9_x86_64.whl size=2488439 sha256=dc8df8fd005dd280a421c9ae3ccf09b85a0a49db4d239d8df709ee4978f2cbd1
  Stored in directory: /private/var/folders/j9/ps8s26f91_1_lgf7mtd4smyr0000gn/T/pip-ephem-wheel-cache-ojer6h4z/wheels/00/d3/b6/ea2d107394c1247867818d1631d0ca312cdb2880da2f77099e
Successfully built enable
Installing collected packages: traits, zipp, importlib-resources, typing-extensions, importlib-metadata, pyface, traitsui, pillow, numpy, fonttools, enable, chaco
    Running setup.py install for traitsui ... done
  Running setup.py develop for chaco
Successfully installed chaco enable-5.1.0.dev286 fonttools-4.23.0 importlib-metadata-4.0.1 importlib-resources-5.1.3 numpy-1.19.5 pillow-8.2.0 pyface-7.3.0 traits-6.2.0 traitsui-7.2.1 typing-extensions-3.10.0.0 zipp-3.4.1

rahulporuri

Looks like this isn't a WIP anymore so LGTM.

I tested this locally and the only error i got was regarding the missing swig. If i fixi that using edm install swig -y, I don't run into any errors and the installation is successfull!

chaco/__init__.py

Co-authored-by: Poruri Sai Rahul <rporuri@enthought.com>

add pyproject.toml specifying numpy and cython as build deps

34acfcc

rahulporuri self-requested a review May 6, 2021 08:49

aaronayres35 changed the title ~~add pyproject.toml specifying numpy and cython as build deps~~ [WIP] add pyproject.toml specifying numpy and cython as build deps May 10, 2021

follow traitsui setup.py

2fc6de5

rahulporuri reviewed May 11, 2021

View reviewed changes

aaronayres35 mentioned this pull request May 12, 2021

Add "examples" section to extras_require in setup.py enthought/envisage#391

Open

aaronayres35 added 2 commits May 13, 2021 07:29

add .gitattributes

b93f3c4

REVERT THIS. Use with enable master to test things work

1d1a505

flake8 so CI passes but this will be reverted as well

e96428a

aaronayres35 requested a review from rahulporuri May 19, 2021 13:59

rahulporuri approved these changes Jun 1, 2021

View reviewed changes

rahulporuri reviewed Jun 1, 2021

View reviewed changes

chaco/__init__.py Outdated Show resolved Hide resolved

rahulporuri changed the title ~~[WIP] add pyproject.toml specifying numpy and cython as build deps~~ dd pyproject.toml specifying numpy and cython as build deps Jun 1, 2021

rahulporuri changed the title ~~dd pyproject.toml specifying numpy and cython as build deps~~ Add pyproject.toml specifying numpy and cython as build deps Jun 1, 2021

dont pull enable source

eaed4cc

Co-authored-by: Poruri Sai Rahul <rporuri@enthought.com>

aaronayres35 merged commit c3820ff into master Jun 1, 2021

aaronayres35 deleted the improve-pypi branch June 1, 2021 12:04

aaronayres35 mentioned this pull request Jun 2, 2021

Add pyproject.toml to MANIFEST.in #747

Merged

mdickinson mentioned this pull request Jun 11, 2021

Built Chaco may be binary-incompatible with older NumPy versions #778

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add pyproject.toml specifying numpy and cython as build deps #730

Add pyproject.toml specifying numpy and cython as build deps #730

aaronayres35 commented May 5, 2021 •

edited

Loading

rahulporuri commented May 10, 2021

rahulporuri May 11, 2021

rahulporuri commented May 11, 2021

aaronayres35 commented May 13, 2021

rahulporuri left a comment

Add pyproject.toml specifying numpy and cython as build deps #730

Add pyproject.toml specifying numpy and cython as build deps #730

Conversation

aaronayres35 commented May 5, 2021 • edited Loading

rahulporuri commented May 10, 2021

rahulporuri May 11, 2021

Choose a reason for hiding this comment

rahulporuri commented May 11, 2021

aaronayres35 commented May 13, 2021

rahulporuri left a comment

Choose a reason for hiding this comment

aaronayres35 commented May 5, 2021 •

edited

Loading