enable windows wheels #24

grlee77 · 2018-11-08T19:31:52Z

This is built on top of #22. The new commits start from 94367ca

grlee77 · 2018-11-08T19:41:22Z

It looks like Appveyor needs to be configured to monitor this repo.

I cancelled Travis to save CI resources.

jakirkham · 2018-11-08T20:17:39Z

Thanks for looking at this, @grlee77.

I looked in AppVeyor to see if I could add this to the org, but seem to lack the permissions. Guessing this is the same for you.

@hgomersall, would you be able to look? FWIW AppVeyor has made some major improvements to their UI.

grlee77 · 2018-11-16T17:00:50Z

Okay, Appveyor is now trying to build this. Thanks @hgomersall.

I will try to take a look at this PR again later today.

hgomersall · 2018-11-16T17:06:39Z

Great, thanks!

grlee77 · 2018-11-16T18:14:22Z

For some reason the submodule initialization on Appveyor is failing with:

git submodule update --init
Submodule 'multibuild' (https://github.com/matthew-brett/multibuild.git) registered for path 'multibuild'
Submodule 'pyFFTW' (https://github.com/pyFFTW/pyFFTW) registered for path 'pyFFTW'
Cloning into 'C:/projects/pyfftw-wheels/multibuild'...
fatal: destination path 'C:/projects/pyfftw-wheels/pyFFTW' already exists and is not an empty directory.
fatal: clone of 'https://github.com/pyFFTW/pyFFTW' into submodule path 'C:/projects/pyfftw-wheels/pyFFTW' failed
Failed to clone 'pyFFTW'. Retry scheduled
fatal: destination path 'C:/projects/pyfftw-wheels/pyFFTW' already exists and is not an empty directory.
fatal: clone of 'https://github.com/pyFFTW/pyFFTW' into submodule path 'C:/projects/pyfftw-wheels/pyFFTW' failed
Failed to clone 'pyFFTW' a second time, aborting
Command exited with code 1

any ideas?

hgomersall · 2018-11-16T18:15:59Z

Is it being created during the build, and then the deployment is failing? I seem to recall something like this on the previous way of doing things - the deploy script needed to delete the directory before building the wheel (or something).

This was based off of a combination of the script at scikit-image-wheels with the scripts in the appveyor subfolder added from the main pyFFTW repository

…d and run_with_env.cmd

grlee77 · 2018-11-16T19:50:01Z

Okay. I see what happened now. The FFTW .dlls were placed in a pyFFTW subdirectory, but submodule init cannot be run after that is done.

grlee77 · 2018-11-16T21:07:30Z

as of commit 4ca0f6e, builds are completing successfully, but there was a remaining issue of trying to make sure we are not doing a pre-release build that I am still working on in d81a121.

I am unable to cancel pending Appveyor builds, but that is probably an issue related to collaborator permissions on Appveyor.

One outstanding question I could use some help with: There is some MSVC configuration going in the main .appveyor.yml that I borrowed from the scikit-image wheel building repo:

  # Fix MSVC builds for 64-bit Python. See:
  # http://stackoverflow.com/questions/32091593/cannot-install-windows-sdk-7-1-on-windows-10
  - echo "C:\Program Files\Microsoft SDKs\Windows\v7.1\Bin\SetEnv.cmd" /x64 > "C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\bin\amd64/vcvars64.bat"

I am not sure if this conflicts with some related settings that are going on within the run_with_env.cmd file I copied from the main repo. There, it seems to be selecting windows SDK version v7.0 for Python 2.7 and v7.1 for the rest. Do you remember why this was and do you think it is still necessary? I am wondering if the stuff in run_with_env.cmd is even necessary to have within this repo?

hgomersall · 2018-11-16T21:15:24Z

This is great stuff, well done!

jakirkham · 2018-11-16T21:16:40Z

AFAIK VS 2010 was only used with Python 3.4. If we are not supporting Python3.4, that line can probably be dropped.

hgomersall · 2018-11-16T21:17:50Z

@jakirkham sorry, can you clarify what you need? Do you want to be made a collaborator too?

jakirkham · 2018-11-16T21:27:02Z

Sure, that sounds good.

Though my comment was more about the issue @grlee77 raised about the .appveyor.yml lines for 64-bit builds. Sorry that was unclear.

…build wheels based on a release tag

grlee77 · 2018-11-16T21:40:18Z

One issue in the d81a121 builds is that the CONTAINER variable was still getting set improperly to "pre-release" instead of "wheels". I think we would have to be building from a different branch than master on pyFFTW-wheels if we want the logic to work properly as it does for scikit-image-wheels. This variable effects where the wheels get uploaded to so it needs to be set to "wheels" so that the wheels don't end up in the pre-release location.

I am going to simplify by removing this pre-release logic as I don't think we need it here. I think just building from release tags is sufficient for now.

grlee77 · 2018-11-16T21:45:29Z

feel free to cancel builds 1.0.11 and 1.0.12 on Appveyor. I only need to see the result for the final commit.

grlee77 · 2018-11-16T22:56:54Z

Okay. This is ready for review.

The final .appveyor.yml here is a mixture of the one from scikit-image-wheels and a slightly simpler one from pywavelets-wheels with download/configuration of the precompiled FFTW libraries done using two scripts in the appveyor subfolder that were copied from the main pyFFTW repo.

If there is still a need for 32-bit windows wheels, they can be added, but I have only built the 64-bit ones for now.

grlee77 · 2018-11-19T22:35:48Z

Okay, this is all green now with 0.11.1.
It just needs someone from @pyfftw/ci to review and merge.

@hgomersall: can you add me to the ci team under the pyFFTW organization on GitHub? I would like to be able to review and merge future PRs from others in this repo.

README.rst

jakirkham · 2018-11-21T21:00:04Z

.appveyor.yml

+  # Check Python version just in case
+  - python --version
+  # Run unit tests with pytest
+  - "python setup.py test"


Should we cleanup the downloaded FFTW DLLs before this step to ensure it is picking the ones in the wheel up?

Looking more closely, the wheel is not actually being used during testing (or even getting installed!). This is because of the following build using:
python setup.py -v build_ext --inplace
that gets used for the tests. This approach was copied from the appveyor script in the main repo.

I can comment out that line installing the wheel as it is not doing anything. The tests do not actually get distributed with the source .tar.gz or the wheels, so I am actually not even sure how to even run the tests without using an inplace build. The comment mentions "pytest", but this was a copy/paste error. Tests effectively get run via:
python -m unittest discover and this only works for me if I do it from the top level pyfftw folder with an in-place build.

I did verify on a Windows 7 machine that downloading the Python 3.5 wheel from the Appveyor artifacts and installing it does import and run basic ffts okay and the expected dynamic libraries are in the .whl.

Usually you can run the tests as long as the package is installed somewhere the Python interpreter can find it. So would try pip installing the wheel, cleaning the source directory, and running the tests with python -m unittest discover.

That said, it's not obvious to me if we are running the tests for the other platforms. So maybe this is good enough as is.

Tests are not currently being run for OS X or linux. That is issue #12 and was being worked on in #11.

I think we have more variants in this repo than we test in the main pyFFTW one, so it would probably be worth having the tests all running here as well at some point.

I tried testing as you suggest locally (i.e. installing from the wheel, cleaning the source directory with git clean -xfd and then running tests via python -m unittest discover. It finds the tests, but I get a ton of ImportErrors.

C:\Users\Greg\src\pyFFTW>python -m unittest discover EEEEEEEEEEEEEEEEEE ====================================================================== ERROR: pyfftw (unittest.loader._FailedTest) ---------------------------------------------------------------------- ImportError: Failed to import test module: pyfftw Traceback (most recent call last): File "F:\Miniconda3\lib\unittest\loader.py", line 462, in _find_test_path package = self._get_module_from_name(name) File "F:\Miniconda3\lib\unittest\loader.py", line 369, in _get_module_from_name __import__(name) File "C:\Users\Greg\src\pyFFTW\pyfftw\__init__.py", line 18, in <module> from .pyfftw import ( ImportError: No module named 'pyfftw.pyfftw' ====================================================================== ERROR: test.test_pyfftw_base (unittest.loader._FailedTest) ---------------------------------------------------------------------- ImportError: Failed to import test module: test.test_pyfftw_base Traceback (most recent call last): File "F:\Miniconda3\lib\unittest\loader.py", line 428, in _find_test_path module = self._get_module_from_name(name) File "F:\Miniconda3\lib\unittest\loader.py", line 369, in _get_module_from_name __import__(name) File "C:\Users\Greg\src\pyFFTW\test\test_pyfftw_base.py", line 35, in <module> from pyfftw import FFTW, _supported_types, _all_types_human_readable File "C:\Users\Greg\src\pyFFTW\pyfftw\__init__.py", line 18, in <module> from .pyfftw import ( ImportError: No module named 'pyfftw.pyfftw'

If I first cd test to move out of the root source directory and into the test subfolder. I get similar errors, but this time about invalid relative imports.

====================================================================== ERROR: test_pyfftw_builders (unittest.loader._FailedTest) ---------------------------------------------------------------------- ImportError: Failed to import test module: test_pyfftw_builders Traceback (most recent call last): File "F:\Miniconda3\lib\unittest\loader.py", line 428, in _find_test_path module = self._get_module_from_name(name) File "F:\Miniconda3\lib\unittest\loader.py", line 369, in _get_module_from_name __import__(name) File "C:\Users\Greg\src\pyFFTW\test\test_pyfftw_builders.py", line 38, in <module> from .test_pyfftw_base import run_test_suites, require ImportError: attempted relative import with no known parent package

.appveyor.yml

jakirkham · 2018-11-21T21:04:19Z

Looks pretty good to me.

I don't totally understand everything going on here, but IDK if it's necessary for me to understand it if you do. Did ask a couple questions above though.

For now tests are run from an inplace build performed immediately after the wheel build. It is probably preferable to install the wheel and use that for running the tests, but this may require some changes to upstream pyFFTW before it can be implemented here.

grlee77 · 2018-12-01T09:44:32Z

Can someone please merge this if there are no strong objections? Not having wheels on PyPI is resulting in a lot of issues being raised at the main repo.

grlee77 · 2018-12-06T18:27:17Z

ping. can we merge this soon and work out improvements to running the tests in the future?

hgomersall

I'm happy

hgomersall · 2018-12-06T18:33:34Z

Apologies for the delay on this, I'm rather busy at the moment!

grlee77 · 2018-12-06T19:45:55Z

no problem. thanks @hgomersall

grlee77 · 2018-12-07T04:05:58Z

I do not see the Windows wheels on Rackspace, but they were built and are available via the Appveyor artifacts tab. Unfortunately the linux/OS X wheels got built from the latest commit and not the 0.11.1 tag, so I have a new PR at #27 to try and fix that prior to uploading everything to PyPI.

jakirkham mentioned this pull request Nov 8, 2018

Migrating Windows wheels build/deploy process #23

Open

This was referenced Nov 9, 2018

try building v0.11.0 #22

Merged

planning the next pyFFTW release pyFFTW/pyFFTW#187

Open

grlee77 closed this Nov 16, 2018

grlee77 reopened this Nov 16, 2018

grlee77 added 5 commits November 16, 2018 14:47

add .appveyor.yaml

85ee081

This was based off of a combination of the script at scikit-image-wheels with the scripts in the appveyor subfolder added from the main pyFFTW repository

update the README.rst to mention the Appveyor builds

1bf162a

define PYTHON_ARCH and PYTHON_VERSION as needed by setup_fftw_dlls.cm…

2f12e1f

…d and run_with_env.cmd

fix BUILD_COMMIT in .appveyor.yml

ea9f399

minor refactoring in .appveyor.yaml

3514a30

grlee77 force-pushed the add_windows branch from f10230e to 3514a30 Compare November 16, 2018 19:47

grlee77 added 4 commits November 16, 2018 14:54

move call to setup_fftw_dlls.cmd after the git submodule update

8514749

fix path to requirements.txt

e806742

fix path to requirements.txt (again)

3bf0cdc

add additional statements for debugging and set PYFFTW_LIB_DIR

c89111e

grlee77 closed this Nov 16, 2018

grlee77 reopened this Nov 16, 2018

grlee77 added 5 commits November 16, 2018 15:21

fix typo in PYTHON_VERSION

3e356fb

additional debugging

b9e237b

extract FFTW libs from within REPO_DIR

4ca0f6e

try removing hard-coded tag in git checkout

d81a121

minor cleanup

e1b28d6

grlee77 added 2 commits November 16, 2018 16:35

remove pre-release wheel building logic. most likely we only want to …

571197d

…build wheels based on a release tag

remove unneeded MSVC10-specific code since we don't build for Python 3.4

4c4c394

wheels -> --no-update-index wheels

700c5be

grlee77 changed the title ~~WIP: try adding windows builds~~ enable windows wheels Nov 16, 2018

This was referenced Nov 19, 2018

remove pypi deployment in favor of pyFFTW-wheels pyFFTW/pyFFTW#251

Merged

bump version to v0.11.1 #26

Closed

bump version

9ff52cb

grlee77 mentioned this pull request Nov 20, 2018

Pip install broken since v0.11.1 pyFFTW/pyFFTW#252

Closed

jakirkham reviewed Nov 21, 2018

View reviewed changes

README.rst Show resolved Hide resolved

jakirkham reviewed Nov 21, 2018

View reviewed changes

.appveyor.yml Show resolved Hide resolved

grlee77 mentioned this pull request Dec 1, 2018

Could not find any FFTW libraries pyFFTW/pyFFTW#255

Closed

hgomersall approved these changes Dec 6, 2018

View reviewed changes

grlee77 merged commit 3893c02 into pyFFTW:master Dec 6, 2018

grlee77 deleted the add_windows branch February 3, 2020 17:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

enable windows wheels #24

enable windows wheels #24

grlee77 commented Nov 8, 2018

grlee77 commented Nov 8, 2018

jakirkham commented Nov 8, 2018

grlee77 commented Nov 16, 2018

hgomersall commented Nov 16, 2018

grlee77 commented Nov 16, 2018

hgomersall commented Nov 16, 2018

grlee77 commented Nov 16, 2018

grlee77 commented Nov 16, 2018

hgomersall commented Nov 16, 2018

jakirkham commented Nov 16, 2018

hgomersall commented Nov 16, 2018

jakirkham commented Nov 16, 2018

grlee77 commented Nov 16, 2018

grlee77 commented Nov 16, 2018

grlee77 commented Nov 16, 2018

grlee77 commented Nov 19, 2018

jakirkham Nov 21, 2018

grlee77 Nov 22, 2018 •

edited

Loading

jakirkham Nov 27, 2018

jakirkham Nov 27, 2018

grlee77 Nov 28, 2018

grlee77 Nov 28, 2018

grlee77 Nov 28, 2018 •

edited

Loading

jakirkham commented Nov 21, 2018

grlee77 commented Dec 1, 2018

grlee77 commented Dec 6, 2018

hgomersall left a comment

hgomersall commented Dec 6, 2018

grlee77 commented Dec 6, 2018

grlee77 commented Dec 7, 2018

enable windows wheels #24

enable windows wheels #24

Conversation

grlee77 commented Nov 8, 2018

grlee77 commented Nov 8, 2018

jakirkham commented Nov 8, 2018

grlee77 commented Nov 16, 2018

hgomersall commented Nov 16, 2018

grlee77 commented Nov 16, 2018

hgomersall commented Nov 16, 2018

grlee77 commented Nov 16, 2018

grlee77 commented Nov 16, 2018

hgomersall commented Nov 16, 2018

jakirkham commented Nov 16, 2018

hgomersall commented Nov 16, 2018

jakirkham commented Nov 16, 2018

grlee77 commented Nov 16, 2018

grlee77 commented Nov 16, 2018

grlee77 commented Nov 16, 2018

grlee77 commented Nov 19, 2018

jakirkham Nov 21, 2018

Choose a reason for hiding this comment

grlee77 Nov 22, 2018 • edited Loading

Choose a reason for hiding this comment

jakirkham Nov 27, 2018

Choose a reason for hiding this comment

jakirkham Nov 27, 2018

Choose a reason for hiding this comment

grlee77 Nov 28, 2018

Choose a reason for hiding this comment

grlee77 Nov 28, 2018

Choose a reason for hiding this comment

grlee77 Nov 28, 2018 • edited Loading

Choose a reason for hiding this comment

jakirkham commented Nov 21, 2018

grlee77 commented Dec 1, 2018

grlee77 commented Dec 6, 2018

hgomersall left a comment

Choose a reason for hiding this comment

hgomersall commented Dec 6, 2018

grlee77 commented Dec 6, 2018

grlee77 commented Dec 7, 2018

grlee77 Nov 22, 2018 •

edited

Loading

grlee77 Nov 28, 2018 •

edited

Loading