Windows wheel for Python 3.9

[richli]

I see on PyPi that Windows wheels are available for Python 3.7 and 3.8, but not Python 3.9:

https://pypi.org/project/pyshtools/4.9.1/#files

Would it be possible to build and upload a wheel for Python 3.9?

[richli]

It looks like it failed because Python 3.9 wasn't available in the Appveyor environment at the time:

https://ci.appveyor.com/project/MarkWieczorek/build-shtools/builds/40518221/job/vwuhkxhuqw2rkhle

[MarkWieczorek]

Is it available now?

[richli]

Yes, according to this:

https://www.appveyor.com/docs/windows-images-software/#python

[MarkWieczorek]

I just tried rebuilding the 3.9 version, but it failed. If you have any ideas on what the problem is, let me know!

[richli]

I have no experience with AppVeyor, but from sleuthing, it looks like you have to manually specify the "Visual Studio 2019" image:

appveyor/ci#3541 (comment)

I assume it currently defaults to one of the older images.

[richli]

Yes, if image is unspecified in the appveyor.yml file, it defaults to "Visual Studio 2015":

https://www.appveyor.com/docs/build-environment/#choosing-image-for-your-builds

[MarkWieczorek]

Adding

image:
  - Visual Studio 2019

Breaks all the appveyor builds: https://ci.appveyor.com/project/MarkWieczorek/build-shtools

I think that I put this is in the right place. If you don't have any ideas on what is happening, I'll just revert this change.

[richli]

So it's failing here because this path doesn't exist in that image:

dir "C:/Program Files/Microsoft SDKs/Windows"

It doesn't seem like this is used for anything in the script other than displaying information, so it could probably be removed. But then there may be other things later on that also have different paths...

Looking at the upstream version, they use the Visual Studio 2017 image, which doesn't include Python 3.9, but it appears they install Python 3.9 using multibuild.

[MarkWieczorek]

I remove the dir statements, but there more errors:

https://ci.appveyor.com/project/MarkWieczorek/build-shtools/build/job/uausw0qoyfd2cyqg

[richli]

Okay, so it can't find libgfortran-3.dll. But, isn't it odd that DLLs for GCC and others are used from MinGW 8.1.0, except for gfortran which uses MinGW 6.3.0? It seems like mixing and matching like that would cause issues.

[richli]

According to AppVeyor's software page, MinGW 8.1 is not in the Visual Studio 2019 image, yet apparently version 8.1.0 is present since the other DLLs were successfully copied.

I tried installing MinGW 6.3.0 and 8.1.0 in a VM and found that while libgfortran-3.dll exists in 6.3.0, there is no libgfortran-*.dll in 8.1.0. gfortran.exe exists, though, so it's not like it's not present. Maybe now it's configured so it statically links the gfortran library?

[MarkWieczorek]

Just let me know if you want me to make any specific changes (I don't have the time to debug this, and don't understand windows....).

[richli]

I also am not that familiar with doing all this on Windows. I'll try out some things and get back to you -- I suspect that libgfortran-3.dll may not be used within the wheel and so it doesn't need to be bundled.

[richli]

I fiddled around with things and can report back: no, libgfortran-3.dll is required due to the specific pre-built LAPACK DLLs used.

I see a couple of possible options:

1. Continue with the "Visual Studio 2019" AppVeyor image and either download and extract that one gfortran DLL from the mingw-w64 archive (e.g., download and extract that one element from the x86_64-6.4.0-release-posix-seh-rt_v5-rev0.7z file, or use chocolatey to install the older mingw-w64 version, like how the scipy-wheels appveyor.yml does).

2. Revert back to the "Visual Studio 2015" AppVeyor image. Match what the scipy-wheels appveyor.yml is doing to call multibuild/install_python.ps1 so it installs Python, if missing, into the image. In this case, only Python 3.9 would actually need to be installed since the AppVeyor image contains the older versions already.

I think the second option may be a little cleaner, and it should make it easier to support Python 3.10 that way, since none of the current AppVeyor images include it yet. (Although other dependencies, such as scipy, don't have wheels for Python 3.10 up on PyPI yet, so it won't work until later...)

[richli]

If I submit a pull request on https://github.com/SHTOOLS/build-shtools, will AppVeyor run on that branch?

[MarkWieczorek]

You can try, I have no idea. If it doesn't, I'll see what I need to do to trigger it.

I would really like to get the windows CI to work with github actions, but I don't know enough about windows to do this myself.

[richli]

I have no experience with Github Actions, but FYI I see that they do support Windows:

https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners#preinstalled-software

[richli]

In the PR I submitted, the wheel for Python 3.9 is successfully generated, although the jobs fail during the PyPI upload. I presume that once it gets merged in, the next build will be fully successful.

[MarkWieczorek]

It works! Thanks for looking into this. If you have any other suggestions for the windows builds, feel free to make a new PR!