Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI(OSGeo4W): Run pytest tests on Windows #4366

Merged
merged 4 commits into from
Sep 23, 2024

Conversation

echoix
Copy link
Member

@echoix echoix commented Sep 23, 2024

This is the next step of testing consolidation across our platforms. After that this PR is merged, we will now have pytest tests running on all our supported platforms. It will finally unleash possibilities to use pytest as a runner for our unittest (gunittest) tests.

To keep this PR concise and limited to only the enablement of pytest, I didn't fix any of the three failing test files here. I have them fixed in echoix#220 (permalink to a commit since I will probably rebase later on echoix@cb97aef), where it was running directly on an installed grass-dev package from OSGeo4W instead of building each time. There are a couple differences, that's why I rebuilt the changes in this branch for this PR. The three fixes will probably need three separate discussions, so it isn't the place here to discuss them.

I also kept the pytest-xdist out of it, but also validated that it worked in one of the sessions where I connected to the runner by ssh to iterate faster and find where the files were and what env vars were present.

A particularity of both our current nonstandard osgeo4w build scripts and the official osgeo4w builds is that the bat files loading env vars wipe overwrite the path but do not include the lib folder (both), nor the bin folder (at least for our), so pytest (or any python script) cannot find the grass_gis.8.5. Note that after Python 3.8, the dll search path has changed, see https://docs.python.org/3/library/os.html#os.add_dll_directory, https://docs.python.org/3/whatsnew/3.8.html#bpo-36085-whatsnew. However, we have changes in our copy of ctypesgen for placing all entries of path as a search path (a bit overkill maybe), that aren't upstream. I didn't have the change to try out pypdfium2's maintained fork ctypesgen, as it has a newer and simpler loader, that might have solved me weeks (months) of headaches https://github.com/pypdfium2-team/ctypesgen/tree/pypdfium2.
The PYTHONPATH is also needed in order to find a module named "grass", that isn't set in these scripts (except the OSGeo4W's python-grass.bat, but we don't use it here).

@echoix echoix added this to the 8.5.0 milestone Sep 23, 2024
@echoix echoix requested a review from wenzeslaus September 23, 2024 01:07
@github-actions github-actions bot added CI Continuous integration Python Related code is in Python libraries module general notebook labels Sep 23, 2024
Copy link
Member

@wenzeslaus wenzeslaus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great to me! As long as these strange cmd commands work :-)

Just as a side note, I wonder if we should make python3-jupyter a dependency on Windows or for the Windows standalone installer.

@echoix
Copy link
Member Author

echoix commented Sep 23, 2024

Looks great to me! As long as these strange cmd commands work :-)

If you're talking about the run: , it mostly came from the existing scripts, but adapted or extracted some contents because I didn't need to actually start grass (nor wanted that).
If you're talking about the shell: it comes from what is used by default when using cmd instead of pwsh on GitHub Actions, but using a custom script wrapper, which is the osgeo4w shell. It's written somewhere in the docs. Like the msys2 or micromamba wrappers we use. Maybe some day I could make a cleaner version in setup-osgeo4w action. I've been using that pattern for a long while, for like 250-300 runs for now...

Just as a side note, I wonder if we should make python3-jupyter a dependency on Windows or for the Windows standalone installer.

Ideally no, reading from the Jupyter pip page,
https://pypi.org/project/jupyter/, they suggest using the dependencies you need. Also the osgeo4w Jupyter package is really old, so I'm not sure it points to exactly the best versions of each dependency. But individual dependencies seem ok in osgeo4w. I could've installed by pip (or even uv), but why not use what may be used by users?

Copy link
Member

@wenzeslaus wenzeslaus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good. I meant to approve the first time.

@echoix echoix merged commit 6325a18 into OSGeo:main Sep 23, 2024
28 checks passed
@neteler neteler modified the milestone: 8.5.0 Sep 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CI Continuous integration general libraries module notebook Python Related code is in Python
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants