Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI(OSGeo4W): Use setup-OSGeo4W action to parametrize install options #4290

Merged
merged 4 commits into from
Sep 20, 2024

Conversation

echoix
Copy link
Member

@echoix echoix commented Sep 8, 2024

Lately I’ve been working to get pytest running on Windows too. I end up needing some more changes to make it work, however using directly the same script that OSGeo4W uses brings me closer. The changes I would need would be reduced by having some common install options (including the OSGeo4W root) refactored out. I will also need #4121 in a near future too. And I find installation a bit long too.

So, I ended up creating an action similar to other setup-* actions, that easily allow to instal and configure a software or language. I longly thought out the pros and cons of having the action inside this repo or as a separate repo. I wrote to @wenzeslaus too during that.
I finally settled on creating a clean version of the action in a separate repo, for a couple reasons:

  • It has a value for users and projects outside our repo, any project that is distributed by OSGeo4W might want to use it
  • It can be released faster, and is not tied to our release cadence
  • Being written mainly in powershell (that is cross platform, just not the OSGeo4W installer), it is quite different than what we have here
  • The step in the workflows written are nicer with that kind of name instead of a long path.
  • It will greatly help backports, since only version changes can be copied back when needed
  • Same thing for grass-addons: Having the action inside the grass repo would make it really hard to actually use it, since we not only check the current main, but also another release, that wouldn’t have the action available.
  • It also prevents to have to pull the complete grass repo if wanting to reference that action.
  • There’s a possibility of doing advanced intelligent caching and pruning of downloaded packages in the future. I already did a proof of concept that parsed the ini package database and was able to collect all pacakages and dependencies and download them multithreaded and asynchronous (powershell is very close to dotnet, and we can use everything from dotnet if we want). I ended up only 1 second faster than the installer, on less than 20 seconds download.

Improvements on using that action instead of jamming everything in the same step inline:

  • We can easily change the location of the download folder (a package cache), as well as the install root.
  • It is important, as the D:\ drive in Windows runner has better IO characteristics (faster).
  • The C:\ drive is not only slower, but also the performance varies noticeably more (throughout the 235+ runs I made in the month)
  • At a first stage, without changing the OSGeo4W root yet, I still get 30-40 seconds improvement over a 2min-2min10 step, and with a more consistent duration taken.
  • Using that action allows to have a nice logic for listing packages like we naturally do in GitHub actions/yaml, avoiding the particularities of the osgeo4w installer (it absolutely needs commas and no space ever in the package list for example). A single space or wrong comma ends up in a silent failure, as not all packages are installed without knowing it didn’t install.
  • We can have packages on a multiple lines, sorted, and be able to have clear diffs when changing the packages installed.
  • Using an other method of calling the GUI installer while remaining in command line, it is now possible to display the logs and see the progress and what packages are being downloaded, extracted, or installed. I already used it to track down a single slow package that caused more slowdown than others, and see that only a small fraction of the time is taken for the download step.

@github-actions github-actions bot added the CI Continuous integration label Sep 8, 2024
@echoix
Copy link
Member Author

echoix commented Sep 8, 2024

I accidentally created the PR directly in grass, so I edited the PR description instead of leaving it empty

@echoix echoix force-pushed the setup-osgeo4w-action branch from 9e73eae to 253284d Compare September 8, 2024 10:35
@echoix echoix changed the title CI(OSGeo4W): Use setup-OSGeo4W action to parametrize instal options CI(OSGeo4W): Use setup-OSGeo4W action to parametrize install options Sep 8, 2024
@echoix echoix requested a review from wenzeslaus September 9, 2024 22:14
Copy link
Member

@wenzeslaus wenzeslaus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like this!

I would merge this as is. We can consider what to do with echoix/setup-OSGeo4W. Perhaps it should live under OSGeo, but we do that later.

However, perhaps one of the people who worked on the workflow or Windows side in general should comment here. @ninsbl @HuidaeCho @landam @hellik

@echoix
Copy link
Member Author

echoix commented Sep 18, 2024

For the last three weekends I've been using it (or what it was before being finished), and it was pretty convenient to be able to change the parameters and explore it. Counting from the beginning, it approaches the 250 runs I've made with this solution and works well without hiccups

@echoix
Copy link
Member Author

echoix commented Sep 19, 2024

I'm waiting for comments for a last day before merging

Copy link
Member

@hellik hellik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't review deeply as very limited internet access these days. The approach looks good.

@echoix echoix merged commit 6bc69e3 into OSGeo:main Sep 20, 2024
28 checks passed
@echoix echoix deleted the setup-osgeo4w-action branch September 20, 2024 11:46
@neteler neteler added this to the 8.5.0 milestone Sep 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CI Continuous integration
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants