Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

win: patch from OSGeo4W applied #4121

Merged
merged 27 commits into from
Dec 15, 2024
Merged

win: patch from OSGeo4W applied #4121

merged 27 commits into from
Dec 15, 2024

Conversation

landam
Copy link
Member

@landam landam commented Jul 31, 2024

@landam landam self-assigned this Jul 31, 2024
@landam landam marked this pull request as draft July 31, 2024 17:01
@landam landam added this to the 8.4.1 milestone Jul 31, 2024
@landam landam added the windows Microsoft Windows specific label Jul 31, 2024
@github-actions github-actions bot added the CI Continuous integration label Jul 31, 2024
@neteler
Copy link
Member

neteler commented Jul 31, 2024

A dependency yet missing?

From the CI log:

...
configure: error: *** couldn't find libpng-config
Error: Process completed with exit code 1.

@nilason
Copy link
Contributor

nilason commented Jul 31, 2024

A dependency yet missing?

From the CI log:

...
configure: error: *** couldn't find libpng-config
Error: Process completed with exit code 1.

Not a dependency, but https://github.com/OSGeo/grass/blob/main/mswindows/osgeo4w/libpng-config (see #2679).

ldesousa and others added 4 commits July 31, 2024 21:46
Add basic tests for GeoPackage and Shapefile export which use import to test the result (so they test round trip but focus on the export).
@jef-n
Copy link
Contributor

jef-n commented Aug 1, 2024

Note that build_osgeo4w.sh and osgeo4w.yml are GRASS things, that are not used in osgeo4w - they were just aligned with package.sh, which actually is.

@landam landam changed the title Patch from OSGeo4W applied win: patch from OSGeo4W applied Aug 4, 2024
@landam
Copy link
Member Author

landam commented Aug 4, 2024

After reintroducing mingw-w64-x86_64-libpng the CI pass successfully.

@jef-n Please could you elaborate why you are removing this dependency in the workflow (https://github.com/jef-n/OSGeo4W/blob/master/src/grass/osgeo4w/patch#L9)? On the other side it's installed in package.sh (https://github.com/jef-n/OSGeo4W/blob/master/src/grass/osgeo4w/package.sh#L101)

@jef-n
Copy link
Contributor

jef-n commented Aug 5, 2024

After reintroducing mingw-w64-x86_64-libpng the CI pass successfully.

@jef-n Please could you elaborate why you are removing this dependency in the workflow (https://github.com/jef-n/OSGeo4W/blob/master/src/grass/osgeo4w/patch#L9)? On the other side it's installed in package.sh (https://github.com/jef-n/OSGeo4W/blob/master/src/grass/osgeo4w/package.sh#L101)

um, osgeo4w has libpng. not sure why it was only removed from build_osgeo4w.sh. As said I don't use that. Maybe I wanted to replace msys libpng with's OSGeo4W's png, replaced it in both and failed to re-add it in build_osgeo4w.sh, when I found that it was not working. But I don't recall… In OSGeo4W it's just a patch to an unused file ;)

jef-n added a commit to jef-n/OSGeo4W that referenced this pull request Aug 5, 2024
@jef-n
Copy link
Contributor

jef-n commented Aug 5, 2024

There was a change in libpng that broke our copy of libpng-config - adapted.

@landam
Copy link
Member Author

landam commented Aug 22, 2024

Any idea why osgeo4w CI is failing on tests while compilation is successful? It seems to be unrelated to this PR...

@echoix
Copy link
Member

echoix commented Aug 22, 2024

There were a lot more failed files, like 58% of files passed, instead of 81%

@github-actions github-actions bot added GUI wxGUI related docker Docker related vector Related to vector data processing raster Related to raster data processing temporal Related to temporal data processing Python Related code is in Python labels Aug 22, 2024
@echoix
Copy link
Member

echoix commented Nov 10, 2024

I solved the conflicts by sorting the two conflicting files of the PR in same way as #4563, then comparing the two sides of the merge making sure changes present in the PR (on GitHub) were still present even if moved. It was all ok, nothing is lost.

@echoix
Copy link
Member

echoix commented Nov 11, 2024

osgeo4w installs libomp.dll:
https://github.com/jef-n/OSGeo4W/blob/f8ada2fe6637b1abde0a361fa8b5b2089a82e33d/src/grass/osgeo4w/package.sh#L104

which is probably why configure hooks up on it.

Perhaps

-/mingw64/bin/libgomp-1.dll
+/mingw64/bin/libomp.dll

in

/mingw64/bin/libgomp-1.dll

could be the solution?

I've reread on a fresh mind. Isn't libomp the llvm OpenMP runtime library (https://packages.msys2.org/packages/mingw-w64-x86_64-openmp, files section at the very end of https://packages.msys2.org/packages/mingw-w64-x86_64-llvm-openmp)

And libgomp the gcc OpenMP library
(https://packages.msys2.org/packages/mingw-w64-x86_64-omp, files section at the very end of https://packages.msys2.org/packages/mingw-w64-x86_64-gcc-libs)

(And libiomp the intel OpenMP one, not in msys2).

So, following the often linked advice that multiple OpenMP runtimes shouldn't be mixed (https://cpufun.substack.com/p/is-mixing-openmp-runtimes-safe), and that we are using the gcc toolchain, I think we should be removing all references to libomp.dll, and go all-in with libgomp-1.dll, and work until fixing all the issues.

@echoix
Copy link
Member

echoix commented Nov 12, 2024

I've implemented the suggested changes here: echoix#285

But since our CI doesn't use the package.sh script, I've reimplemented the patches that I've been waiting and wanting to do after this PR is merged here: echoix#286

It showed that we were missing some dlls for lapack and libblas. I didn't find (only on my phone) what branch I had solved the issue that the msys2 couldn't run the Linux print_versions.sh, where some extra paths need to be added so an msys2 shell finds inside the OSGeo4W install.

At least, in these CI runs, it uploads the compiled package that can be downloaded and installed with the OSGeo4W installer (pointing the download dir to a dir with the package from the artifact) for local testing. We could test if the two or three bugs with OpenMP linked are solved by that

@nilason
Copy link
Contributor

nilason commented Nov 19, 2024

OpenMP related fix (hopefully) is added with jef-n/OSGeo4W@06d4d83 and jef-n/OSGeo4W@c678c71.

@echoix
Copy link
Member

echoix commented Nov 19, 2024

Conflicts to solve

@nilason nilason modified the milestones: 8.4.1, 8.5.0 Nov 19, 2024
@nilason nilason removed the backport to 8.4 PR needs to be backported to release branch 8.4 label Nov 19, 2024
@nilason
Copy link
Contributor

nilason commented Nov 19, 2024

OpenBLAS issue for main has been addressed with 9984205, which doesn't need to backport.

OpenMP issues have been addressed (as mentioned above), but doesn't need changes in GRASS source (and thus no need to backport).

I removed backport label for this PR, as the build issues have likely been addressed.

@nilason
Copy link
Contributor

nilason commented Nov 19, 2024

@landam Could you please check if the win build server is working, as there has been no build log since end of August September.

@echoix
Copy link
Member

echoix commented Nov 23, 2024

I'm still processing the changed patch 16 hours ago in the OSGeo4W repo

@echoix
Copy link
Member

echoix commented Nov 23, 2024

In the last commit I'm keeping (for now) the --with-libpng=${SRC}/mswindows/osgeo4w/libpng-config \ line that was removed. I think it might be a merging/rebase conflict that lost it, as I would have expected that line to be changed to use ${SRC} instead of ${PWD} like the other places in the file that got updated for that. If it is expected, then we should also remove it from our parallel script.

Copy link
Member

@echoix echoix left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've gone through the remaining conversations to resolve. Like I mention in these, I would really like to have this finished, so I could resume some other work that I rebased on this (as a couple weeks before this PR I already did a similar effort and continued on this), but I've been waiting 4 months since. Recently we solved one of the blockers, so there's not much left to think here.

mswindows/osgeo4w/env.bat.tmpl Show resolved Hide resolved
mswindows/osgeo4w/package.sh Show resolved Hide resolved
mswindows/osgeo4w/package.sh Show resolved Hide resolved
The dll files are copied with the following line, which doesn't need to use that list anymore

# copy dependencies
cp -uv $(/usr/bin/find apps/grass/grass$POSTFIX -iname "*.dll" -o -iname "*.exe" | PATH=$PWD/apps/grass/grass$POSTFIX/lib:$PWD/bin:/mingw64/bin:/usr/bin /usr/bin/xargs /usr/bin/ldd | /usr/bin/sed -ne 's#^.* => \(/mingw64/bin/.*\) (.*)$#\1#p' | /usr/bin/sort -u) apps/grass/grass$POSTFIX/bin
Copy link
Member

@echoix echoix left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did a final review, and all conversations are addressed.

@echoix echoix enabled auto-merge (squash) December 15, 2024 20:53
@hellik
Copy link
Member

hellik commented Dec 15, 2024

Thanks for checking all these improvements.

@echoix echoix merged commit 161658b into OSGeo:main Dec 15, 2024
27 checks passed
@echoix
Copy link
Member

echoix commented Dec 22, 2024

@jef-n The daily grass-dev builds seems to have stopped after this got merged, no new builds were made since december 15, 2024. I wanted to let you know

@landam landam deleted the osgeo4w_patch branch February 11, 2025 08:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CI Continuous integration windows Microsoft Windows specific
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants