gradle: auto-detect native version and patch file-events #277721

liff · 2023-12-30T12:54:20Z

Description of changes

Drop the requirement to specify the version of the native-platform library at Nix level. We use the non-platform-specific native-platform-<version>.jar to resolve the version of that library at fixup.

Hoping that we might be able to use nix-update-script to automate updates, but have not tested that yet.

I also noticed there is a bundled file-events library that requires libstdc++, so added a similar patch for that.

Things done

Add a 👍 reaction to pull requests you find important.

liff · 2024-02-01T13:55:53Z

@lorenzleutgeb sorry to bother, but pinging in case you have time to take a peek.

lorenzleutgeb

Builds and works on my machine. Thanks! I suggest to remove some redundancy (DRY, see comment) but could be merged as is.

lorenzleutgeb · 2024-02-13T13:53:49Z

pkgs/development/tools/build-managers/gradle/default.nix

          popd
        done

+        # The file-events library _seems_ to follow the native-platform version, but
+        # we won’t assume that.
+        fileEventsJar=$(basename $(ls -1 $out/lib/gradle/lib/file-events-*.jar | grep -Ev '.-(linux|osx|freebsd|windows)-'))


Maybe define a function to get the version like this? It seems to be the same code in lines 81-83 above.

Added a helper and switched to using autoPatchelf, since it does exactly what we need here.

lorenzleutgeb · 2024-05-19T21:27:24Z

Result of nixpkgs-review pr 277721 run on x86_64-linux 1

3 packages marked as broken and skipped:

frostwire
javaPackages.openjfx15
openjfx15

11 packages failed to build:

bisq-desktop
corretto11
corretto11.debug
corretto17
corretto17.debug
corretto19
corretto19.debug
ganttproject-bin
gscan2pdf
gscan2pdf.man
moneydance

65 packages built:

apkleaks
apkleaks.dist
apksigcopier
apksigcopier.dist
apksigner
armitage
bluej
cie-middleware-linux
cryptomator
diffoscope
diffoscope.dist
diffoscope.man
experienced-pixel-dungeon
fastddsgen
fdroidserver
fdroidserver.dist
freenet
freeplane
ghidra
gradle (gradle_8)
gradle_6
gradle_7
greenfoot
jabref
jadx
openjfx11 (javaPackages.openjfx11)
openjfx (javaPackages.openjfx17 ,openjfx17)
openjfx19 (javaPackages.openjfx19)
openjfx20 (javaPackages.openjfx20)
openjfx21 (javaPackages.openjfx21)
openjfx22 (javaPackages.openjfx22)
jd-gui
jextract
jextract-21
key
kotlin-language-server
ma1sd
maptool
mcaselector
microsoft-identity-broker
mindustry
mindustry-server
mindustry-wayland
mucommander
ns-usbloader
olvid
pattypan
pdf-sign
pdfchain
pdfsam-basic
pdftk
pidginPackages.purple-signald
quark-goldleaf
rat-king-adventure
rkpd2
scenebuilder
scenic-view
shattered-pixel-dungeon
shorter-pixel-dungeon
signald
signaturepdf
sparrow
sparrow-unwrapped
stirling-pdf
summoning-pixel-dungeon

lorenzleutgeb · 2024-05-19T22:01:40Z

Successes

Results gradle{,_{6,7}}/bin/gradle all execute fine.

Failures

`gscan2pdf`

2/1171 tests failed, I don't see how they are related. See gscan2pdf.log.

`corretto*`

Compilations warnings (errors?) don't see how these are related:

`ganttproject-bin`, `moneydance`

Logs are empty.

lorenzleutgeb · 2024-05-19T22:02:51Z

@ofborg build gscan2pdf bisq-desktop corretto11 corretto17 corretto19 ganttproject-bin moneydance

raphiz

I had a look at the code and tried it out and it looks great 👍️

I also grepped through all jar files in the latest gradle distribution (gradle-8.7-bin.zip) and found some other native libraries - not sure if we should/must patch them as well:

libjansi.so in jansi-1.18.jar and kotlin-compiler-embeddable-1.9.22.jar
liblz4-java.so in kotlin-compiler-embeddable-1.9.22.jar

I haven't had any issues so far without it being patched 🤷

raphiz · 2024-05-30T05:58:21Z

pkgs/development/tools/build-managers/gradle/patching.sh

+    shift
+    local candidates="$@"
+
+    jar="$(basename $(ls -1 "$@" | grep -Ev '.-(linux|osx|freebsd|windows)-'))"


raphiz · 2024-05-30T06:07:20Z

pkgs/development/tools/build-managers/gradle/default.nix

-          popd
+          autoPatchelfInJar \
+            $out/lib/gradle/lib/native-platform-linux-${arch}$variant-''${nativeVersion}.jar \
+            "${stdenv.cc.cc.lib}/lib64:${lib.makeLibraryPath [ stdenv.cc.cc ncurses5 ncurses6 ]}"


I don't know too much about elf patching - but this looks reasonable to me🤷

raphiz · 2024-05-30T06:12:18Z

pkgs/development/tools/build-managers/gradle/default.nix

-          mv native-platform-linux-${arch}$variant-${nativeVersion}.jar $out/lib/gradle/lib/
-          popd
+          autoPatchelfInJar \
+            $out/lib/gradle/lib/native-platform-linux-${arch}$variant-''${nativeVersion}.jar \


Just for the record - this is "just" patching the linux version (as it was done before).
As others told me, the resulting nix package works on macOS too

Yes, patchelf is a Linux-only thing, AFAIK.

liff · 2024-05-30T17:20:15Z

As far as I can tell, patchelf doesn’t do anything to the dynamic libraries bundled with the Kotlin compiler. I think the reason is that those libraries only require glibc.

lorenzleutgeb · 2024-06-03T19:15:57Z

I'd like to get this in but am a little hesitant because of the build failures. Re-running nixpkgs-review at the moment, it's not done but gscan2pdf tests fail and one probably unrelated failure for OpenJFX:

error: builder for '/nix/store/w0r4l7pk1n1dqfh3s5rxx77nilc29d5x-openjfx-modular-sdk-17.0.11-ga.drv' failed with exit code 1;
       last 30 log lines:
       > * Where:
       > Build file '/build/source/build.gradle' line: 3513
       >
       > * What went wrong:
       > A problem occurred evaluating root project 'source'.
       > > Could not resolve all files for configuration ':web:icu'.
       >    > Could not resolve :icu4c-73_1-data-bin-l.
       >      Required by:
       >          project :web
       >       > Could not resolve :icu4c-73_1-data-bin-l.
       >          > Could not get resource 'https://download.eclipse.org/eclipse/updates/4.6/R-4.6.3-201703010400/plugins/icu4c-73_1-data-bin-l.zip'.
       >             > Could not HEAD 'https://download.eclipse.org/eclipse/updates/4.6/R-4.6.3-201703010400/plugins/icu4c-73_1-data-bin-l.zip'.
       >                > download.eclipse.org

Edit: Results are in. I don't really understand what's the problem with gscan2pdf and corretto11. But corretto17 builds, so maybe we can ignore it?

Result of nixpkgs-review pr 277721 run on x86_64-linux 1

8 packages marked as broken and skipped:

corretto19
corretto19.debug
javaPackages.openjfx15
javaPackages.openjfx19
javaPackages.openjfx20
openjfx15
openjfx19
openjfx20

7 packages failed to build:

bluej
corretto11
corretto11.debug
greenfoot
gscan2pdf
gscan2pdf.man
scenic-view

71 packages built:

apkleaks
apkleaks.dist
apksigcopier
apksigcopier.dist
apksigner
armitage
autopsy
bisq-desktop
cie-middleware-linux
corretto17
corretto17.debug
corretto21
corretto21.debug
cryptomator
diffoscope
diffoscope.dist
diffoscope.man
experienced-pixel-dungeon
fastddsgen
fdroidserver
fdroidserver.dist
freenet
freeplane
ganttproject-bin
ghidra
ghidra-extensions.gnudisassembler
ghidra-extensions.machinelearning
ghidra-extensions.sleighdevtools
gradle (gradle_8)
gradle_6
gradle_7
jabref
jadx
openjfx11 (javaPackages.openjfx11)
openjfx (javaPackages.openjfx17 ,openjfx17)
openjfx21 (javaPackages.openjfx21)
openjfx22 (javaPackages.openjfx22)
jd-gui
jextract
jextract-21
key
kotlin-language-server
ma1sd
maptool
mcaselector
microsoft-identity-broker
mindustry
mindustry-server
mindustry-wayland
moneydance
mucommander
ns-usbloader
olvid
pattypan
pdf-sign
pdfchain
pdfsam-basic
pdftk
pidginPackages.purple-signald
quark-goldleaf
rat-king-adventure
rkpd2
scenebuilder
shattered-pixel-dungeon
shorter-pixel-dungeon
signald
signaturepdf
sparrow
sparrow-unwrapped
stirling-pdf
summoning-pixel-dungeon

scenic-view.log
gscan2pdf.log
corretto11.log

adamcstephens · 2024-09-08T22:43:17Z

Will need a rebase if still desired.

Instead of having to know the version of native-platform at the Nix level, we use the file `lib/native-platform-<version>.jar` to figure out the version of the library JAR to patch.

It depends on libstdc++, so add that to its RPATH.

liff · 2024-09-09T16:30:43Z

@adamcstephens rebased, thanks.

liff requested a review from lorenzleutgeb December 30, 2023 12:54

ofborg bot added 11.by: package-maintainer This PR was created by the maintainer of the package it changes 10.rebuild-darwin: 11-100 10.rebuild-linux: 11-100 labels Dec 30, 2023

liff force-pushed the gradle/no-nativeVersion branch from daec92d to f7c0dfd Compare February 7, 2024 16:38

lorenzleutgeb approved these changes Feb 13, 2024

View reviewed changes

liff force-pushed the gradle/no-nativeVersion branch from f7c0dfd to 1bb4d47 Compare February 13, 2024 20:48

liff requested a review from lorenzleutgeb February 13, 2024 20:50

lorenzleutgeb approved these changes Mar 3, 2024

View reviewed changes

wegank added 12.approvals: 1 This PR was reviewed and approved by one reputable person 12.approved-by: package-maintainer This PR was reviewed and approved by a maintainer listed in the package labels Mar 8, 2024

liff force-pushed the gradle/no-nativeVersion branch from 1bb4d47 to c326ee4 Compare March 25, 2024 07:24

ofborg bot requested a review from lorenzleutgeb March 25, 2024 07:48

liff force-pushed the gradle/no-nativeVersion branch from c326ee4 to 8e9523e Compare March 25, 2024 17:15

lorenzleutgeb approved these changes Mar 27, 2024

View reviewed changes

liff force-pushed the gradle/no-nativeVersion branch from 8e9523e to 5cfb74b Compare April 3, 2024 10:07

ofborg bot requested a review from lorenzleutgeb April 3, 2024 10:33

wegank removed the 12.approvals: 1 This PR was reviewed and approved by one reputable person label Apr 4, 2024

liff mentioned this pull request May 29, 2024

gradle: determine native version automatically #315611

Closed

13 tasks

raphiz approved these changes May 30, 2024

View reviewed changes

liff force-pushed the gradle/no-nativeVersion branch from 5cfb74b to 4a7a7ab Compare May 30, 2024 17:24

liff requested a review from raphiz May 30, 2024 17:25

liff mentioned this pull request Jun 3, 2024

gradle: 8.7 → 8.8 #316849

Merged

13 tasks

raphiz approved these changes Jun 3, 2024

View reviewed changes

lorenzleutgeb approved these changes Jun 3, 2024

View reviewed changes

lorenzleutgeb added needs_merger (old Marvin label, do not use) 6.topic: java Including JDK, tooling, other languages, other VMs 12.approvals: 2 This PR was reviewed and approved by two reputable people 0.kind: enhancement Add something new labels Jun 3, 2024

wegank added the 2.status: merge conflict This PR has merge conflicts with the target branch label Jul 4, 2024

liff force-pushed the gradle/no-nativeVersion branch from 4a7a7ab to 9e7243c Compare July 20, 2024 09:26

ofborg bot removed the 2.status: merge conflict This PR has merge conflicts with the target branch label Jul 20, 2024

ofborg bot requested a review from lorenzleutgeb July 20, 2024 10:01

lorenzleutgeb approved these changes Jul 20, 2024

View reviewed changes

liff added 2 commits September 9, 2024 18:21

gradle: auto-detect nativeVersion

11692ab

Instead of having to know the version of native-platform at the Nix level, we use the file `lib/native-platform-<version>.jar` to figure out the version of the library JAR to patch.

gradle: also patch the native file-events library

bd244e7

It depends on libstdc++, so add that to its RPATH.

liff force-pushed the gradle/no-nativeVersion branch from 9e7243c to bd244e7 Compare September 9, 2024 15:21

ofborg bot requested a review from lorenzleutgeb September 9, 2024 16:24

lorenzleutgeb approved these changes Sep 11, 2024

View reviewed changes

adamcstephens merged commit 392bc9a into NixOS:master Sep 11, 2024
24 of 26 checks passed

liff deleted the gradle/no-nativeVersion branch November 14, 2024 10:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gradle: auto-detect native version and patch file-events #277721

gradle: auto-detect native version and patch file-events #277721

liff commented Dec 30, 2023 •

edited

Loading

liff commented Feb 1, 2024

lorenzleutgeb left a comment

lorenzleutgeb Feb 13, 2024

liff Feb 13, 2024

lorenzleutgeb commented May 19, 2024

lorenzleutgeb commented May 19, 2024

lorenzleutgeb commented May 19, 2024

raphiz left a comment

raphiz May 30, 2024

liff May 30, 2024

raphiz May 30, 2024

raphiz May 30, 2024

liff May 30, 2024

liff commented May 30, 2024

lorenzleutgeb commented Jun 3, 2024 •

edited

Loading

adamcstephens commented Sep 8, 2024

liff commented Sep 9, 2024

	jar="$(basename $(ls -1 "$@" \| grep -Ev '.-(linux\|osx\|freebsd\|windows)-'))"
	jar="$(basename -a $candidates \| sort \| head -n1)"

gradle: auto-detect native version and patch file-events #277721

gradle: auto-detect native version and patch file-events #277721

Conversation

liff commented Dec 30, 2023 • edited Loading

Description of changes

Things done

liff commented Feb 1, 2024

lorenzleutgeb left a comment

Choose a reason for hiding this comment

lorenzleutgeb Feb 13, 2024

Choose a reason for hiding this comment

liff Feb 13, 2024

Choose a reason for hiding this comment

lorenzleutgeb commented May 19, 2024

lorenzleutgeb commented May 19, 2024

Successes

Failures

gscan2pdf

corretto*

ganttproject-bin, moneydance

lorenzleutgeb commented May 19, 2024

raphiz left a comment

Choose a reason for hiding this comment

raphiz May 30, 2024

Choose a reason for hiding this comment

liff May 30, 2024

Choose a reason for hiding this comment

raphiz May 30, 2024

Choose a reason for hiding this comment

raphiz May 30, 2024

Choose a reason for hiding this comment

liff May 30, 2024

Choose a reason for hiding this comment

liff commented May 30, 2024

lorenzleutgeb commented Jun 3, 2024 • edited Loading

adamcstephens commented Sep 8, 2024

liff commented Sep 9, 2024

liff commented Dec 30, 2023 •

edited

Loading

`gscan2pdf`

`corretto*`

`ganttproject-bin`, `moneydance`

lorenzleutgeb commented Jun 3, 2024 •

edited

Loading