dotnet-sdk_3: 3.1.19 -> 3.1.21 + dotnet-sdk_5: 5.0.10 -> 5.0.12 #142468

prusnak · 2021-10-21T12:35:01Z

Motivation for this change

added print-hashes script which makes further updates very easy (not totally automatic though)
sdk_3 and sdk_5 version updates
5.0.11 contains security fixes, so that commit needs to be backported to stable too

Things done

felschr

10 packages failed to build:
discordchatexporter-cli eventstore github-runner msbuild omnisharp-roslyn osu-lazer python-language-server roslyn ryujinx wasabibackend

17 packages built:
azure-functions-core-tools btcpayserver depotdownloader dotnet-aspnetcore dotnet-runtime dotnet-sdk dotnet-sdk_3 dotnetCorePackages.aspnetcore_3_1 dotnetCorePackages.runtime_3_1 jellyfin jetbrains.rider nbxplorer
opentabletdriver prowlarr radarr vscode-extensions.ms-vsliveshare.vsliveshare wasabiwallet

As far as I can see all the errors, like the one below, are the same kind of dependency errors:

       > /build/source/Ryujinx/Ryujinx.csproj : error NU1102: Unable to find package Microsoft.AspNetCore.App.Runtime.osx-x64 with version (= 5.0.11) [/build/source/Ryujinx.sln]
       > /build/source/Ryujinx/Ryujinx.csproj : error NU1102:   - Found 1 version(s) in nixos [ Nearest version: 5.0.10 ] [/build/source/Ryujinx.sln]

You need to update the dependencies for those packages like in previous updates:
#138296

Also, the commits should follow this naming instead:
dotnetCorePackages.*_5*: 5.0.10 -> 5.0.11

jonringer · 2021-10-25T03:06:14Z

yea, that's the painful part of dotnet+nix, is that some dependent packages need to be manually re-generated. :(

avdv · 2021-11-02T13:13:25Z

Result of nixpkgs-review pr 142468 run on x86_64-linux 1

10 packages failed to build:

discordchatexporter-cli
eventstore
github-runner
msbuild
omnisharp-roslyn
osu-lazer
python-language-server
roslyn
ryujinx
wasabibackend

17 packages built:

azure-functions-core-tools
btcpayserver
depotdownloader
dotnet-aspnetcore (dotnetCorePackages.aspnetcore_5_0)
dotnet-runtime (dotnetCorePackages.runtime_5_0)
dotnet-sdk (dotnet-sdk_5 ,dotnetCorePackages.sdk_5_0)
dotnet-sdk_3 (dotnetCorePackages.sdk_3_1)
dotnetCorePackages.aspnetcore_3_1
dotnetCorePackages.runtime_3_1
jellyfin
jetbrains.rider
nbxplorer
opentabletdriver
prowlarr
radarr
vscode-extensions.ms-vsliveshare.vsliveshare
wasabiwallet

avdv · 2021-11-09T09:51:21Z

@prusnak I see that you updated this PR to upgrade to later versions. Are you working on updating the hashes of dependent packages or do you need a helping hand?

prusnak · 2021-11-09T09:56:11Z

Are you working on updating the hashes of dependent packages or do you need a helping hand?

Helping hand would be great! I don't have access to my x86_64 machine at the moment, so it would be very problematic for me to update these.

pkgs/development/compilers/dotnet/print-hashes.sh

avdv · 2021-11-10T06:57:44Z

Helping hand would be great! I don't have access to my x86_64 machine at the moment, so it would be very problematic for me to update these.

@prusnak I have started to update dependent packages. Got stuck on msbuild where I still need to figure out how the update script should be called.

See prusnak/nixpkgs@dotnet-update...avdv:dotnet-update

TODO:

22 packages built:
ArchiSteamFarm azure-functions-core-tools btcpayserver depotdownloader discordchatexporter-cli dotnet-aspnetcore dotnet-netcore dotnet-sdk dotnet-sdk_3 dotnetCorePackages.aspnetcore_3_1 dotnetCorePackages.runtime_3_1 jellyfin jetbrains.rider nbxplorer netcoredbg opentabletdriver osu-lazer prowlarr radarr ryujinx vscode-extensions.ms-vsliveshare.vsliveshare wasabiwallet

avdv · 2021-11-11T15:53:55Z

Result of nixpkgs-review run on x86_64-linux 1 (:information_source: from this branch https://github.com/avdv/nixpkgs/commits/dotnet-update)

30 packages built:

ArchiSteamFarm
alttpr-opentracker
azure-functions-core-tools
btcpayserver
depotdownloader
discordchatexporter-cli
dotnet-aspnetcore
dotnet-netcore (dotnet-runtime)
dotnet-sdk
dotnet-sdk_3
dotnetCorePackages.aspnetcore_3_1
dotnetCorePackages.runtime_3_1
eventstore
github-runner
jellyfin
jetbrains.rider
msbuild
nbxplorer
netcoredbg
omnisharp-roslyn
opentabletdriver
osu-lazer
prowlarr
python-language-server
radarr
roslyn
ryujinx
vscode-extensions.ms-vsliveshare.vsliveshare
wasabibackend
wasabiwallet

@prusnak I have updated my branch, please have a look and feel free to push these changes to your branch.

prusnak · 2021-11-11T17:09:41Z

@avdv Thank you very much for your work! Pulled in changes from your branch into this PR.

prusnak · 2021-11-11T18:36:51Z

I updated commit c72bbcd3f5f62ad4eefe259435e1efbbea3ed8b2 to enable linux-arm64 build of github-runner

corngood · 2021-11-11T19:53:22Z

@avdv I'm sorry it's such a pain to update the dependencies for msbuild. You basically need to unpack, patch and build, but without the local nuget sources, then run create-deps.sh.

I threw together something quick that should work as an example:

corngood@msbuild-deps

nix-shell -A msbuild --run createDeps

should create ./deps.nix

So much of the build process is required that I'm not sure how best to split this between the derivation and the shell script. Any ideas?

avdv · 2021-11-11T22:27:22Z

I'm sorry it's such a pain to update the dependencies for msbuild. You basically need to unpack, patch and build, but without the local nuget sources, then run create-deps.sh.

Thank you @corngood! I think this is what I ended up doing somehow, but the deps.nix created were identical to the existing deps.nix... and then re-trying to build msbuild simply succeeded. 🤷

I threw together something quick that should work as an example:

corngood@msbuild
nix-shell -A msbuild --run createDeps
should create ./deps.nix

Thanks, I'll give this a spin...

and do not add nuget.build.tasks.pack ``` /build/source/src/NuGet/Microsoft.Net.Compilers.Toolset/Microsoft.Net.Compilers.Toolset.Package.csproj : error NU1100: Unable to resolve nuget.build.tasks.pack (>= 5.10.0-preview.2.7169)' for '.NETCoreApp,Version=v3.1'. /build/source/src/NuGet/Microsoft.Net.Compilers.Toolset/Microsoft.Net.Compilers.Toolset.Package.csproj : error NU1100: Unable to resolve 'nuget.build.tasks.pack (>= 5.10.0-preview.2.7169)' for '.NETFramework,Version=v4.7.2'. ```

The old version of the script tried to parse the output of the dotnet tool, but apparently, that output changed and thus the list of dependencies was empty.

prusnak · 2021-11-16T15:27:28Z

Thanks @avdv !

Going to merge this as msbuild builds just fine.

avdv · 2021-11-17T21:12:29Z

Thank you @prusnak !

@corngood I ran the nix-shell -A msbuild --run createDeps incorporating your changes into the current branch and the dependencies were again identical. The only thing I noticed is that sometimes the server returned an HTTP error code and the createDeps process stopped, and I had to re-try it a couple of times...

corngood · 2021-11-17T21:25:24Z

Could you tell what URL it was fetching, and what the error code was? If we're using unreliable sources that's going to be a problem for builds as well down the road. It would be nice to have a proper mirror list, but I've found that in practice the different mirrors don't have identical content for the packages.

At some point I'd like to do some work to unify all the nuget dependency gathering logic for all these packages.

#133588 might be interesting

avdv · 2021-11-18T10:10:34Z

Could you tell what URL it was fetching, and what the error code was?

The only diagnostic output from the script was:

couldn't find microsoft.netframework.referenceassemblies.net45 1.0.0

and

couldn't find nuget.librarymodel 5.9.1-rc.8

Looking these up in the deps.nix file yields:

https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/45bacae2-5efb-47c8-91e5-8ec20c22b4f8/nuget/v3/flat2/microsoft.netframework.referenceassemblies.net45/1.0.0/microsoft.netframework.referenceassemblies.net45.1.0.0.nupkg

https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/d1622942-d16f-48e5-bc83-96f4539e7601/nuget/v3/flat2/nuget.librarymodel/5.9.1-rc.8/nuget.librarymodel.5.9.1-rc.8.nupkg

To avoid fetching the artifacts twice in the create-deps script I modified it thusly:

diff --git a/pkgs/development/tools/build-managers/msbuild/create-deps.sh b/pkgs/development/tools/build-managers/msbuild/create-deps.sh
index c9bd4ba7eb6..15b997b2af5 100755
--- a/pkgs/development/tools/build-managers/msbuild/create-deps.sh
+++ b/pkgs/development/tools/build-managers/msbuild/create-deps.sh
@@ -15,9 +15,21 @@ mapfile -t repos < <(
         done
     )
 
-find .packages fake-home/.nuget/packages -name \*.nupkg -printf '%P\n' | sort -u |
-    while IFS= read file
-    do
+
+find .packages fake-home/.nuget/packages -name \*.nupkg -printf '%p\n' |
+    (
+     declare -A nupkg
+
+     while IFS= read file
+     do
+        ponly="${file#*?packages/}"
+        nupkg["$ponly"]="$file"
+     done
+
+     declare -a sorted
+     IFS=$'\n' sorted=( $(sort -u <<<"${!nupkg[*]}") )
+
+     for file in "${sorted[@]}"; do
         packagedir=$(dirname $file)
         version=$(basename $packagedir)
         package=$(dirname $packagedir)
@@ -38,8 +50,8 @@ find .packages fake-home/.nuget/packages -name \*.nupkg -printf '%P\n' | sort -u
             echo "couldn't find $package $version" >&2
             exit 1
         fi
+        sha256="$(nix-hash --type sha256 --flat --base32 "${nupkg["$file"]}")"
 
-        sha256=$(nix-prefetch-url "$url" 2>/dev/null)
         cat << EOL
   {
     name = "$package";
@@ -50,7 +62,8 @@ find .packages fake-home/.nuget/packages -name \*.nupkg -printf '%P\n' | sort -u
     };
   }
 EOL
-    done
+     done
+  )
 
 cat << EOL
 ]

which uses the already downloaded files instead of fetching it again.

It would be nice to have a proper mirror list, but I've found that in practice the different mirrors don't have identical content for the packages.

At least, that explains why my modified script turned up a different hash for a single dependency, probably because it was downloaded from a different mirror before:

--- deps.nix	2021-11-11 16:22:53.500281686 +0100
+++ /tmp/newdeps	2021-11-12 09:37:45.439595602 +0100
@@ -364,7 +364,7 @@
     version = "16.6.1";
     src = fetchurl {
       url = "https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/45bacae2-5efb-47c8-91e5-8ec20c22b4f8/nuget/v3/flat2/microsoft.net.test.sdk/16.6.1/microsoft.net.test.sdk.16.6.1.nupkg";
-      sha256 = "0jjdg468jc6pv2z764f3xc19lcr772nzjm9cjfqq3bqw8vkpzmhv";
+      sha256 = "188dd2imzd6iwhfxz5b8hfis3i0xgwjkpf3hd08vcvxy095ixb2b";
     };
   }
   {

avdv · 2021-11-18T10:16:05Z

At some point I'd like to do some work to unify all the nuget dependency gathering logic for all these packages.

Also on my TODO list 😄

#133588 might be interesting

Yes, but it does not support different sources, AFIACS.

legendofmiracles · 2021-11-20T15:05:13Z

It would be great if you could also update the dependencies for ArchiSteamFarm in the future when updating dotnet, which was recently converted to not build from release binaries. #145542

avdv · 2021-11-21T16:21:30Z

It would be great if you could also update the dependencies for ArchiSteamFarm in the future when updating dotnet, which was recently converted to not build from release binaries. #145542

Of course. This will be the case in the future, since all dependent packages will be rebuild with nixpkgs-review. It just broke on Hydra because both modifications were done in parallel.

prusnak force-pushed the dotnet-update branch from 8c4652e to 8c27f5b Compare October 21, 2021 12:39

prusnak added 9.needs: port to stable A PR needs a backport to the stable release. 1.severity: security Issues which raise a security issue, or PRs that fix one labels Oct 21, 2021

ofborg bot added 10.rebuild-darwin: 11-100 10.rebuild-linux: 11-100 labels Oct 21, 2021

prusnak requested review from avdv, felschr and jonringer October 23, 2021 22:24

felschr requested changes Oct 24, 2021

View reviewed changes

prusnak force-pushed the dotnet-update branch from 8c27f5b to ce7adaa Compare October 24, 2021 13:35

prusnak force-pushed the dotnet-update branch from ce7adaa to 9befd96 Compare November 9, 2021 09:47

prusnak changed the title ~~dotnet-sdk_3: 3.1.19 -> 3.1.20 + dotnet-sdk_5: 5.0.10 -> 5.0.11~~ dotnet-sdk_3: 3.1.19 -> 3.1.21 + dotnet-sdk_5: 5.0.10 -> 5.0.12 Nov 9, 2021

SuperSandro2000 reviewed Nov 9, 2021

View reviewed changes

pkgs/development/compilers/dotnet/print-hashes.sh Outdated Show resolved Hide resolved

pkgs/development/compilers/dotnet/print-hashes.sh Outdated Show resolved Hide resolved

prusnak force-pushed the dotnet-update branch from 9befd96 to a3cda2c Compare November 9, 2021 10:45

prusnak requested review from mmahut and RaghavSood as code owners November 11, 2021 17:08

prusnak removed request for mmahut and RaghavSood November 11, 2021 17:21

ofborg bot requested a review from corngood November 11, 2021 17:25

prusnak force-pushed the dotnet-update branch from 2b2b66f to c72bbcd Compare November 11, 2021 18:36

prusnak and others added 3 commits November 16, 2021 10:44

dotnetCorePackages:*_3_1: 3.1.19 -> 3.1.21

a018a79

dotnetCorePackages.*_5*: 5.0.10 -> 5.0.12

c94cfb4

ryujinx: Update dependencies

86d5dd0

prusnak force-pushed the dotnet-update branch from c72bbcd to 2d66b3c Compare November 16, 2021 09:45

avdv added 9 commits November 16, 2021 15:54

osu-lazer: Update dependencies

f201ba1

discordchatexporter-cli: Update dependencies

b3d87e4

wasabibackend: Use nixpkgs-fmt instead of nixfmt

caea821

wasabibackend: Update dependencies

8f25617

python-language-server: Update dependencies

be738c1

eventstore: Fix create-deps.sh script and update dependencies

80cf9cd

The old version of the script tried to parse the output of the dotnet tool, but apparently, that output changed and thus the list of dependencies was empty.

alttpr-opentracker: Update dependencies

9a1ee52

github-runner: Update dependencies

5913fdb

prusnak force-pushed the dotnet-update branch from 2d66b3c to 5913fdb Compare November 16, 2021 14:55

prusnak merged commit 1663708 into NixOS:master Nov 16, 2021

prusnak deleted the dotnet-update branch November 16, 2021 15:27

corngood mentioned this pull request Nov 17, 2021

[Guix] msbuild outdated mono dependencies? #81194

Closed

IvarWithoutBones mentioned this pull request Nov 20, 2021

ArchiSteamFarm: 4.3.1.0 -> 5.1.5.3, use buildDotnetModule #145542

Merged

12 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

dotnet-sdk_3: 3.1.19 -> 3.1.21 + dotnet-sdk_5: 5.0.10 -> 5.0.12 #142468

dotnet-sdk_3: 3.1.19 -> 3.1.21 + dotnet-sdk_5: 5.0.10 -> 5.0.12 #142468

prusnak commented Oct 21, 2021 •

edited

Loading

felschr left a comment

jonringer commented Oct 25, 2021

avdv commented Nov 2, 2021

avdv commented Nov 9, 2021

prusnak commented Nov 9, 2021 •

edited

Loading

avdv commented Nov 10, 2021 •

edited

Loading

avdv commented Nov 11, 2021 •

edited

Loading

prusnak commented Nov 11, 2021

prusnak commented Nov 11, 2021

corngood commented Nov 11, 2021

avdv commented Nov 11, 2021

prusnak commented Nov 16, 2021

avdv commented Nov 17, 2021

corngood commented Nov 17, 2021 •

edited

Loading

avdv commented Nov 18, 2021

avdv commented Nov 18, 2021

legendofmiracles commented Nov 20, 2021

avdv commented Nov 21, 2021

dotnet-sdk_3: 3.1.19 -> 3.1.21 + dotnet-sdk_5: 5.0.10 -> 5.0.12 #142468

dotnet-sdk_3: 3.1.19 -> 3.1.21 + dotnet-sdk_5: 5.0.10 -> 5.0.12 #142468

Conversation

prusnak commented Oct 21, 2021 • edited Loading

Motivation for this change

Things done

felschr left a comment

Choose a reason for hiding this comment

jonringer commented Oct 25, 2021

avdv commented Nov 2, 2021

avdv commented Nov 9, 2021

prusnak commented Nov 9, 2021 • edited Loading

avdv commented Nov 10, 2021 • edited Loading

avdv commented Nov 11, 2021 • edited Loading

prusnak commented Nov 11, 2021

prusnak commented Nov 11, 2021

corngood commented Nov 11, 2021

avdv commented Nov 11, 2021

prusnak commented Nov 16, 2021

avdv commented Nov 17, 2021

corngood commented Nov 17, 2021 • edited Loading

avdv commented Nov 18, 2021

avdv commented Nov 18, 2021

legendofmiracles commented Nov 20, 2021

avdv commented Nov 21, 2021

prusnak commented Oct 21, 2021 •

edited

Loading

prusnak commented Nov 9, 2021 •

edited

Loading

avdv commented Nov 10, 2021 •

edited

Loading

avdv commented Nov 11, 2021 •

edited

Loading

corngood commented Nov 17, 2021 •

edited

Loading