libsodium NuGet package unusable on win-arm64

[nil4]

It appears that the NuGet package (at least version 1.0.20) includes an invalid libsodium.dll for win-arm64. I'm not sure if this is because it was built with Zig, but this is reproducible on two separate machines, a Windows VM running under Parallels on macOS, and a Lenovo laptop with a Qualcomm ARM64 CPU.

(I found a few issues and discussions around this topic already, and I hope I can contribute to fixing this issue by providing the information below.)

I would be happy to help in any way that I can to find a solution to this issue. At the very least, I hope some of the information below might help improve the build/CI or testing strategy for future releases. For instance, actions/partner-runner-images#19 gives some hope that an earlier announcement may soon materialize:

Get started using Arm-hosted runners today!

These runners are available to our customers on our GitHub Team and Enterprise Cloud plans. We expect to begin offering Arm runners for open source projects by the end of the year.

Steps to reproduce with libsodium NuGet package

Run the following steps on a Windows machine with ARM64 CPU. (To keep it brief, I'll use [...] below to elide some irrelevant bits of output.)

Create a new .NET console app and add the libsodium package reference:

> dotnet --version
9.0.101

> mkdir repro && cd repro

> dotnet new console
The template "Console App" was created successfully.
[...]

> dotnet add repro.csproj package libsodium
[...]
info : PackageReference for package 'libsodium' version '1.0.20' added to file 'repro.csproj'

Edit repro.csproj and make the following change:

 <Project Sdk="Microsoft.NET.Sdk">

    <PropertyGroup>
        <OutputType>Exe</OutputType>
        <TargetFramework>net9.0</TargetFramework>
        <ImplicitUsings>enable</ImplicitUsings>
        <Nullable>enable</Nullable>
+       <AllowUnsafeBlocks>true</AllowUnsafeBlocks>
    </PropertyGroup>

    <ItemGroup>
        <PackageReference Include="libsodium" Version="1.0.20" />
    </ItemGroup>

 </Project>

Edit Program.cs and replace its entire content with:

using System.Runtime.CompilerServices;
using System.Runtime.InteropServices;

int result = LibSodium.sodium_init();
Console.WriteLine("sodium_init() returned {0}", result);

static partial class LibSodium
{
    [LibraryImport("libsodium.dll"), UnmanagedCallConv(CallConvs = [typeof(CallConvCdecl)])]
    internal static partial int sodium_init();
}

Compile and run the app for win-arm64, note that libsodium.dll fails to load at runtime:

> dotnet run -r win-arm64

Unhandled exception. System.BadImageFormatException: An attempt was made to load a program with an incorrect format. (0x8007000B)
    at LibSodium.sodium_init()
    at Program.<Main>$(String[] args) in C:\dev\repro\Program.cs:line 4

Testing a local VC++ build

Clone the libsodium repo locally:

> git clone https://github.com/jedisct1/libsodium.git libsodium_localbuild 

> cd libsodium_localbuild

> git checkout stable
branch 'stable' set up to track 'originstable'.
Switched to a new branch 'stable'

As a quick & dirty test, apply the following patch to builds\msvc\build\buildbase.bat to only build the Release DLL for ARM64 and then immediately exit:

diff --git a/builds/msvc/build/buildbase.bat b/builds/msvc/build/buildbase.bat
index 93352732..554c1852 100644
--- a/builds/msvc/build/buildbase.bat
+++ b/builds/msvc/build/buildbase.bat
@@ -49,6 +49,13 @@ IF NOT EXIST !environment! GOTO no_tools
 
 ECHO Building: %solution%
 
+CALL !environment! ARM64 > nul
+ECHO Platform=ARM64
+
+ECHO Configuration=DynRelease
+msbuild /m /v:n /p:Configuration=DynRelease /p:Platform=ARM64 %solution% >> %log%
+goto end
+
 CALL !environment! x86 > nul
 ECHO Platform=x86
 

Run the batch file and note it succeds:

> pushd builds\msvc\build

> buildbase.bat ..\vs2022\libsodium.sln 17
Environment: "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat"
Building: ..\vs2022\libsodium.sln
Platform=ARM64
Configuration=DynRelease

> popd

Confirm that libsodium.dll was indeed compiled successfully:

> dir bin\ARM64\Release\v143\dynamic\libsodium.dll

 Directory of C:\dev\libsodium_localbuild\bin\ARM64\Release\v143\dynamic

2024-12-04  12:19           222,720 libsodium.dll
               1 File(s)        222,720 bytes

Note that the DLL built with VC++ is quite a bit smaller (222,720 bytes) compared to the one provided in the libsodium package (391,168 bytes)

Finally, update the repro\Program.cs to load the locally-built DLL, making the following change:

 using System;
 using System.Runtime.CompilerServices;
 using System.Runtime.InteropServices;
 
 int result = LibSodium.sodium_init();
 Console.WriteLine(result);
 
 static partial class LibSodium
 {
-     [LibraryImport("libsodium"), UnmanagedCallConv(CallConvs = [typeof(CallConvCdecl)])]
+     [LibraryImport("C:\\dev\\libsodium_localbuild\\bin\\ARM64\\Release\\v143\\dynamic\\libsodium.dll"), UnmanagedCallConv(CallConvs = [typeof(CallConvCdecl)])]
     internal static partial int sodium_init();
 }

Compile and run the .NET program again, noting it now succeeds:

> dotnet run -r win-arm64
sodium_init() returned 0

Detailed version information

> dotnet --info
.NET SDK:
 Version:           9.0.101
 Commit:            eedb237549
 Workload version:  9.0.100-manifests.4ae55677
 MSBuild version:   17.12.12+1cce77968

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.26100
 OS Platform: Windows
 RID:         win-arm64
 Base Path:   C:\Program Files\dotnet\sdk\9.0.101\

[...]

Host:
  Version:      9.0.0
  Architecture: arm64
  Commit:       9d5a6a9aa4

> cl
Microsoft (R) C/C++ Optimizing Compiler Version 19.42.34435 for ARM64
Copyright (C) Microsoft Corporation.  All rights reserved.

> ver
Microsoft Windows [Version 10.0.26100.2314]

[nil4]

It seems that the runtimes\win-arm64\native\libsodium.dll file delivered with the NuGet package isn't actually compatible with Windows running on ARM64.

Ruling out .NET being related to, or the actual root case here -- ref. #1431 (comment) loading this DLL using plain Windows APIs from C and Zig fails very similarly, reporting that it is not a valid executable format.

[jedisct1]

Is there a way to test this without a Windows machine?

[jedisct1]

By the way, is this only on Windows ARM64? Does a library compiled with zig build -Doptimize=ReleaseFast -Dtarget=x86_64-windows work?

[nil4]

I'm not aware of one; I believe a Windows machine or VM with an ARM64 CPU is required AFAIK.
I don't know of any virtualization that could run ARM64 windows on non-arm hardware.

[nil4]

Yes, this is solely about Windows ARM64.

The binary runtimes\win-x64\native\libsodium.dll inside the NuGet package works without any issue on x64 Windows PCs. It seems that something is special (or off) about ARM64 or Zig compiling for ARM64, or something along those lines.

Hence my testing that a VC++ build for ARM64 Windows does indeed work, and proposing the PR.

[nil4]

Is there a way to test this without a Windows machine?

https://github.com/utmapp/UTM seems to be able to run Windows VMs on Apple arm hardware -- if that's available.

ref. utmapp/UTM#6816 (comment)

The Windows ARM64 ISO is publicly-downloadable here: https://www.microsoft.com/en-us/software-download/windows11arm64

[jedisct1]

A new version of the NuGet package has been uploaded.

[nil4]

Much appreciated, thank you!