Linux .Net - Unable to load shared library 'Cntk.Core.CSBinding-2.6.dll'

[foldedwave]

I am having issues running any CNTK programs on linux using .net core.

Steps to reproduce

• Create a new vm and install ubuntu 16.04
• Install CNTK using the scripted install method on https://docs.microsoft.com/en-us/cognitive-toolkit/Setup-Linux-Binary-Script
• Install dotnet core 2.1 using instructions on https://www.microsoft.com/net/download/linux-package-manager/ubuntu16-04/sdk-2.1.301
• Create a new console app using
  dotnet new console
• Add CNTK NuGet to the app using
  dotnet add package CNTK.CPUOnly
• Update program.cs with the following code

using System;
using CNTK;

namespace cntktest
{
    class Program
    {
        static void Main(string[] args)
        {
            var device = DeviceDescriptor.CPUDevice;
        }
    }
}

• Run the program using
  dotnet run

Expected output

Program completes successfully.

Actual Output

Program terminates with the following exception.
Unhandled Exception: System.TypeInitializationException: The type initializer for 'CNTK.CNTKLibPINVOKE' threw an exception. ---> System.TypeInitializationException: The type initializer for 'SWIGExceptionHelper' threw an exception. ---> System.DllNotFoundException: Unable to load shared library 'Cntk.Core.CSBinding-2.6.dll' or one of its dependencies. In order to help diagnose loading problems, consider setting the LD_DEBUG environment variable: libCntk.Core.CSBinding-2.6.dll: cannot open shared object file: No such file or directory
   at CNTK.CNTKLibPINVOKE.SWIGExceptionHelper.SWIGRegisterExceptionCallbacks_CNTKLib(ExceptionDelegate applicationDelegate, ExceptionDelegate arithmeticDelegate, ExceptionDelegate divideByZeroDelegate, ExceptionDelegate indexOutOfRangeDelegate, ExceptionDelegate invalidCastDelegate, ExceptionDelegate invalidOperationDelegate, ExceptionDelegate ioDelegate, ExceptionDelegate nullReferenceDelegate, ExceptionDelegate outOfMemoryDelegate, ExceptionDelegate overflowDelegate, ExceptionDelegate systemExceptionDelegate)

Impact

Cannot develop using .net on Linux.

Notes

I have tried adding the following to the csproj file, inside the propertygoup element, to ensure the app runs in 64bit
<Platforms>x64</Platforms>

[waynemunro]

I am developing on Windows and getting the same issue

DllNotFoundException: Unable to load DLL 'Cntk.Core.CSBinding-2.6.dll': The specified module could not be found. (Exception from HRESULT: 0x8007007E)

[aaad]

Same on Windows for the nightly build: Unable to load DLL 'Cntk.Core.CSBinding-2.7-rc0.dev20181116.dll': The specified module could not be found. (Exception from HRESULT: 0x8007007E)

[kanadaj]

@waynemunro @aaad on Windows the issue happens because you need to separately install the CNTK binaries (or add the binaries to your project itself) as the NuGet package only contains the managed portion of the framework. See https://docs.microsoft.com/en-us/cognitive-toolkit/Setup-Windows-Binary-Script

@foldedwave I'm faced with the same issue; trying to run CNTK in a Linux based Docker container based on Microsoft's CNTK image. The problem is that P/Invoke is trying to link against the Windows .dll instead of the Linux .so file. Have you tried installing the 2.7 preview and try to run against that?

[kanadaj]

@foldedwave Okay, so apparently for Linux, instead of the NuGet package you want to download the complete Linux binary package and use the managed DLL from that. You can grab it at https://github.com/Microsoft/CNTK/releases

[kanadaj]

Though the error remains somehow, even if I add the .so files to the project. Fairly sure my container should also have the files since it's the CNTK image

[kanadaj]

It seems DLLImport doesn't seem to find the CNTK libraries even though they are on the LD_LIBRARY_PATH:

DllNotFoundException: Unable to load shared library 'Cntk.Core.CSBinding-2.6.so' or one of its dependencies. In order to help diagnose loading problems, consider setting the LD_DEBUG environment variable: libCntk.Core.CSBinding-2.6.so.so: cannot open shared object file: No such file or directory

The error message is rather weird though - what's the random .so.so? Anyway, this is using the official docker image for CNT 2.6 Python 3 CPU-Only

[Rleahy]

I'm having the same issue trying to get the CNTK C# interface working on the CNTK docker, did you find a solution?

All though my error references the DLL rather than so.

System.DllNotFoundException : Unable to load shared library 'Cntk.Core.CSBinding-2.6.dll' or one of its dependencies. In order to help diagnose loading problems, consider setting the LD_DEBUG environment variable: libCntk.Core.CSBinding-2.6.dll: cannot open shared object file: No such file or directory

[kanadaj]

No, personally I ended up using Python in a docker container with Tensorflow backend since CNTK simply wouldn't work - but I could only do this since I was using Keras in the first place.

[kikniknik]

I faced the same error while trying to use CNTK with dotnet core 3.0 on linux and finally, the following procedure worked:

Install Open MPI following Step 1 from Setup Linux Binary Manual (https://docs.microsoft.com/en-us/cognitive-toolkit/Setup-Linux-Binary-Manual):

$ sudo apt-get install g++
$ https://www.open-mpi.org/software/ompi/v1.10/downloads/openmpi-1.10.3.tar.gz
$ tar -xzf ./openmpi-1.10.3.tar.gz
$ cd openmpi-1.10.3
$ ./configure --prefix=/usr/local/mpi
$ make -j all
$ sudo make install

Download CNTK linux release from https://github.com/Microsoft/CNTK/releases and add lib directories in $LD_LIBRARY_PATH (eg in ~/.bashrh):

export LD_LIBRARY_PATH=cntk_path/cntk/lib:cntk_path/cntk/dependencies/lib:$LD_LIBRARY_PATH

Build project targeting x64 platform:

$ dotnet build -o bin/Debug/netcoreapp3.0 -p:Platform=x64

bin/Debug/netcoreapp3.0 directory should now already include CNTK dlls, but it seems like Cntk.Core.CSBinding-2.7.dll contains native Windows code, so we must copy Cntk.Core.CSBinding-2.7.so from linux release as a DLL:

$ cp cntk_path/cntk/lib/Cntk.Core.CSBinding-2.7.so bin/Debug/netcoreapp3.0/Cntk.Core.CSBinding-2.7.dll

Project should now run with dotnet run.

[songshizhao]

same error on windows just using Nuget pkg，setting the Architecture from anyCpu to x64 fixed it
cntk only suppport x64 (excpt 4 UWP version,maybe support arm)

[kMutagene]

For future reference, i had the same issue vor v 2.8 (2.8.0-rc0.dev20200201) on Ubuntu 20.04 LTS.

I needed to compile opemmpi from source as pointed out by @kikniknik and download the cntk libraries as well (basically follow these steps: https://docs.microsoft.com/en-us/cognitive-toolkit/setup-cntk-on-linux#open-mpi) make sure to install g++ AND make before trying to compile the binaries.

The problem was still not solved, also using the rename hint did not work.

What does work for me is renaming the Cntk.Core.CSBinding.so file to libCntk.Core.CSBinding-2.8-rc0.dev20200201.dll (watch out for the lib prefix here).

So on linux that makes the following steps:

• Create a project (i tested it with net5.0 as well as netcoreapp3.1)
• reference the Cntk CPUOnly nuget package
• download CNTK libs (https://cntk.azurewebsites.net/dllc-2.7.html)
• compile and install OpenMPI from source (https://docs.microsoft.com/en-us/cognitive-toolkit/setup-cntk-on-linux#open-mpi)
• rename /cntk/lib/Cntk.Core.CSBinding.so to libCntk.Core.CSBinding-<your-cntk-version-here>.dll

For windows you just need to set a configuration for x64 on solution level and reference the nuget package.