Unable to load shared library 'OpenCvSharpExtern' VS for Mac Web App deployed using Docker compose

[Lerseb]

Summary of your issue

Environment

Mac Osx 10.15.7
Visual Studio 2019 For Mac
Project is built in Docker container on local Mac using VS Docker compose feature.
installed packages
OpenCvSharp4 4.5.0 20201013
OpenCvSharp4.runtime.osx.10.15-64 4.5.0 20201013
Just installed in case it could help
OpenCvSharp4.runtime.debian.10-amd64 4.3.0 20200424
OpenCvSharp4.runtime.multi-arch 4.3.0 20200527

What did you do when you faced the problem?

The project (Aspnetcore web application) compile properly. It crashes when I pass the code below because OpenCvSharpExtern is missing.
var org = new Mat(image);

Example code:

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Drawing;
using System.IO;
using System.Linq;
using OpenCvSharp;
using OpenCvSharp.Dnn;

namespace YoloOpenCVSharp
{
    public class Inference 
    {
        #region const/readonly


        //random assign color to each label
        private static readonly Scalar[] Colors = Enumerable.Repeat(false, 80).Select(x => Scalar.RandomColor()).ToArray();

        //get labels from coco.names
        private static readonly string[] Labels = File.ReadAllLines(Path.Combine(InferenceConstants.Location, InferenceConstants.Names)).ToArray();
        #endregion

        public static (Mat[], Mat) LoadModelAndImage(string imagePath, string webHostPath)
        {
            var image = imagePath;
            var cfg = Path.Combine(webHostPath, InferenceConstants.Location, InferenceConstants.Cfg);
            var model = Path.Combine(webHostPath, InferenceConstants.Location, InferenceConstants.Weight);
            var threshold = InferenceConstants.Confidence;       //for confidence 
            const float nmsThreshold = 0.3f;    //threshold for nms
           

            //get image
            var org = new Mat(image);

            //setting blob, size can be:320/416/608
            //opencv blob setting can check here https://github.com/opencv/opencv/tree/master/samples/dnn#object-detection
            var blob = CvDnn.BlobFromImage(org, 1.0 / 255, new OpenCvSharp.Size(416, 416), new Scalar(), true, false);

            //load model and config, if you got error: "separator_index < line.size()", check your cfg file, must be something wrong.
            var net = CvDnn.ReadNetFromDarknet(cfg, model);
            #region set preferable
            net.SetPreferableBackend(Net.Backend.OPENCV);
            /*
            0:DNN_BACKEND_DEFAULT 
            1:DNN_BACKEND_HALIDE 
            2:DNN_BACKEND_INFERENCE_ENGINE
            3:DNN_BACKEND_OPENCV 
             */
            net.SetPreferableTarget(0);
            /*
            0:DNN_TARGET_CPU 
            1:DNN_TARGET_OPENCL
            2:DNN_TARGET_OPENCL_FP16
            3:DNN_TARGET_MYRIAD 
            4:DNN_TARGET_FPGA 
             */
            #endregion

            //input data
            net.SetInput(blob);

            //get output layer name
            var outNames = net.GetUnconnectedOutLayersNames();
            //create mats for output layer
            var outs = outNames.Select(_ => new Mat()).ToArray();

            #region forward model
            Stopwatch sw = new Stopwatch();
            sw.Start();

            net.Forward(outs, outNames);

            sw.Stop();
            //Console.WriteLine($"Runtime:{sw.ElapsedMilliseconds} ms");
            #endregion
            return (outs, org);
            //get result from all output
            /*GetResult(outs, org, threshold, nmsThreshold);

            using (new Window("died.tw", org))
            {
                Cv2.WaitKey();
            }*/
        }
}
}

Output:

An unhandled exception occurred while processing the request.

DllNotFoundException: Unable to load shared library 'OpenCvSharpExtern' or one of its dependencies. In order to help diagnose loading problems, consider setting the DYLD_PRINT_LIBRARIES environment variable: dlopen(libOpenCvSharpExtern, 1): image not found
OpenCvSharp.NativeMethods.redirectError(CvErrorCallback errCallback, IntPtr userdata, ref IntPtr prevUserdata)

TypeInitializationException: The type initializer for 'OpenCvSharp.NativeMethods' threw an exception.
OpenCvSharp.NativeMethods.imgcodecs_imread(string fileName, int flags, out IntPtr returnValue)

What did you intend to be?

Inference on the provided image using my trained Yolo model (cfg, weights and names)
Image is provided in the attached files

[ArXen42]

I'd recommend checking issue I created some time ago for this problem:
#920

In short: the only workaround I've found is to build OpenCV and OpenCVSharpExtern right inside your container. Works quite well.

[Lerseb]

Thanks for the answer.
I am trying to build opencvsharp_base container using the Dockerfile you provide and I am getting this error.

make[2]: *** [OpenCvSharpExtern/CMakeFiles/OpenCvSharpExtern.dir/build.make:154: OpenCvSharpExtern/CMakeFiles/OpenCvSharpExtern.dir/features2d.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [CMakeFiles/Makefile2:94: OpenCvSharpExtern/CMakeFiles/OpenCvSharpExtern.dir/all] Error 2
make: *** [Makefile:130: all] Error 2
The command '/bin/sh -c cd /opencvsharp/make && cmake /opencvsharp/src && make -j12 && make install' returned a non-zero code: 2

I tried updating CMake and Make but it didn't solve the issue.
I am just starting with Docker so it might be something obvious.
Thanks.
Sebastien

[Lerseb]

I just did another try with the dockerfile below but didn't work:
Same issue with OpenCvSharpExtern missing. I don't know how to check that everything is installed properly in the container.

FROM mcr.microsoft.com/dotnet/core/aspnet:3.1-buster-slim AS base
WORKDIR /app
EXPOSE 80

FROM mcr.microsoft.com/dotnet/core/sdk:3.1-buster AS build

ENV OPENCV_VERSION=4.4.0

RUN apt-get update && apt-get install -y
apt-transport-https
software-properties-common
wget
unzip
curl
ca-certificates
#bzip2
#grep sed dpkg

Install opencv dependencies

RUN cd ~
RUN apt-get update && apt-get install -y
build-essential
cmake
git
gfortran
libjpeg8-dev
libpng-dev
software-properties-common
RUN add-apt-repository "deb http://security.ubuntu.com/ubuntu xenial-security main" && apt-get update && apt-get install -y
libjasper1
libtiff-dev
libavcodec-dev
libavformat-dev
libswscale-dev
libdc1394-22-dev
libxine2-dev
libv4l-dev

RUN cd /usr/include/linux
RUN ln -s -f ../libv4l1-videodev.h videodev.h
RUN cd ~
RUN apt-get install -y
libgtk2.0-dev libtbb-dev qt5-default
libatlas-base-dev
libfaac-dev
libmp3lame-dev
libtheora-dev
libvorbis-dev
libxvidcore-dev
libopencore-amrnb-dev
libopencore-amrwb-dev
libavresample-dev
x264
v4l-utils

Setup OpenCV source

RUN wget https://github.com/opencv/opencv/archive/${OPENCV_VERSION}.zip &&
unzip ${OPENCV_VERSION}.zip &&
rm ${OPENCV_VERSION}.zip &&
mv opencv-${OPENCV_VERSION} opencv

Setup opencv-contrib Source

RUN wget https://github.com/opencv/opencv_contrib/archive/${OPENCV_VERSION}.zip &&
unzip ${OPENCV_VERSION}.zip &&
rm ${OPENCV_VERSION}.zip &&
mv opencv_contrib-${OPENCV_VERSION} opencv_contrib

Build OpenCV

RUN cd opencv && mkdir build && cd build &&
cmake
-D OPENCV_EXTRA_MODULES_PATH=/opencv_contrib/modules
-D CMAKE_BUILD_TYPE=RELEASE
-D BUILD_SHARED_LIBS=OFF
-D ENABLE_CXX11=ON
-D BUILD_EXAMPLES=OFF
-D BUILD_DOCS=OFF
-D BUILD_PERF_TESTS=OFF
-D BUILD_TESTS=OFF
-D BUILD_JAVA=OFF
-D BUILD_opencv_app=OFF
-D BUILD_opencv_java=OFF
-D BUILD_opencv_python=OFF
-D BUILD_opencv_ts=OFF
-D BUILD_opencv_js=OFF
-D WITH_GSTREAMER=OFF \
-D OPENCV_ENABLE_NONFREE=ON
.. && make -j4 && make install && ldconfig

WORKDIR /

Download OpenCvSharp

RUN git clone https://github.com/shimat/opencvsharp.git
RUN cd opencvsharp && git fetch --all --tags --prune

Install the Extern lib.

WORKDIR /opencvsharp/src
RUN mkdir /opencvsharp/make
RUN cd /opencvsharp/make && cmake -D CMAKE_INSTALL_PREFIX=/opencvsharp/make /opencvsharp/src && make -j4 && make install
RUN ls /opencvsharp/make

FROM mcr.microsoft.com/dotnet/core/sdk:3.1 AS build-dotnet-env
COPY --from=build-native-env /opencvsharp/make/OpenCvSharpExtern/libOpenCvSharpExtern.so ./
RUN git clone https://github.com/shimat/opencvsharp.git
RUN pwd
RUN ls

Install Build the C# part of OpenCvSharp

WORKDIR /opencvsharp/src/OpenCvSharp
RUN cd /opencvsharp/src/OpenCvSharp
RUN dotnet build -c Release -f netstandard2.1

WORKDIR /opencvsharp/src/OpenCvSharp.Blob
RUN cd /opencvsharp/src/OpenCvSharp.Blob
RUN dotnet build -c Release -f netstandard2.1

WORKDIR /opencvsharp/src/OpenCvSharp.Extensions
RUN cd /opencvsharp/src/OpenCvSharp.Extensions
RUN dotnet build -c Release -f netstandard2.1

RUN mkdir /opencvsharp/build
WORKDIR /opencvsharp/build
RUN cp /libOpenCvSharpExtern.so .
RUN cp /opencvsharp/src/OpenCvSharp/bin/Release/netstandard2.1/* .
RUN cp /opencvsharp/src/OpenCvSharp.Blob/bin/Release/netstandard2.1/* .
RUN cp /opencvsharp/src/OpenCvSharp.Extensions/bin/Release/netstandard2.1/* .
RUN pwd
RUN ls

WORKDIR /src
COPY ObserVisionWeb/ObserVisionWeb.csproj ObserVisionWeb/
COPY BizLogic/BizLogic.csproj BizLogic/
COPY BizDbAccess/BizDbAccess.csproj BizDbAccess/
COPY DataLayer/DataLayer.csproj DataLayer/
COPY PermissionParts/PermissionParts.csproj PermissionParts/
COPY RefreshClaimsParts/RefreshClaimsParts.csproj RefreshClaimsParts/
COPY DataKeyParts/DataKeyParts.csproj DataKeyParts/
COPY ServiceLayer/ServiceLayer.csproj ServiceLayer/
COPY AuthorizeSetup/AuthorizeSetup.csproj AuthorizeSetup/
COPY DataAuthorize/DataAuthorize.csproj DataAuthorize/
COPY FeatureAuthorize/FeatureAuthorize.csproj FeatureAuthorize/
COPY UserImpersonation/UserImpersonation.csproj UserImpersonation/
COPY BlobAccess/BlobAccess.csproj BlobAccess/
COPY OnnxObjectDetection/OnnxObjectDetection.csproj OnnxObjectDetection/
COPY YoloOpenCVSharp/YoloOpenCVSharp.csproj YoloOpenCVSharp/
RUN dotnet restore "ObserVisionWeb/ObserVisionWeb.csproj"
COPY . .

WORKDIR "/src/ObserVisionWeb"
RUN dotnet build "ObserVisionWeb.csproj" -c Release -o /app/build

FROM build AS publish
RUN dotnet publish "ObserVisionWeb.csproj" -c Release -o /app/publish

FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "ObserVisionWeb.dll"]

[ArXen42]

Here is the Dockerfile we currently use. The only change from the one I posted is that OpenCV version set to 4.4.0 (latest is 4.5, didn't have time to upgrade and test yet, so be sure to use 4.4.0 NuGet package in your C# app):

FROM mcr.microsoft.com/dotnet/core/sdk:3.1-focal

ENV OPENCV_VERSION=4.4.0

RUN apt-get update && apt-get install -y \
    apt-transport-https \
    software-properties-common \
    wget \
    unzip \
    curl \
    ca-certificates
    #bzip2 \
    #grep sed dpkg 

# Install opencv dependencies
RUN cd ~
RUN apt-get update && apt-get install -y \
    build-essential \
    cmake \
    git \
    gfortran \
    libjpeg8-dev \
    libpng-dev \
    software-properties-common
RUN add-apt-repository "deb http://security.ubuntu.com/ubuntu xenial-security main" && apt-get update && apt-get install -y \
    libjasper1 \
    libtiff-dev \
    libavcodec-dev \
    libavformat-dev \
    libswscale-dev \
    libdc1394-22-dev \
    libxine2-dev \
    libv4l-dev

RUN cd /usr/include/linux
RUN ln -s -f ../libv4l1-videodev.h videodev.h
RUN cd ~
RUN apt-get install -y \
    libgtk2.0-dev libtbb-dev qt5-default \
    libatlas-base-dev \
    libfaac-dev \
    libmp3lame-dev \
    libtheora-dev \
    libvorbis-dev \
    libxvidcore-dev \
    libopencore-amrnb-dev \
    libopencore-amrwb-dev \
    libavresample-dev \
    x264 \
    v4l-utils

# Setup OpenCV source
RUN wget https://github.com/opencv/opencv/archive/${OPENCV_VERSION}.zip && \
    unzip ${OPENCV_VERSION}.zip && \
    rm ${OPENCV_VERSION}.zip && \
    mv opencv-${OPENCV_VERSION} opencv

# Setup opencv-contrib Source
RUN wget https://github.com/opencv/opencv_contrib/archive/${OPENCV_VERSION}.zip && \
    unzip ${OPENCV_VERSION}.zip && \
    rm ${OPENCV_VERSION}.zip && \
    mv opencv_contrib-${OPENCV_VERSION} opencv_contrib

# Build OpenCV
RUN cd opencv && mkdir build && cd build && \
    cmake \
    -D OPENCV_EXTRA_MODULES_PATH=/opencv_contrib/modules \
    -D CMAKE_BUILD_TYPE=RELEASE \
    -D BUILD_SHARED_LIBS=OFF \
    -D ENABLE_CXX11=ON \
    -D BUILD_EXAMPLES=OFF \
    -D BUILD_DOCS=OFF \
    -D BUILD_PERF_TESTS=OFF \
    -D BUILD_TESTS=OFF \
    -D BUILD_JAVA=OFF \
    -D BUILD_opencv_app=OFF \
    -D BUILD_opencv_java=OFF \
    -D BUILD_opencv_python=OFF \
    -D BUILD_opencv_ts=OFF \
    -D BUILD_opencv_js=OFF \
    -D WITH_GSTREAMER=OFF \ 
    -D OPENCV_ENABLE_NONFREE=ON \
    .. && make -j12 && make install && ldconfig

WORKDIR /

# Download OpenCvSharp
RUN git clone https://github.com/shimat/opencvsharp.git
RUN cd opencvsharp && git fetch --all --tags --prune

# Install the Extern lib.
WORKDIR /opencvsharp/src
RUN mkdir /opencvsharp/make
RUN cd /opencvsharp/make && cmake /opencvsharp/src && make -j12 && make install

ENV LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:/usr/local/share/opencv4/lib/:/usr/local/lib/:/lib/x86_64-linux-gnu/"

I have just re-checked, it builds and works fine (with the docker-compose.yml similar to the one I posted in #920 ). My docker version:

[user@archlinux ~]$ docker -v
Docker version 19.03.13-ce, build 4484c46d9d

If it still can't build the image (getting compilation error), I would recommend to "nuke" your docker by removing every container, image and volume you might have (of course, if there isn't anything valuable there):

#!/usr/bin/env bash

docker rm -f -v $(docker ps -aq)
docker rmi -f  $(docker images -q)
docker volume rm $(docker volume ls -q)
docker network rm $(docker network ls -q)

[Lerseb]

Hi ArXen42,
Thank you very much for helping me.
As I said, I am new in Docker containers (I was previously just adding "Docker support" using a feature in Visual Studio).
In your solution, It looks like you create a first image called opencvsharp_base and you aspnetcore project image depend on the first image.
I have been including opencvsharp_base building in Dockerfile of my AspNetCore project but it must be wrong and I have no "verbose" available to check what is happening when I build the Dockercompose in my project.
Sorry to bother you again but can you detail the steps to create the separate initial opencvsharp_base image and how to make it available for the Dockerfile of my AspNetCore project.
Thanks so much for helping.

[Lerseb]

I have moved forward a little bit but I am still stuck:
When I use the docker-compose.yml file you provide, it doesn't find the context OpenCVSharpEnvironment so I replaced it with :
opencvsharp_base:
image: opencvsharp_base
build:
context: .
cache_from:
- opencvsharp_base
opencvsharp_base image exists and its container is running in my docker dashboard.

When I compile from Docker-Compose :
But now the error is "TypeError: can only concatenate str (not dict) to str"
I tried restarting Docker but it doesnt change anything.

When I compile from my web app project it runs and I can access it but I get the opencvsharpextern library missing.
Thanks for your help

[ArXen42]

@Lerseb OpenCVSharpEnvironment is just a directory with Dockerfile that builds this image, didn't notice I included it and didn't mention. I wouldn't use cache_from some existing image - the whole purpose of docker-compose is to build your whole set of containers at once and guarantee that everything is in-sync. So, it will build opencvsharp_base image first, then it will build everything that depends on it.

Regarding Visual Studio feature for adding docker support: it looked somewhat magical to me when I tried it for the first time myself.
I ended up re-creating all docker stuff myself using VS generated code only as a reference (also I was doing it under linux), initially writing it in the simpliest way possible and gradually adding features as needed - that way it actually settled in my head how these things work.

Regarding your issue - I will try to create sample repo soon.

[Lerseb]

I can build the docker images provided on this page https://github.com/shimat/opencvsharp/blob/master/docker/ubuntu.18.04-x64/Dockerfile
or here : #920
But when I include this image in the FROM command of my dockerfile created by visual studio, none of the opencv nor opencvsharp dependencies are carried over to my final image. And I still miss the opencvsharpextern to use opencvsharp in my web app.
Is there a COPY command to execute at some point to get the opencv and opencvsharp libraries imported to my final image.
Any help would be awesome.

[ArXen42]

@Lerseb sorry for late response, forgot to check my github notifications, but here is example repo:
https://github.com/ArXen42/opencvsharp-docker-example

You can clone it and run docker-compose up --build from the solution directory.
It should build basic web app and run it on port 7000 (which you can change in docker-compose.yml of course) with example image generated at the endpoint:
http://localhost:7000/Example/example.webp

I recommend you to check if it works and compare your solution structure with this example. It may help to just create your project from scratch and copy-paste docker files (Dockerfile and docker-compose.yml) from my repo, changing project name there accordingly.

[Lerseb]

Thank you @ArXen42.
I managed to use OpenCvSharp in my web app yesterday evening.
I also wanted to share the docker image built with your dockerfile for opencvsharp_base. It is available as a public image at : docker pull sebnaucet2002/opencvsharp_base
As an example, my docker-compose is

version: '3.4'

services:
opencvsharp_base:
image: opencvsharp_base
container_name: opencvsharp_observision
build:
dockerfile: DockerfileOpenCv
context: OpenCVSharpEnvironment

observisionweb:
    image: ${DOCKER_REGISTRY-}observisionweb
    restart: unless-stopped
    container_name: webapp_observision
    build:
        context: .
        dockerfile: ObserVisionWeb/DockerfileWeb
    depends_on:
        - postgres_db
        - opencvsharp_base
    ports:
        - 5000:80

postgres_db:
    image: ${DOCKER_REGISTRY-}postgres
    container_name: postgres_observision
    restart: unless-stopped
    environment:
        POSTGRES_USER: "postgres"
        POSTGRES_PASSWORD: "your_password"
    ports:
        - 5432:5432

And my DockerfileWeb (the one building my web app) is
Note that "ENV ASPNETCORE_URLS http://+:80" was very important in my case to open up the webapp in the browser directly when compiling from Visual Studio.

FROM opencvsharp_base

ENV ASPNETCORE_URLS http://+:80

WORKDIR /src

EXPOSE 80

COPY ObserVisionWeb/ObserVisionWeb.csproj ObserVisionWeb/
COPY BizLogic/BizLogic.csproj BizLogic/
#COPY here your other libraries

RUN dotnet restore "ObserVisionWeb/ObserVisionWeb.csproj" --verbosity detailed

COPY . .

WORKDIR /src/ObserVisionWeb

RUN dotnet build "ObserVisionWeb.csproj" -c Release -o /app/build --verbosity detailed
RUN dotnet publish "ObserVisionWeb.csproj" -c Release -o /app --verbosity detailed

WORKDIR /app

ENTRYPOINT ["dotnet", "ObserVisionWeb.dll"]

I would like to thank everybody who helped me troubleshouting this.
As it is solved for me, I will close the issue.