System.DllNotFoundException: Unable to load shared library 'lmdb' or one of its dependencies

[fscavo]

Version Information
Version of Akka.NET?
1.14.22
Which Akka.NET Modules?
Akka.Cluster
Akka.Cluster.Sharding

Describe the bug
The service used to work fine with the version 1.14.21. After having updated to 1.14.22 the service stopped working and kept throwing System.DllNotFoundException with the following details:
Unable to load shared library 'lmdb' or one of its dependencies. In order to help diagnose loading problems, consider setting the LD_DEBUG environment variable: liblmdb: cannot open shared object file: No such file or directory

[Aaronontheweb]

We just received another bug report via our paid support channel about this issue. This appears to be a regression in our HOCON configuration that didn't get caught because we have the https://www.nuget.org/packages/Akka.DistributedData.LightningDB/ package installed in our test suites.

You can work around this issue, in the short run, by just installing that package.

However, we're working on an emergency patch to roll this back as it represents a breaking change. We're also going to add a regression test to block for this in the future.

[Aaronontheweb]

Actually, it looks like we are referencing it: https://www.nuget.org/packages/Akka.Cluster.Sharding/1.4.22

[Aaronontheweb]

@fscavo is this on Linux or Windows?

[fscavo]

@fscavo is this on Linux or Windows?

This is on Linux and the docker image is mcr.microsoft.com/dotnet/aspnet:5.0.7-focal

[Aaronontheweb]

Could be related to CoreyKaylor/Lightning.NET#141

[andyfurnival]

I’ve always installed lmdb libraries via apt-get since I started using ddata. I always assumed that was a prerequisite

[Aaronontheweb]

Looks like this was even previously reported and fixed CoreyKaylor/Lightning.NET#133

Not sure why this is happening now with v1.4.22.

[Arkatufus]

Workaround

@fscavo For a workaround, if you're not using durable ddata, you can set akka.cluster.sharding.distributed-data.durable.keys=[] to make sure that LightningDB isn't loaded

[Aaronontheweb]

We just received another bug report via our paid support channel about this issue. This appears to be a regression in our HOCON configuration that didn't get caught because we have the https://www.nuget.org/packages/Akka.DistributedData.LightningDB/ package installed in our test suites.

You can work around this issue, in the short run, by just installing that package.

However, we're working on an emergency patch to roll this back as it represents a breaking change. We're also going to add a regression test to block for this in the future.

Looks like none of this is the case. Just spot checked all of our HOCON configuration changes, library updates, and settings class changes. All of them appear to be ok - wouldn't have effected this in any meaningful way.

[Aaronontheweb]

This should stop LMDB from loading automatically for users who don't want to use it intentionally #5175

[Aaronontheweb]

@fscavo we'll publish a fix to resolve this issue for users upgrading, but I want to keep this issue open until CoreyKaylor/Lightning.NET#141 is resolved as that will be a footgun for users in perpetuity until a new version is pushed.

[Aaronontheweb]

I'm amazed you didn't have this issue on v1.4.21 though - the offending HOCON was added back then.

[fscavo]

I'm amazed you didn't have this issue on v1.4.21 though - the offending HOCON was added back then.

The previous version was running with the base image mcr.microsoft.com/dotnet/aspnet:3.1-focal. Could it be possible that aspnet:3.1-focal had liblmdb-dev installed by default?

[Aaronontheweb]

@fscavo I was just able to reproduce your error using https://github.com/petabridge/akkadotnet-cluster-workshop, but I had to set akka.cluster.sharding.remember-entities=on and akka.cluster.sharding.state-store-mode=ddata to reproduce it with v1.4.22.

The issue does not occur when akka.cluster.sharding.remember-entities=off, FYI.

I'm going to try with v1.4.21 and see if that comes up.

[Aaronontheweb]

Version of the SDK we're using: https://github.com/petabridge/akkadotnet-cluster-workshop/blob/dev/src/Akka.CQRS.Pricing.Service/Dockerfile#L1

[Aaronontheweb]

Yeah I'm definitely hitting this problem on v1.4.21 as well. It's possible that you're only hitting it now because of the underlying changes in the base image.

[Aaronontheweb]

Decided on a different approach for fixing this issue and reverted our planned PR. Stay tuned.

[fscavo]

@fscavo I was just able to reproduce your error using https://github.com/petabridge/akkadotnet-cluster-workshop, but I had to set akka.cluster.sharding.remember-entities=on and akka.cluster.sharding.state-store-mode=ddata to reproduce it with v1.4.22.

The issue does not occur when akka.cluster.sharding.remember-entities=off, FYI.

I'm going to try with v1.4.21 and see if that comes up.

I found out that the previous version of Akka was 1.14.20 and not 1.14.21. The other services run Akka 1.14.21 with no issue, they use StateStoreMode.Persistence and base image is mcr.microsoft.com/dotnet/aspnet:3.1-focal. My bad :-(

[Aaronontheweb]

No problem, this change was introduced in v1.4.21:

akka.net/src/contrib/cluster/Akka.Cluster.Sharding/reference.conf

Lines 184 to 195 in 7997b58

	distributed-data {
	# minCap parameter to MajorityWrite and MajorityRead consistency level.
	majority-min-cap = 5
	durable.keys = ["shard-*"]
	# When using many entities with "remember entities" the Gossip message
	# can become to large if including to many in same message. Limit to
	# the same number as the number of ORSet per shard.
	max-delta-elements = 5
	}
	}

This is what's causing the system to attempt to load LMDB

[Aaronontheweb]

Have a solution pending here: #5180

We're going to put this up into a nightly package after it's merged. I just verified a few moments ago that this ran fine inside https://github.com/petabridge/akkadotnet-cluster-workshop without having to install any custom packages inside our Dockerfiles - all of the native DLLs are contained inside https://github.com/akkadotnet/Lightning.NET, which contains the most recent sources for https://github.com/CoreyKaylor/Lightning.NET that are not yet published on NuGet.

This is called a "vendored" package approach; it's fairly common in the JVM ecosystem; and it's, IMHO, a bit of a hack so I don't want this to be a "forever" thing. I've made our LightningDB.Vendored.Akka package hidden on NuGet because I'd rather have everyone in the ecosystem use the original, but I always don't want to leave our own users in the lurch while we wait for an official release. I will be very happy to drop our vendored version of the package and switch back to the original as soon as it's possible.

I'll post an update on this thread when there is a nightly available - please give that a try @fscavo and let us know if that runs in your environment. If it does, we'll push v1.4.23 on Monday.

[Arkatufus]

I ran apt search liblmdb on some of the dotnet docker containers and none of them have lmdb installed, but the WSL image for Windows 10 does, so this might cause a build that runs on local WSL linux to fail to run inside a docker container.

[Aaronontheweb]

I've been able to validate in local Docker builds that #5180 resolved the issue, but in post-mortem analysis with @Arkatufus this morning we determined that this issue really slipped through our testing matrix because of #3573 and we will need to add MNTR support to Linux going forward to catch these types of issues. I've begun preliminary work on that here #5182 but it will probably take a while to get right.

[Aaronontheweb]

Was able to validate that Windows still works with the latest nightlies here as well: https://github.com/Aaronontheweb/Akka.Cluster.Sharding.DDataDemo

Going to open a separate issue for this, but that code sample definitely exposed some bottlenecks with ShardRegion that need to be resolved

[Aaronontheweb]

Ok, pushing Akka.NET v1.4.23 live with this fix included. Validated inside Docker, Ubuntu WSL, and Windows desktop.