-
Notifications
You must be signed in to change notification settings - Fork 3.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Sqlite Spatial: dotnet crashes when loading Spatial Extension (segmentation fault) #16667
Comments
/cc @bricelam |
Updated above with SQLite version. |
UpdateAdded a Steps to reproduce for Dockerfile (assumes docker is setup on machine)
|
Also, here is my Mac
|
@bricelam do you need anything else from me? |
I don't think so. I just need to repro it locally and dig into the failure. |
Can I help in anyway to dig into the failure @bricelam ? I know you guys are busy and this issue is blocking me. |
Does it repro with something as trivial as this: using static SQLitePCL.raw;
class Program
{
static void Main()
{
SQLitePCL.Batteries_V2.Init();
sqlite3_open(":memory:", out var db);
sqlite3_enable_load_extension(db, 1);
sqlite3_exec(db, "SELECT load_extension('mod_spatialite')");
}
} |
Looks like our CI has been hitting this too. 😁 |
It appears to be in issue with the latest revision of the Homebrew package. WorkaroundUse the previous version: brew install https://raw.githubusercontent.com/Homebrew/homebrew-core/835ae521c065524abdf66578e68032fa24bce514/Formula/libspatialite.rb |
@fxcoudert What's the proper channel for reporting this? |
It may (or may not) be related to the fact that our native SQLite library is named libe_sqlite3.dylib not libsqlite3.dylib |
Amazing @bricelam - looking now :) |
Feedback@bricelam - sorry for the delay. I reverted the version back using the workaround, but this has caused a new set of errors explained now. New Errors
Interestingly now when I run
Could it be some
I even updated my <Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>netcoreapp3.0</TargetFramework>
<IsPackable>false</IsPackable>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Microsoft.Data.Sqlite" Version="3.0.0-preview7.19362.6" />
<PackageReference Include="Microsoft.EntityFrameworkCore" Version="3.0.0-preview7.19362.6" />
<PackageReference Include="Microsoft.EntityFrameworkCore.Sqlite" Version="3.0.0-preview7.19362.6" />
<PackageReference Include="Microsoft.EntityFrameworkCore.Sqlite.NetTopologySuite" Version="3.0.0-preview7.19362.6" />
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.3.0-preview-20190715-02" />
<PackageReference Include="xunit" Version="2.4.0" />
<PackageReference Include="xunit.runner.visualstudio" Version="2.4.0" />
<PackageReference Include="SQLitePCLRaw.bundle_e_sqlite3" Version="2.0.0-pre20190628101813" />
<PackageReference Include="NETStandard.Library" Version="1.6.1" />
</ItemGroup>
</Project>
|
Unfortunately, we are out of ideas here. We are going to put this on the backlog, but we are likely to need expert help to make real progress here. |
This is a critical blocker for us. As a community effort, even with everyone trying their best, crash bugs thrown into the "too hard" bucket mean it can't be taken seriously as something a company's income depends upon. Hard-crash issues need proper engineering resources from Microsoft dedicating to them so that people who are actually able to diagnose and fix the issue are promptly assigned. Perhaps the bottom line is that although EntityFrameworkCore is cross-platform, a crash on a non-Windows platform isn't where the money is and isn't going to get much traction. If there's anything that can be done to escalate this I'm sure others would appreciate it. |
@pond What version of EF Core are you using? |
@pond the crash is caused by changes made to the libspatialite Homebrew package. I've filed Homebrew/homebrew-core#43203. See my comment above for a workaround. |
@bricelam The work-around doesn't help I'm afraid - it was the first thing I tried. It doesn't matter what version of that package I install; the crash happens anyway. It looks likely to be a combination of specific versions of several packages. I have isolated a single machine which works, and will try to replicate that particular configuration on another machine as soon as I have time. @ajcvickers Unsure how to get a version for that particular component; .Net SDK 2.2.4 ("latest download" at the time of writing) and again "most recent" Nuget packages related to EFC & SQLite... Looks like there are references to Microsoft.EntityFrameworkCore.Sqlite 2.2.6 and Microsoft.EntityFrameworkCore.Sqlite.NetTopologySuite 2.2.6 in the |
By carefully unpicking all newer versions and installing specific versions of the runtime and related libraries, I have my own test suite running on SQLite that used to crash. Notably, the version of
It's quite an ugly process. I imagine that most of these carefully chosen versions are not necessary, so I'll edit and update this answer as I narrow things down, until I run out of time for it anyway. First:
Next, remove a few Homebrew packages: brew uninstall postgis
brew uninstall gdal
brew uninstall libspatialite
brew uninstall libgeotiff
brew uninstall proj
brew uninstall sqlite ...resolving dependencies (or ignoring them via SQLite 3.28.0 brew install https://raw.githubusercontent.com/Homebrew/homebrew-core/4fa421cfc34548eb48bc236460584c9f4fa16b22/Formula/sqlite.rb Proj 6.1.0
Libgeotiff seems unrelated, so we just install the most recent version. For this and further calls, Homebrew doesn't seem to recognise the from-SHA installations of dependencies so halts; you have to install with brew install --ignore-dependencies libgeotiff Gdal 2.4.1_1 brew install --ignore-dependencies https://raw.githubusercontent.com/Homebrew/homebrew-core/37ce68573be9fe921ef65d421da4657e2f6d0270/Formula/gdal.rb Libspatialite 4.3.0a_7 (the most recent version at the time of writing, but locking to SHA for safety here) brew install --ignore-dependencies https://raw.githubusercontent.com/Homebrew/homebrew-core/1bbba12ca10e05e7194a6233d4bf5bf56677939f/Formula/libspatialite.rb Postgis 2.5.2_2 (the most recent version at the time of writing, but locking to SHA for safety here) brew install --ignore-dependencies https://raw.githubusercontent.com/Homebrew/homebrew-core/0a2e72798e72e3accead7e793d6cccf4ff89dd70/Formula/postgis.rb |
I too am starting to think this is a problem with PROJ. It looks like it links directly to libspatialite3-0 which breaks SQLite extensions. |
(From @michaeldtaylor on #18068) If I take the SpatiaLiteLoader out into my own app I can get loading the module by explicitly setting the environment variable: DYLD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/ and doing: SELECT load_extension('mod_spatialite.so') |
Any news on when this will be fixed for Macs? |
I didn't get a response from SpatiaLite. I'm working on just building it myself without PROJ over in bricelam/mod_spatialite-NuGet |
Today I learned that our SpatialiteLoader won't actually work on Linux. Apparently modifying LD_LIBRARY_PATH after the first call to dlopen() has no affect. This drastically minimizes the usefulness of distributing Linux binaries of SpatiaLite in our NuGet package. This may also be true for macOS. I'm investigating other possible workarounds to the PROJ issue. |
I was able to confirm my suspicions. This issue stems from PROJ using its own version of SQLite. Here are the relevant parts of the LD_DEBUG log:
Calling load_extension('mod_spatialite') from e_sqlite3 causes the system sqlite3 library to be loaded as a dependency. Then it tries to call sqlite3_extension_init() in mod_spatialite, but it can't reconcile the two versions of SQLite and crashes. If I take e_sqlite3 out of the picture and only use the system sqlite3 version, everything works as expected. (ericsink/SQLitePCL.raw#325 would make this easier) <ItemGroup>
<PackageReference Include="Microsoft.Data.Sqlite.Core" Version="3.1.0" />
<PackageReference Include="SQLitePCLRaw.provider.dynamic_cdecl" Version="2.0.2" />
</ItemGroup> static void Main()
{
SQLite3Provider_dynamic_cdecl
.Setup("sqlite3", new NativeLibraryAdapter("libsqlite3.so.0"));
SQLitePCL.raw.SetProvider(new SQLite3Provider_dynamic_cdecl());
var connection = new SqliteConnection("Data Source=:memory:");
connection.LoadExtension("mod_spatialite");
var command = connection.CreateCommand();
command.CommandText = "SELECT spatialite_version()";
connection.Open();
var version = command.ExecuteScalar();
Console.WriteLine("SpatiaLite Version: " + version);
}
class NativeLibraryAdapter : IGetFunctionPointer
{
readonly IntPtr _library;
public NativeLibraryAdapter(string name)
=> _library = NativeLibrary.Load(name);
public IntPtr GetFunctionPointer(string name)
=> NativeLibrary.TryGetExport(_library, name, out var address)
? address
: IntPtr.Zero;
} |
Another workaround is to manually build and install SpatiaLite without PROJ support: curl https://www.gaia-gis.it/gaia-sins/libspatialite-4.3.0a.tar.gz | tar -xz
cd libspatialite-4.3.0a
if [[ `uname -s` == Darwin* ]]; then
# Minor patch required on Mac OS
sed -i "" "s/shrext_cmds='\`test \\.\$module = .yes && echo .so \\|\\| echo \\.dylib\`'/shrext_cmds='.dylib'/g" configure
fi
./configure --disable-proj
make
make install |
After discussing this with @bricelam and the team, we have decided to close this as external. Brice has done extensive research into making the experience better, but ultimately this comes down to better documentation of workarounds. |
@bricelam Thanks for all your research! |
Take heart and have a look at https://github.com/ericsink/SQLitePCL.raw/blob/master/src/SQLitePCLRaw.nativelibrary/for_netstandard2.cs |
Oh wait, if you're using 2.2, you can just do this: <PackageReference Include="SQLitePCLRaw.provider.sqlite3.netstandard11" Version="1.1.12" /> SQLitePCL.raw.SetProvider(new SQLite3Provider_sqlite3()); |
I'm trying to test out the Spatial Types functionality in EF Core for SQLite In Memory provider.
Now when I run an Xunit test the dotnet executable crashes and I will receive a crash dump report (No application exceptions are thrown).
It seems to crash when trying to access the
mod_spatialite
extension. I followed the following advice from this issue #14402, as my original problem was with receiving a SQLite Exception with a no such function: InitSpatialMetaData error to begin with.Steps to reproduce for Mac
git clone
https://github.com/StratumFive/EFCore.SQLite.SpatialCrash.brew install libspatialit
dotnet test xunit
There is even a console application there to prove it isn't just a Xunit issue.
Further technical details
EF Core version: 2.2.6
Database Provider: Microsoft.EntityFrameworkCore.Sqlite / Microsoft.EntityFrameworkCore.Sqlite.NetTopologySuite
Operating system: Mac Mojave 10.14.5
IDE: Visual Studio Code
SQLite version: 3.24.0 2018-06-04 14:10:15
dotnet info:
.NET Core SDK (reflecting any global.json):
Version: 2.2.300
Commit: 73efd5bd87
Runtime Environment:
OS Name: Mac OS X
OS Version: 10.14
OS Platform: Darwin
RID: osx.10.14-x64
Base Path: /usr/local/share/dotnet/sdk/2.2.300/
Host (useful for support):
Version: 2.2.5
Commit: 0a3c9209c0
.NET Core SDKs installed:
2.2.105 [/usr/local/share/dotnet/sdk]
2.2.106 [/usr/local/share/dotnet/sdk]
2.2.107 [/usr/local/share/dotnet/sdk]
2.2.300 [/usr/local/share/dotnet/sdk]
.NET Core runtimes installed:
Microsoft.AspNetCore.All 2.2.3 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.2.4 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.2.5 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.2.3 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.2.4 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.2.5 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.NETCore.App 2.2.3 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 2.2.4 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 2.2.5 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
The text was updated successfully, but these errors were encountered: