[Feature request] auto-detect missing libstdc++.so.6 issue, and suggest fix or auto-apply fix

[aidigital]

EDIT by @savil:
we resolved the immediate bug via being able to add stdenv.cc.cc.lib. However, the developer-UX here could be improved and this task is now tracking that:

• we could auto-detect this scenario and suggest a fix or auto-apply the fix.
• we also need a better story around locking and updating stdenv.cc.cc.lib.

Original bug report:

Current Behavior (bug)
Most (python) packages won't suffer from the problem I'm describing below, but some will.
Still, I think the problem can be addressed by devbox.

dagster is a popular python framework, whose installation provides a package and a executable.
What works:

• first of all, dagster can be installed with your python pip example in the venv
• it can be imported from the venv (i.e. import dagster works)

What doesn't work:

• running the executable fails, due to not finding libstdc++.so.6.
  In my experience, not finding that library is a pretty common error that executables installed via nix run into.

# in the venv
$ dagster
Traceback (most recent call last):
  File "/tmp/tmp.Y07SaqoKmV/.devbox/virtenv/python310Packages.pip/.venv/bin/dagster", line 5, in <mod
ule>
    from dagster.cli import main
  File "/tmp/tmp.Y07SaqoKmV/.devbox/virtenv/python310Packages.pip/.venv/lib/python3.10/site-packages/
dagster/_module_alias_map.py", line 72, in exec_module
    base_module = importlib.import_module(self.base_spec.name)
  File "/nix/store/qpwrwhkz7hkp8069v369mclp3yd93p6v-python3-3.10.13/lib/python3.10/importlib/__init__
.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/tmp/tmp.Y07SaqoKmV/.devbox/virtenv/python310Packages.pip/.venv/lib/python3.10/site-packages/
dagster/_cli/__init__.py", line 4, in <module>
    from .api import api_cli
  File "/tmp/tmp.Y07SaqoKmV/.devbox/virtenv/python310Packages.pip/.venv/lib/python3.10/site-packages/
dagster/_cli/api.py", line 14, in <module>
    from dagster._cli.workspace.cli_target import (
  File "/tmp/tmp.Y07SaqoKmV/.devbox/virtenv/python310Packages.pip/.venv/lib/python3.10/site-packages/
dagster/_cli/workspace/__init__.py", line 1, in <module>
    from .cli_target import (
  File "/tmp/tmp.Y07SaqoKmV/.devbox/virtenv/python310Packages.pip/.venv/lib/python3.10/site-packages/
dagster/_cli/workspace/cli_target.py", line 29, in <module>
    from dagster._core.host_representation.code_location import CodeLocation
  File "/tmp/tmp.Y07SaqoKmV/.devbox/virtenv/python310Packages.pip/.venv/lib/python3.10/site-packages/
dagster/_core/host_representation/__init__.py", line 9, in <module>
    from .external import (
  File "/tmp/tmp.Y07SaqoKmV/.devbox/virtenv/python310Packages.pip/.venv/lib/python3.10/site
dagster/_core/host_representation/external.py", line 38, in <module>
    from dagster._core.host_representation.origin import (
  File "/tmp/tmp.Y07SaqoKmV/.devbox/virtenv/python310Packages.pip/.venv/lib/python3.10/site
dagster/_core/host_representation/origin.py", line 16, in <module>
    import grpc
  File "/tmp/tmp.Y07SaqoKmV/.devbox/virtenv/python310Packages.pip/.venv/lib/python3.10/site
grpc/__init__.py", line 22, in <module>
    from grpc import _compression
  File "/tmp/tmp.Y07SaqoKmV/.devbox/virtenv/python310Packages.pip/.venv/lib/python3.10/site
grpc/_compression.py", line 20, in <module>
    from grpc._cython import cygrpc
ImportError: libstdc++.so.6: cannot open shared object file: No such file or directory

Expected Behavior (fix)

• Ideally, install the missing dependecy and/or export some missing Env Vars so it can find it
• or at least to least inform the user: "hey, the dependecy you asked for installed a pkg (which will work), and an executable, and we're detecting that the executable will not work"

Why do I think this problem should be addressed by devbox ?
Imagine I didn't use devbox, and I had a global python installation in /nix/store.
I can create a venv off of that global python, run pip install dagster in the venv, and I'd be in a similar place: import dagster would work, but running the executable would fail.
devbox can be a step-up from that experience, and either fix or at least inform the user of the problem and suggest a solution.
As a side note, the above scenario is how I currently do things, except that the global python which I create venvs from is from asdf, because doing that will allow the dagster binary to work.

Additional context

• devbox version

$ devbox version -v
Version:     0.7.1
Platform:    linux_arm64
Commit:      9444baeabcb69c2ec8d67ce57536a44171843d9d
Commit Time: 2023-10-19T19:11:54Z
Go Version:  go1.21.3
Launcher:    0.2.0

• devbox.json

• finding libstdc++.so.6 on my system

$ fd --unrestricted --ignore-case --absolute-path --glob "libstdc++.so.6" /
/usr/lib/aarch64-linux-gnu/libstdc++.so.6 
# it's also in nix store itself, as I've installed gcc

[savil]

@aidigital Thanks for filing this issue. Usually, the workaround for this libstdc++ not found is to do devbox add stdenv.cc.cc.lib. Does that work for you?

Thanks also for your suggestions about improving the overall experience for such a scenario. I agree the developer-UX here can be better.

[aidigital]

@savil Thank you, that worked!

Just for posterity, it turns out there are 2 ways to solve the problem:

• declare stdenv.cc.cc.lib into devbox.json

• if you're a home-manager user, you can declare stdenv.cc.cc.lib in home-manager and export LD_LIBRARY_PATH=$HOME/.nix-profile/lib (in ~/.zshenv or the equivalent depending on your shell).
  This is a bit more generic, as you no longer have to include that dependency into every devbox.json project that might need it.

[savil]

@aidigital great. Glad to hear it worked.

Good point about the home-manager. When developing Devbox, we usually have a frame-of-reference that tries to make the devbox-project be independently runnable, and reproducible. This matters much more for folks collaborating on a project with a team, or want it to be reproducible for CICD. Of course, totally valid to also add it as a global package for convenience as long as you're aware of the trade-off and can say its worth it for your scenario!

Let me rename this task to track your other request of auto-detecting failed libstdc++ import error and suggesting a fix, or auto-applying the fix.

[daniel-fahey]

Try this workaround in your devbox.jsonl to get stdenv.cc.cc.lib from nixpkgs at the nixos-23.05 release version via its flake:

{
  "packages": [
    // ...
    "github:NixOS/nixpkgs/nixos-23.05#stdenv.cc.cc.lib"
  ],
  // ...
}

[aidigital]

I'm not sure if the below is a devbox (latest version, 0.8.1) bug or not.
But basically I found using a specific version of gcc to be more reliable than stdenv.cc.cc.lib.

For example, here I'm using stdenv.cc.cc.lib in the devbox.json file:

{ "packages": [ "stdenv.cc.cc.lib" ] }

The above gives me gcc 12.3:

❯ devbox shell

❯ readlink -f ./.devbox/nix/profile/default/lib/libstdc++.so.6
/nix/store/9fy9zzhf613xp0c3jsjxbjq6yp8afrsv-gcc-12.3.0-lib/lib/libstdc++.so.6.0.30

❯ which gcc
/nix/store/zlzz2z48s7ry0hkl55xiqp5a73b4mzrg-gcc-wrapper-12.3.0/bin/gcc

And if we use strings to look at what libstdc++.so.6 requires, we can see GLIBC_2.38 ...

❯ strings ./.devbox/nix/profile/default/lib/libstdc++.so.6 | grep ^GLIBC | sort | uniq
GLIBC_2.36
GLIBC_2.38  # 👈 oh oh
GLIBCXX_3.4.29
GLIBCXX_3.4.30
# ...

... however my machine has 2.36

❯ ldd --version
ldd (Debian GLIBC 2.36-9+deb12u3) 2.36

So in my scenario, I had stdenv.cc.cc.lib as well as python@3.11.4.
I created a venv, installed dagster in it, activated the venv,

# note from the line below that I'm in both venv and the devbox shell
foo-repo on  master [✓] via 🐍 v3.11.4 (venv) via ❄️  impure (nix-shell-env)
❯ dagster --version
ImportError: /nix/store/ld03l52xq2ssn4x0g5asypsxqls40497-glibc-2.37-8/lib/libc.so.6: version `GLIBC_2.38' not found (required by <...>/foo-repo/.devbox/nix/profile/default/lib/libstdc++.so.6)

If I'm just in the venv but not in the devbox shell, it works, of course.
But I need to be in the devbox shell too, because I have other utilities in it, and that's the whole point of devbox, to open a shell and do your work inside it.

# note I'm just in the venv
foo-repo on  master [✓] via 🐍 v3.11.4 (venv)
❯ dagster --version
dagster, version 1.5.8

---

Fix

Now, if instead of { "packages": [ "stdenv.cc.cc.lib" ] }, I use { "packages": [ "gcc@12.2.0" ] }, the libstdc++.so.6 file will require GLIBC_2.36 instead of GLIBC_2.38, so everything will work:

❯ strings ./.devbox/nix/profile/default/lib/libstdc++.so.6 | grep ^GLIBC | sort | uniq
GLIBC_2.36   # I _do_ have 2.36 on my system, so this will work
GLIBCXX_3.4.29
GLIBCXX_3.4.30
# ...

@savil, instead of downgrading to gcc 12.2, can I specify something else in devbox.json alongside stdenv.cc.cc.lib to also get glibc 2.38 in the devbox shell?

---

Notes:

• this problem (stdenv.cc.cc.lib giving me a version of c++ shared object file which requres a version of c which my system didn't have) happened on this WSL Debian running in an Azure Windows VM.

OS: Debian GNU/Linux 12 (bookworm) on Windows 10.0.19044 x86_64
Kernel: 5.15.90.1-microsoft-standard-WSL2

On a different VM (the below specs) which also has glibc 2.36, until 1-2 weeks ago, stdenv.cc.cc.lib gave me gcc-wrapper-9.5.0, and this worked.
As of the time of this message however, it also gives me gcc 12.3, and same error happens.

OS: Debian GNU/Linux 12 (bookworm) aarch64 
Host: Parallels ARM Virtual Machine Parallels ARM Virtual Platform
Kernel: 6.1.0-13-arm64 

• on both of those machines, totally outside of devbox, using home-manager to install stdenv.cc.cc.lib (correctly, shall we say?) gave me gcc 12.2 not 12.3.
  This may be because my flake.lock file hasn't been updated in a while, so just like devbox last week, it doesn't give me the newest gcc 12.3.
  Hence why I started this message with "this may or may not be a devbox problem".

This also means that in all fairness, I actually don't need either stdenv.cc.cc.lib or gcc in the devbox.json file.
But it feels like having them there shouldn't break things either.

[savil]

@aidigital yeah, this Glibc version mismatch is an issue we've been working on. The underlying cause is a bit involved to explain, but it has to do with what version of glibc the nix packages were built by their maintainers. The glibc version installed on your system isn't relevant: nix packages are designed to be independent of the system by referencing all their dependencies in /nix/store.

We have a preliminary fix that was implemented in #1574. We know that it still has some bugs, so take it with a grain of salt. So:

1. You could try patching your python package with patch_glibc: true as shown in that PR's summary.
2. Note, you'll have to convert the packages in devbox.json from a list to a map, as show in the PR's summary.
3. Unfortunately, this patch_glibc feature currently breaks the Devbox python plugin which normally automatically sets up your venv, so you'll also need to manually do that in a manner similar to the virtenv script in that PR's summary.

Hopefully, that will update your python and dagster to work with the latest glibc 2.38. Let us know if it does?

[aidigital]

@savil, thanks for your suggestion.
I ended up having to temporarily create a venv with asdf due to some problems with the venv cretead by devbox.
Let me explain.

The relavant pkgs are these:

    "python@3.11.5",
    "unixODBC@2.3.12",
    "unixODBCDrivers.msodbcsql18@18.1.1.1-1",
    "stdenv.cc.cc.lib"

I would create a venv, manually, then install dagster and pyodbc.

Here's where it gets good:

• if from my shell I activated the venv and the devbox shell, I would be able to run my dagster pipeline which at some point use pyodbc to read from a mssql db
• but when running the dagster-webserver binary which started the dagster UI and ran the pipeline, no matter in which shells I was running it from, I'd always get

"pyodbc.Error: ('01000', "[01000] [unixODBC][Driver Manager]Can't open lib '/opt/microsoft/msodbcsql18/lib64/libmsodbcsql-18.3.so.2.1' : file not found (0) (SQLDriverConnect)")"

I tried playing with devbox shellenv to try and make it get the correct LD_LIBRARY_PATH etc, but no dice.
Maybe dagster-webserver strips away the env it receives, I don't know.

[jpambrun]

For more context, I just lost 2-3 hours banging my head over this in cgohlke/imagecodecs#87.

Essentially, the python package imagecodecs silently fails with no output making it really difficult to tract to this issue.

[jpambrun]

Welp, that breaks nodejs. I am catching to the other issues. This seems like a very challenging problem..

[aidigital]

Probably it's because stdenv.cc.cc.lib doesn't provide the same libraries that the executable was linked against.

Couple of thoughts:

• can you try installing a different pkg instead of stdenv.cc.cc.lib; e.g. gcc@12.2.0 as I mentioned above ?
• or, as another workaround, you may need to unset the LD_LIBRARY_PATH and set it to what it was set before you ran devbox shell ?

[jpambrun]

gcc doesn't quite help, when ldd the that is breaking I get:

libc.so.6 => /nix/store/gqghjch4p1s69sv4mcjksb2kb65rwqjy-glibc-2.38-23/lib/libc.so.6 (0x00007ff492c16000)
        /nix/store/gqghjch4p1s69sv4mcjksb2kb65rwqjy-glibc-2.38-23/lib64/ld-linux-x86-64.so.2 (0x00007ff492f9b000)
libstdc++.so.6 => not found
...

with stdenv.cc.cc.lib I get

libstdc++.so.6 => /home/jpambrun/work/devboxtest/.devbox/nix/profile/default/lib/libstdc++.so.6 (0x00007f1dc53a0000)

Aslo stdenv.cc.cc.lib breaks nodejs@14, but not nodejs@16

On the positive side, devbox makes this very easy to test.