error: Unable to locate a suitable libpython when running Mojo executable

[scottamain]

Bug description

Compiled Mojo programs that import Python code fail with the message Unable to locate a suitable libpython, please set MOJO_PYTHON_LIBRARY.

This is because Mojo currently does not embed the Python version into Mojo binaries.

As suggested in the error message, the current workaround is to specify the path to your libpython version using the MOJO_PYTHON_LIBRARY environment variable.

UPDATE (Apr 26, 2024):
To get the path to your current Python library and set it in MOJO_PYTHON_LIBRARY, use one of the following commands, depending on whether you've installed the max or mojo package:

# If you installed max:
export MOJO_PYTHON_LIBRARY=$(modular config mojo-max.python_lib)

# If you installed mojo
export MOJO_PYTHON_LIBRARY=$(modular config mojo.python_lib)

To get the path to your current python library, you'll first need to get the location of libpython with this:~

python3 -c 'import sysconfig; print(sysconfig.get_config_var("LIBDIR"))'

Then find the libpython<version>.so filename in that path:

ls $(python3 -c 'import sysconfig; print(sysconfig.get_config_var("LIBDIR"))') | grep libpython

Then, combine the two to create the full path to the .so file (it might be something like /usr/lib/x86_64-linux-gnu/libpython3.10.so) and assign it to the MOJO_PYTHON_LIBRARY variable.

For example, you can set in your .bashrc file and load it:

echo 'export MOJO_PYTHON_LIBRARY="<full_path_to_libpython.so>"' >> ~/.bashrc

source ~/.bashrc

Steps to reproduce

Following code works when running with mojo run:

from python import Python as py
def main():
    foo = py.import_module("builtins").input()
    print(foo)

But when building with mojo build, it complains about MOJO_PYTHON_LIBRARY missing

$ mojo build import_module_input.mojo

 $ ./import_module_input
Mojo/Python interoperability error: Unable to locate a suitable libpython, please set `MOJO_PYTHON_LIBRARY`
fish: Job 1, './import_module_input' terminated by signal SIGSEGV (Address boundary error)

System information

No response

[LiuYuan-SHU]

I encountered same issue, with small difference. On codespace on VSCode, there's only static link lib:

$ ls $(python3 -c 'import sysconfig; print(sysconfig.get_config_var("LIBDIR"))') | grep libpython
libpython3.10.a

Though I still put the absolute path into .bashrc, and run a small piece of code:

# lib.py
def hello():
    print("hello")

from python import Python

def main():
    Python.add_to_path(".")
    let mypython = Python.import_module("lib")

    mypython.hello()

If you run the mojo file with mojo run, everything works fine. Yet if you run with mojo build and ./main:

$ ./main
Segmentation fault (core dumped)

Then I try to find the dumped file:

$ cat /proc/sys/kernel/core_pattern
|/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E
$ ulimit
unlimited
$ pwd
/usr/share/apport
$ ls
package-hooks

There is just nothing.

[sob727]

Same issue here on Debian 12 with Python 3.11

[austinorr]

in the docs here it mentions some workarounds. I use an environment manager (conda) for my python installs and package/dependency mgmt and appended this to my .bashrc file:

...

export MODULAR_HOME=~/.modular
export PATH="$MODULAR_HOME/pkg/packages.modular.com_mojo/bin:$PATH"

find-python-for-mojo() {
    libpath=$(python3 -c 'import sysconfig; print(sysconfig.get_config_var("LIBDIR"))')
    pythonlib=$(ls $libpath | grep "libpython3.*[0-9]\.so$")
    export MOJO_PYTHON_LIBRARY=${libpath}/${pythonlib}
}

now I can activate my env, run find-python-for-mojo and then run my mojo code with the right python linked correctly. I can even do silly things like create a mojo alias that finds the right python lib before running mojo by adding the following to .bashrc:

...
mojopy() {
    find-python-for-mojo
    mojo $@
}

now things work when I test the following mojo file:

hello.mojo

from python import Python

var np: PythonObject = None


fn load():
    try:
        # this doesn't compile outside of a try block.
        # usage of try-except blocks should be documented in the Mojo doc
        # as their usage is enforced by the compiler
        np = Python.import_module("numpy")
    except:
        pass


fn main():
    load()
    np_test()


fn np_test():
    var ar: PythonObject = None
    try:
        ar = np.arange(15).reshape(3, 5)
        print(ar)
        print(ar.shape)
    except:
        pass

$ mamba create -n -y mojo python=3.11 numpy
$ mamba activate mojo
$ mojopy hello.mojo
[[ 0  1  2  3  4]
 [ 5  6  7  8  9]
 [10 11 12 13 14]]
(3, 5)

$ mojopy build hello.mojo
$ ./hello
[[ 0  1  2  3  4]
 [ 5  6  7  8  9]
 [10 11 12 13 14]]
(3, 5)

I'm sure some of these issues around mojo sourcing a hardcoded system pythonlib rather than detecting it from the environment (venv, conda, etc) will be sorted out soon by the developers given that nearly all work in python is done in some type of virtual env. For now, this is helping me try things out. Hope it helps you too!

[vincentsong]

I have same issue in Github Codespaces

[JoaquimCassano]

So, today we cant run compiled apps that requires python tools?

[VMois]

@vincentsong @LiuYuan-SHU The solution for GitHub Codespaces is as follows:

Add conda path:

$ export MOJO_PYTHON_LIBRARY=/opt/conda/lib/libpython3.so

Run your binary:

$ ./llama2

Details of how I got the value. I run mojo with LD_DEBUG to see what library it wants (running llama2.mojo):

LD_DEBUG=libs mojo llama2.mojo stories15M.bin -s 100 -n 256 -t 0.5 -i "Llama is an animal"

Output:

find library=libpython3.10.so.1.0 [0]; searching
     19853:      search path=/opt/conda/lib/./tls/x86_64/x86_64:/opt/conda/lib/./tls/x86_64:/opt/conda/lib/./tls/x86_64:/opt/conda/lib/./tls:/opt/conda/lib/./x86_64/x86_64:/opt/conda/lib/./x86_64:/opt/conda/lib/./x86_64:/opt/conda/lib/.            (RPATH from file /opt/conda/lib/libpython3.so)
     19853:       trying file=/opt/conda/lib/./tls/x86_64/x86_64/libpython3.10.so.1.0
     19853:       trying file=/opt/conda/lib/./tls/x86_64/libpython3.10.so.1.0
     19853:       trying file=/opt/conda/lib/./tls/x86_64/libpython3.10.so.1.0
     19853:       trying file=/opt/conda/lib/./tls/libpython3.10.so.1.0
     19853:       trying file=/opt/conda/lib/./x86_64/x86_64/libpython3.10.so.1.0
     19853:       trying file=/opt/conda/lib/./x86_64/libpython3.10.so.1.0
     19853:       trying file=/opt/conda/lib/./x86_64/libpython3.10.so.1.0
     19853:       trying file=/opt/conda/lib/./libpython3.10.so.1.0
....
....

calling init: /opt/conda/lib/libpython3.so

It looks like libpython3 is omitted in ldd or readelf output. It is loaded dynamically somehow.

[dominic676767]

I've been facing the same error after setting MOJO_PYTHON_LIBRARY=/usr/lib/x86_64-linux-gnu/libpython3.10.so
Same error with miniconda

mojo 0.2.1 (64d14e85)
22.04.1-Ubuntu
Python 3.10.12
num hardware threads: 16 SIMD vector width: 32 checkpoint size: 26954711068 [3490:3490:20230916,103110.135462:ERROR file_io_posix.cc:144] open /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq: No such file or directory (2) [3490:3490:20230916,103110.135514:ERROR file_io_posix.cc:144] open /sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq: No such file or directory (2) Please submit a bug report to https://github.com/modularml/mojo/issues and include the crash backtrace along with all the relevant source codes. Stack dump: 0. Program arguments: mojo llama2.mojo /home/ubu[3490:3491:20230916,103110.136875:ERROR directory_reader_posix.cc:42] opendir /home/ubuntu/.modular/crashdb/attachments/b60f7b88-2ab2-4d1d-bfa8-2679176e946c: No such file or directory (2) ntu/llama2.c/llama2_7b.bin -s 100 -n 256 -t 0.5 -i hello #0 0x000055d7a6b1ef17 (/home/ubuntu/.modular/pkg/packages.modular.com_mojo/bin/mojo+0x5bcf17) #1 0x000055d7a6b1caee (/home/ubuntu/.modular/pkg/packages.modular.com_mojo/bin/mojo+0x5baaee) #2 0x000055d7a6b1f5ef (/home/ubuntu/.modular/pkg/packages.modular.com_mojo/bin/mojo+0x5bd5ef) #3 0x00007f4ff2a42520 (/lib/x86_64-linux-gnu/libc.so.6+0x42520) #4 0x00007f4f4c003740 [3490:3491:20230916,103111.514156:ERROR http_transport_libcurl.cc:483] HTTP status 404 Segmentation fault (core dumped)

[YuhanXu]

austinorr's method works.
For some unknown reasons,
export MOJO_PYTHON_LIBRARY="/usr/lib/x86_64-linux-gnu/libpython3.10.so" doesn't work.
But Just paste find-python-for-mojo() function into bashrc, then
$ find-python-for-mojo
$ echo $MOJO_PYTHON_LIBRARY
/home/myname/anaconda3/lib/libpython3.9.so
$ mojo test_import_numpy.mojo
That works!

Ubuntu-22.04
mojo 0.2.1 (64d14e85)
Python 3.9.13

[austinorr]

@YuhanXu I'm glad the bash function helped you. Are you able to import other Python libraries like pandas and matplotlib? I’m still seeing errors on everything but numpy.

[jackos]

Workaround for this:

First install miniconda and then find the path to libpython:

echo $(ls $(conda info --base)/lib/libpython*)

Copy that into you .bashrc, .zshrc e.g.:

export MOJO_PYTHON_LIBRARY=/home/ubuntu/miniconda/base/lib/libpython3.11.so

[jfuruness]

I'm running ubuntu. I wanted to import a package of mine I had built to see if mojo sped parts of it up. If I import and run that package without setting MOJO_PYTHON_LIBRARY, using mojo test.mojo, it runs fine. However, if I build it, and then run it, I get a segfault.

If I then set the MOJO_PYTHON_LIBRARY path, then BOTH mojo test.mojo and building it and running the build both result in segfaults.

I attempted the fixes described above, unfortunately for me they didn't help

[jfuruness]

Ah I just realized - the automatic fixes above fail when you have multiple python versions installed. If that's the case, you need to manually set the MOJO_PYTHON_PATH (as described in the original answer). I still haven't figured out how to get it set up with local environments, but it now does run on the global python environment (still haven't figured out how to get it working in a venv)

[jackos]

@jfuruness this blog goes over how to setup a venv and link to libpython: https://www.modular.com/blog/using-mojo-with-python

[Rong-Tao]

I kinda work around this by using a Makefile to tidy things up:
`
.PHONY: all build clean run

all: build

build:
mojo build main.🔥

run:
MOJO_PYTHON_LIBRARY=/home/rong/miniconda3/envs/cv/lib/libpython3.11.so ./main

clean:
rm -f main
`

[default-writer]

I kinda work around this by using a Makefile to tidy things up: ` .PHONY: all build clean run

all: build

build: mojo build main.🔥

run: MOJO_PYTHON_LIBRARY=/home/rong/miniconda3/envs/cv/lib/libpython3.11.so ./main

clean: rm -f main `

Thats the key point - you have to setup MOJO_PYTHON_LIBRARY for the process to run anyway, that's not a ideal solution anyway.

Imagine if all Gnome python backend libraries in Ubuntu will require to setup such step for specific lirary not found in system.

Even ln -s would probably with no help whatsoever in such a case. There might be a system solution, at least for each OS to find dynamic-linked library.

It might be better to setup a static linked library bindings inside mojo itself to build, i suppose.

[default-writer]

MOJO_PYTHON_LIBRARY

Can someone just add a -L swithch for lookup path by analogy with -I switch ?

-L "/usr/lib/x86_64-linux-gnu/"

#!/usr/bin/env bash
set -e

cwd=$(pwd)

path=$1
if [[ "${path}" == "" ]]; then
    path=$(dirname 0$)
fi

cd "${path}"

[[ ! -d "${path}/bin" ]] && mkdir "${path}/bin"

# Build the mojo program
mojo build "${path}/src/main.mojo" -o "${path}/bin/main" --debug-level full --no-optimization -L "/usr/lib/x86_64-linux-gnu/"

echo "OK"

cd "${cwd}"

[olmirr]

There's no libpython.so file with .so extension on macOS, how to set MOJO_PYTHON_LIBRARY then?

[default-writer]

There's no libpython.so file with .so extension on macOS, how to set MOJO_PYTHON_LIBRARY then?

Then you have to compile it by yourself if mojo does not officially supports OS X, isn't that hard, OK?

[austinorr]

There's no libpython.so file with .so extension on macOS, how to set MOJO_PYTHON_LIBRARY then?

On Mac look for .dylib extension instead, no need to build it yourself. The mojo blogs and docs all mention Mac, so it’s definitely supported.

[qiuzhanghua]

export MOJO_PYTHON_LIBRARY=/Users/q/anaconda3/envs/all/lib/libpython3.10.dylib

for Mac and anaconda, it is OK.

[HadiSDev]

How would this work in a poetry environment. Where is the path to libpython? Do I have to use conda for this?

[felixf4xu]

How would this work in a poetry environment. Where is the path to libpython? Do I have to use conda for this?

my case seems to be simpler, just use export MOJO_PYTHON_LIBRARY=/usr/lib/x86_64-linux-gnu/libpython3.10.so.1.0 then it's fixed

I did not install conda.

[megamorf]

How would this work in a poetry environment. Where is the path to libpython? Do I have to use conda for this?

my case seems to be simpler, just use export MOJO_PYTHON_LIBRARY=/usr/lib/x86_64-linux-gnu/libpython3.10.so.1.0 then it's fixed

I did not install conda.

Unfortunately this does not explain how mojo can be used with poetry. It looks like there's no shared object in the poetry venv where the binaries and libraries get stored:

projects/mojo/mojo_cli_testproject   
▶ poetry env info                                                                                     

Virtualenv
Python:         3.11.3
Implementation: CPython
Path:           /home/megamorf/.cache/pypoetry/virtualenvs/mojo-cli-testproject-4zCH1SF2-py3.11
Executable:     /home/megamorf/.cache/pypoetry/virtualenvs/mojo-cli-testproject-4zCH1SF2-py3.11/bin/python
Valid:          True

System
Platform:   linux
OS:         posix
Python:     3.11.3
Path:       /home/linuxbrew/.linuxbrew/opt/python@3.11
Executable: /home/linuxbrew/.linuxbrew/opt/python@3.11/bin/python3.11
(mojo-cli-testproject-py3.11) 
projects/mojo/mojo_cli_testproject   
▶ find /home/megamorf/.cache/pypoetry/virtualenvs/mojo-cli-testproject-4zCH1SF2-py3.11 -type f -name '*.so'
(mojo-cli-testproject-py3.11) 
projects/mojo/mojo_cli_testproject   
▶ python3 -c 'import sysconfig; print(sysconfig.get_config_var("LIBDIR"))'                       
/home/linuxbrew/.linuxbrew/opt/python@3.11/lib
▶ ls $(python3 -c 'import sysconfig; print(sysconfig.get_config_var("LIBDIR"))') | grep libpython     
libpython3.11.so
libpython3.11.so.1.0
libpython3.so
(mojo-cli-testproject-py3.11)

Right now this means mojo cannot resolve packages that are installed in my poetry managed virtual environment.

[richardkiss]

On MacOS, I had hard-coded paths break frequently due to brew upgrade swapping in new python versions with slightly different paths, so I finally settled on adding this to my .zshrc, using ideas from the post from @austinorr above:

_T=$(python3 -c 'import sysconfig; print(sysconfig.get_config_var("LIBDIR"))')
export MOJO_PYTHON_LIBRARY=${_T}/$(/bin/ls ${_T}| grep -E "^libpython(.+)(so|dylib)$")
unset _T

Be sure to have your PATH set from brew first so the correct python is located. This works both on MacOS in zsh and on ubuntu in bash. At least, for now, and for me. But maybe it will be helpful to others.

Notes: I use /bin/ls because ls is aliased to ls -G -F which adds a @ to symlinks, and this gets around that. The _T variable is only used here (but twice), so can be unset.

[Levinin]

Same issue with Mojo segfaulting with compiled code needing python packages e.g. numpy.
Similar to some others using conda/mamba, got this working adding export MOJO_PYTHON_LIBRARY=~/mambaforge/lib/libpython3.so to .bashrc.

[jackos]

Resolved by b677622. Mojo now finds libpython using the python executable on the top of PATH.

Please try again with modular install mojo or modular install max, if it still fails try running the script in https://modul.ar/fix-python, if you don't have a compatible python on PATH, you can install conda using that same guide.