diff --git a/docs/source/index.md b/docs/source/index.md index 7a4c20d5..8a20c6ca 100644 --- a/docs/source/index.md +++ b/docs/source/index.md @@ -24,6 +24,7 @@ PyJulia is tested against Python versions 2.7, 3.5, 3.6, and 3.7. usage troubleshooting api + sysimage pytest how_it_works limitations diff --git a/docs/source/limitations.md b/docs/source/limitations.md index eec49436..ea87d33c 100644 --- a/docs/source/limitations.md +++ b/docs/source/limitations.md @@ -21,7 +21,7 @@ example, the Julia method `sum!` can be called in PyJulia using There was a major overhaul in the module loading system between Julia 0.6 and 1.0. As a result, the "hack" supporting the PyJulia to load PyCall stopped working. For the implementation detail of the hack, -see: +see: For the update on this problem, see: diff --git a/docs/source/sysimage.rst b/docs/source/sysimage.rst new file mode 100644 index 00000000..30dc63b5 --- /dev/null +++ b/docs/source/sysimage.rst @@ -0,0 +1,85 @@ +=========================== + Custom Julia system image +=========================== + +.. versionadded:: 0.4 + +If you use standard ``julia`` program, the basic functionalities and +standard libraries of Julia are loaded from so called *system image* +file which contains the machine code compiled from the Julia code. +The Julia runtime can be configured to use a customized system image +which may contain non-standard packages. This is a very effective way +to reduce startup time of complex Julia packages such as PyCall. +Furthermore, it can be used to workaround the problem in statically +linked Python executable if you have the problem described in +:ref:`statically-linked`. + +How to use a custom system image +================================ + +To compile a custom system image for PyJulia, run + +.. code-block:: console + + $ python3 -m julia.sysimage sys.so + +where ``sys.dll`` and ``sys.dylib`` may be used instead of ``sys.so`` +in Windows and macOS, respectively. + +The command line interface `julia.sysimage` will: + +* Install packages required for compiling the system image in an + isolated Julia environment. +* Install PyCall to be compiled into the system image in an isolated + Julia environment. +* Create the system image at the given path (``./sys.so`` in the above + example). + +To use this system image with PyJulia, you need to specify its path +using ``sysimage`` keyword argument of the `Julia` constructor. For +example, if you run `python3` REPL at the directory where you ran the +above `julia.sysimage` command, you can do + +>>> from julia import Julia +>>> jl = Julia(sysimage="sys.so") + +to initialize PyJulia. To check that this Julia runtime is using the +correct system image, look at the output of ``Base.julia_cmd()`` + +>>> from julia import Base +>>> Base.julia_cmd() + + + +Limitations +=========== + +* ``PyCall`` and its dependencies cannot be updated after the system + image is created. A new system image has to be created to update + those packages. + +* The system image generated by `julia.sysimage` uses a different set + of precompilation cache paths for each pair of ``julia-py`` + executable and the system image file. Precompiled cache files + generated by ``julia`` or a different ``julia-py`` executable cannot + be reused by PyJulia when using the system image generated by + `julia.sysimage`. + +* The absolute path of ``julia-py`` is embedded in the system image. + This system image is not usable if ``julia-py`` is removed. + + +Command line interfaces +======================= + +``python3 -m julia.sysimage`` +----------------------------- + +.. automodule:: julia.sysimage + :no-members: + +``julia-py`` +------------ + +.. automodule:: julia.julia_py + :no-members: diff --git a/docs/source/troubleshooting.md b/docs/source/troubleshooting.md deleted file mode 100644 index f7bb8c2c..00000000 --- a/docs/source/troubleshooting.md +++ /dev/null @@ -1,148 +0,0 @@ -Troubleshooting ---------------- - -### Your Python interpreter is statically linked to libpython - -If you use Python installed with Debian-based Linux distribution such -as Ubuntu or install Python by `conda`, you might have noticed that -PyJulia cannot be initialized properly with Julia ≥ 0.7. This is -because those Python executables are statically linked to libpython. -(See [Limitations](limitations.md) for why that's a problem.) - -If you are unsure if your `python` has this problem, you can quickly -check it by: - -```console -$ ldd /usr/bin/python - linux-vdso.so.1 (0x00007ffd73f7c000) - libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f10ef84e000) - libc.so.6 => /usr/lib/libc.so.6 (0x00007f10ef68a000) - libpython3.7m.so.1.0 => /usr/lib/libpython3.7m.so.1.0 (0x00007f10ef116000) - /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f10efaa4000) - libdl.so.2 => /usr/lib/libdl.so.2 (0x00007f10ef111000) - libutil.so.1 => /usr/lib/libutil.so.1 (0x00007f10ef10c000) - libm.so.6 => /usr/lib/libm.so.6 (0x00007f10eef87000) -``` - -in Linux where `/usr/bin/python` should be replaced with the path to -your `python` command (use `which python` to find it out). In macOS, -use `otool -L` instead of `ldd`. If it does not print the path to -libpython like `/usr/lib/libpython3.7m.so.1.0` in above example, you -need to use one of the workaround below. - -#### Turn off compilation cache - -The easiest workaround is to pass `compiled_modules=False` to the -`Julia` constructor. - -```pycon ->>> from julia.api import Julia ->>> jl = Julia(compiled_modules=False) -``` - -This is equivalent to `julia`'s command line option -`--compiled-modules=no` and disables the precompilation cache -mechanism in Julia. Note that this option slows down loading and -using Julia packages especially for complex and large ones. - -See also [low-level API](api.md) - -#### `python-jl`: an easy workaround - -Another easy workaround is to use the `python-jl` command bundled in -PyJulia. This can be used instead of normal `python` command for -basic use-cases such as: - -```console -$ python-jl your_script.py -$ python-jl -c 'from julia.Base import banner; banner()' -$ python-jl -m IPython -``` - -See `python-jl --help` for more information. - -##### How `python-jl` works - -Note that `python-jl` works by launching Python interpreter inside -Julia. Importantly, it means that PyJulia has to be installed in the -Python environment with which PyCall is configured. That is to say, -following commands must work for `python-jl` to be usable: - -```jlcon -julia> using PyCall - -julia> pyimport("julia") -PyObject -``` - -In fact, you can simply use PyJulia inside the Julia REPL, if you are -comfortable with working in it: - -```jlcon -julia> using PyCall - -julia> py""" - from julia import Julia - Julia(init_julia=False) - - from your_module_using_pyjulia import function - function() - """ -``` - -#### Ultimate fix: build your own Python - -Alternatively, you can use [pyenv](https://github.com/pyenv/pyenv) to -build Python with -[`--enable-shared` option](https://github.com/pyenv/pyenv/wiki#how-to-build-cpython-with---enable-shared). -Of course, manually building from Python source distribution with the -same configuration also works. - -```console -$ PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install 3.6.6 -Downloading Python-3.6.6.tar.xz... --> https://www.python.org/ftp/python/3.6.6/Python-3.6.6.tar.xz -Installing Python-3.6.6... -Installed Python-3.6.6 to /home/USER/.pyenv/versions/3.6.6 - -$ ldd ~/.pyenv/versions/3.6.6/bin/python3.6 | grep libpython - libpython3.6m.so.1.0 => /home/USER/.pyenv/versions/3.6.6/lib/libpython3.6m.so.1.0 (0x00007fca44c8b000) -``` - -For more discussion, see: - - -### Segmentation fault in IPython - -You may experience segmentation fault when using PyJulia in old -versions of IPython. You can avoid this issue by updating IPython to -7.0 or above. Alternatively, you can use IPython via Jupyter (e.g., -`jupyter console`) to workaround the problem. - -### Error due to `libstdc++` version - -When you use PyJulia with another Python extension, you may see an -error like ``version `GLIBCXX_3.4.22' not found`` (Linux) or `The -procedure entry point ... could not be located in the dynamic link -library libstdc++6.dll` (Windows). In this case, you might have -observed that initializing PyJulia first fixes the problem. This is -because Julia (or likely its dependencies like LLVM) requires a recent -version of `libstdc++`. - -Possible fixes: - -* Initialize PyJulia (e.g., by `from julia import Main`) as early as - possible. Note that just importing PyJulia (`import julia`) does - not work. -* Load `libstdc++.so.6` first by setting environment variable - `LD_PRELOAD` (Linux) to - `/PATH/TO/JULIA/DIR/lib/julia/libstdc++.so.6` where - `/PATH/TO/JULIA/DIR/lib` is the directory which has `libjulia.so`. - macOS and Windows likely to have similar mechanisms (untested). -* Similarly, set environment variable `LD_LIBRARY_PATH` (Linux) to - `/PATH/TO/JULIA/DIR/lib/julia` directory. Using `DYLD_LIBRARY_PATH` - on macOS and `PATH` on Windows may work (untested). - -See: -, - diff --git a/docs/source/troubleshooting.rst b/docs/source/troubleshooting.rst new file mode 100644 index 00000000..dc6b0838 --- /dev/null +++ b/docs/source/troubleshooting.rst @@ -0,0 +1,167 @@ +Troubleshooting +--------------- + +.. _statically-linked: + +Your Python interpreter is statically linked to libpython +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you use Python installed with Debian-based Linux distribution such as +Ubuntu or install Python by ``conda``, you might have noticed that +PyJulia cannot be initialized properly with Julia ≥ 0.7. This is because +those Python executables are statically linked to libpython. (See +:doc:`limitations` for why that's a problem.) + +If you are unsure if your ``python`` has this problem, you can quickly +check it by: + +.. code-block:: console + + $ ldd /usr/bin/python + linux-vdso.so.1 (0x00007ffd73f7c000) + libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f10ef84e000) + libc.so.6 => /usr/lib/libc.so.6 (0x00007f10ef68a000) + libpython3.7m.so.1.0 => /usr/lib/libpython3.7m.so.1.0 (0x00007f10ef116000) + /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f10efaa4000) + libdl.so.2 => /usr/lib/libdl.so.2 (0x00007f10ef111000) + libutil.so.1 => /usr/lib/libutil.so.1 (0x00007f10ef10c000) + libm.so.6 => /usr/lib/libm.so.6 (0x00007f10eef87000) + +in Linux where ``/usr/bin/python`` should be replaced with the path to +your ``python`` command (use ``which python`` to find it out). In macOS, +use ``otool -L`` instead of ``ldd``. If it does not print the path to +libpython like ``/usr/lib/libpython3.7m.so.1.0`` in above example, you +need to use one of the workaround below. + +Turn off compilation cache +^^^^^^^^^^^^^^^^^^^^^^^^^^ +.. versionadded:: 0.3 + +The easiest workaround is to pass ``compiled_modules=False`` to the +``Julia`` constructor. + +.. code-block:: pycon + + >>> from julia.api import Julia + >>> jl = Julia(compiled_modules=False) + +This is equivalent to ``julia``\ ’s command line option +``--compiled-modules=no`` and disables the precompilation cache +mechanism in Julia. Note that this option slows down loading and using +Julia packages especially for complex and large ones. + +See also API documentation of `Julia`. + +Create a custom system image +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +.. versionadded:: 0.4 + +A very powerful way to avoid this the issue due to precompilation +cache is to create a custom system image. This also has an additional +benefit that initializing PyJulia becomes instant. See +:doc:`sysimage` for how to create and use a custom system image. + +``python-jl``: an easy workaround +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +.. versionadded:: 0.2 + +Another easy workaround is to use the ``python-jl`` command bundled in +PyJulia. This can be used instead of normal ``python`` command for basic +use-cases such as: + +.. code-block:: console + + $ python-jl your_script.py + $ python-jl -c 'from julia.Base import banner; banner()' + $ python-jl -m IPython + +See ``python-jl --help`` for more information. + +How ``python-jl`` works +''''''''''''''''''''''' + +Note that ``python-jl`` works by launching Python interpreter inside +Julia. Importantly, it means that PyJulia has to be installed in the +Python environment with which PyCall is configured. That is to say, +following commands must work for ``python-jl`` to be usable: + +.. code-block:: jlcon + + julia> using PyCall + + julia> pyimport("julia") + PyObject + +In fact, you can simply use PyJulia inside the Julia REPL, if you are +comfortable with working in it: + +.. code-block:: jlcon + + julia> using PyCall + + julia> py""" + from julia import Julia + Julia(init_julia=False) + # Then use your Python module: + from your_module_using_pyjulia import function + function() + """ + +Ultimate fix: build your own Python +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Alternatively, you can use `pyenv `_ +to build Python with ``--enable-shared`` option (see `their Wiki page +`_). +Of course, manually building from Python source distribution with the +same configuration also works. + +.. code-block:: console + + $ PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install 3.6.6 + Downloading Python-3.6.6.tar.xz... + -> https://www.python.org/ftp/python/3.6.6/Python-3.6.6.tar.xz + Installing Python-3.6.6... + Installed Python-3.6.6 to /home/USER/.pyenv/versions/3.6.6 + + $ ldd ~/.pyenv/versions/3.6.6/bin/python3.6 | grep libpython + libpython3.6m.so.1.0 => /home/USER/.pyenv/versions/3.6.6/lib/libpython3.6m.so.1.0 (0x00007fca44c8b000) + +For more discussion, see: https://github.com/JuliaPy/pyjulia/issues/185 + +Segmentation fault in IPython +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +You may experience segmentation fault when using PyJulia in old versions +of IPython. You can avoid this issue by updating IPython to 7.0 or +above. Alternatively, you can use IPython via Jupyter (e.g., +``jupyter console``) to workaround the problem. + +Error due to ``libstdc++`` version +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +When you use PyJulia with another Python extension, you may see an error +like :literal:`version `GLIBCXX_3.4.22' not found` (Linux) or +``The procedure entry point ... could not be located in the dynamic link library libstdc++6.dll`` +(Windows). In this case, you might have observed that initializing +PyJulia first fixes the problem. This is because Julia (or likely its +dependencies like LLVM) requires a recent version of ``libstdc++``. + +Possible fixes: + +- Initialize PyJulia (e.g., by ``from julia import Main``) as early as + possible. Note that just importing PyJulia (``import julia``) does + not work. +- Load ``libstdc++.so.6`` first by setting environment variable + ``LD_PRELOAD`` (Linux) to + ``/PATH/TO/JULIA/DIR/lib/julia/libstdc++.so.6`` where + ``/PATH/TO/JULIA/DIR/lib`` is the directory which has + ``libjulia.so``. macOS and Windows likely to have similar mechanisms + (untested). +- Similarly, set environment variable ``LD_LIBRARY_PATH`` (Linux) to + ``/PATH/TO/JULIA/DIR/lib/julia`` directory. Using + ``DYLD_LIBRARY_PATH`` on macOS and ``PATH`` on Windows may work + (untested). + +See: https://github.com/JuliaPy/pyjulia/issues/180, +https://github.com/JuliaPy/pyjulia/issues/223 diff --git a/setup.py b/setup.py index 52b5fdcf..9e3e0e8d 100644 --- a/setup.py +++ b/setup.py @@ -79,6 +79,7 @@ def pyload(path): }, entry_points={ "console_scripts": [ + "julia-py = julia.julia_py:main", "python-jl = julia.python_jl:main", ], }, diff --git a/src/julia/compile.jl b/src/julia/compile.jl new file mode 100644 index 00000000..bdf8cd40 --- /dev/null +++ b/src/julia/compile.jl @@ -0,0 +1,30 @@ +compiler_env, script, output = ARGS + +if VERSION < v"0.7-" + error("Unsupported Julia version $VERSION") +end + +using Pkg + +Pkg.activate(compiler_env) +@info "Loading PackageCompiler..." +using PackageCompiler + +@info "Installing PyCall..." +Pkg.activate(".") +Pkg.add([ + PackageSpec( + name = "PyCall", + uuid = "438e738f-606a-5dbb-bf0a-cddfbfd45ab0", + ) + PackageSpec( + name = "MacroTools", + uuid = "1914dd2f-81c6-5fcd-8719-6d5c9610ff09", + ) +]) + +@info "Compiling system image..." +sysout, _curr_syso = compile_incremental("Project.toml", script) + +@info "System image is created at $output" +cp(sysout, output, force=true) diff --git a/src/julia/install-packagecompiler.jl b/src/julia/install-packagecompiler.jl new file mode 100644 index 00000000..8583a2b1 --- /dev/null +++ b/src/julia/install-packagecompiler.jl @@ -0,0 +1,38 @@ +compiler_env, = ARGS + +if VERSION < v"0.7-" + error("Unsupported Julia version $VERSION") +end + +using Pkg + +function cat_build_log(pkg) + modpath = Base.locate_package(pkg) + if modpath !== nothing + logfile = joinpath(dirname(modpath), "..", "deps", "build.log") + if isfile(logfile) + print(stderr, read(logfile, String)) + return + end + end + @error "build.log for $pkg not found" +end + +Pkg.activate(compiler_env) +@info "Installing PackageCompiler..." + +Pkg.add([ + PackageSpec( + name = "PackageCompiler", + uuid = "9b87118b-4619-50d2-8e1e-99f35a4d4d9d", + version = "0.6", + ) +]) +cat_build_log(Base.PkgId( + Base.UUID("9b87118b-4619-50d2-8e1e-99f35a4d4d9d"), + "PackageCompiler")) + +@info "Loading PackageCompiler..." +using PackageCompiler + +@info "PackageCompiler is successfully installed at $compiler_env" diff --git a/src/julia/julia_py.py b/src/julia/julia_py.py new file mode 100644 index 00000000..38d16f3c --- /dev/null +++ b/src/julia/julia_py.py @@ -0,0 +1,86 @@ +""" +Launch Julia through PyJulia. + +Currently, `julia-py` is primary used internally for supporting +`julia.sysimage` command line interface. Using `julia-py` like normal +Julia program requires `--sysimage` to be set to the system image +created by `julia.sysimage`. + +Example:: + + $ python3 -m julia.sysimage sys.so + $ julia-py --sysimage sys.so +""" + +from __future__ import print_function, absolute_import + +import argparse +import os +import sys + +from .api import LibJulia +from .core import which, enable_debug +from .tools import julia_py_executable + + +def julia_py(julia, pyjulia_debug, jl_args): + if pyjulia_debug: + enable_debug() + + julia = which(julia) + os.environ["_PYJULIA_JULIA"] = julia + os.environ["_PYJULIA_JULIA_PY"] = julia_py_executable() + os.environ["_PYJULIA_PATCH_JL"] = os.path.join( + os.path.dirname(os.path.realpath(__file__)), "patch.jl" + ) + + api = LibJulia.load(julia=julia) + api.init_julia(jl_args) + code = 1 + if api.jl_eval_string(b"""Base.include(Main, ENV["_PYJULIA_PATCH_JL"])"""): + if api.jl_eval_string(b"Base.invokelatest(Base._start)"): + code = 0 + api.jl_atexit_hook(code) + sys.exit(code) + + +class CustomFormatter( + argparse.RawDescriptionHelpFormatter, argparse.ArgumentDefaultsHelpFormatter +): + pass + + +def parse_args(args, **kwargs): + options = dict( + prog="julia-py", + usage="%(prog)s [--julia JULIA] [--pyjulia-debug] [...]", + formatter_class=CustomFormatter, + description=__doc__, + ) + options.update(kwargs) + parser = argparse.ArgumentParser(**options) + parser.add_argument( + "--julia", + default=os.environ.get("_PYJULIA_JULIA", "julia"), + help=""" + Julia `executable` used by PyJulia. + """, + ) + parser.add_argument( + "--pyjulia-debug", + action="store_true", + help=""" + Print PyJulia's debugging messages to standard error. + """, + ) + ns, jl_args = parser.parse_known_args(args) + ns.jl_args = jl_args + return ns + + +def main(args=None, **kwargs): + julia_py(**vars(parse_args(args, **kwargs))) + + +if __name__ == "__main__": + main() diff --git a/src/julia/patch.jl b/src/julia/patch.jl new file mode 100644 index 00000000..3e9ea56b --- /dev/null +++ b/src/julia/patch.jl @@ -0,0 +1,40 @@ +julia_py = ENV["_PYJULIA_JULIA_PY"] + +if Base.julia_cmd().exec[1] == julia_py + @debug "Already monkey-patched. Skipping..." julia_py +else + @debug "Monkey-patching..." julia_py + + # Monkey patch `Base.package_slug` + # + # This is used for generating the set of precompilation cache paths + # for PyJulia different to the standard Julia runtime. + # + # See also: + # * Suggestion: Use different precompilation cache path for different + # system image -- https://github.com/JuliaLang/julia/pull/29914 + # + Base.eval(Base, quote + function package_slug(uuid::UUID, p::Int=5) + crc = _crc32c(uuid) + crc = _crc32c(unsafe_string(JLOptions().image_file), crc) + crc = _crc32c($julia_py, crc) + return slug(crc, p) + end + end) + + # Monkey patch `Base.julia_exename`. + # + # This is required for propagating the monkey patches to subprocesses. + # This is important especially for the subprocesses used for + # precompilation. + # + # See also: + # * Request: Add an API for configuring julia_cmd -- + # https://github.com/JuliaLang/julia/pull/30065 + # + Base.eval(Base, quote + julia_exename() = $julia_py + end) + @assert Base.julia_cmd().exec[1] == julia_py +end diff --git a/src/julia/precompile.jl b/src/julia/precompile.jl new file mode 100644 index 00000000..f80741dd --- /dev/null +++ b/src/julia/precompile.jl @@ -0,0 +1,15 @@ +# Activate ./Project.toml. Excluding `"@v#.#"` from `Base.LOAD_PATH` +# to make compilation more reproducible. +using Pkg +empty!(Base.LOAD_PATH) +append!(Base.LOAD_PATH, ["@", "@stdlib"]) +Pkg.activate(".") + +# Manually invoking `__init__` to workaround: +# https://github.com/JuliaLang/julia/issues/22910 + +import MacroTools +isdefined(MacroTools, :__init__) && MacroTools.__init__() + +using PyCall +PyCall.__init__() diff --git a/src/julia/sysimage.py b/src/julia/sysimage.py new file mode 100644 index 00000000..ef168fc5 --- /dev/null +++ b/src/julia/sysimage.py @@ -0,0 +1,145 @@ +""" +Build system image. + +Example:: + + python3 -m julia.sysimage sys.so + +Generated system image can be passed to ``sysimage`` option of +`julia.api.Julia`. + +.. note:: + + This script is not tested on Windows. +""" + +from __future__ import print_function, absolute_import + +from contextlib import contextmanager +from logging import getLogger +import argparse +import os +import subprocess +import sys +import shutil +import tempfile + +from .core import enable_debug +from .tools import julia_py_executable + + +logger = getLogger("julia.sysimage") + + +class KnownError(RuntimeError): + pass + + +def script_path(name): + return os.path.join(os.path.dirname(os.path.realpath(__file__)), name) + + +def install_packagecompiler_cmd(julia, compiler_env): + cmd = [julia] + if sys.stdout.isatty(): + cmd.append("--color=yes") + cmd.append(script_path("install-packagecompiler.jl")) + cmd.append(compiler_env) + return cmd + + +def build_sysimage_cmd(julia_py, julia, compile_args): + cmd = [julia_py, "--julia", julia] + if sys.stdout.isatty(): + cmd.append("--color=yes") + cmd.append(script_path("compile.jl")) + cmd.extend(compile_args) + return cmd + + +def check_call(cmd, **kwargs): + logger.debug("Run %s", cmd) + subprocess.check_call(cmd, **kwargs) + + +@contextmanager +def temporarydirectory(**kwargs): + path = tempfile.mkdtemp(**kwargs) + try: + yield path + finally: + shutil.rmtree(path, ignore_errors=True) + + +def build_sysimage( + output, + julia="julia", + script=script_path("precompile.jl"), + debug=False, + compiler_env="", +): + if debug: + enable_debug() + + if output.endswith(".a"): + raise KnownError("Output file must not have extension .a") + + julia_py = julia_py_executable() + + with temporarydirectory(prefix="tmp.pyjulia.sysimage.") as path: + if not compiler_env: + compiler_env = os.path.join(path, "compiler_env") + # Not using julia-py to install PackageCompiler to reduce + # method re-definition warnings: + check_call(install_packagecompiler_cmd(julia, compiler_env), cwd=path) + + # Arguments to ./compile.jl script: + compile_args = [ + compiler_env, + # script -- ./precompile.jl by default + os.path.realpath(script), + # output -- path to sys.o file + os.path.realpath(output), + ] + + check_call(build_sysimage_cmd(julia_py, julia, compile_args), cwd=path) + + +class CustomFormatter( + argparse.RawDescriptionHelpFormatter, argparse.ArgumentDefaultsHelpFormatter +): + pass + + +def main(args=None): + parser = argparse.ArgumentParser( + formatter_class=CustomFormatter, description=__doc__ + ) + parser.add_argument("--julia", default="julia") + parser.add_argument("--debug", action="store_true", help="Print debug log.") + parser.add_argument( + "--script", + default=script_path("precompile.jl"), + help="Path to Julia script with precopmile instructions.", + ) + parser.add_argument( + "--compiler-env", + default="", + help=""" + Path to a Julia project with PackageCompiler to be used for + system image compilation. Create a temporary environment with + appropriate PackageCompiler by default or when an empty string + is given. + """, + ) + parser.add_argument("output", help="Path to system image file sys.o.") + ns = parser.parse_args(args) + try: + build_sysimage(**vars(ns)) + except (KnownError, subprocess.CalledProcessError) as err: + print(err, file=sys.stderr) + sys.exit(1) + + +if __name__ == "__main__": + main() diff --git a/src/julia/tests/test_install.py b/src/julia/tests/test_install.py index 65fd9628..6ac00908 100644 --- a/src/julia/tests/test_install.py +++ b/src/julia/tests/test_install.py @@ -1,17 +1,10 @@ import os +import subprocess import pytest - from julia import install -import subprocess - -only_in_ci = pytest.mark.skipif( - os.environ.get("CI", "false").lower() != "true", reason="CI=true not set" -) -""" -Tests that are too destructive to run with casual `tox` call. -""" +from .utils import only_in_ci @only_in_ci diff --git a/src/julia/tests/test_sysimage.py b/src/julia/tests/test_sysimage.py new file mode 100644 index 00000000..0d9805c5 --- /dev/null +++ b/src/julia/tests/test_sysimage.py @@ -0,0 +1,27 @@ +import pytest + +from .test_compatible_exe import runcode +from .utils import only_in_ci, skip_in_appveyor +from julia.sysimage import build_sysimage + + +@pytest.mark.julia +@only_in_ci +@skip_in_appveyor # Avoid "LVM ERROR: out of memory" +def test_build_and_load(tmpdir, juliainfo): + if juliainfo.version_info < (0, 7): + pytest.skip("Julia < 0.7 is not supported") + + sysimage_path = str(tmpdir.join("sys.so")) + build_sysimage(sysimage_path, julia=juliainfo.julia) + + runcode( + """ + from julia.api import Julia + + sysimage_path = {!r} + jl = Julia(sysimage=sysimage_path) + """.format( + sysimage_path + ) + ) diff --git a/src/julia/tests/utils.py b/src/julia/tests/utils.py new file mode 100644 index 00000000..91c2bf52 --- /dev/null +++ b/src/julia/tests/utils.py @@ -0,0 +1,17 @@ +import os + +import pytest + +only_in_ci = pytest.mark.skipif( + os.environ.get("CI", "false").lower() != "true", reason="CI=true not set" +) +""" +Tests that are too destructive or slow to run with casual `tox` call. +""" + +skip_in_appveyor = pytest.mark.skipif( + os.environ.get("APPVEYOR", "false").lower() == "true", reason="APPVEYOR=true is set" +) +""" +Tests that are known to fail in AppVeyor. +""" diff --git a/src/julia/tools.py b/src/julia/tools.py index 424b1c32..d3aa196c 100644 --- a/src/julia/tools.py +++ b/src/julia/tools.py @@ -1,9 +1,10 @@ from __future__ import absolute_import, print_function +import glob import os +import re import subprocess import sys -import re from .core import JuliaNotFound, _enviorn, which from .find_libpython import linked_libpython @@ -134,3 +135,33 @@ def redirect_output_streams(): # TODO: Invoking `redirect_output_streams()` in terminal IPython # terminates the whole Python process. Find out why. + + +def julia_py_executable(executable=sys.executable): + """ + Path to ``julia-py`` executable installed for this Python executable. + """ + stempath = os.path.join(os.path.dirname(executable), "julia-py") + candidates = {os.path.basename(p): p for p in glob.glob(stempath + "*")} + if not candidates: + raise RuntimeError( + "``julia-py`` executable is not found for Python installed at {}".format( + executable + ) + ) + + for basename in ["julia-py", "julia-py.exe", "julia-py.cmd"]: + try: + return candidates[basename] + except KeyError: + continue + + raise RuntimeError( + """\ +``julia-py`` with following unrecognized extension(s) are found. +Please report it at https://github.com/JuliaPy/pyjulia/issues +with the full traceback. +Files found: + """ + + " \n".join(sorted(candidates)) + ) diff --git a/tox.ini b/tox.ini index cdbb8251..472e3d56 100644 --- a/tox.ini +++ b/tox.ini @@ -35,6 +35,9 @@ passenv = TRAVIS TRAVIS_* + # https://www.appveyor.com/docs/environment-variables/ + APPVEYOR + CI [pytest]