Tracking issue for running JPY and Deephaven wheels outside of docker

[niloc132]

With #1270 in progress we're considering the other doors this will open for us, and one of them is using local python virtual environments (instead of ones in the docker image). This ticket is a placeholder to make sure this idea will work, and to consider other tasks.

Basic user story that this presents is that a python developer or data scientist could create a python venv with jpy and deephaven installed, and install/develop other wheels and other python code locally, then run the deephaven server-jetty binary pointing to that venv. No docker setup would be required to run it, just a zip/tar download and configuration to point at the venv.

[jmao-denver]

As a step in this exploration, we want to make sure the deephaven fork of JPY can be built locally on MacOS. There is a compiler option change we need to the build script (setup.py):

diff --git a/py/jpy/setup.py b/py/jpy/setup.py
index bd91d382..3921347f 100644
--- a/py/jpy/setup.py
+++ b/py/jpy/setup.py
@@ -139,6 +139,11 @@ libraries = [jpyutil.JVM_LIB_NAME]
 define_macros = []
 extra_link_args = []
 extra_compile_args = ["-std=c99"]
+# making a big but probably safe assumption here that the compiler used to build Python is the same one we will be using
+compiler_name, compiler_version, *_ = platform.python_compiler().split()
+if compiler_name.upper() == "CLANG" and compiler_version > "10.0.0":
+    extra_compile_args.append("-Wno-error-implicit-function-declaration")
+

 if platform.system() == 'Windows':
     define_macros += [('WIN32', '1')]

With this change, we are able to build the wheel by running the following commands in the py/jpy directory.

export DEEPHAVEN_VERSION=0.7.0
python setup.py bdist_wheel
cp dist/deephaven_jpy-0.7.0-cp37-cp37m-macosx_11_0_x86_64.whl /tmp
cd ../../Integrations/python
python setup.py bdist_wheel
cp dist/deephaven-0.7.0-py2.py3-none-any.whl /tmp

and install it with pip in a python venv as follows:

 ➜ /tmp python -m venv dockerless
 ➜ /tmp source dockerless/bin/activate
(dockerless) ➜  /tmp python -m pip install --upgrade pip
Collecting pip
  Using cached pip-21.3.1-py3-none-any.whl (1.7 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 20.1.1
    Uninstalling pip-20.1.1:
      Successfully uninstalled pip-20.1.1
Successfully installed pip-21.3.1
(dockerless) ➜  /tmp pip install deephaven_jpy-0.7.0-cp37-cp37m-macosx_11_0_x86_64.whl
Processing ./deephaven_jpy-0.7.0-cp37-cp37m-macosx_11_0_x86_64.whl
Installing collected packages: deephaven-jpy
Successfully installed deephaven-jpy-0.7.0
(dockerless) ➜  /tmp pip install deephaven-0.7.0-py2.py3-none-any.whl
Processing ./deephaven-0.7.0-py2.py3-none-any.whl
Collecting wrapt
  Using cached wrapt-1.13.3-cp37-cp37m-macosx_10_9_x86_64.whl (33 kB)
Collecting dill>=0.2.8
  Using cached dill-0.3.4-py2.py3-none-any.whl (86 kB)
Collecting numpy
  Using cached numpy-1.21.4-cp37-cp37m-macosx_10_9_x86_64.whl (16.9 MB)
Collecting numba
  Using cached numba-0.54.1-cp37-cp37m-macosx_10_14_x86_64.whl (2.3 MB)
Collecting pandas
  Using cached pandas-1.3.5-cp37-cp37m-macosx_10_9_x86_64.whl (11.0 MB)
Requirement already satisfied: deephaven-jpy==0.7.0 in ./dockerless/lib/python3.7/site-packages (from deephaven==0.7.0) (0.7.0)
Collecting llvmlite<0.38,>=0.37.0rc1
  Downloading llvmlite-0.37.0-cp37-cp37m-macosx_10_9_x86_64.whl (19.1 MB)
     |████████████████████████████████| 19.1 MB 12.7 MB/s
Requirement already satisfied: setuptools in ./dockerless/lib/python3.7/site-packages (from numba->deephaven==0.7.0) (47.1.0)
Collecting numpy
  Downloading numpy-1.20.3-cp37-cp37m-macosx_10_9_x86_64.whl (16.0 MB)
     |████████████████████████████████| 16.0 MB 10.2 MB/s
Collecting python-dateutil>=2.7.3
  Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting pytz>=2017.3
  Using cached pytz-2021.3-py2.py3-none-any.whl (503 kB)
Collecting six>=1.5
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Installing collected packages: six, pytz, python-dateutil, numpy, llvmlite, wrapt, pandas, numba, dill, deephaven
Successfully installed deephaven-0.7.0 dill-0.3.4 llvmlite-0.37.0 numba-0.54.1 numpy-1.20.3 pandas-1.3.5 python-dateutil-2.8.2 pytz-2021.3 six-1.16.0 wrapt-1.13.3
(dockerless) ➜  /tmp python
Python 3.7.10 (default, Jun 26 2021, 22:27:26)
[Clang 12.0.5 (clang-1205.0.22.9)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import jpy
>>> import jpyutil
>>> jpyutil.init_jvm()
jpyutil - WARNING: Failed to preload JVM shared library. No shared library found.
(None, ['-Djpy.jpyLib=/private/tmp/dockerless/lib/python3.7/site-packages/jpy.cpython-37m-darwin.so', '-Djpy.jdlLib=/private/tmp/dockerless/lib/python3.7/site-packages/jdl.cpython-37m-darwin.so', '-Djpy.pythonPrefix=/private/tmp/dockerless', '-Djpy.pythonExecutable=/private/tmp/dockerless/bin/python'])
>>> jstring_class = jpy.get_type("java.lang.String")
>>> print(jstring_class("abc"))
abc
>>>

[niloc132]

Prerequisite, make sure your python installation comes with libpython, so deephaven can load against it - some distributions do not have this (set PYTHON_CONFIGURE_OPTS="--enabled-shared" before calling pyenv install).

# Make a new directory for a venv to try out
$ mkdir demo && cd demo

# Create and activate a new python venv and give it a name
$ python -m venv demo-venv
$ source demo-venv/bin/activate

# Install the wheels needed for deephaven to use python, and python bindings for deephaven
# These will exist later in pypi, for all platforms as needed
$ cd -
$ pip install deephaven_jpy-0.8.0-cp37-cp37m-macosx_11_0_x86_64.whl 
$ pip install deephaven-0.8.0-py2.py3-none-any.whl 

# Get a deephaven server-jetty zip, unzip it...
# Note that this name will change in the future to better match our usual naming scheme
$ unzip start-0.8.0.zip

# Start the deephaven server 
$ start-0.8.0/bin/start