refactor!: drop RustPython support

[tisonkun]

I hereby agree to the terms of the GreptimeDB CLA.

Refer to a related PR or issue link (optional)

What's changed and what's your intention?

This supersedes and thus closes #3723.

This closes #3063.

Drop RustPython backend for scripting and always build with PyO3. This is an attempt to simplify our scripting functions.

This should involve a lot of docs changes, but we haven't decided yet. Open this patch as a preview.

Checklist

• I have written the necessary rustdoc comments.
• I have added the necessary unit tests and integration tests.
• This PR does not require documentation updates.

[codecov]

Codecov Report

Attention: Patch coverage is 56.11111% with 79 lines in your changes are missing coverage. Please review.

Project coverage is 85.25%. Comparing base (cfed466) to head (46d0194).
Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3749      +/-   ##
==========================================
- Coverage   85.63%   85.25%   -0.38%     
==========================================
  Files         944      937       -7     
  Lines      158776   156542    -2234     
==========================================
- Hits       135966   133464    -2502     
- Misses      22810    23078     +268     

[tisonkun]

@davidhewitt @messense

We are trying to ship the Python script function with PyO3 primarily here. And since statically link to libpython can be complicated, I'm investigating the way to dynamically link to libpython.

So far, it works well and we can see the CI all passed. But this requires a libpython3.8.so can be linked in the system.

Thus, I have two questions:

1. How can I make this binary work well with all the libpython >= 3.8?
2. How can I disable the related functionalities if libpython3.so is not found?

This refers to:

• Static Build of pyo3 (embedding python interpreter) PyO3/pyo3#416
• https://pyo3.rs/v0.21.2/building-and-distribution

[tisonkun]

cc @discord9 any thoughts here?

[discord9]

cc @discord9 any thoughts here?

Maybe we can use bindgen to create a shared library(call it libpython_finder maybe?) with the same symbols as limited python abi, and forward those calls to actual libpython3.* at runtime with libloading, we would also need to export extra functions to check if libpython is found, and config pyo3 to use our shared library, the shared library would not need much effect to mantain after we are done developing it as limited api doesn't change that much.

[discord9]

Rest LGTM

[discord9]

Should we change this comment?

[discord9]

Pyo3 support rich cmp, see more in this link, and the interface is quite similiar:
https://pyo3.rs/v0.14.5/class/protocols.html?highlight=compare#comparison-operators
Edit: maybe change this todo to TODO(discord9): impl SequenceProtocol and rich cmp?

[tisonkun]

Closed as undetermined yet. This patch can be picked up since it touches only the Python script code path.

Before we make the decision to lift PyO3 as the default backend and distribute it in every artifact, we should resolve the issue described in #3777, namely how to support multiple Python versions, or at least disable when Python environment not found, instead of panic.

[davidhewitt]

Sorry for the late reply, I am somewhat behind on OSS so please expect some delay from me at the moment.

Thus, I have two questions:

1. How can I make this binary work well with all the libpython >= 3.8?
2. How can I disable the related functionalities if libpython3.so is not found?

In principle you can link to libpython3.so of any version of 3.8 or higher using the abi3-py38 feature. But my previous experience is that distribution packagers are frequently omitting this file - e.g. PyO3/pyo3#2901 (comment)

Probably you are safer to avoid depending on the system Python and distribute the Python version you want to link with. This means either you:

• statically link (the easiest way I know to do this is with PyOxidizer)
• dynamically link and ship the linked libpython along with your binary. I believe you can use a rpath to ensure that the linked libpython is preferred for loading (but this may affect your other dependencies too).

[tisonkun]

@davidhewitt Thanks a lot! According to your comment, here are my two questions:

libpython3.so of any version of 3.8 or higher using the abi3-py38 feature

How does it work? We currently add the abi3-py37 feature, but it still failed to load shared lib as in https://github.com/GreptimeTeam/greptimedb/actions/runs/8751166135/job/24016840142:

./bins/greptime: error while loading shared libraries: libpython3.8.so.1.0: cannot open shared object file: No such file or directory

You can see the commits above to fight with this issue.

dynamically link and ship the linked libpython along with your binary. I believe you can use a rpath to ensure that the linked libpython is preferred for loading (but this may affect your other dependencies too).

It still has two challenges:

1. Is it possible to opt-out such Python scripting support on missing dylib? Currently, it panics on binary starts up.
2. Either static linking or dynamic linking with bundled dylib, we may meet challenge to ship 3rdparty dependencies like numpy. So it's still desired to leverage the Python installation on user environment. But GreptimeDB wants a way to be compatible with as many as Python versions and gracefully disable the related features if non dylib found.