Make it possible to write code that interacts with OpenSSL in Rust

[alex]

Vision

The ability to write some primitives in Rust code, using the openssl crate.

Motivation

Performance -- Python code interacting with the OpenSSL APIs introduce nontrivial overhead (e.g. #6457). This also has the potential to be safer, because it means that the responsibility for dealing with OpenSSL's memory semantics (e.g. refcounting ownership) is handled in the openssl crate and isn't our responsibility.

Requirements

Must be entirely backwards compatible with our public API. This needs to also be compatible with standard build configurations, that is to say it needs to be possible to link OpenSSL both statically and dynamically. It also needs to continue to support our entire build matrix of OpenSSL versions.

Approach

This is a sketch of how we might do this:

Currently we produce two .so files, one for _openssl.c, which is CFFI based, and _rust.so which is obviously compiled from Rust. We need to get this down to a single .so to support statically linking OpenSSL and ensuring that both Rust and CFFI using code are using the same version of OpenSSL.

To accomplish this we can: Compile _openssl.c to a .o (not a .so!), link it with the Rust object files to produce a single bindings.so. In it's PyInit_bindings it will create two sub-modules, _rust and _openssl from the Rust and CFFI code. bindings.so will naturally also link OpenSSL in, either statically or dynamically as before.

At this point we now have Python code (via CFFI) and Rust code that are interacting with the same OpenSSL and are properly built.

Once we have that we can begin writing new Rust code using the openssl crate that does whatever we want. Any such code will of course need to pass all existing tests.

Known blockers

• Easy support for generating .o files for CFFI modules ([FR] Add the ability for an extension to be compiled to a .o but not linked into a .so pypa/setuptools#2905)
• BoringSSL support in rust-openssl (BoringSSL Support sfackler/rust-openssl#1519)
• Support for accepting buffers as arguments with abi3

[tiran]

Support for accepting buffers as arguments with abi3

Are you referring to `Py_buffer``? They are currently excluded from stable API. I created https://bugs.python.org/issue45459 a month ago to include them.

[alex]

Yes, Py_buffer. It will never (IMO) be part of the stable ABI because it's stack allocated and thus making it stable would require sizeof(Py_buffer) to be stable.

I think what we need is a middle ground, e.g. stable ABI functions to get a contiguous byte sequence from a memoryview. That would solve all of our problems.

[tiran]

I made an experimental PR that adds a heap-allocated Py_buffer * API to Python. The strides/shapes parts are bit a tricky, though. Antoine argues that Py_buffer is mature and stable enough to make the entire struct and sizeof(Py_buffer) part of the stable ABI. I'm not convinced it is the best idea, but it's certainly a pragmatic approach.

Now that you mention it, a simpler approach for one dimensional, contiguous memory makes a lot of sense. The multi dimensional buffer API is useful for NumPy or to express complex TIFF formats in PIL. A fair amount of libraries can get by with simple buffer views.

[messense]

I was thinking that you can generate the cffi C source code in build_openssl.py then compile and link it in Rust side using a build.rs build script.

diff --git a/src/_cffi_src/build_openssl.py b/src/_cffi_src/build_openssl.py
index c5ab3cb3..4141ea99 100644
--- a/src/_cffi_src/build_openssl.py
+++ b/src/_cffi_src/build_openssl.py
@@ -112,3 +112,10 @@ ffi = build_ffi_for_binding(
     libraries=_get_openssl_libraries(sys.platform),
     extra_compile_args=_extra_compile_args(sys.platform),
 )
+
+if __name__ == "__main__":
+    from cffi import recompiler
+
+    module_name, source, source_extension, kwds = ffi._assigned_source
+    c_file = module_name + source_extension
+    updated = recompiler.make_c_source(ffi, module_name, source, c_file)

[messense]

Made some process here in case you're interested: https://github.com/messense/cryptography/tree/rust-openssl

❯ pytest
......
2681 passed, 1654 skipped in 58.20s

[alex]

Oooooh, thanks for working on this, this is exciting!

…

On Fri, Dec 3, 2021 at 3:02 AM messense ***@***.***> wrote: Made some process here in case you're interested: https://github.com/messense/cryptography/tree/rust-openssl — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#6634 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAAGBHDQ5YOOVTN4FEITLTUPB2TDANCNFSM5IOF7LXA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

-- All that is necessary for evil to succeed is for good people to do nothing.

[alex]

@messense do you intend to continue working on this and ultimately send a PR, or is this a PoC for us to adopt?

[messense]

@alex This is currently a PoC, but I can continue work on it. And I think it's better to open a draft PR so we can have discussions on it. What do you think?

[alex]

Draft PR sounds great! Thanks again for your work on this.

…

On Sun, Dec 12, 2021 at 10:21 AM messense ***@***.***> wrote: @alex <https://github.com/alex> This is currently a PoC, but I can continue work on it. And I think it's better to open a draft PR so we can have discussions on it. What do you think? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#6634 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAAGBHA2I4TBL3M62Z3H4TUQS4X7ANCNFSM5IOF7LXA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

-- All that is necessary for evil to succeed is for good people to do nothing.

[alex]

this happened.