Switch "crypto" implementation to Python

[jlaine]

This simplifies the build process and avoids needing to ship a vendor'd OpenSSL.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 100.00%. Comparing base (9bc1e43) to head (7fffb5a).

Additional details and impacted files

@@            Coverage Diff            @@
##              main      #457   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           25        26    +1     
  Lines         5113      5178   +65     
=========================================
+ Hits          5113      5178   +65     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[jlaine]

Initial testing suggest a 10x slowdown of packet protection compared to the current C code.

[rthalley]

I haven't tried measuring it yet. My prior experience is that cryptography is pretty fast, and I'm wondering if the header protection part might be the slow thing. If it were, it might be possible to do the crypto with cryptography and then do the masking with a tiny bit of C code.

[jlaine]

I haven't tried measuring it yet. My prior experience is that cryptography is pretty fast, and I'm wondering if the header protection part might be the slow thing. If it were, it might be possible to do the crypto with cryptography and then do the masking with a tiny bit of C code.

AFAICT it's the actual crypto code unfortunately that is slower.

[rthalley]

Can you say how you are benchmarking? I'm guessing I'm doing something wrong, but I'm testing by using the example client to pull 20MiB from nginx, and while it is slower it's not too bad, only around 1.074 times the non-cryptography version.

Regular aioquic (main from your branch with python-crypto):

2024-02-27 08:04:13,412 INFO client Response received for GET /foo.bin : 20971520 bytes in 0.4 s (428.916 Mbps)
2024-02-27 08:04:13,414 INFO quic [1da750bf1e625c3a] Connection close sent (code 0x100, reason )
python examples/http3_client.py https://localhost:8080/foo.bin  0.46s user 0.11s system 84% cpu 0.674 total

Your python-crypto branch:

024-02-27 08:12:30,026 INFO client Response received for GET /foo.bin : 20971520 bytes in 0.4 s (389.185 Mbps)
2024-02-27 08:12:30,028 INFO quic [254cea9a35deb3ff] Connection close sent (code 0x100, reason )
python examples/http3_client.py https://localhost:8080/foo.bin  0.41s user 0.07s system 66% cpu 0.724 total

[jlaine]

Here is what I use:

import binascii
import time

from aioquic.quic.crypto import CryptoPair

from tests.test_crypto import (
    LONG_CLIENT_PACKET_NUMBER,
    LONG_CLIENT_PLAIN_HEADER,
    LONG_CLIENT_PLAIN_PAYLOAD,
    PROTOCOL_VERSION,
)

pair = CryptoPair()
pair.setup_initial(
    cid=binascii.unhexlify("8394c8f03e515708"),
    is_client=True,
    version=PROTOCOL_VERSION,
)
start = time.time()
for i in range(100000):
    pair.encrypt_packet(
        LONG_CLIENT_PLAIN_HEADER,
        LONG_CLIENT_PLAIN_PAYLOAD,
        LONG_CLIENT_PACKET_NUMBER,
    )
elapsed = time.time() - start
print(elapsed)

A typical run for me puts the C code at about 70ms and the Python code at around 800ms.

Notes:

• Make sure you purge your build directory when switching to the python-crypto branch, otherwise you will still have the C version in the form of a .so :)
• The main branch of cryptography may perform differently as more code has been moved to rust.
• ChaCha20 is going to regress even further, but there is a WIP patch which helps: ChaCha20 API does not allow updating the nonce pyca/cryptography#10193

[rthalley]

Yeah, I discovered the need to purge while I was benchmarking. The timings I reported are after making sure I was testing in clean setups.

[rthalley]

And of course both of our testing could be accurate, as you're testing just the crypto whereas I'm testing what the overall effect in typical use is.

[jlaine]

I wish cryptography 43 were already out, preferably featuring:

pyca/cryptography#10437

As things currently stand, the build of the C code has broken again through no change of our own:

https://github.com/jlaine/aioquic/actions/runs/9150448224