Performance issues using cryptography v42 and above on oracle linux 8

[pete312]

I'm curious what you think of the issue I raised on asyncssh project

ronf/asyncssh#760

In short; hosts built with oracle linux 8 use OpenSSL 1.1.1k, and these have had a 5x fold or more increase in establishing a connection for all versions of cryptography above v41 (i.e v42 - v45). So pip installing v41 clears the issue and pip installing v45 reveals the issue in the tests (details in the asyncssh ticket).

oracle linux 9 however, uses OpenSSL 3.0.7, and we don't see this issue across any version of cryptography.

Many thanks

[alex]

It looks to me like OL 8's OpenSSL 1.1.1k is a red herring -- as far as I can tell you're using the statically linked OpenSSL from our wheels in all cases. Therefore, it's perplexing that the issue would not appear on OL 9.

I think my recommended steps would be either:

1. Running one of the slow versions under a profiler (e.g. samply) to see where time is spent, perhaps there'll be an obvious cause.
2. Building your own copies of the different cryptography versions against a single OpenSSL version so we can isolate if this is about changes in cryptography or in OpenSSL.

[reaperhulk]

Based on the other issue it sounds like the slowdown is being seen when connecting to those OpenSSL 1.1.1 hosts.

A reproducer using a container or a profile is likely to be needed to determine what might be going on.

[pete312]

I think @alex is correct. OpenSSL 1.1.1 is a red herring. To be clear the client box sees the slowness regardless of the remote archtype.

The tests ive done so far dont make sense to me. I see the problem on metal installs only. I tested a few metal installs installed by different means, and a few container installs by just pulling the distro and installing

dnf install -y python3.11 python3.11-pip openssh-clients vim

then i create a venv with python -m venv /tmp/312 and install asynchssh, do tests with the script and check timings. pip install cryptography==41.* just like the other thread describes and checks timings

Would appreciate if someone else would verify or at least take a closer look at this bcs its seems very odd to me.

Fedora41 (metal, vanilla install) slow
Ubuntu24 (metal, vanilla install) slow
Ubuntu24 (container) looks ok.
oracle linux 8 (metal). slow
oracle linux 8 (container) looks ok.
Oracle Linux 9 (metal and container) looks good. (The OL9 container was tested on a OL8 box to to rule out physical issues)

The test script.
I put some logical distro names in the /etc/hosts and the script is randomizing 20 connections to the 3 hosts.

Here is what I'm testing with. Defaults to private key auth with current user. Else you can export $USER and $PASSWORD to override current user and/or default key auth.

import asyncio, asyncssh
from time import time
from typing import List
from random import choice
import os, getpass

USERNAME = os.getenv('USER')
PASSWORD = os.getenv('PASSWORD')
begin = time()

async def run_client(host, command: str, connect_times = {}) -> asyncssh.SSHCompletedProcess:
    start = time()
    async with asyncssh.connect(host, username=USERNAME, password=PASSWORD, known_hosts=None) as conn:
        connect_times.setdefault(host,[]).append(round(time() - start,3) )
        return await conn.run(command)

async def run_multiple_clients(hosts:List[str], connect_times) -> None:
    
    tasks = (run_client(host, 'echo ok', connect_times=connect_times) for host in hosts)
    results = await asyncio.gather(*tasks, return_exceptions=True)
    for result in results:
        if isinstance(result, PermissionError):
            print(dir(result))
            continue
    #assert len([result.stderr for result in results if result and result.exit_status != 0]) == 0
    [print(result) for result in results]


mixed_set = ['fedora37', 'ubuntu24', 'fedora41']
host_sample = [choice(mixed_set) for i in range(20)] 
connect_times = {}
asyncio.run(run_multiple_clients(host_sample, connect_times))

print('no of hosts', len(host_sample))
print(connect_times)
print("total", round(time() - begin,3) )

[alex]

Are different algorithms being negotiated on different OSs? If you're getting different cryptographic algorithms, that would explain considerable performance variation.

[pete312]

I have not had time to do further testing yet, but I do have an update. The affected OL8 boxes needed openssl 3 libs installed for unrelated reasons, and this seems to have got rid of this issue.

The change was to install these libs on all ol8 boxes.

openssl3-devel-3.2.2-7.1.el8.x86_64
openssl3-libs-3.2.2-7.1.el8.x86_64
openssl3-3.2.2-7.1.el8.x86_64

... on top of the preexisting libs

openssl-libs-1.1.1k-14.el8_6.x86_64
openssl-1.1.1k-14.el8_6.x86_64
openssl-devel-1.1.1k-14.el8_6.x86_64

That does not explain why fedora41 has this issue though, which already has

openssl-libs-3.2.4-1.fc41.x86_64
openssl-3.2.4-1.fc41.x86_64
openssl-devel-3.2.4-1.fc41.x86_64

I'm fine if you want to close the issue at this time, otherwise I will update this if I find out anything else.

Thank you.