ssl.SSLWantReadError: The operation did not complete (read)

[FilipDem]

Describe the issue

I got regular ssl.SSLWantReadError on the get_vehicles().
I am using version 0.16.3, but got it also with other versions.
See traceback below.

Traceback (most recent call last):
  File "/home/pi/domoticz/plugins/Bmw/plugin.py", line 276, in handleTasks
    asyncio.run(self.myBMW.get_vehicles())
  File "/usr/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.9/dist-packages/bimmer_connected/account.py", line 112, in get_vehicles
    await self._init_vehicles()
  File "/usr/local/lib/python3.9/dist-packages/bimmer_connected/account.py", line 84, in _init_vehicles
    vehicle_list_response = await client.post(
  File "/usr/local/lib/python3.9/dist-packages/httpx/_client.py", line 1892, in post
    return await self.request(
  File "/usr/local/lib/python3.9/dist-packages/httpx/_client.py", line 1574, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/usr/local/lib/python3.9/dist-packages/httpx/_client.py", line 1661, in send
    response = await self._send_handling_auth(
  File "/usr/local/lib/python3.9/dist-packages/httpx/_client.py", line 1686, in _send_handling_auth
    request = await auth_flow.__anext__()
  File "/usr/local/lib/python3.9/dist-packages/bimmer_connected/api/authentication.py", line 85, in async_auth_flow
    await self.login()
  File "/usr/local/lib/python3.9/dist-packages/bimmer_connected/api/authentication.py", line 134, in login
    token_data = await self._login_row_na()
  File "/usr/local/lib/python3.9/dist-packages/bimmer_connected/api/authentication.py", line 156, in _login_row_na
    r_oauth_settings = await client.get(
  File "/usr/local/lib/python3.9/dist-packages/httpx/_client.py", line 1801, in get
    return await self.request(
  File "/usr/local/lib/python3.9/dist-packages/httpx/_client.py", line 1574, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/usr/local/lib/python3.9/dist-packages/httpx/_client.py", line 1661, in send
    response = await self._send_handling_auth(
  File "/usr/local/lib/python3.9/dist-packages/httpx/_client.py", line 1689, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "/usr/local/lib/python3.9/dist-packages/httpx/_client.py", line 1726, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "/usr/local/lib/python3.9/dist-packages/httpx/_client.py", line 1763, in _send_single_request
    response = await transport.handle_async_request(request)
  File "/usr/local/lib/python3.9/dist-packages/httpx/_transports/default.py", line 373, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/usr/local/lib/python3.9/dist-packages/httpcore/_async/connection_pool.py", line 216, in handle_async_request
    raise exc from None
  File "/usr/local/lib/python3.9/dist-packages/httpcore/_async/connection_pool.py", line 196, in handle_async_request
    response = await connection.handle_async_request(
  File "/usr/local/lib/python3.9/dist-packages/httpcore/_async/connection.py", line 99, in handle_async_request
    raise exc
  File "/usr/local/lib/python3.9/dist-packages/httpcore/_async/connection.py", line 76, in handle_async_request
    stream = await self._connect(request)
  File "/usr/local/lib/python3.9/dist-packages/httpcore/_async/connection.py", line 154, in _connect
    stream = await stream.start_tls(**kwargs)
  File "/usr/local/lib/python3.9/dist-packages/httpcore/_backends/anyio.py", line 80, in start_tls
    raise exc
  File "/usr/local/lib/python3.9/dist-packages/httpcore/_backends/anyio.py", line 71, in start_tls
    ssl_stream = await anyio.streams.tls.TLSStream.wrap(
  File "/usr/local/lib/python3.9/dist-packages/anyio/streams/tls.py", line 132, in wrap
    await wrapper._call_sslobject_method(ssl_object.do_handshake)
  File "/usr/local/lib/python3.9/dist-packages/anyio/streams/tls.py", line 140, in _call_sslobject_method
    result = func(*args)
  File "/usr/lib/python3.9/ssl.py", line 944, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLWantReadError: The operation did not complete (read) (_ssl.c:1123)

Expected behavior

Which Home Assistant version are you using?

(using Domoticz)

What was the last working version of Home Assistant Core?

No response

What is your region?

Rest of world

MyBMW website

• I can still successfully login to the BMW MyBMW website and the car status is available there.
• I have MyBMW enabled for my vehicle.

Number of cars

• I have 2 or more cars linked to the MyBMW account.
• I have a Mini vehicle linked to my account.
• I have a Toyota Supra vehicle linked to my account.

Output of bimmer_connected fingerprint

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

[rikroe]

I guess this is either an issue of the BMW servers or at least your connectivity to them.

Nothing really I would know we can change. By default, the timeout is 5s (which should be plenty).

[FilipDem]

OK, thanks for the answers. It is not at my side... I have now build in retries...
I would have expected that the error was captured inside the get_vehicles(), but all fine & clear with your explanation.

[FilipDem]

I am still getting troubles and after investigating I might have found a track of the origin. As all was working nice a long time and now I am getting the problem, I migth be linked with an update to bullseye (and thus Python 3.9). I found back
python/cpython#88216 wrt to
ssl.SSLWantReadError.

This issue is a regression in Python 3.9. It was recently fixed in main/3.10, but I'm opening this issue to request that it is fixed in 3.9 as well since it breaks certain Python scripts running in WeeChat.

I have a C application which is using the Python/C API and is running multiple python subinterpreters. One of those subinterpreters is running a script which is reading data from a non-blocking ssl socket. When there is no more data to read, trying to read throws ssl.SSLWantReadError which is handled by the script. However, if a script in another subinterpreter imports _ssl, the SSLWantReadError exceptions thrown in the first script now have a different class instance, so they are not catched anymore.

As my Domoticz server is based on Bullseye (Python 3.9), would it be possible to have a workaround? I am willing to implement retries, but it seems that this exception destroys sth fundamentally so that a restart of the of the server is the only way out (which is not practical). So if it would be possible to catch this error it would be nice...

For more info about my implementation (that ran since a long time without any problem): Domoticz Plugin

Thanks

F

[rikroe]

Hi Filip,

I fear I have no clue how to work around this in the library. As we do not build our own HTTP connections but just reusing httpx, it's very far from me.
If this is a Python 3.9 issue, I would add an alternative/second python version to your distribution (so existing default python3.9 is not touched, but you can use python3.10).

See e.g. https://tiltingatwindmills.dev/how-to-install-an-alternate-python-version/.
I only would check to get the latest patch release and ensure to compile with configure --enable-optimizations.

Then run your scripts not via python3 script.py but python3.10 script.py.

Richard

[FilipDem]

The problem comes with the call to get_vehicles() as I am doing it in loop every 10 min. The problem is that a simple retry from my side doesn't work if I capture the problem... Only a complete reset works (even redoing first a MyBMWAccount fails always then). So I thought that there is some "bmw context" lost during the "crash" and if you could capture the problem yourself and make a retry possible, it could help (I can do the retry myself of course)... But not sure, you know better your code.

I also thought about the second python, but Domoticz uses an embedded python... It is not a simple execution of a script. So this makes it all more complex and I am afraid of side effects.

[rikroe]

It seems you loose the BMW context as you set self.myBmw = None (https://github.com/FilipDem/Domoticz-BMW-plugin/blob/main/plugin.py#L302).

If you don't do this, it should continue.

In light of the recent changes with captchas, you should look into storing gcid and refresh_token into Domoticz after every update (and read from there when initializing MyBMWAccount).

[FilipDem]

This line of code is also for a workaround that doesn't happen anymore...
But anyway, indeed I need update with Captchas and will look into the refresh tokens...
Thanks anyway for your help... Now first the Captchas :-)

[FilipDem]

Hi rikroe,
Just a small question... I analyzed a bit the integration in HA to see how to threat the refresh_token... I store indeed the refresh_token and gcid now, but I am wondering how to use... Also based on your remark above, are the following assumptions correct?

• to inialize MyBMWAccount the first timer it is mandatory to deliver a valid captcha (I already did this change and works)
• Because I am storing now the refresh_token and gcid, I can use them at every subsequent initialization (eg a reboot of the domotica server). To initialize MyBMWAccount, I only need the username, password, (region) and the refresh_token/gcid. So no need to supply the Captcha. Right? I deduct this from the __init__ in the class BMWDataUpdateCoordinator. However I don't understand how it works as the set_refresh_token is called after the MyBMWAccount without captcha...
  Do you know in which circumstances the captcha will be requested? After the initialization, I am just updating in loop by calling get_vehicles() (and updating the refresh_token when changed). So wondering where I can expect problems then...
  Your view and guidance how to initialize with and without captcha and refresh_token would be highly appreciated! Thanks in advance.
  Filip

[rikroe]

Hi Filip,

I'll try to anwer below.

• to inialize MyBMWAccount the first timer it is mandatory to deliver a valid captcha (I already did this change and works)

Correct.

• Because I am storing now the refresh_token and gcid, I can use them at every subsequent initialization (eg a reboot of the domotica server).

Correct.

To initialize MyBMWAccount, I only need the username, password, (region) and the refresh_token/gcid. So no need to supply the Captcha. Right? I deduct this from the __init__ in the class BMWDataUpdateCoordinator. However I don't understand how it works as the set_refresh_token is called after the MyBMWAccount without captcha...

Yes, if you have a refresh_token and gcid, there is normally no need for a captcha (except the refresh_token has been already used).
Maybe the following helps to make it clear:

# Initialize MyBMWAccount (but only the object itself, this does login/connect to BMW API )
acc = MyBMWAccount(username, password, region)

# Now we set the previously stored tokens into the account
acc.set_refresh_token(refresh_token=stored_refresh_token, gcid=stored_gcid)

# When getting the vehicles, the library will automatically login using refresh token
await acc.get_vehicles()

# if acc.refresh_token has changed to the previous value, it should be stored again

Do you know in which circumstances the captcha will be requested? After the initialization, I am just updating in loop by calling get_vehicles() (and updating the refresh_token when changed). So wondering where I can expect problems then...

If BMW decides to force-logout a user or for some reason getting the automated refresh token doesn't work.
The library will raise a MyBMWCaptchaMissingError in this case which you can catch and then act accordingly.

Hope this helps!

[FilipDem]

This was of great help and clarified all for me! Thanks! Update for integration in domoticz is on its way...

[FilipDem]

I finished the implementation yesterday evening, but getting now systematically an error after calling self.myVehicle = get_vehicles() and get_vehicle(<VIN>), I got an error on getting the timestamp of the latest data... See below...
I am running back in the problem https://bugs.python.org/issue27400. I also noticed that there was a recent commit that could have an impact on this
In my code, I have a workaround as in https://github.com/FilipDem/Domoticz-BMW-plugin/blob/main/domoticz_tools.py#L160. Would it be possible to include this workaround pls (I think you did it already some years ago somewhere I think)?
Or do you prefer I open a new issue report?

Traceback (most recent call last):
  File "/home/pi/domoticz/plugins/Bmw/plugin.py", line 323, in handleTasks
    self.updateVehicleStatus()
  File "/home/pi/domoticz/plugins/Bmw/plugin.py", line 469, in updateVehicleStatus
    if self.myVehicle.fuel_and_battery.charging_end_time and self.myVehicle.timestamp:
                                                             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/bimmer_connected/vehicle/vehicle.py", line 230, in timestamp
    parse_datetime(str(self.data[ATTR_STATE].get("lastFetched", ""))),
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/bimmer_connected/utils.py", line 39, in parse_datetime
    parsed = datetime.datetime.strptime(date_str, date_format)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: 'NoneType' object is not callable

[rikroe]

Can you please open a separate issue? There's a lot going on still and this would help organizing

[FilipDem]

Perhaps this topic is a bit mixed up, but coming back to the origin of the topic...
I implemented in the meantime the use of the refresh token, and with the new version it was running again perfectly for a while... And recently I got again the annoying The operation did not complete (read) (_ssl.c:1123) error.

And the strange thing, once this error happens once, it is persistant, only starting the whole flow again from authentication etc is solving the problem (which is in practice - a manual intervention to restart the "plugin").

So I was wondering if it could be a problem with the ssl context that gets lost somewhere?
Btw: if I would do the MyBMWAccount with the option "verify=False", would this work?

Filip

[rikroe]

Well, running with verify=False is insecure as you do not get the information of somebody captures your traffic.

However it would be an easy test and then we can try figuring out an alternative.

[FilipDem]

yes, I know about the security impact. I started yesterday with the option and it is still running without error... Yesterday I had twice the problem... But this is not a guarantee as before it ran for some days without problem... So it really seems random... I will monitor longer and will see.