Invalid State error + Closed Client

[timos-flwls]

Inquiry

We had a production incident today where the Valkey client started erroring with glide.exceptions.ClosingError: Unable to execute requests; the client is closed. Please create a new client.

Of note is that the first request that errored had a different error message:

  File "/app/.venv/lib/python3.12/site-packages/glide/async_commands/core.py", line 614, in get
    return cast(Optional[bytes], await self._execute_command(RequestType.Get, args))
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/glide/glide_client.py", line 443, in _execute_command
    return await self._write_request_await_response(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/glide/glide_client.py", line 647, in _write_request_await_response
    await response_future
glide.exceptions.ClosingError: invalid state

1. I am struggling to find information about this error... what would cause the invalid state error?
2. What's the recommended way of creating a new client? It seems like generally you would create a single global client to benefit from multiplexing. Is this something that the client wants to handle internally?

Language

Python

Language Version

3.12

Engine Version

No response

Operating System

Ubuntu

Additional Technical Information

No response

[jamesx-improving]

Hi @timos-flwls , thank you for reporting this issue.

We would like to know a bit more about your deployment environment:

• Valkey-GLIDE version (2.0.1 is the latest)
• OS (saw Ubuntu, please provide version if possible) and CPU arch of the deploy environment
• Any cloud service is used for the deployment (e.g.: AWS Fargate, Lambda, etc.)

Based on information provided, this could be related to the glide-core process being terminated while the python process is still alive. GLIDE use a native core and wrap it with different language bindings, so both processes need to be alive. You are right to point out that it is recommended to create a single global client to benefit from multiplexing, but if any event causing either the core process or the UDS used for communication between them is terminated, there's no way we can restore the current glide client to a usable state, and recreate a new glide client would be the only option.

[timos-flwls]

Hey, sure here is some more info:

• client version is 2.0.1
• Debian bookworm (sorry I was wrong about this in the OP). Specifically using the python:3.12.9-slim docker image..
• Deployed on AWS Fargate.

Let me know if I can be of any further assistance.

[jamesx-improving]

Hi @timos-flwls ,

So we've had reports with similar errors on Fargate before, especially when it is autoscaling. As we know, as part of the Fargate autoscaling process, it terminates the old containers (SIGTERM, then after 30s SIGKILL). And because GLIDE needs two processes working together, if one of them got terminated while the other is still alive, you will see similar errors as you reported here.

Mostly this should be expected and harmless. If this is a concern, please try to close the glide client in your SIGTERM handler.

Feel free to let us know if you have any further question on this. Happy to help!

[timos-flwls]

I don't think this is expected:

1. we explicitly close the client as part of our application shutdown upon receiving a SIGTERM.
2. there was no autoscaling event, the incident was ongoing for multiple hours.
3. this happened after the server instance was running without issue for multiple days.

From my perspective as a user of the library:

• The client closed randomly without any obvious cause.
• The client provided no information about why the client was closed, I can't find any information about the "invalid state" error (in the source code or documentation) and therefore cannot figure out what the cause might be.
• Once the connection was closed the client did not recover and instead lead to a partial outage in our service.

My questions at this point:

• What could be a possible root cause?
• Could the error raised be more specific/descriptive or if its fine maybe could it be documented somewhere what "invalid state" means?
• Is there a world in which the python client could handle this gracefully? You said "there's no way we can restore the current glide client to a usable state" but what other options do I have as a user? I would need to write a wrapper around the client where everytime its called I need to check whether the client is closed, and if it is recreate it.

[timos-flwls]

We've mitigated this on our side by tweaking our health check endpoint to 400 if the valkey client is closed, and let the cluster manage restarting the app and recreating the client. I could not find any sensible way to restart it within the app itself.

A side note on that, it would be nice if there was a way to check whether the client is closed via the public API, currently you have to use _is_closed.

[timos-flwls]

This has happened again, here's the error stack from the initial "invalid state" error in case its helpful in determining a root cause.

Traceback (most recent call last):
  File "/app/.venv/lib/python3.12/site-packages/glide/glide_client.py", line 340, in _write_or_buffer_request
    await self._write_buffered_requests_to_socket()
  File "/app/.venv/lib/python3.12/site-packages/glide/glide_client.py", line 365, in _write_buffered_requests_to_socket
    await self._stream.send(b_arr)
  File "/app/.venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 1402, in send
    raise ClosedResourceError from None
anyio.ClosedResourceError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/.venv/lib/python3.12/site-packages/ddtrace/contrib/internal/asyncio/patch.py", line 61, in traced_coro
    return await coro
           ^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/glide/glide_client.py", line 349, in _write_or_buffer_request
    res_future.set_exception(e)
asyncio.exceptions.InvalidStateError: invalid state

[jamesx-improving]

Hi @timos-flwls , thank you for your reply, and glad to know you've come up with something to make it work for now.

To your point on the error message and the handling of it, as this type of error is unexpected and should not happen in any circumstance, we really didn't have much chance to fine-tune the user experience of it, including the lack of a client status check API, and I'm sorry for that.

As it is the python code reporting the error, despite the TBD root cause, it is definitely either the native core component, or the Unix Domain Socket used for communication between the core and the python side, has had some fatal error, leading to its termination. In either case, there's no way we could restore the existing glide client, the only option is to close the current client, and create a new one.

Going forward, we shall conduct tests in the Fargate environment, to try to reproduce this type of issue, hopefully find the root cause of it, and fix it. In the mean time, we could keep this issue as a feature requirement to add an API to get the status of the client, like exposing the _is_closed state, to the end user. In terms of the error message invalid state, which can't be found in our documentation or code base, is most likely coming from underlying error reported by the UDS. We will refine related error messages to provide more relevant information to the end user as well.

Putting this to 2.2 candidate for now. Any further question please let us know. Happy to help!

[timos-flwls]

Sounds like a great plan, thank you. Will let you know if we find out any more info on our side.

[liorsve]

Hi @timos-flwls, I'm wondering if there was any ECS event happening at the time of the failure (not just autoscaling)? Anything in the ECS logs? Also, if your CloudWatch logs are enabled, could there maybe be a resource exhaustion issue that you can see there?

It looks like the UDS socket was closed unexpectedly. The InvalidStateError: invalid state happens due to an issue in error handling — Glide runs separate reader and writer tasks, and when the socket closes, both tasks detect it around the same time which causes each to try and set an exception on the same future, but by the time the second task does so, the future is already completed, causing the InvalidStateError. As to the reason the UDS closed in the first place, that's where your logs can help us.

Generally, when the client encounters a fatal error it can’t recover from, a ClosingError should be raised (like in the first traceback you shared), signaling that the client should be re-created. In this case though, it seems there was a scenario not covered by that exception, which we’ll be fixing soon.

[timos-flwls]

Hey @liorsve I just saw this now. Unfortunately there is no other info in the logs, the first log we see is the invalid state error. However one thing I have noticed is this happens pretty reliably after cpu hits 100% for a while caused by a burst in traffic before autoscaling can kick in. No ECS events either.

[liorsve]

Interesting. Could you perhaps connect to your container and check the glide logs? By default, they should be saved in the valkey-glide/python/glide-logs directory. That might give us a clue as to whether the core rust thread closed which then caused the UDS to be half closed, or the UDS itself closed for some reason while the core thread was still alive.