Issue: OpenAI Bad Gateway results in Error in on_retry: asyncio.run() cannot be called from a running event loop (coroutine 'AsyncRunManager.on_retry' was never awaited) inside openai.acompletion_with_retry #8462

maspotts · 2023-07-29T17:11:33Z

Issue you'd like to raise.

I just saw a novel error, which appears to be triggered by a failed OpenAI API call (inside an asynchronous block) which is causing an asyncio.run() inside an asyncio.run(). Error pasted below. Is this my (user) error? Or possibly a problem with the acompletion_with_retry() implementation?

2023-07-29 05:53:14,838 INFO     message='OpenAI API response' path=https://api.openai.com/v1/chat/completions processing_ms=None request_id=None response_code=502
2023-07-29 05:53:14,838 INFO     error_code=502 error_message='Bad gateway.' error_param=None error_type=cf_bad_gateway message='OpenAI API error received' stream_error=False
2023-07-29 05:53:14,839 WARNING  Retrying langchain.chat_models.openai.acompletion_with_retry.<locals>._completion_with_retry in 4.0 seconds as it raised APIError: Bad gateway. {"error":{"code":502,"message":"Bad gateway.","param":null,"type":"cf_bad_gateway"}} 502 {'error': {'code': 502, 'message': 'Bad gateway.', 'param': None, 'type': 'cf_bad_gateway'}} <CIMultiDictProxy('Date': 'Sat, 29 Jul 2023 05:53:14 GMT', 'Content-Type': 'application/json', 'Content-Length': '84', 'Connection': 'keep-alive', 'X-Frame-Options': 'SAMEORIGIN', 'Referrer-Policy': 'same-origin', 'Cache-Control': 'private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0', 'Expires': 'Thu, 01 Jan 1970 00:00:01 GMT', 'Server': 'cloudflare', 'CF-RAY': '7ee3120dab9f1084-ORD', 'alt-svc': 'h3=":443"; ma=86400')>.
2023-07-29 05:53:14,839 ERROR    Error in on_retry: asyncio.run() cannot be called from a running event loop
/usr/local/python-modules/tenacity/__init__.py:338: RuntimeWarning: coroutine 'AsyncRunManager.on_retry' was never awaited
  self.before_sleep(retry_state)
RuntimeWarning: Enable tracemalloc to get the object allocation traceback

Suggestion:

No response

The text was updated successfully, but these errors were encountered:

walkward · 2023-08-02T22:49:25Z

@hinthornw Looks like this error was likely introduced in #8053. Any ideas?

ryanstout · 2023-08-05T18:31:44Z

@maspotts I'm seeing similar, what type of code are you running that is causing this? Thanks

kylrth · 2023-08-06T19:19:05Z

The problem is that create_base_retry_decorator tries to asyncio.run something in the before_sleep callback, which breaks things when this is all happening inside an agenerate call.

I'm not familiar enough with the retry decorator design to fix this myself, but it seems like acompletion_with_retry (async) needs an async version of _create_retry_decorator. 🤷

ryanstout · 2023-08-06T22:45:22Z

@kylrth Thanks for the info, yea, I think I'm seeing similar. What kind of code are you running that causes it. For me it's doing a MapReduce.

bent-verbiage · 2023-08-08T04:17:21Z

+1 on the issue. I got it on a chain.arun() where OpenAI returned a 502.

`Error in on_retry: asyncio.run() cannot be called from a running event loop

Retrying langchain.chat_models.openai.acompletion_with_retry.<locals>._completion_with_retry in 4.0 seconds as it raised APIError: HTTP code 502 from API (<html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>cloudflare</center>
</body>
</html>
).
```

kylrth · 2023-08-08T14:42:21Z

I see it with just ChatOpenAI.agenerate when OpenAI returns a 502.

change types on sessoins.$id.$pageNum.tsx

NikitaSemenovAiforia · 2023-08-14T14:46:36Z

Catched this warning on pytest recently. Don't know if this 502 or what.

ShantanuNair · 2023-09-07T11:53:50Z

@hwchase17 Any chance someone's looking at this? It's a source of high billables potentially.

ShantanuNair · 2023-09-13T09:58:15Z

@maspotts I'm looking into this too, do you see the same issue on the latest version? @kylrth can you expand a bit more on why the asyncio.run is problematic here? I am running into this issue with agenerate, and notice my retries don't run after 4/8/10 seconds as they should - they are run after about 6-7 minutes, and I'm wondering if fixing this bug may fix my issue of retries from 502s failing. Maybe I can take a go at tackling this.

ShantanuNair · 2023-09-13T13:32:03Z

@kylrth Thanks for the info, yea, I think I'm seeing similar. What kind of code are you running that causes it. For me it's doing a MapReduce.

Me too. A mapreduce chain via analayzeDocumentsChain

kylrth · 2023-09-13T13:34:09Z

I think this issue is closed by #8659. @hinthornw ?

Could some of you experiencing the original error please test on v0.0.252 or later?

ShantanuNair · 2023-09-13T14:22:09Z

@kylrth Hah, I was just running through the same PR. Looks like the way tenacity's retry decorator works takes care of the async/sync switch. Can I ask - can you verify that on recent langchain that retries do indeed work after X (2/4/8) seconds? For me it's hanging for 6-7 minutes between retries even though it prints our retrying in X time.

ShantanuNair · 2023-09-20T08:43:48Z

So important notes regarding this issue from my investigation:

If using Async calls, when you receive a 502 bad gateway, it will timeout after the whole 600s. Align request_timeout Behavior in Async and Non-Async APIs openai/openai-python#387 Needs to be fixed in openai-python. We need read timeouts and not a total timeout.
After stalling your chain for the entirety of 10 minutes, you WILL BE BILLED for no generation, besides it absolutely destroying experience by forcing one part of the chain to wait an entire 10 minutes.

nfcampos · 2023-09-25T09:43:56Z

@hinthornw this was fixed in #8659 right? If so, lets close it

dosubot bot added the 🤖:bug Related to a bug, vulnerability, unexpected error with an existing feature label Jul 29, 2023

ryanstout added a commit to ryanstout/mrfreeze that referenced this issue Aug 10, 2023

avoid current langchain async bug: langchain-ai/langchain#8462

7e14355

change types on sessoins.$id.$pageNum.tsx

nfcampos closed this as completed Sep 25, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Issue: OpenAI Bad Gateway results in Error in on_retry: asyncio.run() cannot be called from a running event loop (coroutine 'AsyncRunManager.on_retry' was never awaited) inside openai.acompletion_with_retry #8462

Issue: OpenAI Bad Gateway results in Error in on_retry: asyncio.run() cannot be called from a running event loop (coroutine 'AsyncRunManager.on_retry' was never awaited) inside openai.acompletion_with_retry #8462

maspotts commented Jul 29, 2023

walkward commented Aug 2, 2023 •

edited

Loading

ryanstout commented Aug 5, 2023

kylrth commented Aug 6, 2023 •

edited

Loading

ryanstout commented Aug 6, 2023

bent-verbiage commented Aug 8, 2023

kylrth commented Aug 8, 2023

NikitaSemenovAiforia commented Aug 14, 2023 •

edited

Loading

ShantanuNair commented Sep 7, 2023

ShantanuNair commented Sep 13, 2023

ShantanuNair commented Sep 13, 2023

kylrth commented Sep 13, 2023 •

edited

Loading

ShantanuNair commented Sep 13, 2023

ShantanuNair commented Sep 20, 2023

nfcampos commented Sep 25, 2023

Issue: OpenAI Bad Gateway results in Error in on_retry: asyncio.run() cannot be called from a running event loop (coroutine 'AsyncRunManager.on_retry' was never awaited) inside openai.acompletion_with_retry #8462

Issue: OpenAI Bad Gateway results in Error in on_retry: asyncio.run() cannot be called from a running event loop (coroutine 'AsyncRunManager.on_retry' was never awaited) inside openai.acompletion_with_retry #8462

Comments

maspotts commented Jul 29, 2023

Issue you'd like to raise.

Suggestion:

walkward commented Aug 2, 2023 • edited Loading

ryanstout commented Aug 5, 2023

kylrth commented Aug 6, 2023 • edited Loading

ryanstout commented Aug 6, 2023

bent-verbiage commented Aug 8, 2023

kylrth commented Aug 8, 2023

NikitaSemenovAiforia commented Aug 14, 2023 • edited Loading

ShantanuNair commented Sep 7, 2023

ShantanuNair commented Sep 13, 2023

ShantanuNair commented Sep 13, 2023

kylrth commented Sep 13, 2023 • edited Loading

ShantanuNair commented Sep 13, 2023

ShantanuNair commented Sep 20, 2023

nfcampos commented Sep 25, 2023

walkward commented Aug 2, 2023 •

edited

Loading

kylrth commented Aug 6, 2023 •

edited

Loading

NikitaSemenovAiforia commented Aug 14, 2023 •

edited

Loading

kylrth commented Sep 13, 2023 •

edited

Loading