Frequent Request timed out

[kovogo]

After sending several requests to OpenAI, it always encounter request timeouts, accompanied by long periods of waiting.

Env:

OS: Ubuntu 22
Python: 3.10
langchain: 0.0.117

Request time out

WARNING:/home/soda/.local/lib/python3.10/site-packages/langchain/chat_models/openai.py:Retrying langchain.chat_models.openai.acompletion_with_retry.<locals>._completion_with_retry in 4.0 seconds as it raised Timeout: Request timed out.

TimeoutError from None

    answer, _ = await self.combine_documents_chain.acombine_docs(docs, question=question)
  File "/home/soda/.local/lib/python3.10/site-packages/langchain/chains/combine_documents/stuff.py", line 97, in acombine_docs
    return await self.llm_chain.apredict(**inputs), {}
  File "/home/soda/.local/lib/python3.10/site-packages/langchain/chains/llm.py", line 167, in apredict
    return (await self.acall(kwargs))[self.output_key]
  File "/home/soda/.local/lib/python3.10/site-packages/langchain/chains/base.py", line 154, in acall
    raise e
  File "/home/soda/.local/lib/python3.10/site-packages/langchain/chains/base.py", line 148, in acall
    outputs = await self._acall(inputs)
  File "/home/soda/.local/lib/python3.10/site-packages/langchain/chains/llm.py", line 135, in _acall
    return (await self.aapply([inputs]))[0]
  File "/home/soda/.local/lib/python3.10/site-packages/langchain/chains/llm.py", line 123, in aapply
    response = await self.agenerate(input_list)
  File "/home/soda/.local/lib/python3.10/site-packages/langchain/chains/llm.py", line 67, in agenerate
    return await self.llm.agenerate_prompt(prompts, stop)
  File "/home/soda/.local/lib/python3.10/site-packages/langchain/chat_models/base.py", line 103, in agenerate_prompt
    raise e
  File "/home/soda/.local/lib/python3.10/site-packages/langchain/chat_models/base.py", line 97, in agenerate_prompt
    output = await self.agenerate(prompt_messages, stop=stop)
  File "/home/soda/.local/lib/python3.10/site-packages/langchain/chat_models/base.py", line 62, in agenerate
    results = [await self._agenerate(m, stop=stop) for m in messages]
  File "/home/soda/.local/lib/python3.10/site-packages/langchain/chat_models/base.py", line 62, in <listcomp>
    results = [await self._agenerate(m, stop=stop) for m in messages]
  File "/home/soda/.local/lib/python3.10/site-packages/langchain/chat_models/openai.py", line 286, in _agenerate
    async for stream_resp in await acompletion_with_retry(
  File "/home/soda/.local/lib/python3.10/site-packages/openai/api_resources/abstract/engine_api_resource.py", line 230, in <genexpr>
    return (
  File "/home/soda/.local/lib/python3.10/site-packages/openai/api_requestor.py", line 319, in wrap_resp
    async for r in resp:
  File "/home/soda/.local/lib/python3.10/site-packages/openai/api_requestor.py", line 633, in <genexpr>
    return (
  File "/home/soda/.local/lib/python3.10/site-packages/openai/api_requestor.py", line 114, in parse_stream_async
    async for line in rbody:
  File "/home/soda/.local/lib/python3.10/site-packages/aiohttp/streams.py", line 35, in __anext__
    rv = await self.read_func()
  File "/home/soda/.local/lib/python3.10/site-packages/aiohttp/streams.py", line 311, in readline
    return await self.readuntil()
  File "/home/soda/.local/lib/python3.10/site-packages/aiohttp/streams.py", line 343, in readuntil
    await self._wait("readuntil")
  File "/home/soda/.local/lib/python3.10/site-packages/aiohttp/streams.py", line 303, in _wait
    with self._timer:
  File "/home/soda/.local/lib/python3.10/site-packages/aiohttp/helpers.py", line 721, in __exit__
    raise asyncio.TimeoutError from None

Have other people encountered similar errors?

[edouard-blocktool]

Same here. We use Langchain to power a GPT chat bot. When migrating from the Python openai library to langchain, we started seeing more of those TimeoutError as the traffic increases. They happen on other async calls as well. Here's how we use LangChain.

_TEMPLATE = """[...]"""

class EnqueueCallbackHandler(AsyncCallbackHandler):
    def __init__(self, queue: asyncio.Queue):
        self.queue = queue
        self.token_count = 0

    @property
    def always_verbose(self) -> bool:
        return True

    async def on_llm_new_token(self, token: str, **kwargs: Any) -> None:
        for c in token:
            await self.queue.put(c)
        self.token_count += 1

    async def on_llm_end(self, response: LLMResult, **kwargs: Any) -> None:
        await self.queue.put(None)

    async def on_llm_error(
        self, error: Union[Exception, KeyboardInterrupt], **kwargs: Any
    ) -> None:
        await self.queue.put(None)

class Chat(BaseChat):
    def load_memory(self, user: OrmUser):
        [...]
        memory = ConversationEntityMemory(llm=llm, store=store, chat_memory=ChatMessageHistory(messages=messages), return_messages=True)
        return memory

    async def run(self, model_name: str, user: OrmUser, input: str, queue: asyncio.Queue):
        callback_handler = EnqueueCallbackHandler(queue)
        system_prompt = PromptTemplate(
            template=_TEMPLATE,
            input_variables=["entities"],
            partial_variables={"model_name": GPT_MODELS[model_name]},
        )
        prompt = ChatPromptTemplate.from_messages([
            SystemMessagePromptTemplate(prompt=system_prompt),
            MessagesPlaceholder(variable_name="history"),
            HumanMessagePromptTemplate.from_template("{input}"),
        ])
        chain = ConversationChain(
            llm=ChatOpenAI(
                model_name=model_name,
                streaming=True,
                max_tokens=1000,
                temperature=0.7,
                top_p=1,
                frequency_penalty=0,
                presence_penalty=0,
                callback_manager=AsyncCallbackManager([callback_handler]),
            ), 
            verbose=True,
            prompt=prompt,
            memory=self.load_memory(user),
        )
        answer = await chain.arun(input=input)
        return answer, callback_handler.token_count

[edouard-blocktool]

Answering my own question. ConversationEntityMemory doesn't work well in an async context. Under the hood, the entity resolution is done synchronously. Which breaks async application with high traffic.

[jackbai233]

have you ever used a proxy? I also encountered the same async request problem after using a proxy. if this, you can test adding openai proxy in your main.py:

import openai
openai.proxy = {
            "http": "http://127.0.0.1:7890",
            "https": "http://127.0.0.1:7890"
        }

it works fine for me. refer to langchain-ai/chat-langchain#30

[kovogo]

have you ever used a proxy? I also encountered the same async request problem after using a proxy. if this, you can test adding openai proxy in your main.py:

import openai
openai.proxy = {
            "http": "http://127.0.0.1:7890",
            "https": "http://127.0.0.1:7890"
        }

it works fine for me. refer to hwchase17/chat-langchain#30

My program run on Azure, there is not need to use proxy。I fix this problem but rewrite all sync code to async。

[sgch1982]

have you ever used a proxy? I also encountered the same async request problem after using a proxy. if this, you can test adding openai proxy in your main.py:

import openai
openai.proxy = {
            "http": "http://127.0.0.1:7890",
            "https": "http://127.0.0.1:7890"
        }

it works fine for me. refer to hwchase17/chat-langchain#30

这个好使