Concurrent API requests to Gemini vision model cause non-responsive behavior and block all tasks forever

[Allamss]

Self Checks

• This is only for bug report, if you would like to ask a question, please head to Discussions.
• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
• Please do not modify this template :) and fill in all the required fields.

Dify version

0.9.1 / 0.9.2 / 0.11.2

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

1. Configure Gemini 1.5 Pro 001

2. Create a new workflow using LLM with visual capabilities to briefly analyze images

3. Upload an image via API, then make 50 concurrent API calls to execute the image analysis workflow (I m using java, and 20 concurrent calls is fine). The API calls will hang indefinitely without returning, and all LLM operations on the site will become blocked
   image:
   

4. According to the workflow logs, LLM requests are successfully returned
   

5. This issue does not occur with GPT vision model

✔️ Expected Behavior

Results should be returned normally

❌ Actual Behavior

Results are not returned as expected, and the entire site becomes globally blocked. The site can only be restored by restarting

[dosubot]

The issue you're experiencing with the Gemini vision model might be related to how it handles concurrency. Here are a few steps you can take to troubleshoot and potentially resolve the problem:

1. Concurrency Limits: Check if there are any concurrency limits configured for the Gemini model. If not documented, you might need to experiment with reducing the number of concurrent requests to see if it alleviates the issue.

2. Error Handling: Ensure that your application is correctly handling any errors returned by the API. The Gemini model has specific error handling mechanisms that map various errors to unified types ^[1].

3. Resource Allocation: Compare the resource allocation settings between the Gemini and GPT models. Differences in resource handling might be causing the blockage.

4. Logs and Monitoring: Utilize Dify's logging and monitoring features to diagnose the issue. The logs can provide insights into what happens during the API calls and why they might be hanging ^[2].

5. Timeouts and Retries: Implement timeouts and retries in your API requests to prevent indefinite hanging. This can help manage requests that take too long to process.

6. Version and Configuration: Ensure that your Dify version and configuration are up to date and correctly set up for high-concurrency scenarios.

If these steps don't resolve the issue, consider isolating the problem by testing with fewer concurrent requests or different configurations to pinpoint the cause.

To continue talking to Dosu, mention @dosu.

[Allamss]

The issue you're experiencing with the Gemini vision model might be related to how it handles concurrency. Here are a few steps you can take to troubleshoot and potentially resolve the problem:

1. Concurrency Limits: Check if there are any concurrency limits configured for the Gemini model. If not documented, you might need to experiment with reducing the number of concurrent requests to see if it alleviates the issue.
2. Error Handling: Ensure that your application is correctly handling any errors returned by the API. The Gemini model has specific error handling mechanisms that map various errors to unified types [1].
3. Resource Allocation: Compare the resource allocation settings between the Gemini and GPT models. Differences in resource handling might be causing the blockage.
4. Logs and Monitoring: Utilize Dify's logging and monitoring features to diagnose the issue. The logs can provide insights into what happens during the API calls and why they might be hanging [2].
5. Timeouts and Retries: Implement timeouts and retries in your API requests to prevent indefinite hanging. This can help manage requests that take too long to process.
6. Version and Configuration: Ensure that your Dify version and configuration are up to date and correctly set up for high-concurrency scenarios.

If these steps don't resolve the issue, consider isolating the problem by testing with fewer concurrent requests or different configurations to pinpoint the cause.

To continue talking to Dosu, mention @dosu.

1. The Gemini model has no extra configuration. Crucially, the logs show that Dify's requests to Gemini are returning (with execution time and token usage parameters), indicating that the issue is not caused by limitations.

2. Firstly, the API requests don't return at all but remain blocked indefinitely. I've checked the logs for the web, API, and sandbox containers, and there are no error messages.

3. As per point 1.

4. As per point 1.

5. After the blockage occurs, any API or site LLM requests become blocked as well.

6. I'll try this suggestion later.
   @dosu

[dosubot]

Given the situation, here are some additional steps you might consider:

1. Server Configuration: Ensure that your server settings are optimized for high concurrency. Adjust settings like SERVER_WORKER_AMOUNT and SQLALCHEMY_POOL_SIZE to handle more simultaneous connections ^[1].

2. Gevent Worker Class: Since you're using Docker, confirm that the SERVER_WORKER_CLASS is set to gevent to efficiently manage multiple connections ^[1].

3. Database Bottlenecks: Check if there are any database bottlenecks. Increasing the database connection pool size might help if the database is a limiting factor.

4. Network and Firewall: Verify that there are no network or firewall settings that might be limiting the number of concurrent connections.

5. Resource Monitoring: Use monitoring tools to check CPU, memory, and network usage during the concurrent requests to identify any resource exhaustion.

6. Isolation Testing: Try running the Gemini model in isolation with a reduced number of requests to see if the issue persists, which might help identify if the problem is with the model or the infrastructure.

These steps should help you further diagnose and potentially resolve the issue.

To continue talking to Dosu, mention @dosu.

[Allamss]

I’ve tried the latest version 0.9.2, and the issue is still reproducible @crazywoola @dosu

[hw872715125]

I also encountered this problem @crazywoola

[dosubot]

Hi, @Allamss. I'm Dosu, and I'm helping the Dify team manage their backlog. I'm marking this issue as stale.

Issue Summary:

• The Gemini vision model becomes non-responsive with 50 concurrent API requests.
• Despite successful log returns, tasks are blocked, unlike the GPT vision model.
• Troubleshooting steps included checking concurrency limits, error handling, and resource allocation.
• You confirmed the issue persists without error messages and affects all API requests post-blockage.
• The issue remains unresolved in version 0.9.2, as confirmed by you and another user.

Next Steps:

• Please let us know if this issue is still relevant to the latest version of the Dify repository. If so, you can keep the discussion open by commenting on this issue.
• Otherwise, this issue will be automatically closed in 15 days.

Thank you for your understanding and contribution!

[Allamss]

@dosu, In the latest version 0.11.2, the problem can still be reproduced