Unable to respond normally to HTTPS requests that use ports other than 443

[h4ckdepy]

Self Checks

• This is only for bug report, if you would like to ask a question, please head to Discussions.
• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
• Please do not modify this template :) and fill in all the required fields.

Dify version

0.15.1

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

Use Http Request Node:

Use Curl:

✔️ Expected Behavior

Can respond normally to HTTPS interfaces that use ports other than 443

❌ Actual Behavior

For interfaces that use the HTTPS protocol, as long as the port number is included, for example https://xx.com:6677 , it will display a 403 forbidden error. I will ensure that the host network is functioning properly and curl access is normal

[dosubot]

Hey @h4ckdepy! 👋 I'm here to help you with bug fixes, questions, and becoming a contributor. Let's squash those bugs together!

I found a similar discussion titled "403 Forbidden" which is still open and was last updated on August 7, 2024 ^[1]. This might be related to the issue you're experiencing with the 403 Forbidden error when using non-standard HTTPS ports.

Additionally, there is a known difference in how Dify's HTTP request node and a standard curl command handle HTTPS requests. In a self-hosted Dify instance, the Dify client might fail due to proxy misconfiguration, leading to plain HTTP requests being sent to an HTTPS port, resulting in a 400 Bad Request error. The curl command succeeds because it correctly handles the HTTPS request without encountering the proxy issue. Ensuring proper proxy configuration is crucial for successful communication with the API ^[2].

You might want to check the proxy settings in your Docker setup, as Dify's Docker configuration can affect HTTP request handling. Ensure that the proxy is properly configured to handle HTTPS requests and maintain the correct protocol when forwarding requests to the Dify API endpoint ^[3].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[h4ckdepy]

What you said seems to be fine. I just did the same operation on dify's official website and it was successfully accessed. It seems to be a problem caused by Docker deployment.

[h4ckdepy]

In addition, curl testing is only to prove that network access is not a problem. Use regular interfaces, such as https://www.baidu.com There is no problem with such a website. Only sites that use HTTPS and require a port will encounter problems. How do I configure Docker? @dosu

[dosubot]

To address the issue with Docker deployment affecting HTTPS requests on non-standard ports, you should ensure that your Docker setup is correctly configured to handle such traffic. Here are some steps you can take:

1. Check Docker Network Settings: Ensure that the Docker container is using the correct network mode. For handling HTTPS traffic, using the host network mode can sometimes resolve issues, as it allows the container to share the host's network stack.

2. Port Mapping: Make sure that the ports required by your application are correctly mapped in your Docker configuration. For example, if your application needs to access port 6677, ensure that this port is exposed and mapped in your Docker docker-compose.yml or Dockerfile.

3. Proxy Configuration: Verify that any proxy settings within your Docker environment are correctly configured to handle HTTPS traffic. Misconfigured proxies can lead to protocol mismatches, causing errors like 403 Forbidden.

4. Firewall and Security Groups: Check if there are any firewall rules or security group settings that might be blocking access to the required ports.

5. SSL/TLS Configuration: Ensure that your SSL/TLS settings are correctly configured to handle requests on non-standard ports.

By ensuring these configurations are correctly set, you should be able to resolve the issues with accessing HTTPS sites on non-standard ports in your Docker deployment.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[h4ckdepy]

I just tested the dify service deployed locally on macOS using Docker and encountered the same issue.

[crazywoola]

Duplicated, check if this domain is listed in squid.conf, you need to run the curl command in docker container instead of running on your laptop.

[h4ckdepy]

@crazywoola

Hi,

I'm sorry, I may not have explained the problem clearly. Let me illustrate this issue with clearer examples:

1. DIFY for Docker deployment
2. I have three APIs belonging to the same domain, two of which are bound with SSL certificates. One for https://xx.com/api One for https://xx.com:7766/api, Another one is the HTTP protocol http://xx.com:8092/api
3. On the HTTP request module of DIFY, only https://xx.com:7766/api Unable to access. Display 403 prohibited
4. But on the official DIFY demo site, all three APIs can be accessed successfully

Due to the difference in response from the official website, I believe it is due to certain mechanisms and configurations of Docker that prevent HTTPS sites with ports from responding properly.

[laipz8200]

Please check if #13108 resolves this issue.

[h4ckdepy]

Hi, @laipz8200

I have upgraded the version to 0.15.3.

Unfortunately, the bug still exists. My interface service is to obtain the export IP, but on the dify deployed in Docker, it is still 403.

Is there a debugging document for Docker services? Can quickly locate the code logic related to HTTP request nodes and debug their entire lifecycle.

[h4ckdepy]

Good evening, let me share some of my findings.

Command: docker exec -it docker-api-1 cat /app/logs/server.log

Show This:

2025-02-07 15:32:42.187 WARNING [Dummy-5] [ssrf_proxy.py:62] - Request to URL https://www.***.com:8821/index/index/getIp?token=*** failed on attempt 1: 403 Forbidden
2025-02-07 15:32:42.188 WARNING [Dummy-5] [node.py:103] - http request node 1738940219471 failed to run: 403 Forbidden

It seems that the problem lies in SSRF-PROXY.PY. According to the code meaning, I used the SSRF_PROXY_HTTP_URL in the system variable which is http://ssrf_proxy:3128 .

Make curl proxy request:

root@99723075c939:/app/api# curl https://www.***.com:8821/index/index/getIp?token=***
120.***.***.27
root@99723075c939:/app/api# curl -x http://ssrf_proxy:3128 https://www.***.com:8821/index/index/getIp?token=***
curl: (56) CONNECT tunnel failed, response 403

After testing a large number of URLs, it was found that only URLs with HTTPS and ports would have 403 errors.

@laipz8200

[h4ckdepy]

Hi, @laipz8200 @crazywoola

Based on my findings above I was able to pinpoint that the issue was with the ssrf_proxy:3128 proxy server, which was unable to respond to data normally.

So I entered the docker-ssrf_proxy-1 container and found that it was using the Squid service for proxy forwarding.

I checked the configuration of the /etc/squid/squid.conf file and found that the acl SSL_ports port was only configured for 443.

So I added a new line: acl SSL_ports port 1025-65535.

Then I used the squid -k reconfigure command to reload the configuration, and now the issue has been resolved.

I'll submit a PR later to update the default squid.conf.template file.

Thank you all once again for your hard work and attention.

[laipz8200]

Excellent troubleshooting!