Upload start failing on internets with 1mbps upload ( Like ADSL ) with XHTTP packet-up over CDN or/and DIRECT. #4100

deepsm0ke · 2024-12-02T23:00:58Z

Integrity requirements

I confirm that I have read the documentation, understand the meaning of all the configuration items I wrote, and did not pile up seemingly useful options or default values.
I provided the complete config and logs, rather than just providing the truncated parts based on my own judgment.
I searched issues and did not find any similar issues.
The problem can be successfully reproduced in the latest Release

Description

The problem that has been and still is that the upload speed ( Download speed is ok ) and stability on ADSL is not good at all (except for Annex M mode, which has relatively good adsl's upload about 3mbps). On the other hand, other transports like Websocket or httpUpgrade Over CDN ( Tested on CF ) have good upload on ADSL (It uploads slowly, but the important thing is that it starts uploading, whereas with splithttp it doesn't even start uploading.), but in XHTTP(splitHTTP), we may have to wait (even in h3 alpn mode or even in h2 mode!) for nearly 2 minutes (Maybe or maybe not) until only 128kb of that file is uploaded!.

Reproduction Method

I really have no idea how to fix this problem and I don't know how to coding, but I was eager to help.

Client config


"inbounds": [
    {
      "listen": "127.0.0.1",
      "port": 10808,
      "protocol": "socks",
      "settings": {
        "udp": true,
        "auth": "noauth"
      },
      "sniffing": {
        "destOverride": [
          "http",
          "tls"
        ],
        "enabled": true,
        "routeOnly": true
      },
      "tag": "socks"
    },
    {
      "listen": "127.0.0.1",
      "port": 10809,
      "protocol": "http",
      "tag": "http"
    }
  ],
  "outbounds": [
    {
      "tag": "proxy",
      "protocol": "vless",
      "settings": {
        "vnext": [
          {
            "address": "CF-IP",
            "port": 443,
            "users": [
              {
                "id": "xxxxxx-xxxx-xxxx-xxxx-xxxxxx",
                "security": "auto",
                "encryption": "none",
                "email": "xxxxxx",
                "alterId": 0,
                "flow": null
              }
            ]
          }
        ]
      },
      "streamSettings": {
        "network": "splithttp",
        "security": "tls",
        "tlsSettings": {
          "serverName": "sub.mycfdomain.com",
          "allowInsecure": false,
          "fingerprint": "chrome",
          "alpn": [
            "h3"
          ],
          "show": false
        },
        "splithttpSettings": {
          "mode": "auto",
          "xmux": {
            "maxConcurrency": 0,
            "maxConnections": 0,
            "cMaxReuseTimes": 0,
            "cMaxLifetimeMs": 60000
          },
          "headers": {
            "Pragma": "no-cache"
          },
          "path": "/",
          "host": "sub.mycfdomain.com",
          "scMaxEachPostBytes": 1000000,
          "scMaxConcurrentPosts": 100,
          "scMinPostsIntervalMs": 30,
          "xPaddingBytes": "100-200"
        },
        "sockopt": {
          "dialerProxy": "dialer"
        }
      }
    },

Server config


{
  "tag": "VLESS + SPLITHTTP + TLS",
  "listen": "0.0.0.0",
  "port": 443,
  "protocol": "vless",
  "settings": {
      "clients": [],
      "decryption": "none"
    },
  "streamSettings": {
    "network": "splithttp",
    "splithttpSettings": {
      "scMaxEachPostBytes": 1000000,
      "scMaxConcurrentPosts": 100,
      "path": "/",
      "xPaddingBytes": "100-200",
      "mode": "auto"
 },
    "security": "tls",
    "tlsSettings": {
      "serverName": "sub.mycfdomain.com",
      "certificates": [
        {
          "ocspStapling": 3600,
          "certificateFile": "/root/cert/cert.crt",
          "keyFile": "/root/cert/private.key"
        }
      ],
      "minVersion": "1.3"
    }
  },
  "sniffing": {
    "enabled": true,
    "destOverride": ["http", "tls"]
  }
}

Client log

No Errors but, when using h3 mode v2rayN show :
connection doesn't allow setting of send buffer size. Not a *net.UDPConn?. See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details.

Server log

No Errors.

The text was updated successfully, but these errors were encountered:

RPRX · 2024-12-03T02:06:38Z

我也觉得这个问题仍存在，上次 #3972 (comment) 只是修好了一部分

你可以换用 XHTTP stream-up H2 over CDN 作为临时解决方案，如果你需要 H3 下行，可以设置上下行分离

deepsm0ke · 2024-12-05T17:51:57Z

我也觉得这个问题仍存在，上次 #3972 (comment) 只是修好了一部分

你可以换用 XHTTP stream-up H2 over CDN 作为临时解决方案，如果你需要 H3 下行，可以设置上下行分离

Does packet-up mode parallelize HTTP packets? If the answer is no, perhaps parallelizing HTTP packets can solve this problem. I said before that I have no understanding of coding, but the things I saw and heard were as follows:

Maybe you can leverage goroutines, which are lightweight concurrent threads, along with channels to manage the flow of data. by using goroutines, you can handle multiple upload tasks concurrently, Then, Synchronize using channels function:we ensure that all goroutines complete before finishing.

dragonbreath2000 · 2024-12-06T01:14:13Z

I did some tests to find the issue
this one i accidently found out,i use splithttp with 2 cdn,cf and some not famous one,when I used it with cf to do a speed test,the upload crashed and ray was not usable for maybe 10 seconds,the interesting thing is after the speedtest crash if i connect to my server through the other cdn,it still does not work for around 10 seconds,suggesting a server side problem
after that i tried to reproduce this locally but I could not,when i ran my splithttp client and server(tested with h1.1 and h2) both on my pc,there was no issue for upload
This is my theory:
I think the server side packet assembly for post request has issues,in my local test all packet were sending in order so there was no issue becuase of that,but with cdn,becuase of congestions( specially with adsl which is really bad) or weird behavior of cdn they will arrive out of order(maybe the assembly code gets surprised when there are too many post requests for each session?)
when doing normal web browsing the post requests are pretty small so they will not be sent in multiple posts that is why its fine but when doing any upload work,it goes crazy
if my theory is correct then for direct comunication it should not be a problem but it is so I could be very wrong(even if there is congestion,there should not be that many out of order requests)

RPRX · 2024-12-06T04:01:28Z

@deepsm0ke XHTTP packet-up 的 POST 请求就是并发的，不等上一个请求收到响应就会发下一个

@dragonbreath2000 可能是服务端重组乱序包的问题或者乱序包太多给服务端整 OOM 了，但直连不应乱序，也可能是中间哪个包丢了没能自动重传，但直连不应丢包，所以的确可能要去其它方面找问题

另外 packet-up 模式默认的 1MB 加最大 100 并发导致上行缓存是 100MB，有些测速会先直接起飞再遇到 back pressure

RPRX · 2024-12-10T22:24:07Z

#4148 (comment) 差点忘了 Xray 的 quic-go 没升所以有可能是它的问题，你们换 packet-up H2 试试

RPRX · 2024-12-10T22:43:07Z

或者你们试试把 quic-go 升到最新版看能不能解决问题

dragonbreath2000 · 2024-12-10T23:10:28Z

I changed quic-go version in go.mod to 0.48.2 and buit ray,no diffrence,
tried h2 as well it was the same
It is really weird,when upload speedtest starts it first starts uploading for a second then it fully stops and fails,I have some theories about it that I will check later

deepsm0ke · 2024-12-10T23:27:37Z

I said it first. This problem exists on packet-up mode, whether without TLS or vice versa, i.e. with TLS on http1.1 or h2 or h3, even if there is no CDN in the middle and the connection is direct (client<->server). If the client's internet upload is higher than 1mbps, even if it does not fail sometimes, it will slowly start uploading in packet-up mode, but if it is less than or equal to 1mbps, it will not. This problem does not exist in stream-up mode and it will easily start uploading, no matter how slow it is, but eventually the upload will start and will not crash or stop.

RPRX · 2024-12-10T23:42:56Z

我觉得有可能是网速跟不上等待太久导致 HTTP 客户端自动重传了就一直堵着然后服务端也傻了，或者服务端缓存太多 OOM 了

你们试试把客户端 scMaxEachPostBytes 改为 100000 即 0.1MB，这样的话最大上传速度应该是 3MB/s

RPRX · 2024-12-10T23:47:39Z

3MB/s 还存在问题的话就试试 10000 即 0.3MB/s，1000 即 0.03MB/s，如果没问题了的话那应该和乱序包无关

RPRX · 2024-12-10T23:50:21Z

总之直觉是 back pressure 没到位的问题

RPRX · 2024-12-10T23:57:08Z

但我查了一下至少 Go 官方 HTTP 库的 POST 不会自动重传，GET 会，~~那有可能是 POST 包丢了的问题？~~

RPRX · 2024-12-11T00:01:48Z

或者可能后调用的 client.Do() 有更高的优先级，导致前面的包还没传完就先传后面的包，前面的包就一直没传到，尤其是小水管

dragonbreath2000 · 2024-12-11T00:02:11Z

But I checked and found that at least the Go official HTTP library does not automatically retransmit POST, but GET does.~~Could it be that the POST packet is lost?~~

some times ray shows me this in logs,is this expected?:
2024/12/11 03:03:35 [Info] [3122916268] transport/internet/splithttp: failed to send upload > Post "https://mydomain.com/2bdae5a3-6bba-42db-90c8-d66bff7b973a/29?x_padding=00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000": http2: Transport: cannot retry err [stream error: stream ID 19; PROTOCOL_ERROR; received from peer] after Request.Body was written; define Request.GetBody to avoid this error

RPRX · 2024-12-11T00:04:19Z

@dragonbreath2000 I'm not sure. Try #4100 (comment)

dragonbreath2000 · 2024-12-11T00:08:33Z

lowering scMaxEachPostBytes does improve it quite a bit ,does not get oomed right away and speedtest goes better,but sometimes it still gets stuck

RPRX · 2024-12-11T00:19:02Z

@dragonbreath2000 调到多少基本上没错误？此时的上行测速结果是？

dragonbreath2000 · 2024-12-11T01:00:40Z

scMaxEachPostBytes of 50000 or 100000 did not fail speedtest mostly,which is good enough with my crappy isp(only for h2,for h3 it did not even begin to upload)

deepsm0ke · 2024-12-11T01:02:20Z

If there is still a problem with 3MB/s, try 10000, which is 0.3MB/s, 1000, which is 0.03MB/s. If there is no problem, it should not be related to out-of-order packets.

When I set the scMaxEachPostBytes value to 1000, things got a lot better ( tested with h2 & h3 ), but sometimes it would get stuck.

RPRX · 2024-12-11T01:23:46Z

那应该就是 back pressure 没到位导致的各种问题，这个本来就应该等上个 POST 把数据写进系统网络栈后再弄下一个 POST，我看了下 Go 的 HTTP 库的 trace 有个 WroteRequest 应该能用，如果 quic-go 也有这个 trace 的话

deepsm0ke · 2024-12-11T01:44:05Z

That should be the various problems caused by the lack of back pressure. This should wait until the previous POST writes the data into the system network stack before making the next POST. I looked at the trace of the Go HTTP library and found a WroteRequest that should be able to be used. If quic-go also has this trace,

Now I tested with a different configuration: I set the scMaxEachPostBytes value to 1000000 as before, but I set the scMinPostsIntervalMs value to 100 (client), and the result was that it worked perfectly (only on h2. On h3 it didn't even start uploading), But it still sometimes gets stuck and won't even start.

RPRX · 2024-12-11T03:31:08Z

#4150 我用了 WroteRequest 以确保“等上个 POST 把数据写进系统网络栈后再弄下一个 POST”，不过 quic-go 貌似也不支持这个 tracepoint，你们试试 h2：https://github.com/XTLS/Xray-core/actions/runs/12269113619

RPRX · 2024-12-11T04:12:48Z

显然今天的版本等不上 #4150 ，我在考虑把 scMaxEachPostBytes 的客户端默认值调到 300000（10MB/s）左右，可能调更小

* Add wroteRequest (waiting for new quic-go) * Use XTLS/quic-go instead * Client doesn't need `scMaxConcurrentPosts` anymore * GotConn is available in H3 * `scMaxConcurrentPosts` -> `scMaxBufferedPosts` (server only, 30 by default) Fixes XTLS#4100

RPRX changed the title ~~Upload start failing on internets with 1mbps upload ( Like ADSL ) with XHTTP over CDN.~~ Upload start failing on internets with 1mbps upload ( Like ADSL ) with XHTTP packet-up over CDN. Dec 3, 2024

RPRX added the bug Something isn't working label Dec 3, 2024

deepsm0ke changed the title ~~Upload start failing on internets with 1mbps upload ( Like ADSL ) with XHTTP packet-up over CDN.~~ Upload start failing on internets with 1mbps upload ( Like ADSL ) with XHTTP packet-up over CDN or/and DIRECT. Dec 5, 2024

RPRX mentioned this issue Dec 11, 2024

XHTTP client: Refactor "packet-up" mode, chasing "stream-up" #4150

Merged

RPRX closed this as completed in 8cd9a74 Dec 11, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upload start failing on internets with 1mbps upload ( Like ADSL ) with XHTTP packet-up over CDN or/and DIRECT. #4100

Upload start failing on internets with 1mbps upload ( Like ADSL ) with XHTTP packet-up over CDN or/and DIRECT. #4100

deepsm0ke commented Dec 2, 2024 •

edited

Loading

RPRX commented Dec 3, 2024

deepsm0ke commented Dec 5, 2024

dragonbreath2000 commented Dec 6, 2024 •

edited

Loading

RPRX commented Dec 6, 2024

RPRX commented Dec 10, 2024

RPRX commented Dec 10, 2024

dragonbreath2000 commented Dec 10, 2024

deepsm0ke commented Dec 10, 2024

RPRX commented Dec 10, 2024 •

edited

Loading

RPRX commented Dec 10, 2024

RPRX commented Dec 10, 2024

RPRX commented Dec 10, 2024

RPRX commented Dec 11, 2024

dragonbreath2000 commented Dec 11, 2024

RPRX commented Dec 11, 2024

dragonbreath2000 commented Dec 11, 2024 •

edited

Loading

RPRX commented Dec 11, 2024

dragonbreath2000 commented Dec 11, 2024

deepsm0ke commented Dec 11, 2024 •

edited

Loading

RPRX commented Dec 11, 2024

deepsm0ke commented Dec 11, 2024 •

edited

Loading

RPRX commented Dec 11, 2024

RPRX commented Dec 11, 2024

Upload start failing on internets with 1mbps upload ( Like ADSL ) with XHTTP packet-up over CDN or/and DIRECT. #4100

Upload start failing on internets with 1mbps upload ( Like ADSL ) with XHTTP packet-up over CDN or/and DIRECT. #4100

Comments

deepsm0ke commented Dec 2, 2024 • edited Loading

Integrity requirements

Description

Reproduction Method

Client config

Server config

Client log

Server log

RPRX commented Dec 3, 2024

deepsm0ke commented Dec 5, 2024

dragonbreath2000 commented Dec 6, 2024 • edited Loading

RPRX commented Dec 6, 2024

RPRX commented Dec 10, 2024

RPRX commented Dec 10, 2024

dragonbreath2000 commented Dec 10, 2024

deepsm0ke commented Dec 10, 2024

RPRX commented Dec 10, 2024 • edited Loading

RPRX commented Dec 10, 2024

RPRX commented Dec 10, 2024

RPRX commented Dec 10, 2024

RPRX commented Dec 11, 2024

dragonbreath2000 commented Dec 11, 2024

RPRX commented Dec 11, 2024

dragonbreath2000 commented Dec 11, 2024 • edited Loading

RPRX commented Dec 11, 2024

dragonbreath2000 commented Dec 11, 2024

deepsm0ke commented Dec 11, 2024 • edited Loading

RPRX commented Dec 11, 2024

deepsm0ke commented Dec 11, 2024 • edited Loading

RPRX commented Dec 11, 2024

RPRX commented Dec 11, 2024

deepsm0ke commented Dec 2, 2024 •

edited

Loading

dragonbreath2000 commented Dec 6, 2024 •

edited

Loading

RPRX commented Dec 10, 2024 •

edited

Loading

dragonbreath2000 commented Dec 11, 2024 •

edited

Loading

deepsm0ke commented Dec 11, 2024 •

edited

Loading

deepsm0ke commented Dec 11, 2024 •

edited

Loading