Memory leak on "observed" node

[sakontwist]

I turned on “observer” on April 30 and memory leak appeared on the nodes being "observed". The nodes under "observing" are using a primitive configuration, no DNS with one direct outbound.
RSS xray graph (green):

What debugging information needs to be attached?

[Fangliding]

Client config and server config just as the template required. Why ignore it

[sakontwist]

I use split and complex routing (homeland via vps balancer) on the client, it will be difficult to give full conifguration, I will give part of the config on the problem:

outbounds for observed node

    {
      "protocol": "shadowsocks",
      "tag": "vps6",
      "settings": {
	    "servers": [
            {
                "address": "x.x.x.x","port": xxxx,
                "method": "2022-blake3-chacha20-poly1305",
                "password": "xxxxxxxxxx",
                "uot":true, "UoTVersion":2
              }
	    ]
      },
      "mux": {
          "enabled": true,
          "concurrency": 8,
          "xudpConcurrency": 0,
          "xudpProxyUDP443": "allow"
      }
    }

observer config

"observatory": {
    "subjectSelector": [
      "vps6",
      "vps5",
      "vps2"
    ],
    "probeURL": "https://www.instagram.com",
    "probeInterval": "30s",
    "enableConcurrency": true
  }

balancer

	"balancers": [
		{
			"tag": "ss-bl",
			"selector": ["vps6","vps5","vps2"],
			"strategy": {
				"type": "leastPing"
			},
			"fallbackTag": "vps6"
		}
	]  

"observed" nodes
inbound:

    {
      "tag":"in-ss",
      "listen": "x.x.x.x","port": xxx,
      "protocol": "shadowsocks",
      "settings": {
        "network":"tcp",
        "method": "2022-blake3-chacha20-poly1305",
        "password":"xxxxxxxxxx"
      },
      "sniffing": {
        "enabled": false,
        "destOverride":["http","tls","quic"],
        "routeOnly": true
      }
    }

routing:

  "routing": {
    "domainStrategy": "AsIs",
    "rules": [
      {
        "inboundTag": ["in_api"],
        "outboundTag": "api"
      },
      {
	"inboundTag": ["metrics-in"],
	"outboundTag": "metrics-out"
      }
    ]
  }

out:

  "outbounds": [
    {
      "tag": "direct",
      "protocol": "freedom",
      "settings": {
        "domainStrategy": "AsIs"
      }
    }
  ]

I assume polls need to be done less frequently than the timeout policy.... (using default values for policy).
I have to disable observer for now so that Xray doesn't crash, but I can reproduce if necessary.

[Fangliding]

Attempting to remove unnecessary options and routes to provide a complete configuration, it is difficult to determine the problem if it cannot be reproduced
And if the observer is turned off, will the problem still be exists？

[yuhan6665]

what if you change observatory to burstObservatory config?

[sakontwist]

1. Which of these do you consider “unnecessary options”?

-rw-r--r-- 1 root root  144 апр 27 09:27 00-log.json
-rw-r--r-- 1 root root  165 апр 27 00:26 05-api-stats-metrics.json
-rw-r--r-- 1 root root  672 апр 29 17:02 10-dns.json
-rw-r--r-- 1 root root 4467 мая  5 20:09 15-routing.json
-rw-r--r-- 1 root root  381 апр 27 00:26 19-in-api-metrics.json
-rw-r--r-- 1 root root  354 апр  7 17:47 20-policy.json
-rw-r--r-- 1 root root 1004 апр 27 09:27 23-in-lan.json
-rw-r--r-- 1 root root 1535 мая  5 22:11 25-in-spec.json
-rw-r--r-- 1 root root 1889 апр 24 12:45 35-in-ext.json
-rw-r--r-- 1 root root  254 апр 24 19:11 40-out-direct.json
-rw-r--r-- 1 root root 1033 апр 27 00:26 45-out-internal-tail.json
-rw-r--r-- 1 root root 2725 апр 24 21:41 50-out-ss-vps-tail.json
-rw-r--r-- 1 root root 4015 апр 25 01:08 60-out-rea-vps-tail.json

2. Turning off the observer completely eliminated the problem:
   

3. The documentation lags significantly behind the development, I did the "observer" customization by finding examples in the issues. I did not try burstObserver, because there is very little information on its configuration.

[sakontwist]

It seems the problem was that I have an entry "geosite:instagram" in the routing section:

  "routing": {
	"domainStrategy":"IPOnDemand",
	"rules": [
                ...
		{
			"inboundTag": ["in_tproxy","in_socks_lan","in_vless","in-ss"],
			"domain": [
				"geosite:instagram",
				"geosite:facebook",
                                ...
			],
			"balancerTag": "ss-bl"
		},

I have changed the ProbeURL to a different address that is not in the routing lists through balancer outbounds and the problem on the observed nodes is not reproduced.
It is not clear how "observer" requests through the routing and balancer is causing the problem on "observed" nodes.

[yuhan6665]

@sakontwist interesting, sounds like the root cause is you created an infinite loop.
Ideally we should fix it.

[Fangliding]

Even if it leads to a loop, it should still occur on the client side. Therefore, this observers requests is different from normal requests, leading to memory leaks on the server side (such as mux sessions that may not close properly)

[sakontwist]

I've done some experiments and the conclusions are as follows:

• using burstObserver does not fix memory problems
• the problem is probably caused by the mux.cool module.

I disabled mux.cool on XRAY - XRAY trunks for testing (it's not very convenient, because RKN detects servers also by the number of sessions) and the problem doesn't appear until clients connect. I know that mostly clients use nekobox and v2rayng.

I've been testing v2rayNG and noticed in the server logs that even when using vless (rprx), mux.cool calls come in if a device makes an udp request (e.g. dns). I managed to forbid mux only after enabling mux in the client by setting “-1” (I apologize for writing here and not to v2ray developers - it's for example).

So far I've recommended to clients not to use mux if possible, but apparently someone is using it because the problem comes back periodically.

I'll try to pick a window so there is no client activity and confirm there is no problem on a clean xray to xray connection.

[sakontwist]

Yesterday I ran a simple scheme with two Xray 1.8.11 45ab4cb (go1.22.2 linux/amd64), with no extraneous clients. After enabling mux, the server started eating memory.

On another similar trunk, the result was the same:

server

{
  "inbounds": [
    {
      "tag":"in-ss",
      "listen": "xx.xx.xx.xx","port": 10051,
      "protocol": "shadowsocks",
      "settings": {
        "network":"tcp",
        "method": "2022-blake3-chacha20-poly1305",
        "password":"xxxxxxxxxxxxxxxxxxxxxx"
      },
      "sniffing": {
        "enabled": false,
        "destOverride":["http","tls","quic"],
        "routeOnly": true
      }
    }
  ],
  "routing": {
    "domainStrategy": "AsIs",
    "rules": [
    ]
  },
  "outbounds": [
    {
      "tag": "direct",
      "protocol": "freedom",
      "settings": {
        "domainStrategy": "AsIs"
      }
    }
  ]
}

client

{
  "inbounds": [
    {
		"tag": "in-socks-vps",
		"listen": "192.168.88.2","port": 1094,
		"protocol": "socks",
		"settings": {
		  "udp": false
		}
	}
  ],
  "routing": {
	"domainStrategy":"IPOnDemand",
	"rules": [
		{
			"inboundTag": ["in-socks-vps"],
			"outboundTag": "ss-vps4"
		}
	]  
  },
  "outbounds": [ 
    {
      "protocol": "shadowsocks",
      "tag": "ss-vps4",
      "settings": {
        "servers": [
            {
                "address": "xx.xx.xx.xx","port": 10051,
                "method": "2022-blake3-chacha20-poly1305",
                "password": "xxxxxxxxxxxxxxxxxxxxxx",
                "uot": true,"UoTVersion": 2
              }
        ]
      },
      "mux": {
          "enabled": true,
          "concurrency": 8,
          "xudpConcurrency": 16,
          "xudpProxyUDP443": "allow"
      }
    }
  ]
}

[wrk6]

Hello everyone,

Let me start by saying that I have read all the articles and issues on similar topics, but I haven’t found a definitive and clear solution to this problem. Therefore, I decided to write here, hoping I’m not bothering developers or users too much.

The issues described in #3221 and #3338 do not provide a clear answer to the problem of memory leaks or OOM (Out of Memory). It’s also unclear whether these issues discuss the same problem or different ones.

Perhaps this can revive discussions on the topic. Thank you in advance!

Xray Core version - 24.11.11