[Docs] [istio mtls] Add warning on sidecar OOM for mTLS (#53385)

We just had a production outage at Roblox due to a large number of
headless services created for Ray overwhelming the service mesh sidecars
and causing ingress gateways to fail.

Another thing we observed is that mTLS slows down some jobs
significantly. So we ended up using istio with interception mode "none"
to only proxy the 8265 port to expose the head node securely, and leave
the head - worker grpc connections unencrypted. But I wasn't sure it's a
common enough issue to mention in the docs.

<!-- Thank you for your contribution! Please review
https://github.com/ray-project/ray/blob/master/CONTRIBUTING.rst before
opening a pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

---------

Signed-off-by: Steve Han <36038610+han-steve@users.noreply.github.com>
Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com>
Co-authored-by: Kai-Hsun Chen <kaihsun@anyscale.com>
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

Author: 4 people

doc/source/cluster/kubernetes/k8s-ecosystem/istio.md

@@ -165,7 +165,7 @@ Kubernetes Service doesn't support specifying ports in ranges. You _must_ set th
 :::
 
 :::{warning}
-The default Ray worker port range, from 10002 to 19999, is too large to specify in the service manifest and can cause memory issues in Kubernetes. Set a smaller `max-worker-port` to work with Istio.
+The default Ray worker port range, from 10002 to 19999, is too large to specify in the service manifest and can cause memory issues in Kubernetes. Set a smaller `max-worker-port` to work with Istio. Note that by default every sidecar in the service mesh caches these ports, which could lead to sidecar OOMs if you create too many headless services. 
 :::
 
 ## Step 4: Create the RayCluster