Track outbound connections #939

howardjohn · 2024-04-18T22:48:49Z

Refactor connection tracker
Track outbound connections as well

Ultimately this enables istioctl to give us:

WORKLOAD                                      DIRECTION LOCAL                                              REMOTE                                             REMOTE TARGET
cartservice-bffc98bf4-bmqsg.anthos            Inbound   cartservice-bffc98bf4-bmqsg.anthos:7070            checkoutservice-c58746cd5-prxg9.anthos:41903
cartservice-bffc98bf4-bmqsg.anthos            Inbound   cartservice-bffc98bf4-bmqsg.anthos:7070            frontend-86c4744d65-f6xwb.anthos:50955
cartservice-bffc98bf4-bmqsg.anthos            Outbound  cartservice-bffc98bf4-bmqsg.anthos:50320           redis-cart-7ff8f4d6ff-lmlwg.anthos:6379            redis-cart.anthos.svc.cluster.local:6379
cartservice-bffc98bf4-bmqsg.anthos            Outbound  cartservice-bffc98bf4-bmqsg.anthos:50330           redis-cart-7ff8f4d6ff-lmlwg.anthos:6379            redis-cart.anthos.svc.cluster.local:6379
productcatalogservice-64898b8c5f-pqx6h.anthos Inbound   productcatalogservice-64898b8c5f-pqx6h.anthos:3550 recommendationservice-85846457-79l2f.anthos:58205
productcatalogservice-64898b8c5f-pqx6h.anthos Inbound   productcatalogservice-64898b8c5f-pqx6h.anthos:3550 checkoutservice-c58746cd5-prxg9.anthos:46941
productcatalogservice-64898b8c5f-pqx6h.anthos Inbound   productcatalogservice-64898b8c5f-pqx6h.anthos:3550 frontend-86c4744d65-f6xwb.anthos:39685
frontend-86c4744d65-f6xwb.anthos              Inbound   frontend-86c4744d65-f6xwb.anthos:8080              loadgenerator-866ccfdfc9-9mhz7.anthos:48071
frontend-86c4744d65-f6xwb.anthos              Outbound  frontend-86c4744d65-f6xwb.anthos:44436             checkoutservice-c58746cd5-prxg9.anthos:5050        checkoutservice.anthos.svc.cluster.local:5050
frontend-86c4744d65-f6xwb.anthos              Outbound  frontend-86c4744d65-f6xwb.anthos:40386             cartservice-bffc98bf4-bmqsg.anthos:7070            cartservice.anthos.svc.cluster.local:7070
frontend-86c4744d65-f6xwb.anthos              Outbound  frontend-86c4744d65-f6xwb.anthos:50976             productcatalogservice-64898b8c5f-pqx6h.anthos:3550 productcatalogservice.anthos.svc.cluster.local:3550
frontend-86c4744d65-f6xwb.anthos              Outbound  frontend-86c4744d65-f6xwb.anthos:37700             recommendationservice-85846457-79l2f.anthos:8080   recommendationservice.anthos.svc.cluster.local:8080
frontend-86c4744d65-f6xwb.anthos              Outbound  frontend-86c4744d65-f6xwb.anthos:40528             adservice-684fbd8977-r6vsh.anthos:9555             adservice.anthos.svc.cluster.local:9555
frontend-86c4744d65-f6xwb.anthos              Outbound  frontend-86c4744d65-f6xwb.anthos:41618             shippingservice-77d6cdd487-v4fj6.anthos:50051      shippingservice.anthos.svc.cluster.local:50051
frontend-86c4744d65-f6xwb.anthos              Outbound  frontend-86c4744d65-f6xwb.anthos:48212             currencyservice-5495597f5-nwnd5.anthos:7000        currencyservice.anthos.svc.cluster.local:7000
emailservice-d74c4c797-xpkp2.anthos           Inbound   emailservice-d74c4c797-xpkp2.anthos:8080           checkoutservice-c58746cd5-prxg9.anthos:54561
recommendationservice-85846457-79l2f.anthos   Inbound   recommendationservice-85846457-79l2f.anthos:8080   frontend-86c4744d65-f6xwb.anthos:54843
recommendationservice-85846457-79l2f.anthos   Outbound  recommendationservice-85846457-79l2f.anthos:36164  productcatalogservice-64898b8c5f-pqx6h.anthos:3550 productcatalogservice.anthos.svc.cluster.local:3550
checkoutservice-c58746cd5-prxg9.anthos        Inbound   checkoutservice-c58746cd5-prxg9.anthos:5050        frontend-86c4744d65-f6xwb.anthos:40593
checkoutservice-c58746cd5-prxg9.anthos        Outbound  checkoutservice-c58746cd5-prxg9.anthos:39576       cartservice-bffc98bf4-bmqsg.anthos:7070            cartservice.anthos.svc.cluster.local:7070
checkoutservice-c58746cd5-prxg9.anthos        Outbound  checkoutservice-c58746cd5-prxg9.anthos:59008       emailservice-d74c4c797-xpkp2.anthos:8080           emailservice.anthos.svc.cluster.local:5000
checkoutservice-c58746cd5-prxg9.anthos        Outbound  checkoutservice-c58746cd5-prxg9.anthos:56736       currencyservice-5495597f5-nwnd5.anthos:7000        currencyservice.anthos.svc.cluster.local:7000
checkoutservice-c58746cd5-prxg9.anthos        Outbound  checkoutservice-c58746cd5-prxg9.anthos:57080       productcatalogservice-64898b8c5f-pqx6h.anthos:3550 productcatalogservice.anthos.svc.cluster.local:3550
checkoutservice-c58746cd5-prxg9.anthos        Outbound  checkoutservice-c58746cd5-prxg9.anthos:41766       paymentservice-5dd485cf9c-ks47b.anthos:50051       paymentservice.anthos.svc.cluster.local:50051
checkoutservice-c58746cd5-prxg9.anthos        Outbound  checkoutservice-c58746cd5-prxg9.anthos:39090       shippingservice-77d6cdd487-v4fj6.anthos:50051      shippingservice.anthos.svc.cluster.local:50051
redis-cart-7ff8f4d6ff-lmlwg.anthos            Inbound   redis-cart-7ff8f4d6ff-lmlwg.anthos:6379            cartservice-bffc98bf4-bmqsg.anthos:41363
currencyservice-5495597f5-nwnd5.anthos        Inbound   currencyservice-5495597f5-nwnd5.anthos:7000        frontend-86c4744d65-f6xwb.anthos:43913
currencyservice-5495597f5-nwnd5.anthos        Inbound   currencyservice-5495597f5-nwnd5.anthos:7000        checkoutservice-c58746cd5-prxg9.anthos:41455
adservice-684fbd8977-r6vsh.anthos             Inbound   adservice-684fbd8977-r6vsh.anthos:9555             frontend-86c4744d65-f6xwb.anthos:59253
loadgenerator-866ccfdfc9-9mhz7.anthos         Outbound  loadgenerator-866ccfdfc9-9mhz7.anthos:60636        frontend-86c4744d65-f6xwb.anthos:8080              frontend.anthos.svc.cluster.local:80
loadgenerator-866ccfdfc9-9mhz7.anthos         Outbound  loadgenerator-866ccfdfc9-9mhz7.anthos:60638        frontend-86c4744d65-f6xwb.anthos:8080              frontend.anthos.svc.cluster.local:80
loadgenerator-866ccfdfc9-9mhz7.anthos         Outbound  loadgenerator-866ccfdfc9-9mhz7.anthos:60586        frontend-86c4744d65-f6xwb.anthos:8080              frontend.anthos.svc.cluster.local:80
loadgenerator-866ccfdfc9-9mhz7.anthos         Outbound  loadgenerator-866ccfdfc9-9mhz7.anthos:40164        frontend-86c4744d65-f6xwb.anthos:8080              frontend.anthos.svc.cluster.local:80
loadgenerator-866ccfdfc9-9mhz7.anthos         Outbound  loadgenerator-866ccfdfc9-9mhz7.anthos:40178        frontend-86c4744d65-f6xwb.anthos:8080              frontend.anthos.svc.cluster.local:80
loadgenerator-866ccfdfc9-9mhz7.anthos         Outbound  loadgenerator-866ccfdfc9-9mhz7.anthos:60628        frontend-86c4744d65-f6xwb.anthos:8080              frontend.anthos.svc.cluster.local:80
loadgenerator-866ccfdfc9-9mhz7.anthos         Outbound  loadgenerator-866ccfdfc9-9mhz7.anthos:60594        frontend-86c4744d65-f6xwb.anthos:8080              frontend.anthos.svc.cluster.local:80
loadgenerator-866ccfdfc9-9mhz7.anthos         Outbound  loadgenerator-866ccfdfc9-9mhz7.anthos:60608        frontend-86c4744d65-f6xwb.anthos:8080              frontend.anthos.svc.cluster.local:80
loadgenerator-866ccfdfc9-9mhz7.anthos         Outbound  loadgenerator-866ccfdfc9-9mhz7.anthos:60612        frontend-86c4744d65-f6xwb.anthos:8080              frontend.anthos.svc.cluster.local:80
loadgenerator-866ccfdfc9-9mhz7.anthos         Outbound  loadgenerator-866ccfdfc9-9mhz7.anthos:60630        frontend-86c4744d65-f6xwb.anthos:8080              frontend.anthos.svc.cluster.local:80
paymentservice-5dd485cf9c-ks47b.anthos        Inbound   paymentservice-5dd485cf9c-ks47b.anthos:50051       checkoutservice-c58746cd5-prxg9.anthos:41735
shippingservice-77d6cdd487-v4fj6.anthos       Inbound   shippingservice-77d6cdd487-v4fj6.anthos:50051      checkoutservice-c58746cd5-prxg9.anthos:35323
shippingservice-77d6cdd487-v4fj6.anthos       Inbound   shippingservice-77d6cdd487-v4fj6.anthos:50051      frontend-86c4744d65-f6xwb.anthos:51271

Or waypoints:

WORKLOAD                       DIRECTION LOCAL                                REMOTE                                  REMOTE TARGET
echo-66d88ff694-tf8bm.default  Inbound   echo-66d88ff694-tf8bm.default:80     waypoint-654dccf598-229fx.default:50712
shell-56bd5dbdbf-24v42.default Outbound  shell-56bd5dbdbf-24v42.default:46034 waypoint-654dccf598-229fx.default:15008 echo.default.svc.cluster.local:80

istio-testing · 2024-04-18T22:48:52Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

ilrudie

This looks really nice. Hides the gritty details of handling the late rbac checks and makes it less fiddly to use.

Unit tests can probably be added for the new code. Maybe that's a follow up.

ilrudie · 2024-04-20T03:05:57Z

src/proxy/connection_manager.rs

+        mut self,
+        send: impl Future<Output = Result<(u64, u64), Error>> + Sized,
+    ) -> Result<(u64, u64), Error> {
+        let watch = self.watch.take().expect("watch cannot be taken twice");


comment would be nice, it's a little subtle why this expect is OK I think.
"watch can't be taken twice because this ConnectionGuard is consumed by handle_connection" maybe

Shouldn't we error instead of crashing here? Maybe debug_assert the err path.

I think because the guard is owned by handle_connection and not returned it shouldn't ever be called twice. It's dropped once this returns.

stevenctl · 2024-04-22T07:01:49Z

src/proxy/outbound.rs

@@ -226,6 +226,11 @@ impl OutboundConnection {
            return;
        }
        let connection_metrics = Self::conn_metrics_from_request(&req);
+        // TODO: should we use the original address or the actual address? Both seems nice!
+        let _conn_guard =


I think definitely both.

stevenctl

LGTM

howardjohn · 2024-04-22T15:07:27Z

/cherrypick release-1.22

istio-testing · 2024-04-22T15:08:05Z

@howardjohn: new pull request created: #948

In response to this:

/cherrypick release-1.22

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

howardjohn added 2 commits April 18, 2024 15:26

Refactor connection tracker

dd8a1b1

Track outbound connections as well

5690f0d

howardjohn added the release-notes-none Indicates a PR that does not require release notes. label Apr 18, 2024

istio-testing added the do-not-merge/work-in-progress Block merging of a PR because it isn't ready yet. label Apr 18, 2024

istio-testing added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Apr 18, 2024

howardjohn changed the title ~~conntrack/outbound~~ Track outbound connections Apr 18, 2024

howardjohn marked this pull request as ready for review April 18, 2024 22:52

howardjohn requested a review from a team as a code owner April 18, 2024 22:52

istio-testing removed the do-not-merge/work-in-progress Block merging of a PR because it isn't ready yet. label Apr 18, 2024

howardjohn requested a review from ilrudie April 18, 2024 22:52

fix test

c680896

This was referenced Apr 18, 2024

istioctl ztunnel: add connections command istio/istio#50541

Merged

Pass original service over HBONE for debugging #942

Merged

ilrudie approved these changes Apr 20, 2024

View reviewed changes

ilrudie mentioned this pull request Apr 20, 2024

iterate on close connections which violate policy after updates #798

Open

3 tasks

stevenctl reviewed Apr 22, 2024

View reviewed changes

stevenctl approved these changes Apr 22, 2024

View reviewed changes

istio-testing merged commit 509bdc1 into istio:master Apr 22, 2024
3 checks passed

istio-testing mentioned this pull request Apr 22, 2024

[release-1.22] Track outbound connections #948

Merged

howardjohn mentioned this pull request May 1, 2024

h2/ready #996

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Track outbound connections #939

Track outbound connections #939

howardjohn commented Apr 18, 2024 •

edited

Loading

istio-testing commented Apr 18, 2024

ilrudie left a comment

ilrudie Apr 20, 2024

stevenctl Apr 22, 2024 •

edited

Loading

ilrudie Apr 22, 2024

stevenctl Apr 22, 2024

stevenctl left a comment

howardjohn commented Apr 22, 2024

istio-testing commented Apr 22, 2024

Track outbound connections #939

Track outbound connections #939

Conversation

howardjohn commented Apr 18, 2024 • edited Loading

istio-testing commented Apr 18, 2024

ilrudie left a comment

Choose a reason for hiding this comment

ilrudie Apr 20, 2024

Choose a reason for hiding this comment

stevenctl Apr 22, 2024 • edited Loading

Choose a reason for hiding this comment

ilrudie Apr 22, 2024

Choose a reason for hiding this comment

stevenctl Apr 22, 2024

Choose a reason for hiding this comment

stevenctl left a comment

Choose a reason for hiding this comment

howardjohn commented Apr 22, 2024

istio-testing commented Apr 22, 2024

howardjohn commented Apr 18, 2024 •

edited

Loading

stevenctl Apr 22, 2024 •

edited

Loading