Skip to content

Add 14 metrics for TCP connections created for tunnels. #9403

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ywkaras merged 5 commits into from
Oct 17, 2023
Merged

Add 14 metrics for TCP connections created for tunnels. #9403

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
e680550
Add 14 metrics for TCP connections created for tunnels.
ywkaras Feb 10, 2023
fc95473
VConnection::make_tunnel_endpoint() -> mark_as_tunnel_endpoint().
ywkaras Jun 27, 2023
a7379ba
Fix error in proxy.process.http.connect_requests.
ywkaras Jun 27, 2023
c7ef93a
Rebase changes for PR 9869.
ywkaras Aug 17, 2023
70239b5
Rebase changes.
ywkaras Oct 14, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 24 additions & 1 deletion doc/admin-guide/monitoring/statistics/core/http-connection.en.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -164,10 +164,33 @@ HTTP Connection

Counts the number of times current parent or next parent was detected

.. ts:stat:: global proxy.process.tunnel.total_client_connections_blind_tcp integer
:type: counter

Total number of non-TLS TCP connections for tunnels where the far end is the client
initiated with an HTTP request (such as a CONNECT or WebSocket request).

.. ts:stat:: global proxy.process.tunnel.current_client_connections_blind_tcp integer
:type: counter

Current number of non-TLS TCP connections for tunnels where the far end is the client
initiated with an HTTP request (such as a CONNECT or WebSocket request).

.. ts:stat:: global proxy.process.tunnel.total_server_connections_blind_tcp integer
:type: counter

Total number of TCP connections for tunnels where the far end is the server,
except for those counted by ``proxy.process.tunnel.total_server_connections_tls``

.. ts:stat:: global proxy.process.tunnel.current_server_connections_blind_tcp integer
:type: counter

Current number of TCP connections for tunnels where the far end is the server,
except for those counted by ``proxy.process.tunnel.current_server_connections_tls``

HTTP/2
------


.. ts:stat:: global proxy.process.http2.total_client_connections integer
:type: counter

Expand Down
60 changes: 60 additions & 0 deletions doc/admin-guide/monitoring/statistics/core/ssl.en.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -238,6 +238,66 @@ SSL/TLS

A gauge of current active SNI Routing Tunnels.

.. ts:stat:: global proxy.process.tunnel.total_client_connections_tls_tunnel integer
:type: counter

Total number of TCP connections for TLS tunnels where the far end is the client
created based on a ``tunnel_route`` key in a table in the :file:`sni.yaml` file.

.. ts:stat:: global proxy.process.tunnel.current_client_connections_tls_tunnel integer
:type: counter

Current number of TCP connections for TLS tunnels where the far end is the client
created based on a ``tunnel_route`` key in a table in the :file:`sni.yaml` file.

.. ts:stat:: global proxy.process.tunnel.total_client_connections_tls_forward integer
:type: counter

Total number of TCP connections for TLS tunnels where the far end is the client
created based on a ``forward_route`` key in a table in the :file:`sni.yaml` file.

.. ts:stat:: global proxy.process.tunnel.current_client_connections_tls_forward integer
:type: counter

Current number of TCP connections for TLS tunnels where the far end is the client
created based on a ``forward_route`` key in a table in the :file:`sni.yaml` file.

.. ts:stat:: global proxy.process.tunnel.total_client_connections_tls_partial_blind integer
:type: counter

Total number of TCP connections for TLS tunnels where the far end is the client
created based on a ``partial_blind_route`` key in a table in the :file:`sni.yaml` file.

.. ts:stat:: global proxy.process.tunnel.current_client_connections_tls_partial_blind integer
:type: counter

Current number of TCP connections for TLS tunnels where the far end is the client
created based on a ``partial_blind_route`` key in a table in the :file:`sni.yaml` file.

.. ts:stat:: global proxy.process.tunnel.total_client_connections_tls_http integer
:type: counter

Total number of TLS connections for tunnels where the far end is the client
initiated with an HTTP request.
Copy link
Contributor

@masaori335 masaori335 Jun 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This covers "HTTP CONNECT" over TLS connection?

Copy link
Contributor Author

@ywkaras ywkaras Jun 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes

Copy link
Member

@maskit maskit Jun 29, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does that include QUIC connection once we support it?

Copy link
Contributor Author

@ywkaras ywkaras Jul 3, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not currently. QUICNetVConnection would have to override _in_context_tunnel(). It could use the same metrics or different metrics.


.. ts:stat:: global proxy.process.tunnel.current_client_connections_tls_http integer
:type: counter

Current number of TLS connections for tunnels where the far end is the client
initiated with an HTTP request.

.. ts:stat:: global proxy.process.tunnel.total_server_connections_tls integer
:type: counter

Total number of TCP connections for TLS tunnels where the far end is the server
created based on a ``partial_blind_route`` key in a table in the :file:`sni.yaml` file.
Copy link
Contributor

@masaori335 masaori335 Jun 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need similar server-side metrics for forward_route?

Copy link
Contributor Author

@ywkaras ywkaras Jul 3, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A partial blind tunnel is the only one for which the outgoing connection corresponds to an SSLNetVConnection object rather than a UnixNetVConnection object. It's not straight forward how to implement separate counters for outgoing/server connections for the other TLS tunnel types, but it may be possible if they're needed.


.. ts:stat:: global proxy.process.tunnel.current_server_connections_tls integer
:type: counter

Current number of TCP connections for TLS tunnels where the far end is the server
created based on a ``partial_blind_route`` key in a table in the :file:`sni.yaml` file.

.. _pre-warming-tls-tunnel-stats:

Pre-warming TLS Tunnel
Expand Down
8 changes: 7 additions & 1 deletion iocore/eventsystem/I_VConnection.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -360,7 +360,13 @@ class VConnection : public Continuation
return false;
}

public:
// This function should be called when the VConnection is a tunnel endpoint. By default, a VConnection does not care if it
// is a tunnel endpoint.
virtual void
mark_as_tunnel_endpoint()
Copy link
Contributor

@masaori335 masaori335 Jun 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need this virtual function in the VConnection class? As you commented, VConnection doesn't care about it's tunnel or something else.

Copy link
Contributor Author

@ywkaras ywkaras Jun 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because it's called through a VConnection pointer at certain places.

Copy link
Contributor

@masaori335 masaori335 Jun 29, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one?

HttpSM::setup_blind_tunnel(bool send_response_hdr, IOBufferReader *initial) {
...
   ua_entry->vc->mark_as_tunnel_endpoint();
   server_entry->vc->mark_as_tunnel_endpoint();

Can we use ua_txn->get_netvc() and server_txn->get_netvc() instead of ua_entry->vc and server_entry->vc here? If possible, I prefer NetVConnection to have this virtual function.

Copy link
Contributor Author

@ywkaras ywkaras Jul 3, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK I just tried that. server_txn is null at that point in the code, so it doesn't work.

Copy link
Contributor

@masaori335 masaori335 Oct 17, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair enough, thanks for checking.

{
}

/**
The error code from the last error.

Expand Down
77 changes: 50 additions & 27 deletions iocore/net/Net.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,33 +82,56 @@ register_net_stats()
{
ts::Metrics &intm = ts::Metrics::getInstance();

net_rsb.accepts_currently_open = intm.newMetricPtr("proxy.process.net.accepts_currently_open");
net_rsb.calls_to_read = intm.newMetricPtr("proxy.process.net.calls_to_read");
net_rsb.calls_to_read_nodata = intm.newMetricPtr("proxy.process.net.calls_to_read_nodata");
net_rsb.calls_to_readfromnet = intm.newMetricPtr("proxy.process.net.calls_to_readfromnet");
net_rsb.calls_to_write = intm.newMetricPtr("proxy.process.net.calls_to_write");
net_rsb.calls_to_write_nodata = intm.newMetricPtr("proxy.process.net.calls_to_write_nodata");
net_rsb.calls_to_writetonet = intm.newMetricPtr("proxy.process.net.calls_to_writetonet");
net_rsb.connections_currently_open = intm.newMetricPtr("proxy.process.net.connections_currently_open");
net_rsb.connections_throttled_in = intm.newMetricPtr("proxy.process.net.connections_throttled_in");
net_rsb.connections_throttled_out = intm.newMetricPtr("proxy.process.net.connections_throttled_out");
net_rsb.default_inactivity_timeout_applied = intm.newMetricPtr("proxy.process.net.default_inactivity_timeout_applied");
net_rsb.default_inactivity_timeout_count = intm.newMetricPtr("proxy.process.net.default_inactivity_timeout_count");
net_rsb.fastopen_attempts = intm.newMetricPtr("proxy.process.net.fastopen_out.attempts");
net_rsb.fastopen_successes = intm.newMetricPtr("proxy.process.net.fastopen_out.successes");
net_rsb.handler_run = intm.newMetricPtr("proxy.process.net.net_handler_run");
net_rsb.inactivity_cop_lock_acquire_failure = intm.newMetricPtr("proxy.process.net.inactivity_cop_lock_acquire_failure");
net_rsb.keep_alive_queue_timeout_count = intm.newMetricPtr("proxy.process.net.dynamic_keep_alive_timeout_in_count");
net_rsb.keep_alive_queue_timeout_total = intm.newMetricPtr("proxy.process.net.dynamic_keep_alive_timeout_in_total");
net_rsb.read_bytes = intm.newMetricPtr("proxy.process.net.read_bytes");
net_rsb.read_bytes_count = intm.newMetricPtr("proxy.process.net.read_bytes_count");
net_rsb.requests_max_throttled_in = intm.newMetricPtr("proxy.process.net.max.requests_throttled_in");
net_rsb.socks_connections_currently_open = intm.newMetricPtr("proxy.process.socks.connections_currently_open");
net_rsb.socks_connections_successful = intm.newMetricPtr("proxy.process.socks.connections_successful");
net_rsb.socks_connections_unsuccessful = intm.newMetricPtr("proxy.process.socks.connections_unsuccessful");
net_rsb.tcp_accept = intm.newMetricPtr("proxy.process.tcp.total_accepts");
net_rsb.write_bytes = intm.newMetricPtr("proxy.process.net.write_bytes");
net_rsb.write_bytes_count = intm.newMetricPtr("proxy.process.net.write_bytes_count");
net_rsb.accepts_currently_open = intm.newMetricPtr("proxy.process.net.accepts_currently_open");
net_rsb.calls_to_read = intm.newMetricPtr("proxy.process.net.calls_to_read");
net_rsb.calls_to_read_nodata = intm.newMetricPtr("proxy.process.net.calls_to_read_nodata");
net_rsb.calls_to_readfromnet = intm.newMetricPtr("proxy.process.net.calls_to_readfromnet");
net_rsb.calls_to_write = intm.newMetricPtr("proxy.process.net.calls_to_write");
net_rsb.calls_to_write_nodata = intm.newMetricPtr("proxy.process.net.calls_to_write_nodata");
net_rsb.calls_to_writetonet = intm.newMetricPtr("proxy.process.net.calls_to_writetonet");
net_rsb.connections_currently_open = intm.newMetricPtr("proxy.process.net.connections_currently_open");
net_rsb.connections_throttled_in = intm.newMetricPtr("proxy.process.net.connections_throttled_in");
net_rsb.connections_throttled_out = intm.newMetricPtr("proxy.process.net.connections_throttled_out");
net_rsb.tunnel_total_client_connections_blind_tcp = intm.newMetricPtr("proxy.process.tunnel.total_client_connections_blind_tcp");
net_rsb.tunnel_current_client_connections_blind_tcp =
intm.newMetricPtr("proxy.process.tunnel.current_client_connections_blind_tcp");
net_rsb.tunnel_total_server_connections_blind_tcp = intm.newMetricPtr("proxy.process.tunnel.total_server_connections_blind_tcp");
net_rsb.tunnel_current_server_connections_blind_tcp =
intm.newMetricPtr("proxy.process.tunnel.current_server_connections_blind_tcp");
net_rsb.tunnel_total_client_connections_tls_tunnel =
intm.newMetricPtr("proxy.process.tunnel.total_client_connections_tls_tunnel");
net_rsb.tunnel_current_client_connections_tls_tunnel =
intm.newMetricPtr("proxy.process.tunnel.current_client_connections_tls_tunnel");
net_rsb.tunnel_total_client_connections_tls_forward =
intm.newMetricPtr("proxy.process.tunnel.total_client_connections_tls_forward");
net_rsb.tunnel_current_client_connections_tls_forward =
intm.newMetricPtr("proxy.process.tunnel.current_client_connections_tls_forward");
net_rsb.tunnel_total_client_connections_tls_partial_blind =
intm.newMetricPtr("proxy.process.tunnel.total_client_connections_tls_partial_blind");
net_rsb.tunnel_current_client_connections_tls_partial_blind =
intm.newMetricPtr("proxy.process.tunnel.current_client_connections_tls_partial_blind");
net_rsb.tunnel_total_client_connections_tls_http = intm.newMetricPtr("proxy.process.tunnel.total_client_connections_tls_http");
net_rsb.tunnel_current_client_connections_tls_http =
intm.newMetricPtr("proxy.process.tunnel.current_client_connections_tls_http");
net_rsb.tunnel_total_server_connections_tls = intm.newMetricPtr("proxy.process.tunnel.total_server_connections_tls");
net_rsb.tunnel_current_server_connections_tls = intm.newMetricPtr("proxy.process.tunnel.current_server_connections_tls");
net_rsb.default_inactivity_timeout_applied = intm.newMetricPtr("proxy.process.net.default_inactivity_timeout_applied");
net_rsb.default_inactivity_timeout_count = intm.newMetricPtr("proxy.process.net.default_inactivity_timeout_count");
net_rsb.fastopen_attempts = intm.newMetricPtr("proxy.process.net.fastopen_out.attempts");
net_rsb.fastopen_successes = intm.newMetricPtr("proxy.process.net.fastopen_out.successes");
net_rsb.handler_run = intm.newMetricPtr("proxy.process.net.net_handler_run");
net_rsb.inactivity_cop_lock_acquire_failure = intm.newMetricPtr("proxy.process.net.inactivity_cop_lock_acquire_failure");
net_rsb.keep_alive_queue_timeout_count = intm.newMetricPtr("proxy.process.net.dynamic_keep_alive_timeout_in_count");
net_rsb.keep_alive_queue_timeout_total = intm.newMetricPtr("proxy.process.net.dynamic_keep_alive_timeout_in_total");
net_rsb.read_bytes = intm.newMetricPtr("proxy.process.net.read_bytes");
net_rsb.read_bytes_count = intm.newMetricPtr("proxy.process.net.read_bytes_count");
net_rsb.requests_max_throttled_in = intm.newMetricPtr("proxy.process.net.max.requests_throttled_in");
net_rsb.socks_connections_currently_open = intm.newMetricPtr("proxy.process.socks.connections_currently_open");
net_rsb.socks_connections_successful = intm.newMetricPtr("proxy.process.socks.connections_successful");
net_rsb.socks_connections_unsuccessful = intm.newMetricPtr("proxy.process.socks.connections_unsuccessful");
net_rsb.tcp_accept = intm.newMetricPtr("proxy.process.tcp.total_accepts");
net_rsb.write_bytes = intm.newMetricPtr("proxy.process.net.write_bytes");
net_rsb.write_bytes_count = intm.newMetricPtr("proxy.process.net.write_bytes_count");
}

void
Expand Down
14 changes: 14 additions & 0 deletions iocore/net/P_Net.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,20 @@ struct NetStatsBlock {
Metrics::IntType *read_bytes;
Metrics::IntType *read_bytes_count;
Metrics::IntType *requests_max_throttled_in;
Metrics::IntType *tunnel_total_client_connections_blind_tcp;
Metrics::IntType *tunnel_current_client_connections_blind_tcp;
Metrics::IntType *tunnel_total_server_connections_blind_tcp;
Metrics::IntType *tunnel_current_server_connections_blind_tcp;
Metrics::IntType *tunnel_total_client_connections_tls_tunnel;
Metrics::IntType *tunnel_current_client_connections_tls_tunnel;
Metrics::IntType *tunnel_total_server_connections_tls;
Metrics::IntType *tunnel_current_server_connections_tls;
Metrics::IntType *tunnel_total_client_connections_tls_forward;
Metrics::IntType *tunnel_current_client_connections_tls_forward;
Metrics::IntType *tunnel_total_client_connections_tls_partial_blind;
Metrics::IntType *tunnel_current_client_connections_tls_partial_blind;
Metrics::IntType *tunnel_total_client_connections_tls_http;
Metrics::IntType *tunnel_current_client_connections_tls_http;
Metrics::IntType *socks_connections_currently_open;
Metrics::IntType *socks_connections_successful;
Metrics::IntType *socks_connections_unsuccessful;
Expand Down
3 changes: 3 additions & 0 deletions iocore/net/P_SSLNetVConnection.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -500,6 +500,9 @@ class SSLNetVConnection : public UnixNetVConnection,
ssl_error_t _ssl_write_buffer(const void *buf, int64_t nbytes, int64_t &nwritten);
ssl_error_t _ssl_connect();
ssl_error_t _ssl_accept();

void _in_context_tunnel() override;
void _out_context_tunnel() override;
};

typedef int (SSLNetVConnection::*SSLNetVConnHandler)(int, void *);
Expand Down
19 changes: 19 additions & 0 deletions iocore/net/P_UnixNetVConnection.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,8 @@ struct PollDescriptor;

enum tcp_congestion_control_t { CLIENT_SIDE, SERVER_SIDE };

// WARNING: many or most of the member functions of UnixNetVConnection should only be used when it is instantiated
// directly. They should not be used when UnixNetVConnection is a base class.
class UnixNetVConnection : public NetVConnection, public NetEvent
{
public:
Expand Down Expand Up @@ -225,10 +227,27 @@ class UnixNetVConnection : public NetVConnection, public NetEvent

friend void write_to_net_io(NetHandler *, UnixNetVConnection *, EThread *);

// set_context() should be called before calling this member function.
void mark_as_tunnel_endpoint() override;

bool
is_tunnel_endpoint() const
{
return _is_tunnel_endpoint;
}

private:
virtual void *_prepareForMigration();
virtual NetProcessor *_getNetProcessor();

bool _is_tunnel_endpoint{false};

// Called by make_tunnel_endpiont() when the far end of the TCP connection is the active/client end.
virtual void _in_context_tunnel();

// Called by make_tunnel_endpiont() when the far end of the TCP connection is the passive/server end.
virtual void _out_context_tunnel();

inline static DbgCtl _dbg_ctl_socket{"socket"};
inline static DbgCtl _dbg_ctl_socket_mptcp{"socket_mptcp"};
};
Expand Down
64 changes: 64 additions & 0 deletions iocore/net/SSLNetVConnection.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1037,6 +1037,29 @@ SSLNetVConnection::free_thread(EThread *t)
}
con.close();

if (is_tunnel_endpoint()) {
ink_assert(get_context() != NET_VCONNECTION_UNSET);

Metrics::decrement(([&]() -> Metrics::IntType * {
if (get_context() == NET_VCONNECTION_IN) {
switch (get_tunnel_type()) {
case SNIRoutingType::BLIND:
return net_rsb.tunnel_current_client_connections_tls_tunnel;
case SNIRoutingType::FORWARD:
return net_rsb.tunnel_current_client_connections_tls_forward;
case SNIRoutingType::PARTIAL_BLIND:
return net_rsb.tunnel_current_client_connections_tls_partial_blind;
default:
return net_rsb.tunnel_current_client_connections_tls_http;
}
}
// NET_VCONNECTION_OUT - Never a tunnel type for out (to server) context.
ink_assert(get_tunnel_type() == SNIRoutingType::NONE);

return net_rsb.tunnel_current_server_connections_tls;
})());
}

#if TS_HAS_TLS_EARLY_DATA
if (_early_data_reader != nullptr) {
_early_data_reader->dealloc();
Expand Down Expand Up @@ -1960,6 +1983,47 @@ SSLNetVConnection::populate(Connection &con, Continuation *c, void *arg)
return EVENT_DONE;
}

void
SSLNetVConnection::_in_context_tunnel()
{
ink_assert(get_context() == NET_VCONNECTION_IN);

Metrics::IntType *t, *c;

switch (get_tunnel_type()) {
case SNIRoutingType::BLIND:
t = net_rsb.tunnel_total_client_connections_tls_tunnel;
c = net_rsb.tunnel_current_client_connections_tls_tunnel;
break;
case SNIRoutingType::FORWARD:
t = net_rsb.tunnel_total_client_connections_tls_forward;
c = net_rsb.tunnel_current_client_connections_tls_forward;
break;
case SNIRoutingType::PARTIAL_BLIND:
t = net_rsb.tunnel_total_client_connections_tls_partial_blind;
c = net_rsb.tunnel_current_client_connections_tls_partial_blind;
break;
default:
t = net_rsb.tunnel_total_client_connections_tls_http;
c = net_rsb.tunnel_current_client_connections_tls_http;
break;
}
Metrics::increment(t);
Metrics::increment(c);
}

void
SSLNetVConnection::_out_context_tunnel()
{
ink_assert(get_context() == NET_VCONNECTION_OUT);

// Never a tunnel type for out (to server) context.
ink_assert(get_tunnel_type() == SNIRoutingType::NONE);

Metrics::increment(net_rsb.tunnel_total_server_connections_tls);
Metrics::increment(net_rsb.tunnel_current_server_connections_tls);
}

void
SSLNetVConnection::increment_ssl_version_metric(int version) const
{
Expand Down
Loading