ATS with certifier

[pokerman79]

Hi,

I would appreciate any assistance to configure ATS (9.1.3) as forwarding proxy with certifier

Configured certifier in plugins and got message that "Dynamic cert generation is enabled".

plugin is set to receive TS_SSL_CERT_HOOK
but whatever I tried I'm unable to see this hook being triggered

some of the config options that would matter:
CONFIG proxy.config.http.server_ports STRING 8080
CONFIG proxy.config.reverse_proxy.enabled INT 0
CONFIG proxy.config.url_remap.remap_required INT 0

ssl_multicert
dest_ip=* ssl_cert_name=myCA.crt

plugins
certifier.so --store /etc/trafficserver/certs --max 1000 --sign-cert /etc/trafficserver/certifier/myCA.crt --sign-key /etc/trafficserver/certifier/myCA.key --sign-serial /etc/trafficserver/certifier/ca-serial.txt

When calling
curl -vI -x 10.10.13.5:8080 "https://ipecho.net"
I get proper response but certifier is not activated and I get real tunnel connection

+++++++++ Proxy's Request +++++++++
-- State Machine Id: 3
CONNECT ipecho.net:443 HTTP/1.1
Host: ipecho.net:443
User-Agent: curl/7.64.1
Via: http/1.1 traffic_server[b4732282-fd4e-4c71-af65-3813df62f127] (ApacheTrafficServer/9.1.3)

[Nov 3 00:32:21.082] [ET_NET 18] DEBUG: <HttpTransact.cc:2208 (LookupSkipOpenServer)> (http_trans) Next action next; HttpTransact::HandleResponse
[Nov 3 00:32:21.082] [ET_NET 18] DEBUG: <HttpSM.cc:7409 (call_transact_and_set_next_state)> (http) [3] State Transition: SM_ACTION_API_OS_DNS -> SM_ACTION_ORIGIN_SERVER_RAW_OPEN
[Nov 3 00:32:21.082] [ET_NET 18] DEBUG: <HttpSM.cc:4944 (do_http_server_open)> (http_track) entered inside do_http_server_open ][ipv4]
[Nov 3 00:32:21.082] [ET_NET 18] DEBUG: <HttpSM.cc:4964 (do_http_server_open)> (http) [3] open connection to ipecho.net: 34.160.111.145:443
[Nov 3 00:32:21.082] [ET_NET 18] DEBUG: <HttpSM.cc:4977 (do_http_server_open)> (http_seq) [HttpSM::do_http_server_open] Sending request to server
[Nov 3 00:32:21.082] [ET_NET 18] DEBUG: <HttpSM.cc:5311 (do_http_server_open)> (http) calling netProcessor.connect_re
[Nov 3 00:32:21.082] [ET_NET 18] DEBUG: <HttpSM.cc:2715 (main_handler)> (http) [3] [HttpSM::main_handler, NET_EVENT_OPEN/TS_EVENT_NET_CONNECT]
[Nov 3 00:32:21.082] [ET_NET 18] DEBUG: <HttpSM.cc:1233 (state_raw_http_server_open)> (http) [3] [&HttpSM::state_raw_http_server_open, NET_EVENT_OPEN/TS_EVENT_NET_CONNECT]
[Nov 3 00:32:21.082] [ET_NET 18] DEBUG: <HttpTransact.cc:3372 (OriginServerRawOpen)> (http_trans) [3] [HttpTransact::OriginServerRawOpen]
[Nov 3 00:32:21.082] [ET_NET 18] DEBUG: <HttpTransactHeaders.cc:1137 (add_server_header_to_response)> (http_trans) Adding Server: ATS/9.1.3
+++++++++ Proxy's Response 2 +++++++++
-- State Machine Id: 3
HTTP/1.1 200 OK
Date: Thu, 03 Nov 2022 00:32:21 GMT
Proxy-Connection: keep-alive
Server: ATS/9.1.3

[Nov 3 00:32:21.082] [ET_NET 18] DEBUG: <HttpTransact.cc:3393 (OriginServerRawOpen)> (http_trans) [3] [OriginServerRawOpen] connection alive. next action is ssl_tunnel
[Nov 3 00:32:21.082] [ET_NET 18] DEBUG: <HttpSM.cc:7409 (call_transact_and_set_next_state)> (http) [3] State Transition: SM_ACTION_ORIGIN_SERVER_RAW_OPEN -> SM_ACTION_SSL_TUNNEL
[Nov 3 00:32:21.082] [ET_NET 18] DEBUG: <HttpTunnel.cc:605 (add_producer)> (http_tunnel) [3] adding producer 'http server - tunnel'
[Nov 3 00:32:21.082] [ET_NET 18] DEBUG: <HttpTunnel.cc:660 (add_consumer)> (http_tunnel) [3] adding consumer 'user agent - tunnel'
[Nov 3 00:32:21.082] [ET_NET 18] DEBUG: <HttpTunnel.cc:605 (add_producer)> (http_tunnel) [3] adding producer 'user agent - tunnel'
[Nov 3 00:32:21.082] [ET_NET 18] DEBUG: <HttpTunnel.cc:660 (add_consumer)> (http_tunnel) [3] adding consumer 'http server - tunnel'
[Nov 3 00:32:21.082] [ET_NET 18] DEBUG: <HttpTunnel.cc:707 (tunnel_run)> (http_tunnel) tunnel_run started, p_arg is NULL
[Nov 3 00:32:21.082] [ET_NET 18] DEBUG: <HttpTunnel.cc:1319 (consumer_handler)> (http_tunnel) [3] consumer_handler [user agent - tunnel VC_EVENT_WRITE_READY/TS_EVENT_VCONN_WRITE_READY]
[Nov 3 00:32:21.090] [ET_NET 18] DEBUG: <HttpTunnel.cc:1319 (consumer_handler)> (http_tunnel) [3] consumer_handler [http server - tunnel VC_EVENT_WRITE_READY/TS_EVENT_VCONN_WRITE_READY]
[Nov 3 00:32:21.096] [ET_NET 18] DEBUG: <HttpTunnel.cc:1113 (producer_handler)> (http_tunnel) [3] producer_handler [user agent - tunnel VC_EVENT_READ_READY/TS_EVENT_VCONN_READ_READY]

[shinrich]

I don't think certifier will trigger on the connect method. The actual TLS handshake will be between origin server and client, not client and proxy. You need the TLS connection to terminate on the proxy for the hooks to go off to engage certifier.

We are using it in transparent mode with GET/POST, etc. The traffic routes through our gateway, and we use iptables to mark and TPROXY the traffic into trafficserver. In that case certifier does generate server certificates to return.

Notes on transparent mode https://docs.trafficserver.apache.org/admin-guide/configuration/transparent-proxy.en.html

You could use certifier in reverse proxy mode, though there isn't much of a point for it. Or at least that is not how I'm used to thinking about it.

[pokerman79]

I'm trying to change squid with ATS with minimal changes to our network.
All our clients have proxy already configured, and this is with squid done seamlessly via SSL bump

Goal is that when CONNECT method comes we route incoming CONNECT request to ssl port on traffic server

as per https://docs.trafficserver.apache.org/admin-guide/plugins/certifier.en.html

to be exact we want as per image to "Route CONNECT back to self "

This was so far unsuccessful mission and documentation is heavily lacking:(

Anyone has any example or directions what else to try, so far we have tried with remap, parent and sni and in best scenario would return "503 Origin server congested"

[SolidWallOfCode]

Yes, documentation is always a problem. I know this has been used, I'll see if I can take a look.

[bneradt]

@pokerman79 : I notice you mention the "503 Origin server congested" reason string via the per_server.connection.max setting. Just checking: does this patch help your issue: #9182

[pokerman79]

@bneradt issue that I'm seeing is that we are unable to do a "self route" so certifier can even be triggered

parents.conf is not being used at all, we are seeing this in log "request not cacheable, so bypass parent"
request in question is CONNECT

every attempt to use localhost loop would return 503
mainly due to
proxy.process.host_status.localhost HOST_STATUS_DOWN,ACTIVE:UP:0:0,LOCAL:UP:0:0,MANUAL:UP:0:0,SELF_DETECT:DOWN:1667546618

even though we have tried in parents to set ignore_self_detect=true

in records changed value for proxy.config.http.parent_proxy.self_detect to 0

for some reason we are absolutely unable to do a self route, do you know of any examples or location, we have searched high and low but no working example was found

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. Marking it stale to flag it for further consideration by the community.

[lenhartd-apex]

@bneradt issue that I'm seeing is that we are unable to do a "self route" so certifier can even be triggered

parents.conf is not being used at all, we are seeing this in log "request not cacheable, so bypass parent" request in question is CONNECT

every attempt to use localhost loop would return 503 mainly due to proxy.process.host_status.localhost HOST_STATUS_DOWN,ACTIVE:UP:0:0,LOCAL:UP:0:0,MANUAL:UP:0:0,SELF_DETECT:DOWN:1667546618

even though we have tried in parents to set ignore_self_detect=true

in records changed value for proxy.config.http.parent_proxy.self_detect to 0

for some reason we are absolutely unable to do a self route, do you know of any examples or location, we have searched high and low but no working example was found

Did you ever get it to work?