Add access logging support

[npolshakova]

Description

Adds support for HTTP access logging: #10507

API changes

Introduces access logging support on the HCM for kgateway. This is based on the Gloo access logging APIs:

• Gloo API: https://docs.solo.io/gloo-edge/latest/guides/security/access_logging/ (this used to be on the Gloo Gateway resource)
• K8s Gateway API: https://docs.solo.io/gateway/latest/security/access-logging/ (this used to be on Listener Options)

Before Gloo would configure both the listener, tcp and HttpConnectionManager (HCM) access logging in one plugin.

The new kgateway configuration is on the HTTPListenerPolicy which only configures the access logging on the HCM to make it clear what specific access logging is being configured. Listener and TCP access logging can be supported in the future, but will live on a separate resource.

How does it work?

The HTTPListenerPolicy selects a Gateway via a targetRef:

apiVersion: gateway.kgateway.dev/v1alpha1
kind: HTTPListenerPolicy
metadata:
  name: accesslog
  namespace: gwtest
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: http-gw-for-test
  accessLog:
    - grpcService:
        logName: "test-accesslog-service"
        backendRef:
          name: log-test
          port: 50051
      filter:
          headerFilter:
              header:
                value: "test"
                name: "x-my-cool-test-filter"
                type: "Exact"

The user must configure where the access logs are sent by either setting:

1. a GRPC service with a backendRef (this can be an Upstream or Kubernetes Service)
2. filter sink config

Optionally, the user can also configure the filter field to filter access logs based on some criteria. We support a variety of envoy's filtering config:

• cel expression filter
• status code
• duration (Filter on total request/response duration)
• not health check (Filter out health check requests)
• traceable (Filter traceable requests)
  - runtime (Filter that uses a runtime feature key to check if the log should be written)
• header
• response flag (Filter requests that had a response with an Envoy response flag set)
• grpc status

Code changes

• Adds the access logging APIs to the http_listener_policy_types.go
• Adds a new httplistener_plugin.go
• ApplyHCM plugins are run in computeNetworkFilters to apply the HCM config before the HCM typedConfig is serialized

CI changes

NONE

Docs changes

Docs changes will need to be added in a follow up. Should be based on scenarios covered in https://docs.solo.io/gateway/latest/security/access-logging/ and include CEL.

Context

A common use case for the API gateway is to produce an access log (sometimes referred to as an audit log). The entries of an access log represent traffic through the proxy. The access log entries can be customized to include data from the request, the routing destination, and the response.

Access logs in Envoy can be written to a file, the stdout stream of the gateway proxy container, or exported to a gRPC server for custom handling.

Envoy also supports configuring the format of the access logs. See: https://www.envoyproxy.io/docs/envoy/v1.10.0/configuration/access_log#format-dictionaries In kgateway, istead of outputting strings, the file sink access logger can be configured to log structured json instead. When configuring structured json, the Envoy fields are referenced in the same way as in the string format, however a mapping to json keys is defined:

  accessLog:
      - fileSink:
          path: /dev/stdout
          jsonFormat:
            start_time: "%START_TIME%"
            method: "%REQ(X-ENVOY-ORIGINAL-METHOD?:METHOD)%"
            path: "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%"
            protocol: "%PROTOCOL%"
            response_code: "%RESPONSE_CODE%"
            response_flags: "%RESPONSE_FLAGS%"
            bytes_received: "%BYTES_RECEIVED%"
            bytes_sent: "%BYTES_SENT%"
            total_duration: "%DURATION%"
            resp_upstream_service_time: "%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%"
            req_x_forwarded_for: "%REQ(X-FORWARDED-FOR)%"
            user_agent: "%REQ(USER-AGENT)%"
            request_id: "%REQ(X-REQUEST-ID)%"
            authority: "%REQ(:AUTHORITY)%"
            upstreamHost: "%UPSTREAM_HOST%"
            upstreamCluster: "%UPSTREAM_CLUSTER%"

More than one access log can be configured for a single Envoy listener. For example, the following configuration includes four different access log outputs: a default string-formatted access log to standard out on the Envoy container, a default string-formatted access log to a file in the Envoy container, a json-formatted access log to a different file in the Envoy container, and an access log to standard out a separate access_log_cluster container.

apiVersion: gateway.kgateway.dev/v1alpha1
kind: HTTPListenerPolicy
metadata:
  name: accesslog
  namespace: gwtest
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: http-gw-for-test
  accessLog:
        - fileSink:
            path: /dev/stdout
            stringFormat: ""
        - fileSink:
            path: /dev/default-access-log.txt
            stringFormat: ""
        - fileSink:
            path: /dev/other-access-log.json
            jsonFormat:
              protocol: "%PROTOCOL%"
              duration: "%DURATION%"
              upstreamCluster: "%UPSTREAM_CLUSTER%"
              upstreamHost: "%UPSTREAM_HOST%"
        - grpcService:
             logName: "test-accesslog-service"
             backendRef:
                name: log
                kind: Upstream
                group: gateway.kgateway.dev

In addition to the output, users can apply different filters on their access logs to reduce and optimize the number of logs that are stored. kgateway supports filtering access logs based on request headers, HTTP response codes, gRPC status codes, request duration, health check status, tracing parameters, response flags, and custom CEL expressions.

HTTP response header filter example:

  accessLog:
      - fileSink:
          path: /dev/stdout
        filter:
            headerFilter:
                header:
                  value: "test"
                  name: "x-my-cool-test-filter"
                  type: "Exact"

CEL filter example:

  accessLog:
      - fileSink:
          path: /dev/stdout
        filter:
          celFilter:
            match: "connection.mtls"

It is also possible to combine multiple filters, and perform AND and OR operations on filter results.

Interesting decisions

The HTTP header match API for the access logging filter is based on the K8s Gateway API. This is the same header filtering supported on the HTTPRoutes for reusability.

Some of the kubebuilder validation oneOf logic was too complex to be evaulated:

Error: failed to install CRD crds/gateway.kgateway.dev_httplistenerpolicies.yaml: 1 error occurred:
        * CustomResourceDefinition.apiextensions.k8s.io "httplistenerpolicies.gateway.kgateway.dev" is invalid: [spec.validation.openAPIV3Schema.properties[spec].properties[accessLoggingConfig].properties[accessLog].items.properties[filter].properties[andFilter].items.x-kubernetes-validations[0].rule: Forbidden: estimated rule cost exceeds budget by factor of 2.8x (try simplifying the rule, or adding maxItems, maxProperties, and maxLength where arrays, maps, and strings are declared), spec.validation.openAPIV3Schema.properties[spec].properties[accessLoggingConfig].properties[accessLog].items.properties[filter].properties[orFilter].items.x-kubernetes-validations[0].rule: Forbidden: estimated rule cost exceeds budget by factor of 2.8x (try simplifying the rule, or adding maxItems, maxProperties, and maxLength where arrays, maps, and strings are declared), spec.validation.openAPIV3Schema.properties[spec].properties[accessLoggingConfig].properties[accessLog].items.properties[filter].x-kubernetes-validations[0].rule: Invalid value: apiextensions.ValidationRule{Rule:"1 == (self.statusCodeFilter != null?1:0) + (self.durationFilter != null?1:0) + (self.notHealthCheckFilter != null?1:0) + (self.traceableFilter != null?1:0) + (self.runtimeFilter != null?1:0) + (self.andFilter.items > 0?1:0) + (self.orFilter.items > 0?1:0) + (self.headerFilter != null?1:0) + (self.responseFlagFilter != null?1:0) + (self.grpcStatusFilter != null?1:0)", Message:"There must one and only one AccessLogFilter type set", MessageExpression:"", Reason:(*apiextensions.FieldValueErrorReason)(nil), FieldPath:"", OptionalOldSelf:(*bool)(nil)}: compilation failed: ERROR: <input>:1:209: type 'list(selfType203865929.andFilter.@idx)' does not support field selection
 | 1 == (self.statusCodeFilter != null?1:0) + (self.durationFilter != null?1:0) + (self.notHealthCheckFilter != null?1:0) + (self.traceableFilter != null?1:0) + (self.runtimeFilter != null?1:0) + (self.andFilter.items > 0?1:0) + (self.orFilter.items > 0?1:0) + (self.headerFilter != null?1:0) + (self.responseFlagFilter != null?1:0) + (self.grpcStatusFilter != null?1:0)
 | 

To keep the oneOf behavior, the plugin accesslog conversion does the validation instead.

Testing steps

Added new tests for:

• Access logging tests at the translator level in ggv2setup_test.go (CEL, headers, fileSink format)
• Unit tests for access logging conversion
• TODO: Follow up issue to add e2e access logging test: Add access logging e2e test #10575

Manually testing steps:

1. Setup env

CONFORMANCE=true ./ci/kind/setup-kind.sh

helm upgrade -i -n kgateway-system kgateway ./_test/kgateway-1.0.0-ci1.tgz 

2. Apply config

I used httpbin in the default namespace along with this config to setup basic access logging with a file sink:

kind: Gateway
apiVersion: gateway.networking.k8s.io/v1
metadata:
  name: http-gw-for-test
  namespace: default
spec:
  gatewayClassName: kgateway
  listeners:
    - protocol: HTTP
      port: 8080
      name: http
      allowedRoutes:
        namespaces:
          from: All
    - protocol: HTTP
      port: 8081
      name: other
      allowedRoutes:
        namespaces:
          from: All
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: httpbin
  namespace: default
spec:
  parentRefs:
    - name: http-gw-for-test
  hostnames:
    - "httpbin"
  rules:
    - backendRefs:
        - name: httpbin
          port: 8000
---
apiVersion: gateway.kgateway.dev/v1alpha1
kind: HTTPListenerPolicy
metadata:
  name: accesslog
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: http-gw-for-test
  accessLog:
  - fileSink:
      path: /dev/stdout
      jsonFormat:
        start_time: "%START_TIME%"
        method: "%REQ(X-ENVOY-ORIGINAL-METHOD?:METHOD)%"
        path: "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%"
        protocol: "%PROTOCOL%"
        response_code: "%RESPONSE_CODE%"
        response_flags: "%RESPONSE_FLAGS%"
        bytes_received: "%BYTES_RECEIVED%"
        bytes_sent: "%BYTES_SENT%"
        total_duration: "%DURATION%"
        resp_upstream_service_time: "%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%"
        req_x_forwarded_for: "%REQ(X-FORWARDED-FOR)%"
        user_agent: "%REQ(USER-AGENT)%"
        request_id: "%REQ(X-REQUEST-ID)%"
        authority: "%REQ(:AUTHORITY)%"
        upstreamHost: "%UPSTREAM_HOST%"
        upstreamCluster: "%UPSTREAM_CLUSTER%"

3. Test

Port-forward the gateway:

k port-forward services/gloo-proxy-http-gw-for-test  8080:8080

Send a request:

curl localhost:8080/ -v -H "host: httpbin"

Look at the access logs in the gateway, you should see the json access log:

{"response_flags":"-","upstreamCluster":"kube_default_httpbin_8000","path":"/status","resp_upstream_service_time":"62","start_time":"2025-02-03T22:37:43.555Z","upstreamHost":"10.244.0.12:80","request_id":"9107ed0c-2a8d-45fe-9086-60c948628b9
a","user_agent":"curl/8.10.1","req_x_forwarded_for":null,"bytes_sent":233,"method":"GET","bytes_received":0,"response_code":404,"total_duration":80,"protocol":"HTTP/1.1","authority":"httpbin"}
{"upstreamHost":"10.244.0.12:80","total_duration":80,"response_code":404,"path":"/status","bytes_sent":233,"request_id":"9107ed0c-2a8d-45fe-9086-60c948628b9a","upstreamCluster":"kube_default_httpbin_8000","method":"GET","response_flags":"-"
,"protocol":"HTTP/1.1","authority":"httpbin","start_time":"2025-02-03T22:37:43.555Z","req_x_forwarded_for":null,"bytes_received":0,"resp_upstream_service_time":"62","user_agent":"curl/8.10.1"}
{"upstreamHost":"10.244.0.12:80","response_code":200,"path":"/","total_duration":87,"protocol":"HTTP/1.1","user_agent":"curl/8.10.1","bytes_received":0,"start_time":"2025-02-03T22:37:52.729Z","resp_upstream_service_time":"85","request_id":"
8b55b705-b365-4764-8226-3c92fd27354c","upstreamCluster":"kube_default_httpbin_8000","req_x_forwarded_for":null,"authority":"httpbin","response_flags":"-","method":"GET","bytes_sent":9593}
{"method":"GET","total_duration":87,"request_id":"8b55b705-b365-4764-8226-3c92fd27354c","bytes_sent":9593,"upstreamHost":"10.244.0.12:80","req_x_forwarded_for":null,"upstreamCluster":"kube_default_httpbin_8000","start_time":"2025-02-03T22:3
7:52.729Z","resp_upstream_service_time":"85","bytes_received":0,"response_flags":"-","response_code":200,"user_agent":"curl/8.10.1","protocol":"HTTP/1.1","path":"/","authority":"httpbin"}
{"resp_upstream_service_time":"9","protocol":"HTTP/1.1","response_flags":"-","upstreamCluster":"kube_default_httpbin_8000","path":"/","bytes_sent":9593,"bytes_received":0,"authority":"httpbin","method":"GET","response_code":200,"upstreamHos
t":"10.244.0.12:80","req_x_forwarded_for":null,"request_id":"8635bfa2-f8af-4bd8-8400-f8d2c3deface","user_agent":"curl/8.10.1","start_time":"2025-02-03T22:37:56.546Z","total_duration":10}
{"path":"/","method":"GET","start_time":"2025-02-03T22:37:56.546Z","total_duration":10,"request_id":"8635bfa2-f8af-4bd8-8400-f8d2c3deface","response_code":200,"bytes_received":0,"protocol":"HTTP/1.1","req_x_forwarded_for":null,"upstreamHost
":"10.244.0.12:80","resp_upstream_service_time":"9","bytes_sent":9593,"upstreamCluster":"kube_default_httpbin_8000","user_agent":"curl/8.10.1","authority":"httpbin","response_flags":"-"}

Checklist:

• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added tests that prove my fix is effective or that my feature works

[nfuden]

any chance we can do something like https://github.com/solo-io/gloo/blob/main/projects/gloo/pkg/plugins/als/converter.go#L56-L64 to validate the contents of the access logging itself?

[npolshakova]

any chance we can do something like https://github.com/solo-io/gloo/blob/main/projects/gloo/pkg/plugins/als/converter.go#L56-L64 to validate the contents of the access logging itself?

I'm hesitant to include those initially for a couple of reasons:

1. envoy go control plane doesn't have those defined anywhere so we'd need to hard code the "unuseful" values
2. It's unclear how Gloo identified the "unuseful" commands from the envoy list. The original issue from when this code was introduced (Support listener access logging #8438) didn't seem to have it in the requirements. A bad entry shouldn't be the end of the world as long as the syntax is enclosed correctly.

Most of the validation should be happening as part of the kubebuilder validation lines. I'm not sure we want to validate on the content of access logs, just the configuration.

[lgadban]

is this an accurate name? the plugin suffix is throwing me off, seems like this is closer to httpListenerOptsPolicy

[npolshakova]

It's matching what we do in the listener plugin:

kgateway/internal/kgateway/extensions2/plugins/listenerpolicy/listener_policy_plugin.go

Line 23 in d68bd4d

type listenerOptsPlugin struct {

Can rename both if we want to be consistent.

[lgadban]

Yes I think we should rename, wdyt @yuval-k ?

[yuval-k]

Looks great!
left some small comments

[lgadban]

I think this struct should still be named routeOptsPluginGwPass.

My point was just the initial struct, which is now routeOptsPolicy, is not specific to an instance of a plugin, i.e. a plugin doesn't have a created time or a policy Spec, but the policy does.

However the routeOptsPluginGwPass is accurate -- it represents a translation pass of a given plugin, in this case the route options plugin

[lgadban]

🚀