Support Audit Logging export in teleport

[QuentinBisson]

Is your feature request related to a problem? Please describe.

Teleport provide audit logs when we access clusters and nodes using teleport but those logs are stored in dynamodb. @ssyno talked to me about this plugin that could be useful https://goteleport.com/docs/reference/helm-reference/teleport-plugin-event-handler/

Describe the solution you'd like

I would like to have a solution to be able to get the teleport audit logs sent to customer endpoints.

To that end, we could use this https://goteleport.com/docs/reference/helm-reference/teleport-plugin-event-handler/ with a custom fluentd/fluent bit instance or maybe another teleport plugin to be able to send the logs to customer endpoints if they require it

Describe alternatives you've considered

Additional context

This is needed by a few customers

cc @gawertm as PO of bigmac. Should we schedule a session to talk about this ?

[ssyno]

As of teleport documentation around teleport-plugin-event-handler.

1. Generate a plugin configuration teleport-event-handler-client-tls

TELEPORT_CLUSTER_ADDRESS=teleport.giantswarm.io:443
docker run -v `pwd`:/opt/teleport-plugin -w /opt/teleport-plugin public.ecr.aws/gravitational/teleport-plugin-event-handler:15.0.2 configure . ${TELEPORT_CLUSTER_ADDRESS?}
kubectl create secret generic teleport-event-handler-client-tls --from-file=ca.crt=ca.crt,client.crt=client.crt,client.key=client.key

2. It is possible to create separate user and role for each cluster in the following pattern:

kind: role
metadata:
  name: teleport-event-handler-gaggle
spec:
  allow:
    rules:
      - resources: ['event', 'session']
        verbs: ['list','read']
        where: 'contains(session.cluster_name, "gaggle")'
version: v5
---
kind: user
metadata:
  name: teleport-event-handler-gaggle
spec:
  roles: ['teleport-event-handler-gaggle']
version: v2

Apply with tctl command (can be included to our Gitops approach).

tctl create -f teleport-event-handler-gaggle-role.yaml

3. Create teleport-event-handler credentials for each user/role:

kind: role
version: v5
metadata:
  name: teleport-event-handler-gaggle-impersonator
spec:
  options:
    max_session_ttl: 10h
  allow:
    impersonate:
      users: ["teleport-event-handler-gaggle"]
      roles: ["teleport-event-handler-gaggle"]

Apply with tctl command (can be included to our Gitops approach).

tctl create teleport-event-handler-gaggle-impersonator.yaml

4. Both tbot-credentials and long lived-credentials are supported to be discussed
5. fluentd.conf has been generated on step no.1,fluentd deployment?
6. Implementation of the handler and all mentioned above to teleport helm chart like:

eventHandler:
  storagePath: "./storage"
  timeout: "10s"
  batch: 20
  namespace: "teleport"

teleport:
  address: "teleport.giantswarm.io:443"
  identitySecretName: teleport-event-handler-gaggle-identity
  identitySecretPath: identity

fluentd:
  url: "https://fluentd.fluentd.svc.cluster.local/events.log"
  sessionUrl: "https://fluentd.fluentd.svc.cluster.local/session.log"
  certificate:
    secretName: "teleport-event-handler-client-tls"
    caPath: "ca.crt"
    certPath: "client.crt"
    keyPath: "client.key"

persistentVolumeClaim:
  enabled: true

[gawertm]

@QuentinBisson do you know how to proceed here?

[QuentinBisson]

Hey @gawertm and @ssyno sorry about the delay :(

Thank you very much for the investigation.

So from what I understand, we could deploy a fluentd now and eventually configure it later to send logs to some custom locations right?

We probably would need to be able to send logs to private MCs as well the customer SIEM tools will probably be private. Do you think that would be possible ? If yes then it would be amazing if you could deploy this on our teleport cluster and then we can configure it per installation.

I think we probably would need those logs in loki as well as in a customer fluent logshipper for some customers

[ssyno]

Hey @QuentinBisson, sorry about the delay,

Deploying Fluentd/Fluent Bit with the teleport-plugin-event-handler configuration for secure log collection as an initial will allow us to collect audit logs securely and forward them to various endpoints, including Loki for internal use and customer-specific SIEM tools.
Then we can set up the necessary roles and permissions for secure access to the logs and configure log forwarding to meet both internal and customer-specific requirements, ensuring privacy and security.

[QuentinBisson]

Awesome then for next steps, I would probably start by sending them to each loki and then we can split it for specific customers SIEM. What do you think?
Should we schedule a call to start this?

[ssyno]

Sure, we can have a call whenever you like.

[QuentinBisson]

Here is the outcome of the call of the day:

Preferred solution is the orange one. Atlas will take care of the fluent bit receiver (issue will come) and Big Mac will take care of the event handlers to be deployed per instalation.

[QuentinBisson]

Here is the receiver issue on our end #3343 :)

[gawertm]

@QuentinBisson @Rotfuks as you have a separate ticket in your board already, do we need this ticket here still?

[Rotfuks]

No, not really :) Guess you can close it then as a duplicate