Rate limit error when using github interceptors

[Allda]

Hello everyone,

my team is using Tekton pipelines in combination with Tekton triggers and from time to time, we see that for a subset of webhook events, the Trigger doesn't run a pipeline. From the logs I see the event listeners are being rate-limited when trying to access a secret for GitHub interceptor. I am unsure if this is a trigger bug or if a cluster can be configured to increase the limit, so any suggestions are appreciated!

Expected Behavior

Every webhook event triggers a pipeline without running into a rate-limiting error.

Actual Behavior

{"severity":"info","timestamp":"2024-06-04T07:59:59.597Z","logger":"eventlistener","caller":"sink/sink.go:420","message":"interceptor stopped trigger processing: rpc error: code = FailedPrecondition desc = error getting secret: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline","commit":"80de3d1","eventlistener":"community-operator-pipeline-github-listener","namespace":"araszka-playground","/triggers-eventid":"6e04a021-6c23-4928-abd1-6fee8705c005","eventlistenerUID":"f50c7e03-7ec2-444e-8f10-5a0bc6d1b6a8","/triggers-eventid":"6e04a021-6c23-4928-abd1-6fee8705c005","/trigger":"github-community-label-listener-hosted"}

Steps to Reproduce the Problem

Create a EventListener with Github interceprot:

triggers:
  # run pipeline on PR opened, reopened, synchronized
  - name: github-pull-request-listener-hosted
    interceptors:
      - ref:
          name: "github"
        params:
          - name: "secretRef"
            value:
              secretName: github-webhook-secret
              secretKey: webhook-secret
          - name: "eventTypes"
            value: ["pull_request"]
          - name: "addChangedFiles"
            value:
              enabled: true
              personalAccessToken:
                secretName: github-bot-token
                secretKey: github_bot_token

Additional Info

• Kubernetes version:

Client Version: 4.14.6
Kustomize Version: v5.0.1
Server Version: 4.14.26
Kubernetes Version: v1.27.13+e709aa5

• Tekton Pipeline version:

Client version: 0.33.0
Chains version: v0.20.1
Pipeline version: v0.56.4
Triggers version: v0.26.2
Operator version: v0.70.3