Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Invalid value in runnerScopeFromEnv crashes keda operator with invalid memory address or nil pointer dereference error #6156

Open
swgriffith opened this issue Sep 11, 2024 · 0 comments · May be fixed by #6173
Labels
bug Something isn't working

Comments

@swgriffith
Copy link

Report

If your ScaledJob manifest uses the runnerScopeFromEnv parameter and the value is not found at runtime, either because the variable is missing or is incorrectly named, the parseGitHubRunnerMetadata method will return a null which will later be referenced which triggers a nil reference exception. This causes a panic which puts the keda operator pod in a crash loop.

Expected Behavior

If an invalid value is provided to this parameter, that value should be evaluated and if missing or invalid it should log an informative error and cleanly exit the process.

Actual Behavior

When an invalid or blank value is provided to the runnerScopeFromEnv metadata value in the ScaledJob manifest, the parseGitHubRunnerMetadata returns a null for the 'meta' object. In a subsequent line of code the values of meta.applicationID, meta.installationID and meta.applicationkey are evaluated. That check against a null meta object triggers a null reference exception. This exception puts the keda operator in a crash loop.

Steps to Reproduce the Problem

  1. Create Kubernetes cluster (tested on version 1.29)
  2. Install keda via helm
  3. Create a manifest like the following, filling in redacted data
apiVersion: v1
kind: Namespace
metadata:
  name: github-runner
---
apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: github-auth
  namespace: github-runner
data:
  appKey: REDACTED
---
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: github-trigger-auth
  namespace: github-runner
spec:
  secretTargetRef:
    - parameter: appKey
      name: github-auth
      key: appKey
---
apiVersion: keda.sh/v1alpha1
kind: ScaledJob
metadata:
  name: scaledjob-github-runner
  namespace: github-runner
spec:
  jobTargetRef:
    template:
      metadata:
        labels:
          app: scaledjob-github-runner
      spec:
        containers:
        - name: scaledjob-github-runner
          image: myoung34/github-runner:2.302.1-ubuntu-focal
          imagePullPolicy: Always
          env:
          - name: EPHEMERAL
            value: "true"
          - name: DISABLE_RUNNER_UPDATE
            value: "true"
          - name: REPO_URL
            value: "REDACTED"
          - name: RUNNER_SCOPE
            value: "repo"
          - name: LABELS
            value: "test-local-runner"
        restartPolicy: Never
  minReplicaCount: 0
  maxReplicaCount: 20
  pollingInterval: 30
  triggers:
  - type: github-runner
    metadata:
      owner: "REDACTED"
      repos: "github-aca-runner"
      labelsFromEnv: "LABELS"
      #runnerScope: "repo"
      runnerScopeFromEnv: "thiswillfail"
      applicationID: "REDACTED"
      installationID: "REDACTED"
    authenticationRef:
      name: github-trigger-auth
  1. Start logging on the keda operator
  2. Deploy the manifest

You should see the logs show the nil reference excption and the keda operator pod enter a crash loop.

  1. Uninstall the manifest and watch the operator pod recover

Logs from KEDA operator

INFO	Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference	{"controller": "scaledjob", "controllerGroup": "keda.sh", "controllerKind": "ScaledJob", "ScaledJob": {"name":"scaledjob-github-runner","namespace":"github-runner"}, "namespace": "github-runner", "name": "scaledjob-github-runner", "reconcileID": "5d7e7d2b-c347-452a-8525-882fd5c10118"}
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x78 pc=0x359848f]

KEDA Version

2.15.1

Kubernetes Version

1.29

Platform

Microsoft Azure

Scaler Details

Github Runner Scaler

Anything else?

Lines of code reference.

getValueFromMetaOrEnv returns null:

if val, err := getValueFromMetaOrEnv("runnerScope", config.TriggerMetadata, config.ResolvedEnv); err == nil && val != "" {

Calls the parse and gets back a null 'meta' object

meta, err := parseGitHubRunnerMetadata(config)

Checks the values in a null 'meta' object which triggers nil reference exception

if meta.applicationID != nil && meta.installationID != nil && meta.applicationKey != nil {

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Status: To Triage
Development

Successfully merging a pull request may close this issue.

1 participant