Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Running a test from within a k8s job on a self-hosted Testkube instance doesn't work #6104

Open
aletv8 opened this issue Dec 26, 2024 · 1 comment
Labels
bug 🐛 Something is not working as should be

Comments

@aletv8
Copy link

aletv8 commented Dec 26, 2024

Describe the bug
When launching a Testkube test from within a Kubernetes job on a self-hosted Testkube instance using cluster client configuration, the operation fails with a permission error.

To Reproduce
Steps to reproduce the behavior:

  1. Define and apply the following job.yaml resource:
apiVersion: batch/v1
kind: Job
metadata:
  name: testkube-run-tests-1
spec:
  template:
    spec:
      containers:
      - name: run-tests
        image: kubeshop/testkube-cli:2.1.19
        imagePullPolicy: IfNotPresent
        command:
        - /bin/sh
        - -c
        - |
          testkube run test testkube-my-test --client cluster -f
      restartPolicy: Never
  backoffLimit: 2
  ttlSecondsAfterFinished: 120
  1. Apply the job: kubectl -n testkube apply -f job.yaml
  2. See error:
error: services "testkube-api-server:8088" is forbidden: User "system:serviceaccount:testkube:default" cannot get resource "services/proxy" in API group "" in the namespace "testkube": Azure does not have opinion for this user.)
  1. Attempt to provide missing access rights by defining the following ClusterRole and ClusterRoleBinding:
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: testkube-clusterrole-test
rules:
- apiGroups: [""]
  resources: ["services/proxy"]
  resourceNames: ["*"]
  verbs: ["get", "list", "watch", "create", "update", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: binding-test
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: testkube-clusterrole-test 
subjects:
- kind: ServiceAccount
  name: testkube-ex-sa
  namespace: rag-test
  1. Re-apply the Job using the service account and observe that the error persists.

Expected behavior
The Job should execute the test successfully, leveraging the provided ClusterRole and ClusterRoleBinding for the permissions.

Version / Cluster

  • Which testkube version? CLI: 2.1.19, Testkube: 1.16.9
  • What Kubernetes cluster? Azure Kubernetes Service (AKS)
@aletv8 aletv8 added the bug 🐛 Something is not working as should be label Dec 26, 2024
@vsukhin
Copy link
Collaborator

vsukhin commented Dec 27, 2024

hey, @aletv8 Testkube version is too old, we don't support such old ones. You need to use either direct option for your client connection or provide enough RBAC for your job to be able to access services/proxy

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug 🐛 Something is not working as should be
Projects
None yet
Development

No branches or pull requests

2 participants