Access denied after context switch

[egvimo]

Describe the bug
If I start K9s everything works fine with the current context, but if I switch the context via K9s the view stays empty.

ERR  Watcher failed for v1/pods  error="[list watch] access denied on resource \"\":\"v1/pods\""

If I exit K9s switch context via kubectl and then enter K9s again. All resources are being displayed correctly.

To Reproduce
Steps to reproduce the behavior:

1. Go to K9s
2. Switch context
3. See error in log

Expected behavior
The resources are shown correctly after context switch.

Versions (please complete the following information):

• OS: Windows
• K9s: 0.25.7
• K8s: 1.20.5

Additional context
This does not happen if I switch to Kind and back. Only if I switch contexts of external Clusters.

[IvanBodnya]

Strange, I've tried reproducing, but context switching works fine for me on v.0.25.7.

[egvimo]

"Access denied" indicates that the user has no permission. Maybe the context switch does not switch the user correctly?
This particular setup has different users and different cluster for each context.

I've tried the --context CLI option. This works fine, but the switching inside K9s does not work.

In another environment context switching works for me too.

[egvimo]

Here is a debug log with the switch. The first log starts with context 1 and switches to context 2. The second log the other way around.

Context 1 -> Context 2

8:15AM INF 🐶 K9s starting up...
8:15AM DBG Active Context "stage-site-ns1"
8:15AM INF ✅ Kubernetes connectivity
8:15AM WRN No context specific skin file found -- C:\Users\myuser\AppData\Local\k9s\stage-site-ns1_skin.yml
8:15AM WRN No skin file found -- C:\Users\myuser\AppData\Local\k9s\skin.yml. Loading stock skins.
8:15AM DBG Factory START with ns `"stage-ns1"
8:15AM WRN Cluster metrics failed error="`list access denied for user on \"\":v1/nodes"
8:15AM DBG Fetching latest k9s rev...
8:15AM DBG K9s latest rev: "v0.25.7"
8:15AM WRN Fail CRDs load error="`list access denied for user on \"\":apiextensions.k8s.io/v1/customresourcedefinitions"
8:15AM DBG CustomView watching `C:\Users\myuser\AppData\Local\k9s\views.yml
8:15AM WRN Custom view load failed C:\Users\myuser\AppData\Local\k9s\views.yml error="open C:\\Users\\myuser\\AppData\\Local\\k9s\\views.yml: The system cannot find the file specified."
8:15AM ERR CustomView watcher failed error="GetFileAttributes: The system cannot find the file specified."
8:15AM WRN Cluster metrics failed error="user is not authorized to list nodes"
8:15AM WRN Fail CRDs load error="[list watch] access denied on resource \"-\":\"apiextensions.k8s.io/v1/customresourcedefinitions\""
8:15AM WRN No nodes metrics access error="`list access denied for user on \"\":metrics.k8s.io/v1beta1/nodes"
8:15AM DBG TABLE-UPDATER canceled -- "v1/pods"
8:15AM DBG SWITCH CTX "contexts"--"stage-site-ns2"
8:15AM DBG Switching context "stage-site-ns2"
8:15AM DBG TABLE-UPDATER canceled -- "contexts"
8:15AM DBG --> Switching Context "stage-site-ns2"--"ctx"
8:15AM DBG ClusterInfo updater canceled!
8:15AM DBG CustomViewWatcher CANCELED `C:\Users\myuser\AppData\Local\k9s\views.yml!!
8:15AM DBG Factory START with ns `"stage-ns1"
8:15AM WRN Fail CRDs load error="`list access denied for user on \"\":apiextensions.k8s.io/v1/customresourcedefinitions"
8:15AM WRN No context specific skin file found -- C:\Users\myuser\AppData\Local\k9s\stage-site-ns2_skin.yml
8:15AM WRN No skin file found -- C:\Users\myuser\AppData\Local\k9s\skin.yml. Loading stock skins.
8:15AM WRN Validation failed for namespace: "default" error="namespaces \"default\" is forbidden: User \"system:serviceaccount:stage-ns2:domain-admin\" cannot get resource \"namespaces\" in API group \"\" in the namespace \"default\""
8:15AM ERR component init failed for "Pod" `list access denied for user on "stage-ns1":v1/pods error="`list access denied for user on \"stage-ns1\":v1/pods"
8:15AM ERR Watcher failed for v1/pods error="`list access denied for user on \"\":v1/pods"
8:15AM WRN Cluster metrics failed error="`list access denied for user on \"\":v1/nodes"
8:15AM DBG CustomView watching `C:\Users\myuser\AppData\Local\k9s\views.yml
8:15AM WRN Custom view load failed C:\Users\myuser\AppData\Local\k9s\views.yml error="open C:\\Users\\myuser\\AppData\\Local\\k9s\\views.yml: The system cannot find the file specified."
8:15AM ERR CustomView watcher failed error="GetFileAttributes: The system cannot find the file specified."
8:15AM WRN No nodes metrics access error="`list access denied for user on \"\":metrics.k8s.io/v1beta1/nodes"
8:15AM WRN Cluster metrics failed error="user is not authorized to list nodes"
8:15AM WRN Fail CRDs load error="[list watch] access denied on resource \"-\":\"apiextensions.k8s.io/v1/customresourcedefinitions\""

Context 2 -> Context 1

8:18AM INF 🐶 K9s starting up...
8:18AM DBG Active Context "stage-site-ns2"
8:18AM INF ✅ Kubernetes connectivity
8:18AM WRN No context specific skin file found -- C:\Users\myuser\AppData\Local\k9s\stage-site-ns2_skin.yml
8:18AM WRN No skin file found -- C:\Users\myuser\AppData\Local\k9s\skin.yml. Loading stock skins.
8:18AM DBG Factory START with ns `"stage-ns2"
8:18AM WRN Cluster metrics failed error="`list access denied for user on \"\":v1/nodes"
8:18AM DBG Fetching latest k9s rev...
8:18AM DBG K9s latest rev: "v0.25.7"
8:18AM WRN Fail CRDs load error="`list access denied for user on \"\":apiextensions.k8s.io/v1/customresourcedefinitions"
8:18AM DBG CustomView watching `C:\Users\myuser\AppData\Local\k9s\views.yml
8:18AM WRN Custom view load failed C:\Users\myuser\AppData\Local\k9s\views.yml error="open C:\\Users\\myuser\\AppData\\Local\\k9s\\views.yml: The system cannot find the file specified."
8:18AM ERR CustomView watcher failed error="GetFileAttributes: The system cannot find the file specified."
8:18AM WRN Cluster metrics failed error="user is not authorized to list nodes"
8:18AM WRN Fail CRDs load error="[list watch] access denied on resource \"-\":\"apiextensions.k8s.io/v1/customresourcedefinitions\""
8:18AM WRN No nodes metrics access error="`list access denied for user on \"\":metrics.k8s.io/v1beta1/nodes"
8:18AM DBG TABLE-UPDATER canceled -- "v1/pods"
8:18AM DBG SWITCH CTX "contexts"--"stage-site-ns1"
8:18AM DBG Switching context "stage-site-ns1"
8:18AM DBG TABLE-UPDATER canceled -- "contexts"
8:18AM DBG --> Switching Context "stage-site-ns1"--"ctx"
8:18AM DBG ClusterInfo updater canceled!
8:18AM DBG CustomViewWatcher CANCELED `C:\Users\myuser\AppData\Local\k9s\views.yml!!
8:18AM DBG Factory START with ns `"stage-ns2"
8:18AM WRN Fail CRDs load error="`list access denied for user on \"\":apiextensions.k8s.io/v1/customresourcedefinitions"
8:18AM WRN No context specific skin file found -- C:\Users\myuser\AppData\Local\k9s\stage-site-ns1_skin.yml
8:18AM WRN No skin file found -- C:\Users\myuser\AppData\Local\k9s\skin.yml. Loading stock skins.
8:18AM WRN Validation failed for namespace: "default" error="namespaces \"default\" is forbidden: User \"system:serviceaccount:stage-ns1:domain-admin\" cannot get resource \"namespaces\" in API group \"\" in the namespace \"default\""
8:18AM ERR component init failed for "Pod" `list access denied for user on "stage-ns2":v1/pods error="`list access denied for user on \"stage-ns2\":v1/pods"
8:18AM ERR Watcher failed for v1/pods error="`list access denied for user on \"\":v1/pods"
8:18AM WRN Cluster metrics failed error="`list access denied for user on \"\":v1/nodes"
8:18AM DBG CustomView watching `C:\Users\myuser\AppData\Local\k9s\views.yml
8:18AM WRN Custom view load failed C:\Users\myuser\AppData\Local\k9s\views.yml error="open C:\\Users\\myuser\\AppData\\Local\\k9s\\views.yml: The system cannot find the file specified."
8:18AM ERR CustomView watcher failed error="GetFileAttributes: The system cannot find the file specified."
8:18AM WRN No nodes metrics access error="`list access denied for user on \"\":metrics.k8s.io/v1beta1/nodes"
8:18AM WRN Cluster metrics failed error="user is not authorized to list nodes"
8:18AM WRN Fail CRDs load error="[list watch] access denied on resource \"-\":\"apiextensions.k8s.io/v1/customresourcedefinitions\""

[egvimo]

After looking closely at the log, I think I've found the problem:

...
8:15AM DBG Active Context "stage-site-ns1"
...
8:15AM DBG SWITCH CTX "contexts"--"stage-site-ns2"
...
8:15AM ERR component init failed for "Pod" `list access denied for user on "stage-ns1":v1/pods error="`list access denied for user on \"stage-ns1\":v1/pods"

The error indicates that K9s tries to get the pod list from the previous namespace stage-ns1.

[egvimo]

Could this be the line 392 from here?

k9s/internal/view/app.go

Lines 387 to 396 in 9b49819

	func (a *App) switchCtx(name string, loadPods bool) error {
	log.Debug().Msgf("--> Switching Context %q--%q", name, a.Config.ActiveView())
	a.Halt()
	defer a.Resume()
	{
	ns, err := a.Conn().Config().CurrentNamespaceName()
	if err != nil {
	log.Warn().Msg("No namespace specified in context. Using K9s config")
	}
	a.initFactory(ns)

Here the current namespace is selected, but the new namespace should be used.

[derailed]

@egvimo Thank you fo the details on this!! Can u give v0.25.8 a rinse and see if we're happier? Tx!!

[egvimo]

@derailed Still the same issue. The namespace specified in the context is ignored.

2:51PM WRN Validation failed for namespace: "default" error="namespaces \"default\" is forbidden: User \"system:serviceaccount:stage-ns2:domain-admin\" cannot get resource \"namespaces\" in API group \"\" in the namespace \"default\""
2:51PM ERR component init failed for "Pod" `list access denied for user on "":v1/pods error="`list access denied for user on \"\":v1/pods"

[derailed]

@egvimo Rats! Tx for reporting back!! Could you add more details here ie the full logs switch context ctx1->ctx2? I have an idea but it could be a shot in the dark as I can't seem to repro locally ;(

[egvimo]

You can find the full log here: #1345 (comment)

The problem is, that the namespace of the previous context is used for the new context.

My current workaround is to switch the context and then manually switch the namespace to the one specified in the kubeconfig file.

[derailed]

@egvimo Let's see if it's still happening on v0.25.10. If not please reopen. Tx!

[egvimo]

@derailed The issue was gone in v0.25.10 and is back again in v0.25.12. Could you please reopen.

[egvimo]

Works with v0.25.18. Thanks

[ecarey-paa]

I'm having this issue with v0.27.3. I'm on macos 12.6 with kubectl 1.25.2. I don't recall switching context to trigger the error though. Also, I'm not finding how to provide the verbose logging as others have. Attempting to execute k9s -l debug will result in the error message and the Usage text (which i've trimmed down below):

➜  ~ k9s -l debug
Error: [list watch] access denied on resource "":"v1/pods"
Usage:
  k9s [flags]
  k9s [command]

... output trimmed for brevity

➜  ~ 

The error initially occurred while using the previous version. I first updated to v0.27.3 then uninstalled and re-installed without success.

[Doofus100500]

Fixed this issue by reboot my mac)

[ecarey-paa]

thx @Doofus100500. I dunno if this was the root cause for me but after explicitly adding a namespace to my kubectl context I now am no longer having the issue.

kubectl config set-context --current --namespace=default

[mathcoder23]

I meet the same issue.

I see this in two ways

• First, the wrong namespace is used
• Second, the current k8s token does not have access to the default namespace

we can try to kubectl get pods -n xxx, confirm xxx can access
and use k9s -n xxx to Specify namespce

[ecarey-paa]

@mathcoder23 thx. including any namespace gets me past the error. k9s -n default works when k9s by itself does not.

[derailed]

@ecarey-paa Please take a peek at your k9s config + kubeconfig.
Guessing either your kubeconfig default ns or the stored k9s config ns could be incorrect?

[jeffkni]

I was also getting the error:

Watcher failed for v1/namespaces -- [list watch] access denied on resource "":"v1/namespaces"

when I try to start k9s with no namepace because, in my case, I lack permissions (cat /tmp/k9s-$(whoami).log):

11:33AM WRN Fail CRDs load error="`list access denied for user on "":apiextensions.k8s.io/v1/customresourcedefinitions"

I can list namespaces (kubectl get ns) but don't have access to CRDs (?) that would apparently allow k9s to list namespaces (?). This seems to put k9s config ~/.config/k9s/config.yml into a bad state where I can't access anything with k9s.

I can work around this by deleting my ~/.config/k9s/config.yml (I'm sure I could edit the offending lines too) and restarting with an explicit namespace:

 k9s -n my-namespace-123

If I later try to issue a :namespaces command within with k9s', I'm stuck again and have to reset the config.

[startio-dmitry-k]

I have the same issue. Switching between contexts using k9s isn't working. My assumption is that it tries to list all namespaces and if it lacks permissions it fails. It should use namespaces defined in kubeconfig instead without listing all namespaces.
The error is:
list access denied for user on "default":v1/pods
I have no idea where "default" is coming from

[ecarey-paa]

@startio-dmitry-k I dunno if this has anything to do with the root cause for you but one thing different I'm doing now (and maybe you're already doing 🤷 ) is to set the KUBECONFIG env var. I recently learned that this env var is a list of colon separated configs. Adding that to my shell's run command file made context switching in k9s work. For example, I have 3 contexts and my KUBECONFIG looks like this now:

KUBECONFIG=/Users/evan/.kube/anna-dev:/Users/evan/.kube/anna-prod:/Users/evan/.kube/config

Without exporting that env var, executing a bare k9s is working for me now (I'm guessing by updating k9s)... but only the one context defined in ~/.kube/config will be available in k9s.
I'm using k9s v0.32.5. Mac OS v14.5. k8s v1.29.2 (via docker-desktop v4.32).