Stacked kubeconfigs don't work

[nightkr]

The Go client (including the kubectl tool) supports merging multiple kubeconfigs, as described at https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/#merging-kubeconfig-files.

Repro steps

1. Check out 9741360 (also happens with the 0.25.0 release)
2. Create two empty stacked kubeconfigs: touch /tmp/{a,b}
3. Run the node_reflector example: env KUBECONFIG=/tmp/a:/tmp/b cargo run --features openapi --example node_reflector

Expected behaviour

/tmp/a and /tmp/b are loaded and merged. node_reflector crashes because the merged kubeconfig still doesn't say which cluster to connect to.

Actual behaviour

node_reflector tries to load the literal path /tmp/a:/tmp/b, and crashes because it does not exist.

Workaround

Use kubectl to flatten the kubeconfig: kubectl config --flatten > tmp-kubeconfig && env KUBECONFIG=tmp-kubeconfig cargo run ....

[clux]

Ugh, yeah, there's definitely not logic in here to do that. That's also a bit hairy.

Really happy you have a workaround here :-)

[MOZGIII]

I found out there's a crate specifically for loading kubeconfig with all the associated complexity: https://github.com/esphen/kube-conf
Does it look good? Can we use it to fix this?

[clux]

I found out there's a crate specifically for loading kubeconfig with all the associated complexity: https://github.com/esphen/kube-conf
Does it look good? Can we use it to fix this?

Looks pretty minimal. Smaller than what we got already herein, and it doesn't look like it deals with the logic needed for this ^

[MOZGIII]

Yes, I was under the impression they already had support for multiple files. Sorry to bother.

[clux]

A Config::from_custom_kubeconfig is added in 0.34.0. This will at least make it possible to do the any custom config parsing externally. Of course this isn't the ideal solution because it still allows a lot of work on the users side, but it is at least something that works while we wait for proper support.

That said, the workaround posted in the main post is still the recommended workaround.

[justinturpin]

Depending on your needs you might not need a full merging solution - where I'm at, we just have a config file for each environment, KUBECONFIG is a colon-separated list of paths and the kubectl command will set the context in one of the files. So all I need to do is figure out which context is the current context, and then load the config that contains the current context:

fn find_kubeconfig() -> Option<kube::config::Kubeconfig> {
        let mut current_context: Option<String> = None;

        // Unpack the colon-separated KUBECONFIG env var
        let paths: Vec<String> = std::env::var("KUBECONFIG").unwrap().split(":")
            .map(|x| String::from(x)).collect();

        // First pass: find the first non-empty current_context
        for path in paths.iter() {
            let context = kube::config::Kubeconfig::read_from(path).unwrap().current_context;

            if !context.is_empty() {
                current_context = Some(context.clone());

                break;
            }
        }

        // Second pass: actually load the kubeconfig and set the current_context equal
        // to the context we found in the first pass
        for path in paths.iter() {
            let mut kubeconfig = kube::config::Kubeconfig::read_from(path).unwrap();

            let context_names: Vec<std::string::String> = kubeconfig.contexts.iter().map(|x| x.name.clone()).collect();

            if context_names.contains(current_context.as_ref().unwrap()) {
                // Set the current context equal to the found context, which may be in a totally
                // different file. The only real "merging" we do here
                kubeconfig.current_context = current_context.unwrap();

                return Some(kubeconfig);
            }
        }

        None
    }

This could probably be adapted to be a full merging solution but this works for me and hopefully others will find it useful.

[clux]

Fixed in 0.50.0