odo logs too slow for non-local clusters

[dharmit]

What versions of software are you using?

Operating System: all

Output of odo version: main

How did you run odo exactly?

odo logs against a cluster that's not minikube/CRC. For example, one spun up using ClusterBot

Actual behavior

Takes many seconds to show the logs

Expected behavior

Take lesser time

Any logs, error output, etc.?

Originally discussed at #5846 (comment).

/kind bug
/area log

[rm3l]

Expected behavior

Take lesser time

To ensure we won't have regressions in the future, I think it could be great to include some kind of performance testing for this, with an acceptable threshold for what we mean by "lesser time". Relates to #5830, but for odo logs.

[dharmit]

Self-note:

• Look at client-go caching mentioned by Tomas
• By threshold, Armel referred to how long odo should wait till odo logs exits if it doesn't get a response it's looking for.
• Check with Dev Sandbox cluster

[dharmit]

Tasks pending on this issue:

• Address odo logs --follow issue explained in Grab pod logs concurrently #5942 (comment).
• In a further PR, modify the code to first fetch the resources most likely to have Pods of interest for odo. This would include Deployments, ReplicaSets and Pods itself. Check other resources after that.
• Use channels instead of sharing a slice+mutex in the go routines, as is the case here:

  odo/pkg/kclient/all.go

  Lines 29 to 57 in e7588e3

  	var mu sync.Mutex
  	var wg sync.WaitGroup
  	var out []unstructured.Unstructured
  	start := time.Now()
  	klog.V(2).Infof("starting to query %d APIs in concurrently", len(apis))
  	var errResult error
  	for _, api := range apis {
  	if !api.r.Namespaced {
  	klog.V(4).Infof("[query api] api (%s) is non-namespaced, skipping", api.r.Name)
  	continue
  	}
  	wg.Add(1)
  	go func(a apiResource) {
  	defer wg.Done()
  	klog.V(4).Infof("[query api] start: %s", a.GroupVersionResource())
  	v, err := queryAPI(client, a, ns, selector)
  	if err != nil {
  	klog.V(4).Infof("[query api] error querying: %s, error=%v", a.GroupVersionResource(), err)
  	errResult = err
  	return
  	}
  	mu.Lock()
  	out = append(out, v...)
  	mu.Unlock()
  	klog.V(4).Infof("[query api] done: %s, found %d apis", a.GroupVersionResource(), len(v))
  	}(api)
  	}

• Anything else that we might be able to think of while working on the above.

We have discussed adding tests that would help us track the performance of odo logs so that we can know if it slows down in the future. This is to avoid regression, or at least finding it ourselves before it becomes a problem for the users. I prefer it be done in a separate issue than this.

[rm3l]

We have discussed adding tests that would help us track the performance of odo logs so that we can know if it slows down in the future. This is to avoid regression, or at least finding it ourselves before it becomes a problem for the users. I prefer it be done in a separate issue than this.

@dharmit Just so we don't forget, I've created this user story to track the performance of odo logs: #6014
Feel free to add any additional info you might have.