Skaffold apply deployments never stabilize on Skaffold 1.35

[edwardthiele]

Using the manifest file "manifest.xml"

apiVersion: v1
kind: Pod
metadata:
  name: getting-started
spec:
  containers:
  - image: k8s.gcr.io/echoserver:1.4
    name: echoserver

with the command:

/usr/local/bin/skaffold apply 'manifest.yaml'

skaffold apply 'manifest.yaml'
Starting deploy...

• pod/getting-started configured
  Waiting for deployments to stabilize...
• pods: could not stabilize within 10m0s
• pods failed. Error: could not stabilize within 10m0s.

The actual deployment appears to work as expected

Expected behavior

The deployment reports success

Actual behavior

The command times out

Information

• Skaffold version: 1.35
• Operating system: Linux
• Installed via: skaffold.dev

[briandealwis]

I've been able to reproduce this by deploying to a GKE Autopilot cluster. In digging into the issue, I've found that we call into checkStandalonePodsStatus and len(r.resources) == 0 and so we cycle:

skaffold/pkg/skaffold/kubernetes/status/resource/deployment.go

Lines 144 to 147 in 9f56bbd

	func (r *Resource) checkStandalonePodsStatus(ctx context.Context, cfg kubectl.Config) *proto.ActionableErr {
	if len(r.resources) == 0 {
	return &proto.ActionableErr{ErrCode: proto.StatusCode_STATUSCHECK_STANDALONE_PODS_PENDING}
	}

DEBU[0015] about to call CheckStatus on pods             subtask=-1 task=DevLoop
DEBU[0015] checkStandalonePodsStatus                     subtask=-1 task=DevLoop
DEBU[0015] update status: errCode:STATUSCHECK_STANDALONE_PODS_PENDING  subtask=-1 task=DevLoop
DEBU[0015] finished call to CheckStatus on pods          subtask=-1 task=DevLoop
DEBU[0016] about to call CheckStatus on pods             subtask=-1 task=DevLoop
DEBU[0016] checkStandalonePodsStatus                     subtask=-1 task=DevLoop
DEBU[0016] update status: errCode:STATUSCHECK_STANDALONE_PODS_PENDING  subtask=-1 task=DevLoop
DEBU[0016] finished call to CheckStatus on pods          subtask=-1 task=DevLoop
DEBU[0017] about to call CheckStatus on pods             subtask=-1 task=DevLoop
DEBU[0017] checkStandalonePodsStatus                     subtask=-1 task=DevLoop
DEBU[0017] update status: errCode:STATUSCHECK_STANDALONE_PODS_PENDING  subtask=-1 task=DevLoop
DEBU[0017] finished call to CheckStatus on pods          subtask=-1 task=DevLoop

[briandealwis]

Steps to reproduce:

1. Use an Autopilot cluster
2. cd examples/getting-started
3. copy the manifest above into echo.yaml
4. skaffold apply -vdebug echo.yaml

[gsquared94]

I can actually consistently repro this locally against a kind cluster. I was able to debug it to be a skaffold apply issue that isn't setting namespaces correctly. For instance, running skaffold dev on Edward's manifest.yaml file sets the namespace "default" in the status.Monitor

But running skaffold apply manifest.yaml sets the namespace "". Which gets filtered out here, so status check just hangs.

[gsquared94]

The difference between apply and dev is being caused in these lines:

skaffold/pkg/skaffold/deploy/kubectl/kubectl.go

Lines 344 to 346 in 9f56bbd

	if !offline {
	return k.kubectl.ReadManifests(ctx, manifests)
	}

In apply we don't run kubectl create --dry-run so the ManifestList doesn't have metadata.namespace populated.