request ip return 500 no address allocated to pod $POD_NAME provider ovn, please see kube-ovn-controller logs to find errors

[hezhangjian]

Expected Behavior

The pod can successful created and start

Actual Behavior

the nginx pod shows that

  Warning  FailedCreatePodSandBox  4m7s (x92 over 54m)  kubelet            (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "b0ce178c18403cbe12d02b36ef6588e2cd6c1dfc53703d1c35b23ada614d5282": plugin type="kube-ovn" failed (add): RPC failed; request ip return 500 no address allocated to pod default/nginx-5b4664bcd4-hk4s6 provider ovn, please see kube-ovn-controller logs to find errors

kube-ovn-controller logs that

I1222 17:59:54.688648       7 pod.go:319] handle add pod default/nginx-5b4664bcd4-hk4s6
I1222 17:59:54.688693       7 ipam.go:51] allocate v4 10.16.0.16 v6  mac 00:00:00:0F:9C:A4 for default/nginx-5b4664bcd4-hk4s6 from subnet ovn-default
E1222 17:59:54.712915       7 pod.go:331] error syncing 'default/nginx-5b4664bcd4-hk4s6': map: map[] does not contain declared merge key: name, requeuing
I1222 17:59:54.718920       7 pod.go:319] handle add pod default/nginx-5b4664bcd4-hk4s6
I1222 17:59:54.718973       7 ipam.go:51] allocate v4 10.16.0.16 v6  mac 00:00:00:0F:9C:A4 for default/nginx-5b4664bcd4-hk4s6 from subnet ovn-default
E1222 17:59:54.732737       7 pod.go:331] error syncing 'default/nginx-5b4664bcd4-hk4s6': map: map[] does not contain declared merge key: name, requeuing
I1222 17:59:54.743283       7 pod.go:319] handle add pod default/nginx-5b4664bcd4-hk4s6
I1222 17:59:54.743350       7 ipam.go:51] allocate v4 10.16.0.16 v6  mac 00:00:00:0F:9C:A4 for default/nginx-5b4664bcd4-hk4s6 from subnet ovn-default
E1222 17:59:54.754261       7 pod.go:331] error syncing 'default/nginx-5b4664bcd4-hk4s6': map: map[] does not contain declared merge key: name, requeuing
I1222 17:59:54.774560       7 pod.go:319] handle add pod default/nginx-5b4664bcd4-hk4s6
I1222 17:59:54.774602       7 ipam.go:51] allocate v4 10.16.0.16 v6  mac 00:00:00:0F:9C:A4 for default/nginx-5b4664bcd4-hk4s6 from subnet ovn-default
E1222 17:59:54.780337       7 pod.go:331] error syncing 'default/nginx-5b4664bcd4-hk4s6': map: map[] does not contain declared merge key: name, requeuing
I1222 17:59:54.820879       7 pod.go:319] handle add pod default/nginx-5b4664bcd4-hk4s6
I1222 17:59:54.820938       7 ipam.go:51] allocate v4 10.16.0.16 v6  mac 00:00:00:0F:9C:A4 for default/nginx-5b4664bcd4-hk4s6 from subnet ovn-default
E1222 17:59:54.827388       7 pod.go:331] error syncing 'default/nginx-5b4664bcd4-hk4s6': map: map[] does not contain declared merge key: name, requeuing
I1222 17:59:54.908164       7 pod.go:319] handle add pod default/nginx-5b4664bcd4-hk4s6
I1222 17:59:54.908213       7 ipam.go:51] allocate v4 10.16.0.16 v6  mac 00:00:00:0F:9C:A4 for default/nginx-5b4664bcd4-hk4s6 from subnet ovn-default
E1222 17:59:54.913282       7 pod.go:331] error syncing 'default/nginx-5b4664bcd4-hk4s6': map: map[] does not contain declared merge key: name, requeuing
I1222 17:59:55.073611       7 pod.go:319] handle add pod default/nginx-5b4664bcd4-hk4s6
I1222 17:59:55.073693       7 ipam.go:51] allocate v4 10.16.0.16 v6  mac 00:00:00:0F:9C:A4 for default/nginx-5b4664bcd4-hk4s6 from subnet ovn-default
E1222 17:59:55.079675       7 pod.go:331] error syncing 'default/nginx-5b4664bcd4-hk4s6': map: map[] does not contain declared merge key: name, requeuing
I1222 17:59:55.400343       7 pod.go:319] handle add pod default/nginx-5b4664bcd4-hk4s6
I1222 17:59:55.400407       7 ipam.go:51] allocate v4 10.16.0.16 v6  mac 00:00:00:0F:9C:A4 for default/nginx-5b4664bcd4-hk4s6 from subnet ovn-default
E1222 17:59:55.406840       7 pod.go:331] error syncing 'default/nginx-5b4664bcd4-hk4s6': map: map[] does not contain declared merge key: name, requeuing
I1222 17:59:56.046868       7 pod.go:319] handle add pod default/nginx-5b4664bcd4-hk4s6
I1222 17:59:56.046925       7 ipam.go:51] allocate v4 10.16.0.16 v6  mac 00:00:00:0F:9C:A4 for default/nginx-5b4664bcd4-hk4s6 from subnet ovn-default
E1222 17:59:56.052742       7 pod.go:331] error syncing 'default/nginx-5b4664bcd4-hk4s6': map: map[] does not contain declared merge key: name, requeuing
I1222 17:59:57.333755       7 pod.go:319] handle add pod default/nginx-5b4664bcd4-hk4s6
I1222 17:59:57.333811       7 ipam.go:51] allocate v4 10.16.0.16 v6  mac 00:00:00:0F:9C:A4 for default/nginx-5b4664bcd4-hk4s6 from subnet ovn-default
E1222 17:59:57.339608       7 pod.go:331] error syncing 'default/nginx-5b4664bcd4-hk4s6': map: map[] does not contain declared merge key: name, requeuing
I1222 17:59:59.899676       7 pod.go:319] handle add pod default/nginx-5b4664bcd4-hk4s6
I1222 17:59:59.899717       7 ipam.go:51] allocate v4 10.16.0.16 v6  mac 00:00:00:0F:9C:A4 for default/nginx-5b4664bcd4-hk4s6 from subnet ovn-default
E1222 17:59:59.906024       7 pod.go:331] error syncing 'default/nginx-5b4664bcd4-hk4s6': map: map[] does not contain declared merge key: name, requeuing
I1222 18:00:05.026723       7 pod.go:319] handle add pod default/nginx-5b4664bcd4-hk4s6
I1222 18:00:05.026782       7 ipam.go:51] allocate v4 10.16.0.16 v6  mac 00:00:00:0F:9C:A4 for default/nginx-5b4664bcd4-hk4s6 from subnet ovn-default
E1222 18:00:05.032407       7 pod.go:331] error syncing 'default/nginx-5b4664bcd4-hk4s6': map: map[] does not contain declared merge key: name, requeuing
I1222 18:00:15.272715       7 pod.go:319] handle add pod default/nginx-5b4664bcd4-hk4s6
I1222 18:00:15.272809       7 ipam.go:51] allocate v4 10.16.0.16 v6  mac 00:00:00:0F:9C:A4 for default/nginx-5b4664bcd4-hk4s6 from subnet ovn-default
E1222 18:00:15.279510       7 pod.go:331] error syncing 'default/nginx-5b4664bcd4-hk4s6': map: map[] does not contain declared merge key: name, requeuing
I1222 18:00:35.760030       7 pod.go:319] handle add pod default/nginx-5b4664bcd4-hk4s6
I1222 18:00:35.760102       7 ipam.go:51] allocate v4 10.16.0.16 v6  mac 00:00:00:0F:9C:A4 for default/nginx-5b4664bcd4-hk4s6 from subnet ovn-default
E1222 18:00:35.766227       7 pod.go:331] error syncing 'default/nginx-5b4664bcd4-hk4s6': map: map[] does not contain declared merge key: name, requeuing
I1222 18:01:16.727042       7 pod.go:319] handle add pod default/nginx-5b4664bcd4-hk4s6
I1222 18:01:16.727101       7 ipam.go:51] allocate v4 10.16.0.16 v6  mac 00:00:00:0F:9C:A4 for default/nginx-5b4664bcd4-hk4s6 from subnet ovn-default
E1222 18:01:16.732675       7 pod.go:331] error syncing 'default/nginx-5b4664bcd4-hk4s6': map: map[] does not contain declared merge key: name, requeuing
I1222 18:02:38.652971       7 pod.go:319] handle add pod default/nginx-5b4664bcd4-hk4s6
I1222 18:02:38.653040       7 ipam.go:51] allocate v4 10.16.0.16 v6  mac 00:00:00:0F:9C:A4 for default/nginx-5b4664bcd4-hk4s6 from subnet ovn-default
E1222 18:02:38.659001       7 pod.go:331] error syncing 'default/nginx-5b4664bcd4-hk4s6': map: map[] does not contain declared merge key: name, requeuing
I1222 18:05:22.499650       7 pod.go:319] handle add pod default/nginx-5b4664bcd4-hk4s6
I1222 18:05:22.499856       7 ipam.go:51] allocate v4 10.16.0.16 v6  mac 00:00:00:0F:9C:A4 for default/nginx-5b4664bcd4-hk4s6 from subnet ovn-default
E1222 18:05:22.507852       7 pod.go:331] error syncing 'default/nginx-5b4664bcd4-hk4s6': map: map[] does not contain declared merge key: name, requeuing
I1222 18:06:26.045239       7 gc.go:356] gc logical switch port nginx-5b4664bcd4-hk4s6.default
I1222 18:06:26.045244       7 ovn-nbctl-legacy.go:112] delete lsp nginx-5b4664bcd4-hk4s6.default
I1222 18:06:26.057211       7 subnet.go:460] release v4 10.16.0.16 mac 00:00:00:0F:9C:A4 for default/nginx-5b4664bcd4-hk4s6, add ip to released list
I1222 18:10:50.188109       7 pod.go:319] handle add pod default/nginx-5b4664bcd4-hk4s6
I1222 18:10:50.188184       7 ipam.go:51] allocate v4 10.16.0.23 v6  mac 00:00:00:8C:14:94 for default/nginx-5b4664bcd4-hk4s6 from subnet ovn-default
E1222 18:10:50.203880       7 pod.go:331] error syncing 'default/nginx-5b4664bcd4-hk4s6': map: map[] does not contain declared merge key: name, requeuing
I1222 18:18:26.181381       7 gc.go:356] gc logical switch port nginx-5b4664bcd4-hk4s6.default
I1222 18:18:26.181386       7 ovn-nbctl-legacy.go:112] delete lsp nginx-5b4664bcd4-hk4s6.default
I1222 18:18:26.193591       7 subnet.go:460] release v4 10.16.0.23 mac 00:00:00:8C:14:94 for default/nginx-5b4664bcd4-hk4s6, add ip to released list
I1222 18:21:45.563920       7 pod.go:319] handle add pod default/nginx-5b4664bcd4-hk4s6
I1222 18:21:45.563998       7 ipam.go:51] allocate v4 10.16.0.30 v6  mac 00:00:00:65:A9:91 for default/nginx-5b4664bcd4-hk4s6 from subnet ovn-default
E1222 18:21:45.578536       7 pod.go:331] error syncing 'default/nginx-5b4664bcd4-hk4s6': map: map[] does not contain declared merge key: name, requeuing
I1222 18:30:26.316763       7 gc.go:356] gc logical switch port nginx-5b4664bcd4-hk4s6.default
I1222 18:30:26.316768       7 ovn-nbctl-legacy.go:112] delete lsp nginx-5b4664bcd4-hk4s6.default
I1222 18:30:26.345989       7 subnet.go:460] release v4 10.16.0.30 mac 00:00:00:65:A9:91 for default/nginx-5b4664bcd4-hk4s6, add ip to released list
I1222 18:38:25.578563       7 pod.go:319] handle add pod default/nginx-5b4664bcd4-hk4s6
I1222 18:38:25.578644       7 ipam.go:51] allocate v4 10.16.0.36 v6  mac 00:00:00:03:20:2B for default/nginx-5b4664bcd4-hk4s6 from subnet ovn-default
E1222 18:38:25.593687       7 pod.go:331] error syncing 'default/nginx-5b4664bcd4-hk4s6': map: map[] does not contain declared merge key: name, requeuing
I1222 18:48:26.445566       7 gc.go:356] gc logical switch port nginx-5b4664bcd4-hk4s6.default
I1222 18:48:26.445582       7 ovn-nbctl-legacy.go:112] delete lsp nginx-5b4664bcd4-hk4s6.default
I1222 18:48:26.456952       7 subnet.go:460] release v4 10.16.0.36 mac 00:00:00:03:20:2B for default/nginx-5b4664bcd4-hk4s6, add ip to released list

Steps to Reproduce the Problem

I don't know how to reproduce yet. That's my first try to kube-ovn

Additional Info

• Kubernetes version:

1.26

• kube-ovn version:

v1.11.0

• operation-system/kernel version:

  Output of awk -F '=' '/PRETTY_NAME/ { print $2 }' /etc/os-release:
  Output of uname -r:

Ubuntu 20.04.4 LTS
5.4.0-100-generic

[zhangzujian]

Please attach yaml of the pod/workload.

[oilbeater]

@Shoothzj can you attach the related pod yaml?

[hezhangjian]

I can't attact it now. I will try it again in the future versions. Thanks for your reply. Much appreciated.

[hongzhen-ma]

确认1.9 分支，没有合入使用 GenerateStrategicMergePatchPayload 接口的代码
最初替换该接口的原因是，原因的patch 操作，会导致pod annotation 信息丢失

1.9 替换之前的 patch 操作

修改 1.9 分支代码，使用 GenerateStrategicMergePatchPayload 接口测试，存在报错

使用1.10 分支代码测试，存在同样的问题

报错的原因，就是 GenerateStrategicMergePatchPayload 接口，最终调用的 k8s 接口实现中，对map value 为空的情况进行了校验，不允许这种配置。

作为对比，GenerateMergePatchPayload 是使用 json patch，这个接口经测试没有报错。

[hongzhen-ma]

kubernetes/kubernetes#98963