Updating ConfigMap always fail due to "_example" modified error

[nak3]

Description

• When updating config-map, it always fail with error validation failed: "_example" modified, you likely wanted to create an unindented configuration.
• I know Validate _examples block and warn the user if they changed it seemingly by accident. #8123 validates the _example field, but it fails even though non-_example block field with correct indent.
• The e2e script updates the config-map but passed, so it looks like my env specific?

In what area(s)?

/area API

What version of Knative?

Build on top of 4aedf5f

Expected Behavior

$ kubectl patch configmap config-network -n knative-serving -p '{"data":{"autoTLS":"Disabled"}}'

should work.

Actual Behavior

$ kubectl patch configmap config-network -n knative-serving -p '{"data":{"autoTLS":"Disabled"}}'
Error from server (BadRequest): admission webhook "config.webhook.istio.networking.internal.knative.dev" denied the request: validation failed: "_example" modified, you likely wanted to create an unindented configuration

Steps to Reproduce the Problem

• Update configmap like just run patch command:

$ kubectl patch configmap config-network -n knative-serving -p '{"data":{"autoTLS":"Disabled"}}'

or

$ kubectl edit configmap config-network -n knative-serving 

and add a new config in non-_example block field with correct indent.

[vagababov]

I reproduce it easily.
I think there's difference in the two strings the parsers generate (because those are two different YAML parsers). I think a few spaces here and there or a \n is all that matters.

[vagababov]

Ok, found it
/assign

[markusthoemmes]

Ugh, sorry for missing this. My first version checked for this but I lost that in rewriting it to a go tool. My bad 👼

[vagababov]

Mainly I think it's because we're using two different yaml parsers :| which is always inviting trouble in subtle differences :)

[munjalpatel]

Hello,

I am running into this issue while trying to configure the domain as described here:
https://knative.dev/development/install/installing-istio/

$ kubectl patch configmap/config-domain \
        -n knative-serving \
        --type merge -p '{"data":{"localhost.xip.io":""}}'

Error from server (BadRequest): admission webhook
"config.webhook.istio.networking.internal.knative.dev" denied the request: 
validation failed:  "_example" modified, you likely wanted to create an 
unindented configuration

How do I proceed forward?

[vagababov]

Did you update your serving?

[munjalpatel]

I am using the vision 0.15 as documented on the knative docs site. Since this is fairly new, I am sure I am not using the latest version. Which version should I be upgrading to?

[zzvara]

The same problem appears on knative 1.2.0:

{"level":"info","ts":"2022-03-15T08:31:00.443Z","logger":"controller.kustomization","msg":"Starting workers","reconciler group":"kustomize.toolkit.fluxcd.io","reconciler kind":"Kustomization","worker count":4}
{"level":"info","ts":"2022-03-15T08:31:10.443Z","logger":"controller.kustomization","msg":"server-side apply completed","reconciler group":"kustomize.toolkit.fluxcd.io","reconciler kind":"Kustomization","name":"flux-system","namespace":"flux-system","output":{"CustomResourceDefinition/alerts.notification.toolkit.fluxcd.io":"unchanged","CustomResourceDefinition/buckets.source.toolkit.fluxcd.io":"unchanged","CustomResourceDefinition/gitrepositories.source.toolkit.fluxcd.io":"unchanged","CustomResourceDefinition/helmcharts.source.toolkit.fluxcd.io":"unchanged","CustomResourceDefinition/helmreleases.helm.toolkit.fluxcd.io":"unchanged","CustomResourceDefinition/helmrepositories.source.toolkit.fluxcd.io":"unchanged","CustomResourceDefinition/imagepolicies.image.toolkit.fluxcd.io":"unchanged","CustomResourceDefinition/imagerepositories.image.toolkit.fluxcd.io":"unchanged","CustomResourceDefinition/imageupdateautomations.image.toolkit.fluxcd.io":"unchanged","CustomResourceDefinition/knativeeventings.operator.knative.dev":"unchanged","CustomResourceDefinition/knativeservings.operator.knative.dev":"unchanged","CustomResourceDefinition/kustomizations.kustomize.toolkit.fluxcd.io":"unchanged","CustomResourceDefinition/providers.notification.toolkit.fluxcd.io":"unchanged","CustomResourceDefinition/receivers.notification.toolkit.fluxcd.io":"unchanged","Namespace/flux-system":"unchanged"}}
{"level":"error","ts":"2022-03-15T08:31:11.872Z","logger":"controller.kustomization","msg":"Reconciliation failed after 11.42867209s, next try in 10m0s","reconciler group":"kustomize.toolkit.fluxcd.io","reconciler kind":"Kustomization","name":"flux-system","namespace":"flux-system","revision":"master/2a7ae64191fa4e30882097f46d3c04f5b294b156","error":"ConfigMap/system/config-logging dry-run failed, reason: BadRequest, error: admission webhook \"config.webhook.serving.knative.dev\" denied the request: validation failed: the update modifies a key in \"_example\" which is probably not what you want. Instead, copy the respective setting to the top-level of the ConfigMap, directly below \"data\"\n","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/internal/controller/controller.go:227"}
{"level":"debug","ts":"2022-03-15T08:31:11.872Z","logger":"events","msg":"Warning","object":{"kind":"Kustomization","namespace":"flux-system","name":"flux-system","uid":"b36d31d3-66b3-4545-a9bc-b67a574df99d","apiVersion":"kustomize.toolkit.fluxcd.io/v1beta2","resourceVersion":"11670096"},"reason":"error","message":"ConfigMap/system/config-logging dry-run failed, reason: BadRequest, error: admission webhook \"config.webhook.serving.knative.dev\" denied the request: validation failed: the update modifies a key in \"_example\" which is probably not what you want. Instead, copy the respective setting to the top-level of the ConfigMap, directly below \"data\"\n"}

[zzvara]

The same problem appears with knative 1.3.1 installed as operator.

[dfsdevops]

I am on 1.7.2 and I can't figure out how to get around this issue. Whats the actual resolution here? I am trying to modify the ingress class and I cannot, not even if by deleting and recreating from the configMap. I'm not sure what it thinks I changed.

EDIT: ok, turns out a few weeks ago when I was messing with this I must have modified the initial config, and when I went to apply it from scratch it created the configMap before it created the validating webhook, therefore it was already "modified" from the example but I thought it was only my newer modifications that were breaking thing. I pulled the right example block from the upstream and fixed the issue.

[gabo1208]

For anyone having this issue, this snippet can help yo redo your example checksum in case you need it:

import (
	"fmt"
	"hash/crc32"
	"os"
	"regexp"
	"strings"

	"gopkg.in/yaml.v2"
)

var (
	// Allows for normalizing by collapsing newlines.
	sequentialNewlines = regexp.MustCompile("(?:\r?\n)+")
)

type ConfigMap struct {
	Data map[string]string `json:"data,omitempty" protobuf:"bytes,2,rep,name=data"`
}

func Checksum(value string) string {
	return fmt.Sprintf("%08x", crc32.ChecksumIEEE([]byte(sequentialNewlines.ReplaceAllString(strings.TrimSpace(value), `\n`))))
}

func main() {
	contents, _ := os.ReadFile("./config/config-leader-election.yaml")
	var cm ConfigMap
	if err := yaml.Unmarshal(contents, &cm); err != nil {
		fmt.Println("cannot decode incoming new object:")
	}

	exampleData := cm.Data["_example"]
	chks := Checksum(exampleData)
	exampleChecksum := "a2f53e10"
	fmt.Println(chks, exampleChecksum)
}