Finalizer for KafkaChannel throws "no topic annotated on channel"

[mgencur]

Describe the bug
When a KafkaChannel is created with Knative version older than 1.10 and after upgrade to 1.10 the channel or namespace is deleted quickly the reconciler hangs and can't delete the KafkaChannel.

The KafkaChanel is missing annotations in status such as this one because the reconciler didn't have enough time to add it.

status:
  annotations:
    default.topic: knative-messaging-kafka.test-jvbjbuag.channel-0000-bvaktsis 

As a result, this error is repeatedly thrown in kafka-controller here

{"level":"error","ts":"2023-08-10T22:38:23.841Z","logger":"kafka-broker-controller","caller":"kafkachannel/reconciler.go:303","msg":"Returned an error","commit":"bf8f4e6-dirty","knative.dev/pod":"kafka-controller-566c979d5f-pb5jj","knative.dev/controller":"knative.dev.eventing-kafka-broker.control-plane.pkg.reconciler.channel.Reconciler","knative.dev/kind":"messaging.knative.dev.KafkaChannel","knative.dev/traceid":"328ee1ee-b430-4600-8406-d0c34ef068f0","knative.dev/key":"test-pmkfshbk/kcksvcksvcn","targetMethod":"FinalizeKind","error":"no topic annotated on channel","stacktrace":"knative.dev/eventing-kafka-broker/control-plane/pkg/client/injection/reconciler/messaging/v1beta1/kafkachannel.(*reconcilerImpl).Reconcile\n\t/go/src/github.com/openshift/origin/control-plane/pkg/client/injection/reconciler/messaging/v1beta1/kafkachannel/reconciler.go:303\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\t/go/src/github.com/openshift/origin/vendor/knative.dev/pkg/controller/controller.go:542\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\t/go/src/github.com/openshift/origin/vendor/knative.dev/pkg/controller/controller.go:491"}

When the KafkaChanel is marked for deletion the reconciler doesn't try to reconcile the kind anymore and wants to finalize/delete it. So it will never add the annotations in status and will also hang on deleting the resource due to the error. These annotations are new in 1.10.

Expected behavior
When KafkaChannel is created before 1.10, I can upgrade to 1.10 and delete the namespace at any time. The resource should not hang there.

To Reproduce

• Install KafkaChannel with Knative 1.9
• Upgrade to 1.10
• Quickly delete the namespace with the resource (kubectl delete ns ZYZ)

Knative release version
1.10

Additional context
Add any other context about the problem here such as proposed priority

[Cali0707]

/assign

[Cali0707]

/triage accepted
/area control-plane

[pierDipi]

If the annotation is not present we should fallback to "guess the topic name"