Knative Eventing requires CNI mode set to true

[lionelvillard]

Describe the change you'd like to see
When installing Knative Eventing, the prerequisite section should mention that the CNI mode should be set to true.

Additional context
Add any other context or screenshots about the feature request here.

knative/eventing#2039

[abrennan89]

@lionelvillard it looks like this is a specific requirement for minikube? In that case, I'm not sure it belongs in our core docs.

[Shashankft9]

I had to enable hairpin mode in the CNI (kube-router) I am using in my k8s cluster. Hairpin mode was specifically required for using Sequence, where the imc-dispatcher pod was trying to connect to itself through the clusterIP of http://sequence-kn-sequence-1-kn-channel.serverless.svc.cluster.local".
I am not sure how the behavior is in the other CNIs, but yes for kube-router, I had to exclusively turn this Hairpin mode on, otherwise following error would be there in the imc-dispatcher logs:

{"level":"error","ts":"2021-06-16T12:55:30.242Z","logger":"inmemorychannel-dispatcher","caller":"fanout/fanout_message_handler.go:282","msg":"Fanout had an error","commit":"39d1977","knative.dev/pod":"imc-dispatcher-7d469f8459-pqz8g","knative.dev/controller":"knative.dev.eventing.pkg.reconciler.inmemorychannel.dispatcher.Reconciler","knative.dev/kind":"messaging.knative.dev.InMemoryChannel","knative.dev/traceid":"bece0523-af75-4881-869d-b8a1b380ca0a","knative.dev/key":"serverless/sequence-kn-sequence-0","error":"failed to forward reply to http://sequence-kn-sequence-1-kn-channel.serverless.svc.cluster.local: Post \"http://sequence-kn-sequence-1-kn-channel.serverless.svc.cluster.local\": dial tcp 10.244.39.43:80: i/o timeout","stacktrace":"knative.dev/eventing/pkg/channel/fanout.(*FanoutMessageHandler).dispatch\n\tknative.dev/eventing/pkg/channel/fanout/fanout_message_handler.go:282\nknative.dev/eventing/pkg/channel/fanout.createMessageReceiverFunction.func1.1\n\tknative.dev/eventing/pkg/channel/fanout/fanout_message_handler.go:192"}

[snneji]

/priority high

[abrennan89]

I think more info is required here, since I could only find the comment about fixing it for minikube: kubernetes/minikube#1568 (comment)

@lionelvillard can you please provide an explanation of what CNI is, why it needs to be set, and how to set it? Thanks.

[lionelvillard]

I'm not a networking guy. Maybe @markusthoemmes can help?

[csantanapr]

I had to enable hairpin mode in the CNI (kube-router) I am using in my k8s cluster. Hairpin mode was specifically required for using Sequence, where the imc-dispatcher pod was trying to connect to itself through the clusterIP of http://sequence-kn-sequence-1-kn-channel.serverless.svc.cluster.local".

@lionelvillard Why is the dispatcher need to connect to itself in the first place? Are all dispatchers required this?

[abrennan89]

cc @nak3 can you help with this one maybe?

[nak3]

I think we should specifically ask users to enable hairpin traffic rather than CNI. And the setting for the hairpin traffic depends on their Cluster/CNI.

So, I'd add the following sentence to the eventing installation docs:

Some eventing components need "hairpin" traffic. Please verify that your pod can reach itself via the service IP.
If the hairpin traffic is not available, It is a cluster level (typically CNI) setting so you can reach out to your cluster administrator.

For "Some eventing components", we can replace it with imc-dispatcher if we know only imc-dispatcher needs it as @csantanapr asked 😸

@abrennan89 @lionelvillard @Shashankft9 What do you think?

[Shashankft9]

@csantanapr its mostly related to how sequences work, so in the case of sequence following the example here: https://knative.dev/docs/eventing/flows/sequence/sequence-reply-to-event-display/, if I create a sequence and check the services created:

[root@k8s-master01 sequence]# kubectl get svc -n serverless | grep sequence
sequence-kn-sequence-0-kn-channel   ExternalName   <none>          kafka-ch-dispatcher.knative-eventing.svc.cluster.local   <none>                                       2m34s
sequence-kn-sequence-1-kn-channel   ExternalName   <none>          kafka-ch-dispatcher.knative-eventing.svc.cluster.local   <none>                                       2m34s
sequence-kn-sequence-2-kn-channel   ExternalName   <none>          kafka-ch-dispatcher.knative-eventing.svc.cluster.local   <none>                                       2m33s

so the dispatcher runs the first sink in the sequence, then for the second sink it tries to connect to sequence-kn-sequence-1-kn-channel which basically is mapped to its own service and similarly for the third - and for this particular thing I needed to turn on the hairpin mode (a pod connects to itself). You can read more about this here (I was using imc dispatcher that time): https://knative.slack.com/archives/C017X0PFC0P/p1623848192086100

So what @nak3 sugggested for the installation docs makes sense, or particularly add this in the sequence docs. I am not aware if this could be required anywhere else, but atleast putting this in sequence would be a good start.

[github-actions]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.