failed to start eventsource server: timed out waiting for the condition #1351
Comments
|
What's the difference between your testing container and the github eventsource? Different namespace? Are they both with the Istio sidecar? |
|
This is a shot in the dark, but is it possible that the Istio Sidecar is not ready? (My go-to question with all problems Istio...) In my applications I set the
The documentation suggests this can be enabled on a per-pod basis with the annotation, but I have not tested: annotations:
proxy.istio.io/config: |
holdApplicationUntilProxyStarts: false |
both are with istio-sidecar. No difference at all. They are in same namespace.
it is ready. In fact I was able to execute curl from the sidecar of failing container: |
|
Another shot in the dark but, what about a rollout redeployment of the eventbus service? |
|
It didn't made any difference. |
|
@antoniomo - does it work in your case? |
|
Yep, we have a bunch of argo and argo-sensor installations working with Istio with a similar change. Our argo version is 3.0.8, and argo-events is at 1.4.0. We make use of the SQS and Webhook event sources. We only have Istio on the webhook ones, for ingress (virtual service) purposes. Some more information on our setup, excerpts from the manifests coming. On the SQS event-sources we disable it: apiVersion: argoproj.io/v1alpha1
kind: EventSource
metadata:
name: sqs
namespace: foobar
spec:
template:
serviceAccountName: foobar
metadata:
annotations:
sidecar.istio.io/inject: "false"
sqs:
...The eventbus service and the event-sources sit on the same namespace, as we are using namespaced argo and argo-events installations, which forces this setup. The namespace is istio-enabled: labels:
istio-injection: enabledThe eventbus service: apiVersion: v1
kind: Service
metadata:
labels:
controller: eventbus-controller
eventbus-name: default
owner-name: default
stan: "yes"
name: eventbus-default-stan-svc
namespace: foobar
spec:
clusterIP: None
ports:
- name: tcp-client
port: 4222
protocol: TCP
targetPort: 4222
- name: cluster
port: 6222
protocol: TCP
targetPort: 6222
- name: monitor
port: 8222
protocol: TCP
targetPort: 8222
selector:
controller: eventbus-controller
eventbus-name: default
owner-name: default
sessionAffinity: None
type: ClusterIPNot sure if any of that might help. |
|
@antoniomo does your eventbus pods are started with envoy injected? |
Actually nope, I should have given our manifest instead of describe from the service! Here it is: apiVersion: argoproj.io/v1alpha1
kind: EventBus
metadata:
name: default
spec:
nats:
native:
metadata:
annotations:
sidecar.istio.io/inject: "false"
replicas: 3 # optional, defaults to 3, and requires minimal 3
auth: token # optional, default to none
persistence: # optional
# storageClassName: standard # Default of the cluster
accessMode: ReadWriteOnce
volumeSize: 10Gi
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchLabels:
controller: eventbus-controller
eventbus-name: default
topologyKey: kubernetes.io/hostname
weight: 100 # Best-effort anti-affinityWe only have the istio sidecar in the webhook event source (as we need a virtual service there for ingress purposes). |
|
Can you check if it works when you have istio sidecar enabled? |
|
Not during the week, this is a production setup, I would have to replicate locally. However, we don't have istio anywhere where it is not required as it consumes quite a bit of resources. Hence, we only run it in the webhook event source, as explained. The only reason why that event source is in the same namespace as the eventbus is due to a namespaced argo-events installation, that forces us to use this setup. |
|
Can you test with/without the annotation to disable istio sidecar on the event bus, while keeping istio in your event source? |
|
Yes, without istio sidecar in event bus it's connecting fine. Unfortunately in our organization istio is a requirement :( |
|
Unfortunately #1311 won't fix it for you then, the Basically that enables istio/istio#28623, that is, calling NATS (the EventBus) from an Istio-enabled namespace/from your event sources. But that doesn't enable NATS itself (the EventBus) to have an istio proxy.
Unfortunately, NATS and Istio don't play so well together, that's just one of several threads about it :/ |
|
Issue was caused by a destinationrule that was disabling mTLS. |
|
Awesome! I'm sure this thread will help more people in the future, as well! |
Describe the bug
I'm unable to start github-eventsource pod on a Kubernetes with istio enabled.
Environment (please complete the following information):
Additional context
Originally I was unable to access eventbus-default-stan service, but as suggested in #1311 I updated the service to:
Now I'm able to access it using curl from testing container:
but github-eventsource is still unable to start:
Can you please help me with that?
Message from the maintainers:
If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.
The text was updated successfully, but these errors were encountered: