Argo workflows server not able to port forward using auth=server

[piedrahitapablo]

Describe the bug

I’m using kind to create a local cluster so I can test everything easier. If I install Argo Workflows using the steps in the quick start (using kubectl apply) everything works as expected, but if I try to do it using the helm chart I’m not able to access the workflows dashboard using port forwarding.

This is what I’m doing:

To create the cluster and install argo-workflows:

kind create cluster

kubectl create namespace argo-events

helm install argo-workflows argo/argo-workflows \
  -n argo-workflows \
  --create-namespace \
  --set server.replicas=1 \
  --set-json 'server.extraArgs=["--auth-mode=server"]' \
  --set singleNamespace=false \
  --set-json 'controller.workflowNamespaces=["argo-events","argo-workflows"]'

After everything is installed, I run:

kubectl port-forward deployment/argo-workflows-server 2746:2746 -n argo-workflows

And when I go to https://localhost:2746/ I see this in my console:

Handling connection for 2746
E0714 10:08:42.034383    5447 portforward.go:406] an error occurred forwarding 2746 -> 2746: error forwarding port 2746 to pod f5a3066cfb1b060465ac151546af01cb4e7f9a47656eb3244225f6a09e7b4d46, uid : failed to execute portforward in network namespace "/var/run/netns/cni-9f2cbdca-2348-e6f4-0536-9af24019a96a": read tcp4 127.0.0.1:60790->127.0.0.1:2746: read: connection reset by peer
E0714 10:08:42.037543    5447 portforward.go:234] lost connection to pod

And these are the logs for the deployment:

time="2023-07-14T15:08:23.973Z" level=info msg="not enabling pprof debug endpoints"
time="2023-07-14T15:08:23.974Z" level=info authModes="[server]" baseHRef=/ managedNamespace= namespace=argo-workflows secure=false ssoNamespace=argo-workflows
time="2023-07-14T15:08:23.974Z" level=warning msg="You are running in insecure mode. Learn how to enable transport layer security: https://argoproj.github.io/argo-workflows/tls/"
time="2023-07-14T15:08:23.974Z" level=warning msg="You are running without client authentication. Learn how to enable client authentication: https://argoproj.github.io/argo-workflows/argo-server-auth-mode/"
time="2023-07-14T15:08:23.974Z" level=info msg="SSO disabled"
time="2023-07-14T15:08:23.980Z" level=info msg="Starting Argo Server" instanceID= version=v3.4.8
time="2023-07-14T15:08:23.980Z" level=info msg="Creating event controller" asyncDispatch=false operationQueueSize=16 workerCount=4
time="2023-07-14T15:08:23.982Z" level=info msg="GRPC Server Max Message Size, MaxGRPCMessageSize, is set" GRPC_MESSAGE_SIZE=104857600
time="2023-07-14T15:08:23.982Z" level=info msg="Argo Server started successfully on http://localhost:2746/" url="http://localhost:2746/"
time="2023-07-14T15:08:34.773Z" level=info duration="337.333µs" method=GET path=index.html size=473 status=0
time="2023-07-14T15:08:34.774Z" level=info duration="137.75µs" method=GET path=index.html size=473 status=0
time="2023-07-14T15:08:54.768Z" level=info duration="295.292µs" method=GET path=index.html size=473 status=0

I tried checking the pods, but everything seems to be running fine. Finally, sorry if this is not a bug but an error on my end, I already tried asking on the argo-workflows slack channel but had no luck.

Thanks in advance, let me know if I can provide more information on this issue.

Related helm chart

argo-workflows

Helm chart version

0.30.0

To Reproduce

1. kind create cluster --name tinybeans
   

2. kubectl create namespace argo-events
   

3. helm install argo-workflows argo/argo-workflows \
     -n argo-workflows \
     --create-namespace \
     --set server.replicas=1 \
     --set-json 'server.extraArgs=["--auth-mode=server"]' \
     --set singleNamespace=false \
     --set-json 'controller.workflowNamespaces=["argo-events","argo-workflows"]'
   

4. kubectl port-forward deployment/argo-workflows-server 2746:2746 -n argo-workflows
   

5. Go to https://localhost:2746

Expected behavior

Be able to access the workflows dashboard using port forwarding and server auth

Screenshots

No response

Additional context

Not sure if related to #2159

[ghostheory]

I am also running into this exact same error while attempting port-forward of port 2746. I have stood up a local Argo Workflows deployment via helm (as per the quick start guide on the Argo Workflows site). I have attempted this in both KinD and in K3d with the same results. My local setup is Ubuntu in WSL2.

I have tried port-forward directly referencing the pod and the service, and I think also the deployment--all with the same error message that happens only once I actually try to hit localhost:2746 in the browser.

I suspect that it has something to do with the container/pod per I cannot generate IT to the pods or kubectl exec any commands at all.

To sanity check I am able to port-forward other services (local dev of Hashicorp Vault for example) just fine.

Any help would be absolutely amazing here as this would unblock me to continue learning Argo Workflows so that I can help implement it here at my place of employment. Thanks.

[ghostheory]

Of course... to follow up here for anyone else who comes across this. I was able to get this to work when I ran:

kubectl -n argo port-forward service/argo-server 2746:2746

And then in browser hit the endpoint specifically with https:

https://localhost:2746

And reviewing the quick start page it calls this out as well 🤦🏻‍♂️

Pay close attention to the URI. It uses https and not http. Navigating to http://localhost:2746 result in server-side error that breaks the port-forwarding.

So my issue was not the same as the OP. I am just an assumptive idiot who didn't read the doc.

[yu-croco]

And when I go to https://localhost:2746/ I see this in my console:

I think http://localhost:2746/ works well (it worked in my local with kind cluster).

I wonder upstream's info is wrong in the latest version, or there are something different between upstream's manifest and Helm Chart.

Since it looks that it's not related to Helm Chart, I think you can get more help in upstream.

[jmeridth]

I've confirmed what @yu-croco said. Use http://localhost:2746 and it works. Upstream is using straight kubectl and not helm install. I'm not sure of why there is a difference but it works. IF I figure it out, I'll update here. Closing for now.