Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WebSocket connection fails on ALB #2749

Open
vlasov-y opened this issue Jan 14, 2025 · 9 comments
Open

WebSocket connection fails on ALB #2749

vlasov-y opened this issue Jan 14, 2025 · 9 comments
Labels
bug Something isn't working EKS Related to Amazon Elastic Kubernetes Service

Comments

@vlasov-y
Copy link

Describe the bug

Websocket does not work for installation I have in EKS 1.31 (latest stable version at the moment).
I use ALB with stickiness enabled. Generally, headlamp works, and probably I am doing something wrong, but maybe it has issues with AWS?

image

ALB listener configuration goes next.
image

Target group attributes goes next.
image

@vlasov-y vlasov-y added the bug Something isn't working label Jan 14, 2025
@dosubot dosubot bot added the EKS Related to Amazon Elastic Kubernetes Service label Jan 14, 2025
@illume
Copy link
Collaborator

illume commented Jan 14, 2025

Thanks for the report.

I wonder if this is in main branch or another version?

We recently merged a big refactor of websocket handling code into main branch. So I’m wondering if it’s related to these websocket changes.

@vlasov-y
Copy link
Author

Thanks for the report.

I wonder if this is in main branch or another version?

We recently merged a big refactor of websocket handling code into main branch. So I’m wondering if it’s related to these websocket changes.

It happened with a build from my fork which is aligned with a master and make no changes to websockets functionality.
https://github.com/vlasov-y/headlamp/tree/main
I had this issue with a build from 06 Jan and now have with today's build that is up to date with master.

@vlasov-y
Copy link
Author

vlasov-y commented Jan 14, 2025

@illume this is important to mention: I have headlamp installed in RKE2 on-premise installations with Canal CNI and haproxy-ingress - no issues.
image
I am pretty sure, that I am missing something in AWS setup specifically, but cannot understand what exactly.
P.S. Same build from the fork

@illume
Copy link
Collaborator

illume commented Jan 15, 2025

I had this issue with a build from 06 Jan and now have with today's build that is up to date with master.

Ok, so I think these websocket changes were merged after Jan 6. So they are not related to your issue.
#2563

Yeah, I guess there’s some issue with the websocket code with the proxy and web sockets in the load balancer.

The trace back is in production mode which is minimized and without source maps. So it’s hard to see what the issue is. To debug maybe you can step through it in the debugger or make a dev build without source maps (I haven’t done this myself so can’t give instructions on how to do it).

Maybe we should have a mode to disable websockets, which could be a work around?

@illume
Copy link
Collaborator

illume commented Jan 15, 2025

Reading some ALB docs... here's a few things I'd check:

  • I wonder if the Listener configuration is setup for the right port?
  • I don't see a health check for the websocket connections configured... is there one?

@vlasov-y
Copy link
Author

vlasov-y commented Jan 15, 2025

@illume

  • I don't see a health check for the websocket connections configured... is there one?

image

  • I wonder if the Listener configuration is setup for the right port?

I have permanent redirect from 80 to 443 and have 443 HTTPS with a valid certificate. There are other services beside of Headlamp and they work. Headlamp works too, not considering the websocket issue.

@vlasov-y
Copy link
Author

Maybe we should have a mode to disable websockets, which could be a work around?

I think we have to fix websockets, because no workaround is needed. Headlamp work with websocket errors as well.

@illume
Copy link
Collaborator

illume commented Jan 15, 2025

I wonder if adding a https health check would help?

I see it's "wss://" in the error screenshot. Therefore I think ALB needs a https health check to work.

@vlasov-y
Copy link
Author

I wonder if adding a https health check would help?

I see it's "wss://" in the error screenshot. Therefore I think ALB needs a https health check to work.

Health check works fine, it is not related IMHO. Otherwise, I won't be able to open it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working EKS Related to Amazon Elastic Kubernetes Service
Projects
Development

No branches or pull requests

2 participants