Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Live Logs connecting to wrong port on workers #426

Open
rrusso1982 opened this issue Jul 8, 2021 · 5 comments
Open

Live Logs connecting to wrong port on workers #426

rrusso1982 opened this issue Jul 8, 2021 · 5 comments

Comments

@rrusso1982
Copy link

Summary

When attempting to view live logs on worker nodes The system attempts to connect to https://:3012. However 3012 is the http port on the worker and not the https port. This causes live logs to not be streamed to the job as it runs. Logs become available after the job has completed.

Steps to reproduce the problem

Run any job in the system and attempt to view the live logs. Instead of logs you will get messages of the type: Log Watcher: Server Connect Error: Error: websocket error (https://10.75.6.150:3012)

Your Setup

Single master with many workers. The master is behind a load balancer. The LB has an SSL certificate installed to present https to the end user. Master and Workers are configured with http only.

Operating system and version?

Amazon Linux 2

Node.js version?

v14.17.1

Cronicle software version?

0.8.62

Are you using a multi-server setup, or just a single server? Multi

Are you using the filesystem as back-end storage, or S3/Couchbase? S3

Can you reproduce the crash consistently? Yes

Log Excerpts

Log Watcher: Connecting to server: https://10.75.6.150:3012...
Log Watcher: Server Connect Error: Error: websocket error (https://10.75.6.150:3012)
Log Watcher: Server Connect Error: Error: websocket error (https://10.75.6.150:3012)

@justinmdudley
Copy link

Yes! I have this same exact issue! Would love to see this resolved! :)

@mikeTWC1984
Copy link

I'm afraid it's not possible in this configuration. Downgrading websockets from https to http is blocked by design.
Hopefully in the next version live log would go through the master node (not via direct connection).
As a workaround you can rewrite live log function just to use http polling vs websockets, it's actually not that complicated, see my answer from here:
#391 (comment)

@jhuckaby
Copy link
Owner

jhuckaby commented Jul 8, 2021

Hopefully in the next version live log would go through the master node (not via direct connection).

It will 😊, and also HTTPS/SSL will be fully supported 100%.

@rrusso1982
Copy link
Author

I have worked to put together this PR that implements live log proxying through the master. Please take a look and let me know what you think.
PR# 427

Also much thanks to @mikeTWC1984 for the inspiration and base code that I worked from for this.

@muety
Copy link

muety commented Jan 5, 2022

I'd highly appreciate to have this merged!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants