Allow websockets to connect on same hostname / port as reverse proxy

[alex-phillips]

Summary

Feature request: Allow websockets to use the reverse proxy hostname / port

Steps to reproduce the problem

Your Setup

I'm running traefik as a reverse proxy which supports websocket upgrades to the same hostname / port HTTP(s) requests are on. Can support be added so the websockets (for live logs) connect to the hostname / port the UI is running on?

Operating system and version?

Ubuntu 20.04

Node.js version?

14.16.0

Cronicle software version?

0.8.56

Are you using a multi-server setup, or just a single server?

Single

Are you using the filesystem as back-end storage, or S3/Couchbase?

Filesystem

[mikeTWC1984]

I believe single node should work fine under reverse proxy if you are using same protocol on both sides. But if Cronicle is http but proxy is https it won't work because websockets do not allow downgrade.
There are some cases when it's tricky/imposible to configure live log. The only ultimate solution for that would be centralized api (rather than direct ws connection), which I think was planned for v2.0. I managed to create a workaround, using dead simple api that polls last N lines from job log file. Surprisingly it's quite simple update, if you are willing to update source code I can explain how to achieve that.

[alex-phillips]

@mikeTWC1984 Sure, happy to give it a try.

[krazyjakee]

I have the exact same setup/issue and am also interested in trying this out.

[mikeTWC1984]

OK, here is the steps you need to take.
General idea:
Backend: create 2 new api endpoints - one to read last N lines from log file on the node where job is running (internal api) and another one on manager node, that invoke the first api as per client request (needs jobid and proper session).
Frontend: redefine start_live_log_watcher to poll above api (instead of using direct ws connection). So on UI you only see those last N lines

1. Install read-last-lines module (npm i read-last-lines).
2. Open htdocs/lib/api/jobs.js file.
   add this at the top:

const readLastLines = require('read-last-lines');

then add new api definitions (in the same file):

	api_get_live_console: async function (args, callback) {
		// runs on manager 
		const self = this;

		self.loadSession(args, function (err, session, user) {
			if (err) return self.doError('session', err.message, callback);
			if (!self.requireValidUser(session, user, callback)) return;

			//let query = args.query;
			let params = Tools.mergeHashes(args.params, args.query);

			if (!self.requireParams(params, {
				id: /^\w+$/
			}, callback)) return;

			let activeJobs = self.getAllActiveJobs(true)

			let job = activeJobs[params.id]

			if (!job) return self.doError('job', "Invalid or Completed job", callback);

			//self.server.config.get('WebServer').http_port

			let port = self.server.config.get('WebServer').http_port;
			let tailUrl = `http://${job.hostname}:${port}/api/app/get_live_log_tail` //?id=${job.id}
			let tailSize = parseInt(params.tail) || 80;
			let auth = Tools.digestHex(params.id + self.server.config.get('secret_key'))
			let reqParams = { id: job.id, tail: tailSize, download: params.download || 0, auth: auth }

			self.request.json(tailUrl, reqParams, (err, resp, data) => {
				if (err) return self.doError('job', "Failed to fetch live job log: " + err.message, callback);
				data.hostname = job.hostname;
				data.event_title = job.event_title;
				callback(data);
			});
		});
	},

	api_get_live_log_tail: function (args, callback) {
		// internal api,  runs on target machine
		const self = this;

		let params = Tools.mergeHashes(args.params, args.query);

		if (!this.requireParams(params, {
			id: /^\w+$/,
			auth: /^\w+$/
		}, callback)) return;

		if (params.auth != Tools.digestHex(params.id + self.server.config.get('secret_key'))) {
			return callback("403 Forbidden", {}, "Authentication failure.\n");
		}

		// see if log file exists on this server
		var log_file = self.server.config.get('log_dir') + '/jobs/' + params.id + '.log';

		let tailSize = parseInt(params.tail) || 80;
		if (params.download == 1) { // read entire file
			fs.readFile(log_file, { encoding: 'utf-8' }, (err, data) => {
				if (err) return self.doError('job', "Failed to fetch job log: invalid or completed job", callback);
				callback({ data: data });
			});

		}
		else {
			readLastLines.read(log_file, tailSize )
			.then( lines => callback({data: lines}))
			.catch(e => { return self.doError('job', "Failed to fetch job log: invalid or completed job", callback)})
		}
	},

note: if you want you can use "tail" shell command (using fs/childProcess) instead of read-last-lines. I used it for POC.

3. Now locate start_live_log_watcher function. In the source code (before cronicle install) it should be updated in htdocs/js/pages/JobDetails.class.js, after installation in _combo.js file

	start_live_log_watcher: function (job) {
		let self = this;
		self.curr_live_log_job = job.id;

		//let ansi_up = new AnsiUp;
		webConsole = document.getElementById('d_live_job_log');
		// poll live_console api until job is running or some error occur
		function refresh() {
			if(self.curr_live_log_job != job.id) return; // prevent double logging
			app.api.post('/api/app/get_live_console', { id: job.id, tail: parseInt(app.config.ui.live_log_tail_size) || 80 }
				, (data) => {  // success callback
					if (!data.data) return; // stop polling if no data
					// webConsole.innerHTML = `<pre>${ansi_up.ansi_to_html(data.data.replace(/\u001B=/g, ''))} </pre>`;
                                        webConsole.innerHTML = `<pre>${data.data} </pre>`;
					pollInterval = parseInt(app.config.ui.live_log)
					if(!pollInterval || pollInterval < 1000) pollInterval = 1000;
					setTimeout(refresh,  1000);
				}
				// stop polling on error, report unexpected errors
				, (e) => {			
					if(e.code != 'job') console.error('Live log poll error: ', e)
					return
				}
			)
		}

		refresh();

	},

I think that should be pretty much it. I implemented this some time ago in my experimental fork, you can try it out for a quick test: https://github.com/cronicle-edge/cronicle-edge

[alex-phillips]

@mikeTWC1984 Nice, I'll have to check out your fork. Is this a drop-in replacement? How would I go about migrating my existing cronicle installation to your edge fork? And if for some reason I wanted to go back, would the process be the same?

Also, you mention that it looks to add features that are in Orchestra (the upcoming v2 of Cronicle). It doesn't look like there's much info on Orchestra yet, I imagine this will still be open source and available for homelab users such as myself.

[mikeTWC1984]

Yes, the fork is backwards compatible (obviously new features won't work if you port them back).
You can search for "Orchestra" in issues of this repo for more info. I believe it will opensource too.

[alex-phillips]

@mikeTWC1984 How would I go about moving to cronicle-edge? What files to I need to keep / migrate to this new repo checkout? (hoping to preserve as much as possible)

[mikeTWC1984]

You can use output of "storage-cli.js export" command to import events/schedules/categories/users, you can do it from UI (import button). To carry over run history and job details, I think you can copy data/logs and data/jobs to the new data folder location (assuming you are using file system storage)

[krazyjakee]

I've switched to cronicle-edge but sadly I get stuck waiting for master server out of the box.

[opswhisperer]

Any updates on this?

[jhuckaby]

@opswhisperer This change would be too difficult to retrofit onto Cronicle v0 (although somehow @mikeTWC1984 did it in his fork -- he's some kind of wizard tho). I have completely redesigned the way sockets work in Cronicle v2 (Orchestra) so that the UI, by design, only ever makes one single websocket connection to the main server, and live logs go through that connection.

Orchestra development continues, but life keeps getting in the way. I don't think I'll make my "late 2023" estimate. My new estimate is early 2024 for release. Sorry for the delay on this.

[opswhisperer]

Sounds good, this is not critical path for my use case so I can wait. Appreciate the project 💯 . Oddly I found some reference to custom_live_log_socket_url but it didnt help.