Direct passthru of stdout/stderr from tasks to parent (for Docker)

[BusterNeece]

Hello!

I recently switched over to using Jobber for scheduled task management on my FOSS application, and I'm very happy with the flexibility and ease of configuration it offers. I'm especially a fan of the per-second resolution, as my project depended on several "every x seconds" tasks that crond simply couldn't do. Thank you for your work and contributions to the FOSS community.

I'm not sure if you already plan to cover this in #180, but I can't seem to find any setting that would allow me to basically pipe all the output from tasks (or just the success/fail status) directly out to the jobbermanager's own stdout/stderr. This is important for my Docker-based setup, as currently those logs are only visible by exec'ing into the running container and tailing the files as they're written to disk.

This may be something that the Jobber Docker image already built to do, but unfortunately I couldn't just use that image as the one part of my stack that would use Jobber is still based on Ubuntu 16.04 images instead of Alpine. I'm currently installing the packaged .deb file and manually calling /usr/libexec/jobbermaster as the command Docker runs.

[dshearer]

Thank you for the kind words!

I am happy you brought this up, because I have been wondering how best to do #180. What you're asking for is certainly one way of putting Jobber into a Docker container... but it may not meet everyone's needs.

[dshearer]

Could you provide more details about how you are using Jobber? Are you running one or many jobs in one container? What do the jobs do?

My idea was that the Jobber Docker container would listen on some port for commands --- "list jobs", "add job", "show run log", etc. To see the results of job runs from outside the container, you'd have to send the "show run log" command.

It sounds like you'd prefer "push" style notifications about job runs. Maybe this would work: you connect to the container on some TCP port, and then you'll receive "job status notifications" on that connection. What do you think?

[BusterNeece]

@dshearer Sorry, I meant to get back to this issue sooner rather than later, but it's been a busy week.

I was thinking that since Jobber already has the ability to specify "stdoutHandler" and "stderrHandler", if it would be possible to create, basically, "passthru" handlers for both, that just bounces the stdout/stderr messages sent to it "up the chain" to the task manager, which then outputs them via its own stdout/stderr.

If you like, I would be happy to PR in some relevant code that would accomplish this.

As for your second question, I'm running a number of jobs that do routine maintenance/synchronization tasks behind the scenes for a web application. There are longer-running, less frequent tasks scheduled every hour or 5 minutes, and some tasks that are super fast but need very granular data scheduled every 15 seconds. One single Jobber container manages the whole set of tasks, as a drop-in replacement for a single crond-running container doing the same thing.

You mentioned an API through which you could interact with the Jobber manager and view the status of recent commands and/or control the processes themselves...I think that would be an excellent idea as well, and I'd love to see it. I use a similar project, supervisord, for some other tasks in my application specifically because it has this level of API-level control, so my web application can manage processes without directly needing to spawn them itself.

I think both are probably separate approaches, but they would both make progress toward making Jobber more Docker-friendly.

[dshearer]

I'm not a big fan of the idea of reporting job results in jobberrunner's stdout, because jobberrunner's stdout is currently used to print troubleshooting messages.

What do you think about the idea I described above, of reporting job results on a dedicated socket?

[BusterNeece]

@dshearer I don't think the passthru option should be on by default or anything, but as dirty as it may seem, this is really the best way to work with Docker's built-in log management features. If I can see the full diagnostic report of jobber tasks just by running docker-compose logs -f jobber, that makes it predictable and exactly the same as every other Docker container I use, rather than having to invent some sort of new interaction style and remember it specifically for this one container.

If you were to set up an API, while a socket would be fine, I'd be looking for a little mini-REST API that could respond to HTTP requests with job status information.

Even then, these are two different solutions that help in two different ways. The API would let me integrate notifications and error checking directly into my application (which would be nice), but if I'm actually sitting there on the SSH terminal on a machine, I want to be able to get at the raw results of task execution via docker-compose logs -f the same as I can, say, nginx or mariadb or influxdb or anything else.

[dshearer]

Okay, I think I can add this passthru option.

As for the API, I'd like to use JSON-RPC 1.0. What do you think about that?

[dshearer]

PR #188

[BusterNeece]

@dshearer Thank you for resolving this issue and merging updates to the application.

Unfortunately, I'm not clear from the documentation how one would go about just piping the stdout/stderr results from the jobber tasks themselves out to the parent stdout/stderr of the jobbermaster process, which is largely what the subject matter of this request was: being able to see a rolling log of the jobber tasks' output via docker-compose logs -f cron_service or similar.

[BusterNeece]

@dshearer I've been able to extrapolate from the unit tests what a possible jobber.yml file may look like:

version: 1.4

prefs:
  runLog:
    type: memory
    maxLen: 500

resultSinks:
  - &stdoutSink
    type: stdout
    data: [stdout, stderr]

jobs:
- name: LongSync
  cmd: php foo.php
  time: 0 0 * * * *
  notifyOnSuccess: [*stdoutSink]
  notifyOnError: [*stdoutSink]

...the only problem is that the stdout/stderr of each jobberrunner doesn't appear to 'bubble up' through the jobbermaster, and I'm still not seeing any input on the master Docker log.

I'm not sure how to run an individual jobberrunner by itself (it doesn't seem to be a binary I can individually call) though.

Any insights on this? Thank you very much for your help.

[BusterNeece]

@dshearer I think I've figured out the problem here...the 1.3.4 release that's tagged as the latest release doesn't appear to have the new "result sink" features (or the new YAML parsing) in it.

Sorry I'm having so much trouble crawling through the documentation myself. I'm also not seeing a simple way of building the application binary from scratch in order to take advantage of the master branch features. Is there a file with a straightforward guide to building from source?

[dshearer]

Yeah, these features are only in master currently.

In Docker, the best way to run jobber is just by running a jobberrunner process. To compile from scratch, follow the instructions here, but skip the git checkout v1.3.3 and sudo make install parts. You can then find jobberrunner in /path/to/your/workspace/bin/jobberrunner.

[BusterNeece]

@dshearer Thank you! Got it all figured out and working, now running the jobberrunner directly and dumping the output directly into the Docker logs.

Quick question: is there any way to turn down the verbosity of the jobberrunner task, so it doesn't fill up the stdout log? I'm talking about this stuff:

cron_1         | It is now Oct 15 21:18:30, which is NOT before Oct 15 21:18:30
cron_1         | Running SyncName
cron_1         | sync-command
cron_1         | Next job to run is SyncName, at Oct 15 21:18:45.
cron_1         | It is now Oct 15 21:18:30.
cron_1         | Sleeping for 14.9907878s.

[dshearer]

There is not right now. I really do need to fix that. Those were added to troubleshoot this horrible Golang time bug.

…

On Oct 15, 2018, at 2:21 PM, SlvrEagle23 ***@***.***> wrote: @dshearer Thank you! Got it all figured out and working, now running the jobberrunner directly and dumping the output directly into the Docker logs. Quick question: is there any way to turn down the verbosity of the jobberrunner task, so it doesn't fill up the stdout log? I'm talking about this stuff: cron_1 | It is now Oct 15 21:18:30, which is NOT before Oct 15 21:18:30 cron_1 | Running SyncName cron_1 | sync-command cron_1 | Next job to run is SyncName, at Oct 15 21:18:45. cron_1 | It is now Oct 15 21:18:30. cron_1 | Sleeping for 14.9907878s. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[BusterNeece]

@dshearer Should I make a new issue for that?

[dshearer]

Sure

…

On Oct 15, 2018, at 4:54 PM, SlvrEagle23 ***@***.***> wrote: @dshearer Should I make a new issue for that? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.