[JENKINS-38381] Pipeline remote build logging

[jglick]

JENKINS-38381

Implements an SPI in jenkinsci/workflow-job-plugin#27 to provide remote logging of builds.

Mostly the work of others; my addition is mainly in 7d92624, with some notes.

Demo is in Docker Compose format in the demo/ subdirectory. Instructions

@reviewbybees esp. @oleg-nenashev

[jglick]

It will make running job on detached agents possible.

@fengxx I am not sure what use case you are thinking of here, but it is not goal of this set of changes. Here the goal is simply to allow build messages which are currently streamed from the agent back to the Jenkins master to instead be sent directly to Logstash.

[touchbeamTeamcity]

any news about this ?

[oleg-nenashev]

@touchbeamTeamcity I still think we need some time to finally test it and to re-integrate the Freestyle project support. Since there is a new maintainer of this plugin, I suppose we would be able to discuss the next steps on this front with him once he unravels other stuff. CC @jakub-bochenski

[jakub-bochenski]

@oleg-nenashev thanks, I'm definitely interested in this PR as we are slowly migrating our jobs to pipeline.

I haven't had time to look into this yet though and I'm not too familiar with pipeline-related Jenkins code in general so this might take some time.

[jglick]

Well for this to work with Pipeline

• a series of PRs associated with JENKINS-38381 need to be merged & released; currently stalled pending tests on performance impacts
• this patch, and perhaps this whole plugin, would need to be made more product-quality—I consider this a proof of concept, not optimized in the least

[jglick]

imitate the demo with a similarly configured Jenkins container

Do the instructions in demo/README.md not work as advertised?

[paulwilljones]

The demo works, however using the plugin with ELK 5.X yields an error when making a get request for the logs from ES due to the request fields changing between versions.

Is there any incentive to support this plugin / functionality?

[oleg-nenashev]

@paulwilljones Do you mean compatibility of this PR with Elasticsearch 5? If somebody finds time to finalize these changes, I think it will definitely be in the TODO list.

[jglick]

@paulwilljones to set expectations, I know next to nothing about ELK and care little about this particular plugin (certainly the code added here is not even close to production quality). I filed this as a proof of concept to demonstrate that systems developed in the upstream PRs can really work “end-to-end”; could just as well have connected to any other service which allows character/byte streams to be incrementally uploaded and downloaded, but this plugin happened to be handy.

[oleg-nenashev]

BTW the renewed demo has been also packaged as WAR in https://github.com/jenkinsci/custom-war-packager/tree/master/demo/external-logging-elasticsearch (as a demo for Custom WAR Packager). I doubt this PR is going to be integrated in such way anyway tho. A clean implementation in a separate plugin may be preferable, especially for log browsing

[jglick]

This is now out of date not just with respect to master but with major refactorings in the upstream PRs, so it is pretty much useless. If we want an ELK-based implementation it will be easier to write one from scratch at this point. (Which would be better anyway, as it would not pick up unrelated and unwanted historical features from this plugin.)