Skip to content

Commit dedca4d

Browse files
oleg-nenashevoleg-nenashev
authored
Merge pull request #1 from jenkinsci/JENKINS-52693-plugin-skeleton
[JENKINS-52693] - External Build Logging. Reference Implementation for Elasticsearch
2 parents 18c013e + c790f6e commit dedca4d

32 files changed

+2204
-0
lines changed

‎Jenkinsfile

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
buildPlugin(platforms: ['linux'])

‎demo/Makefile

+33
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
# Just a Makefile for manual testing
2+
.PHONY: all
3+
4+
ARTIFACT_ID = jenkins-external-task-logging-elk-demo
5+
VERSION = 2.131-elk-SNAPSHOT
6+
CWP_VERSION= 1.0
7+
8+
all: clean build
9+
10+
clean:
11+
rm -rf tmp
12+
mkdir tmp
13+
14+
build: tmp/output/target/${ARTIFACT_ID}-${VERSION}.war
15+
16+
tmp/output/target/${ARTIFACT_ID}-${VERSION}.war:
17+
mvn com.googlecode.maven-download-plugin:download-maven-plugin:1.4.0:artifact \
18+
-DgroupId=io.jenkins.tools.custom-war-packager \
19+
-DartifactId=custom-war-packager-cli \
20+
-Dclassifier=jar-with-dependencies \
21+
-Dversion=${CWP_VERSION} \
22+
-DoutputDirectory=tmp \
23+
-DoutputFileName=cwp-cli.jar
24+
java -jar tmp/cwp-cli.jar \
25+
-configPath packager-config.yml -version ${VERSION}
26+
27+
run: tmp/output/target/${ARTIFACT_ID}-${VERSION}.war
28+
docker-compose rm -fv
29+
docker-compose up --build --force-recreate jenkins elk
30+
31+
debug: tmp/output/target/${ARTIFACT_ID}-${VERSION}.war
32+
docker-compose rm -fv
33+
docker-compose up --build --force-recreate jenkinsDebug elk

‎demo/README.md

+72
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
External Task Logging to Elasticsearch Demo
2+
===
3+
4+
This demo packages Jenkins WAR for External Task Logging to Elasticsearch with help of Logstash plugin.
5+
6+
This demo includes [Logstash Plugin PR#18](https://github.com/jenkinsci/logstash-plugin/pull/18) and
7+
all its upstream dependencies.
8+
It also bundles auto-configuration System Groovy scripts, so that the WAR file starts
9+
up with pre-configured Logstash plugin settings and some other configs.
10+
11+
Features of the demo:
12+
13+
* Pipeline jobs logging goes to Elasticsearch
14+
* When tasks are executed on agents, the logs get posted to Elasticsearch directly
15+
without passing though the master and causing scalability issues
16+
* Pipeline jobs override standard Log actions in the Jenkins core, so the
17+
underlying implementation is transparent to users
18+
* Secrets are escaped in stored/displayed logs when running on master and agents.
19+
* Console annotations work as they work for common Jenkins instances
20+
* Log blocks are collapsible in the _Console_ screen
21+
* Origin container ID of every message is visible in Kibana (if you have set that up) via sender field
22+
23+
The demo can be run in Docker Compose,
24+
ELK stack is provided by the [sebp/elk](https://hub.docker.com/r/sebp/elk/) image in this case.
25+
26+
## Prerequisites
27+
28+
* Docker and Docker Compose are installed
29+
30+
## Building demo
31+
32+
To build the demo...
33+
34+
1. Go to the repository root, run `mvn clean package` to build Jenkins Custom WAR Packager
35+
2. Change directory to the demo root
36+
3. Run `make build`
37+
38+
First build may take a while, because the packager will need to checkout and build
39+
many repositories.
40+
41+
## Running demo
42+
43+
1. Run `make run`. It will spin up the demo with predefined environment.
44+
Jenkins will be available on the port 8080, credentials: `admin/admin`
45+
2. If you want to run demo jobs on the agent,
46+
also run `docker-compose up agent` in a separate terminal window
47+
3. In order to access the instance, use the "admin/admin" credentials.
48+
4. Run one of the demo jobs.
49+
5. Browse logs
50+
* Classic Log action queries data from Elasticsearch
51+
* There is a _Log (Kibana)_ action in runs, which shows Kibana.
52+
* In order to see Kibana logs, you will need to configure the default index in the
53+
embedded page once Jenkins starts up. Use `logstash/` as a default index and
54+
`@timestamp` as data source
55+
56+
## Manual run
57+
58+
This guideline allows to run the demo locally.
59+
Only Logstash will be preconfigured.
60+
61+
1. Run `docker run -p 5601:5601 -p 9200:9200 -p 5044:5044 -it --name elk sebp/elk:es241_l240_k461`
62+
to start the Docker container to to expose ports
63+
2. Run Jenkins using `JENKINS_HOME=$(pwd)/work java -jar tmp/output/target/external-task-logging-elk-2.107.3-elk-SNAPSHOT.war --httpPort=8080 --prefix=/jenkins`
64+
(or just `run run.sh`).
65+
* If needed, the demo can be configured by setting system properties
66+
* `elasticsearch.host` - host, defaults to `http://elk`
67+
* `elasticsearch.port` - Elasticsearch port, defaults to `9200`
68+
* `logstash.key` - Path to the root index/key for logging, defaults to `/logstash/logs`
69+
* `elasticsearch.username` and `elasticsearch.password` -
70+
3. Pass through the installation Wizard
71+
4. Create a Pipeline job with some logging (e.g. `echo` commands), run it
72+
5. Browse logs (see above)

‎demo/docker-compose.yml

+51
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
version: '3'
2+
3+
services:
4+
elk:
5+
#TODO: update to ES 5.0
6+
image: sebp/elk:es241_l240_k461
7+
container_name: elk
8+
ports:
9+
- "5601:5601" # Kibana
10+
- "9200:9200" # Elasticsearch
11+
- "5044:5044" # Logstash
12+
expose:
13+
- "5601"
14+
- "9200"
15+
- "5044"
16+
jenkins:
17+
image: jenkins/demo-external-task-logging-elk:latest
18+
container_name: jenkins
19+
links:
20+
- elk
21+
environment:
22+
- JAVA_OPTS=-Dio.jenkins.demo.external-task-logging-elk.enabled=true -Djenkins.install.runSetupWizard=false
23+
ports:
24+
- "8080:8080"
25+
- "9000:9000"
26+
expose:
27+
- "8080"
28+
- "9000"
29+
jenkinsDebug:
30+
image: jenkins/demo-external-task-logging-elk:latest
31+
container_name: jenkins
32+
links:
33+
- elk
34+
environment:
35+
- JAVA_OPTS=-Dio.jenkins.demo.external-task-logging-elk.enabled=true -Djenkins.install.runSetupWizard=false
36+
- DEBUG=true
37+
ports:
38+
- "8080:8080"
39+
- "9000:9000"
40+
- "5005:5005"
41+
expose:
42+
- "8080"
43+
- "9000"
44+
- "5005"
45+
agent:
46+
image: cloudbees/jnlp-slave-with-java-build-tools:2.2.0
47+
links:
48+
- elk
49+
- jenkins
50+
# TODO there is no -noreconnect yet this does not work when first started; you need to relaunch it; work around with wait-for-it: https://docs.docker.com/compose/startup-order/
51+
command: -url http://jenkins:8080/ f9bf0c290371481814f8bc235e3c53736ea9cd9f11b466b76ad794b91cb57a0b -workDir "/home/jenkins" agent

‎demo/external-logging-demo.iml

+15
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<module org.jetbrains.idea.maven.project.MavenProjectsManager.isMavenModule="true" type="JAVA_MODULE" version="4">
3+
<component name="NewModuleRootManager" LANGUAGE_LEVEL="JDK_1_8">
4+
<output url="file://$MODULE_DIR$/target/classes" />
5+
<output-test url="file://$MODULE_DIR$/target/test-classes" />
6+
<content url="file://$MODULE_DIR$">
7+
<sourceFolder url="file://$MODULE_DIR$/src/main/groovy" isTestSource="false" />
8+
<sourceFolder url="file://$MODULE_DIR$/src/main/java" isTestSource="false" />
9+
<sourceFolder url="file://$MODULE_DIR$/target/generated-test-sources/injected" isTestSource="true" generated="true" />
10+
<excludeFolder url="file://$MODULE_DIR$/target" />
11+
</content>
12+
<orderEntry type="inheritedJdk" />
13+
<orderEntry type="sourceFolder" forTests="false" />
14+
</component>
15+
</module>

‎demo/packager-config.yml

+99
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,99 @@
1+
bundle:
2+
groupId: "io.jenkins.tools.war-packager.demo"
3+
artifactId: "jenkins-external-task-logging-elk-demo"
4+
vendor: "Jenkins project"
5+
title: "Jenkins External Task Logging demo for Elasticsearch"
6+
description: "Jenkins External Task Logging demo for Elasticsearch, packaged as a single WAR file. It automatically configures logging on startup"
7+
buildSettings:
8+
docker:
9+
base: "jenkins/jenkins:2.130"
10+
tag: "jenkins/demo-external-task-logging-elk"
11+
build: true
12+
war:
13+
groupId: "org.jenkins-ci.main"
14+
artifactId: "jenkins-war"
15+
source:
16+
version: 2.131-SNAPSHOT
17+
plugins:
18+
- groupId: "org.jenkins-ci.plugins"
19+
artifactId: "structs"
20+
source:
21+
version: 1.14
22+
- groupId: "org.jenkins-ci.plugins"
23+
artifactId: "scm-api"
24+
source:
25+
version: 2.2.7
26+
- groupId: "org.jenkins-ci.plugins.workflow"
27+
artifactId: "workflow-aggregator"
28+
source:
29+
version: 2.5
30+
- groupId: "org.jenkins-ci.plugins.workflow"
31+
artifactId: "workflow-api"
32+
source:
33+
version: 2.29-rc219.239019e84015
34+
build:
35+
buildOriginalVersion: true
36+
- groupId: "org.jenkins-ci.plugins.workflow"
37+
artifactId: "workflow-step-api"
38+
source:
39+
version: 2.15
40+
- groupId: "org.jenkins-ci.plugins.workflow"
41+
artifactId: "workflow-support"
42+
source:
43+
version: 2.19-rc265.3e5e4aeecfff
44+
build:
45+
buildOriginalVersion: true
46+
- groupId: "org.jenkins-ci.plugins.workflow"
47+
artifactId: "workflow-job"
48+
source:
49+
version: 2.22-rc311.5616213fbed0
50+
build:
51+
buildOriginalVersion: true
52+
- groupId: "org.jenkins-ci.plugins.workflow"
53+
artifactId: "workflow-durable-task-step"
54+
source:
55+
version: 2.20-rc333.74dc7c303e6d
56+
build:
57+
buildOriginalVersion: true
58+
- groupId: "org.jenkins-ci.plugins"
59+
artifactId: "logstash"
60+
source:
61+
version: 2.1.1-SNAPSHOT
62+
- groupId: "io.jenkins.plugins.external-logging"
63+
artifactId: "external-logging-api"
64+
source:
65+
version: 1.0-alpha-1-SNAPSHOT
66+
- groupId: "io.jenkins.plugins.external-logging"
67+
artifactId: "external-logging-elasticsearch"
68+
source:
69+
version: 1.0-alpha-1-SNAPSHOT
70+
71+
# Security warnings
72+
- groupId: "org.jenkins-ci.plugins"
73+
artifactId: "junit"
74+
source:
75+
version: 1.24
76+
- groupId: "org.jenkins-ci.plugins"
77+
artifactId: "mailer"
78+
source:
79+
version: 1.21
80+
- groupId: "org.jenkins-ci.plugins"
81+
artifactId: "git-client"
82+
source:
83+
version: 2.7.2
84+
- groupId: "org.jenkins-ci.plugins"
85+
artifactId: "credentials-binding"
86+
source:
87+
version: 1.16
88+
- groupId: "org.jenkins-ci.plugins"
89+
artifactId: "docker-commons"
90+
source:
91+
version: 1.13
92+
systemProperties: {
93+
jenkins.model.Jenkins.slaveAgentPort: "9000",
94+
jenkins.model.Jenkins.slaveAgentPortEnforce: "true"}
95+
groovyHooks:
96+
- type: "init"
97+
id: "initScripts"
98+
source:
99+
dir: src/main/groovy

‎demo/pom.xml

+88
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
3+
<modelVersion>4.0.0</modelVersion>
4+
5+
<parent>
6+
<groupId>org.jenkins-ci.plugins</groupId>
7+
<artifactId>plugin</artifactId>
8+
<version>3.15</version>
9+
<relativePath />
10+
</parent>
11+
12+
<groupId>io.jenkins.plugins.external-logging</groupId>
13+
<artifactId>external-logging-demo</artifactId>
14+
<name>External Logging for Elasticsearch/Logstash Demo</name>
15+
<description>The plugin provides API to simplify external logging implementations for Jenkins</description>
16+
<url>https://wiki.jenkins.io/display/JENKINS/External+Logging+API+Plugin</url>
17+
<version>${revision}${changelist}</version>
18+
<packaging>hpi</packaging>
19+
20+
<properties>
21+
<revision>1.0-alpha-1</revision>
22+
<changelist>-SNAPSHOT</changelist>
23+
<jenkins.version>2.131-SNAPSHOT</jenkins.version>
24+
<java.level>8</java.level>
25+
<useBeta>true</useBeta>
26+
</properties>
27+
28+
<licenses>
29+
<license>
30+
<name>MIT License</name>
31+
<url>https://opensource.org/licenses/MIT</url>
32+
</license>
33+
</licenses>
34+
35+
<scm>
36+
<connection>scm:git:git://github.com/jenkinsci/${project.artifactId}-plugin.git</connection>
37+
<developerConnection>scm:git:git@github.com:jenkinsci/${project.artifactId}-plugin.git</developerConnection>
38+
<url>https://github.com/jenkinsci/${project.artifactId}-plugin</url>
39+
<tag>${scmTag}</tag>
40+
</scm>
41+
42+
<dependencies>
43+
<dependency>
44+
<groupId>io.jenkins.plugins.external-logging</groupId>
45+
<artifactId>external-logging-elasticsearch</artifactId>
46+
<version>1.0-alpha-1-SNAPSHOT</version>
47+
</dependency>
48+
</dependencies>
49+
50+
<build>
51+
<plugins>
52+
<plugin>
53+
<groupId>org.codehaus.gmaven</groupId>
54+
<artifactId>gmaven-plugin</artifactId>
55+
<version>1.5-jenkins-3</version>
56+
<dependencies>
57+
<dependency>
58+
<groupId>org.codehaus.gmaven.runtime</groupId>
59+
<artifactId>gmaven-runtime-1.8</artifactId>
60+
<version>1.5-jenkins-3</version>
61+
</dependency>
62+
</dependencies>
63+
<executions>
64+
<execution>
65+
<goals>
66+
<goal>compile</goal>
67+
<goal>testCompile</goal>
68+
</goals>
69+
</execution>
70+
</executions>
71+
</plugin>
72+
</plugins>
73+
</build>
74+
75+
<repositories>
76+
<repository>
77+
<id>repo.jenkins-ci.org</id>
78+
<url>https://repo.jenkins-ci.org/public/</url>
79+
</repository>
80+
</repositories>
81+
<pluginRepositories>
82+
<pluginRepository>
83+
<id>repo.jenkins-ci.org</id>
84+
<url>https://repo.jenkins-ci.org/public/</url>
85+
</pluginRepository>
86+
</pluginRepositories>
87+
88+
</project>

0 commit comments

Comments
 (0)