automate the ironbank docker context generation #32251

v1v · 2022-07-07T12:38:12Z

Motivation/summary

Automate the docker context generation for the IronBank releases, this will allow us to move away from creating those docker context manually in a different repository, since the Platform Release team already provide the automation.

Implementation details

A similar approach was done for Kibana and Elasticsearch

I don't know if other teams are actually building those docker, but for now this is the very first iteration to move away from the manual steps required.

What projects are the ones?

auditbeat
filebeat
metricbeat
packetbeat

DoD build context content

As required the build context contains the following files:

Dockerfile with specific requirements for DoD (registry args, no internet dependencies, healthcheck, ...)
hardening_manifest.yaml with all required dependencies
LICENSE specific for DoD
README.md with specific content for DoD
config/ (optional) directory containing all config files to include into the Docker image
scripts/ directory containing all scripts to include into the Docker image (example: entrypoint...)

Tasks

Enable configuration in the DRA so it can be automated.
Gather feedback from the teams.

Test

if you mage ironbank then you will see the .tar.gz in the build/distributions folder for each subscribed beat, for instance:

$ mage ironbank 
>> prepareIronbankBuild: prepare the IronBank container context.
>> saveIronbank: save the IronBank container context.
>> creating TAR file from directory: build/auditbeat-ironbank-8.4.0-docker-build-context, target: build/distributions/auditbeat-ironbank-8.4.0-docker-build-context.tar.gz
>> skipping root directory
ironbank ran for 19.374167ms
$ ls -ltra build 
total 0
drwxr-xr-x 25 vmartinez staff 800 Jul 19 11:43 ..
drwxr-xr-x  7 vmartinez staff 224 Jul 19 11:43 auditbeat-ironbank-8.4.0-docker-build-context
drwxr-xr-x  4 vmartinez staff 128 Jul 19 11:43 .
drwxr-x---  4 vmartinez staff 128 Jul 19 11:43 distributions
$ ls -l auditbeat-ironbank-8.4.0-docker-build-context 
total 32
-rw-r--r-- 1 vmartinez staff  3754 Jul  7 13:33 Dockerfile
-rw-r--r-- 1 vmartinez staff 16642 Jul  7 13:33 LICENSE
-rw-r--r-- 1 vmartinez staff  1399 Jul  7 13:33 README.md
drwxr-xr-x 3 vmartinez staff    96 Jul  7 13:33 config
-rw-r--r-- 1 vmartinez staff  2256 Jul  7 13:33 hardening_manifest.yaml
$ cd build/distributions
$ tar -xvf auditbeat-ironbank-8.4.0-docker-build-context.tar.gz 
x Dockerfile
x LICENSE
x README.md
x config
x config/auditbeat.yml
x hardening_manifest.yaml
$ ls -ltra
total 48
-rw-r--r-- 1 vmartinez staff  2256 Jul 19 11:43 hardening_manifest.yaml
drwxr-xr-x 3 vmartinez staff    96 Jul 19 11:43 config
-rw-r--r-- 1 vmartinez staff  1399 Jul 19 11:43 README.md
-rw-r--r-- 1 vmartinez staff 16642 Jul 19 11:43 LICENSE
-rw-r--r-- 1 vmartinez staff  3754 Jul 19 11:43 Dockerfile
drwxr-xr-x 4 vmartinez staff   128 Jul 19 11:43 ..
-rw-r--r-- 1 vmartinez staff 12088 Jul 19 11:43 auditbeat-ironbank-8.4.0-docker-build-context.tar.gz
-rw-r--r-- 1 vmartinez staff   182 Jul 19 11:43 auditbeat-ironbank-8.4.0-docker-build-context.tar.gz.sha512

If it's not supported then mage ironbank will print a message:

$ mage ironbank
>> Ironbank: this module is not subscribed to the IronBank releases.

CI

ironbank is now enabled in the package stage and artifacts are generated:

This build

executed the ironbank step:

$ gsutil list gs://beats-ci-artifacts/beats/pull-requests/pr-32251/metricbeat | grep ironban
gs://beats-ci-artifacts/beats/pull-requests/pr-32251/metricbeat/metricbeat-ironbank-8.4.0-SNAPSHOT-docker-build-context.tar.gz
gs://beats-ci-artifacts/beats/pull-requests/pr-32251/metricbeat/metricbeat-ironbank-8.4.0-SNAPSHOT-docker-build-context.tar.gz.sha512

Issue

Similar to:

Requires #32274

Follow ups

Automate the heartbeats one, this is one is a bit more special since all the package dependencies need to be declared, and we might need to figure a way to automate this earlier than failing when generating the package.
Backport to 7.17, will require to automate the elastic-agent too.

mage: use BeatName

even though they are not subscribed to the ironbank releases this should help the calls in the package to just run it

elasticmachine · 2022-07-07T12:57:07Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-07-20T11:55:46.403+0000
Duration: 157 min 46 sec

Test stats 🧪

Test	Results
Failed	0
Passed	22480
Skipped	1937
Total	24417

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

…tomation * upstream/main: [main](backport elastic#32235) docs: Prepare Changelog for 8.3.2 (elastic#32253) Revert "Fix flags for dev build (elastic#31955)" (elastic#32250) [Automation] Update elastic stack version to 8.4.0-d0a4da44 for testing (elastic#32243) x-pack/filebeat/module/cisco: fix handling of user parsing with sgt fields (elastic#32196) packetbeat/route: make use of newly added GetBestInterfaceEx in x/sys/windows (elastic#32180)

v1v · 2022-07-07T16:36:53Z

There are some issues with the packaging in the main branch after bumping 1.17.11. Blocked until it's fixed

v1v · 2022-07-11T08:03:45Z

/test

dev-tools/packaging/templates/ironbank/auditbeat/Dockerfile

cachedout · 2022-07-18T14:08:50Z

dev-tools/packaging/templates/ironbank/auditbeat/Dockerfile

+CMD ["-environment", "container"]
+
+# see https://www.elastic.co/guide/en/beats/auditbeat/current/http-endpoint.html
+HEALTHCHECK --interval=10s --timeout=5s --start-period=1m --retries=5 CMD curl -I -f --max-time 5 --unix-socket '/usr/share/auditbeat/data/auditbeat.sock' 'http:/stats?pretty'


Where does this come from originally? I am surprised by the single slash in the final string.

This Dockerfile is exactly the one published in ironbank: https://repo1.dso.mil/dsop/elastic/beats/auditbeat/-/blob/development/Dockerfile#L81

dev-tools/packaging/templates/ironbank/auditbeat/hardening_manifest.yaml

v1v · 2022-07-19T14:18:21Z

/test

v1v · 2022-07-20T13:49:24Z

Test

➜  beats gsutil cp gs://beats-ci-artifacts/beats/pull-requests/pr-32251/auditbeat/auditbeat-ironbank-8.4.0-SNAPSHOT-docker-build-context.tar.gz .
Copying gs://beats-ci-artifacts/beats/pull-requests/pr-32251/auditbeat/auditbeat-ironbank-8.4.0-SNAPSHOT-docker-build-context.tar.gz...
/ [1 files][ 11.8 KiB/ 11.8 KiB]                                                
Operation completed over 1 objects/11.8 KiB
➜  beats tar xvf auditbeat-ironbank-8.4.0-SNAPSHOT-docker-build-context.tar.gz
x Dockerfile
x LICENSE
x README.md
x config
x config/auditbeat.yml
x hardening_manifest.yaml
➜  beats grep 8.4 *
Dockerfile:ARG ELASTIC_STACK=8.4.0
README.md:License](https://github.com/elastic/beats/blob/8.4/licenses/ELASTIC-LICENSE.txt),
README.md:[here](https://www.elastic.co/guide/en/beats/libbeat/current/release-notes-8.4.0.html).
auditbeat-ironbank-8.4.0-SNAPSHOT-docker-build-context.tar.gz.sha512:544fbef4e3fe6c4fd5495446211d42cb3e26ba2bcace54b932ab9f887a1b683eb3b368f94878e46ea5bebdb0a9b300c26171a548fe177d1a56e845f56eb6467a  auditbeat-ironbank-8.4.0-SNAPSHOT-docker-build-context.tar.gz
grep: config: Is a directory
hardening_manifest.yaml:  - "8.4.0"
hardening_manifest.yaml:  ELASTIC_STACK: "8.4.0"
hardening_manifest.yaml:  org.opencontainers.image.version: "8.4.0"
hardening_manifest.yaml:  - filename: "auditbeat-8.4.0-linux-x86_64.tar.gz"
hardening_manifest.yaml:    url: "<artifact_path>/auditbeat-8.4.0-linux-x86_64.tar.gz"

mdelapenya

Awesome addition!

(cherry picked from commit e41dc18) # Conflicts: # .ci/packaging.groovy # dev-tools/mage/pkg.go # x-pack/winlogbeat/magefile.go

(cherry picked from commit e41dc18) # Conflicts: # x-pack/winlogbeat/magefile.go

#32420)

…on (#32419)

v1v added 9 commits July 6, 2022 15:36

ironbank: added templates for some beats

03268c1

mage: initial support for Ironbank in auditbeat and generally

4aa48fe

mage: fix paths to be relative to the root folder

fb5c620

ironbank: support auditbeat with config

dfda93d

mage: use BeatName

chore: update docs

993e892

chore: update docs

d8f0cd1

ironbank: add support for filebeat metricbeat and packetbeat

6e7b9d9

ironbank: add goal for the remaining beats

0f15745

even though they are not subscribed to the ironbank releases this should help the calls in the package to just run it

ci: enable ironbank goal in the packaging

d8e953b

v1v added automation Team:Automation Label for the Observability productivity team backport-v8.3.0 Automated backport with mergify backport-7.17 Automated backport to the 7.17 branch with mergify labels Jul 7, 2022

v1v requested review from kuisathaverat and a team July 7, 2022 12:38

v1v requested review from a team as code owners July 7, 2022 12:38

v1v self-assigned this Jul 7, 2022

v1v requested review from belimawr and rdner and removed request for a team July 7, 2022 12:38

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jul 7, 2022

make: enable ironbank as part of the release

dd13abc

v1v added 4 commits July 7, 2022 14:36

lint: fix import

93161f6

fix: linting

8ddfaf1

mage: add ironbank goal even if they are not subscribed

683f7e0

kuisathaverat approved these changes Jul 7, 2022

View reviewed changes

v1v marked this pull request as draft July 8, 2022 14:36

v1v marked this pull request as ready for review July 11, 2022 08:03

cachedout reviewed Jul 18, 2022

View reviewed changes

v1v added 2 commits July 19, 2022 11:47

mage: support Tar without nested folders

8e45ee3

chore: fix comment

92fbb1c

fix linting

7eba9db

mdelapenya approved these changes Jul 20, 2022

View reviewed changes

pazone approved these changes Jul 20, 2022

View reviewed changes

v1v merged commit e41dc18 into elastic:main Jul 20, 2022

mergify bot pushed a commit that referenced this pull request Jul 20, 2022

automate the ironbank docker context generation (#32251)

b13342d

(cherry picked from commit e41dc18) # Conflicts: # .ci/packaging.groovy # dev-tools/mage/pkg.go # x-pack/winlogbeat/magefile.go

mergify bot mentioned this pull request Jul 20, 2022

[7.17](backport #32251) automate the ironbank docker context generation #32419

Merged

mergify bot pushed a commit that referenced this pull request Jul 20, 2022

automate the ironbank docker context generation (#32251)

7ad98f4

(cherry picked from commit e41dc18) # Conflicts: # x-pack/winlogbeat/magefile.go

mergify bot mentioned this pull request Jul 20, 2022

[8.3](backport #32251) automate the ironbank docker context generation #32420

Merged

v1v pushed a commit that referenced this pull request Jul 21, 2022

[8.3](backport #32251) automate the ironbank docker context generation (

1f6f7ac

#32420)

v1v pushed a commit that referenced this pull request Jul 21, 2022

[7.17](backport #32251) automate the ironbank docker context generati…

8bf2b59

…on (#32419)

This was referenced Jul 21, 2022

ironbank: support for elastic-agent in 7.17 #32455

Merged

ironbank: support for heartbeat #32502

Closed

v1v mentioned this pull request Sep 1, 2022

ironbank: support for heartbeat #32954

Merged

2 tasks

chrisberkhout pushed a commit that referenced this pull request Jun 1, 2023

automate the ironbank docker context generation (#32251)

8c0e9c8

v1v mentioned this pull request Aug 8, 2023

ironbank: update the list of maintainers for the ironbank #36254

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

automate the ironbank docker context generation #32251

automate the ironbank docker context generation #32251

v1v commented Jul 7, 2022 •

edited

Loading

elasticmachine commented Jul 7, 2022 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

v1v commented Jul 7, 2022

v1v commented Jul 11, 2022

cachedout Jul 18, 2022

v1v Jul 19, 2022 •

edited

Loading

v1v commented Jul 19, 2022

v1v commented Jul 20, 2022

mdelapenya left a comment

automate the ironbank docker context generation #32251

automate the ironbank docker context generation #32251

Conversation

v1v commented Jul 7, 2022 • edited Loading

Motivation/summary

Implementation details

DoD build context content

Tasks

Test

CI

Issue

Follow ups

elasticmachine commented Jul 7, 2022 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

v1v commented Jul 7, 2022

v1v commented Jul 11, 2022

cachedout Jul 18, 2022

Choose a reason for hiding this comment

v1v Jul 19, 2022 • edited Loading

Choose a reason for hiding this comment

v1v commented Jul 19, 2022

v1v commented Jul 20, 2022

Test

mdelapenya left a comment

Choose a reason for hiding this comment

v1v commented Jul 7, 2022 •

edited

Loading

elasticmachine commented Jul 7, 2022 •

edited by jenkins-beats-ci bot

Loading

v1v Jul 19, 2022 •

edited

Loading