Make Wolfi a part of Docker packaging #5062

rdner · 2024-07-04T17:00:22Z

What does this PR do?

New images with -wolfi suffix are created
The -cloud image is now based on Wolfi
Refactored the packaging spec for better readability
Fixed ignored architecture flags when building Docker images. Building ARM64 on AMD64 and AMD64 or ARM64 is now possible

Why is it important?

Wolfi is a security-focused Linux image for containers, see more here https://github.com/wolfi-dev/

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
~~- [ ] I have made corresponding changes to the documentation~~
~~- [ ] I have made corresponding change to the default configuration files~~
~~- [ ] I have added tests that prove my fix is effective or that my feature works~~
I have added an entry in ./changelog/fragments using the changelog tool
~~- [ ] I have added an integration test or an E2E test~~

How to test this PR locally

Images

x86

SNAPSHOT=true PLATFORMS=linux/amd64 PACKAGES=docker mage package

Before this change:

REPOSITORY                                       TAG                   IMAGE ID       CREATED         SIZE
docker.elastic.co/beats-ci/elastic-agent-cloud   8.16.0-SNAPSHOT       c3521379c659   4 minutes ago   982MB
docker.elastic.co/beats/elastic-agent            8.16.0-SNAPSHOT       79526721a60b   4 minutes ago   610MB
docker.elastic.co/beats/elastic-agent-complete   8.16.0-SNAPSHOT       5958311a5565   3 minutes ago   2.11GB
docker.elastic.co/beats/elastic-agent-ubi        8.16.0-SNAPSHOT       30c624ba0e7a   4 minutes ago   607MB

After this change:

REPOSITORY                                       TAG                   IMAGE ID       CREATED         SIZE
docker.elastic.co/beats-ci/elastic-agent-cloud   8.16.0-SNAPSHOT       0badce97e1fd   5 minutes ago   897MB
docker.elastic.co/beats/elastic-agent            8.16.0-SNAPSHOT       18bab22a67ba   5 minutes ago   635MB
docker.elastic.co/beats/elastic-agent-complete   8.16.0-SNAPSHOT       a392ec2eb0bd   3 minutes ago   2.1GB
docker.elastic.co/beats/elastic-agent-ubi        8.16.0-SNAPSHOT       4ce09f39a9f6   5 minutes ago   603MB
docker.elastic.co/beats/elastic-agent-wolfi      8.16.0-SNAPSHOT       b1a78ff97f30   5 minutes ago   526MB

Run this command to verify the correct architecture and layers

for id in $(docker image ls --format json | jq .ID -r); do docker inspect $id | jq '{"id":.[].Id, "tags":.[].RepoTags, "arch":.[].Architecture, "layers":.[].RootFS.Layers}'; done;

Output for X86

{
  "id": "sha256:a392ec2eb0bd34a2f28653fe0aa8c696da519d003f87d947c1706a4d52209366",
  "tags": [
    "docker.elastic.co/beats/elastic-agent-complete:8.16.0-SNAPSHOT"
  ],
  "arch": "amd64",
  "layers": [
    "sha256:3ec3ded77c0ce89e931f92aed086b2a2c774a6fbd51617853decc8afa4e1087a",
    "sha256:2bc7e199fd03b7fa1a5ccdd3b40620f4cad3c31cddd4ba35418b749aa4b56a7a",
    "sha256:034383ef36a09b960ecccdf6447f0ef91865b9c143b7cb813970c7cff6f2d486",
    "sha256:c7e981d93236a530ed1f85b1a6a48106ada33ecd5e00ccf441e73d276aed1e0e",
    "sha256:3a3aee489da2e892d6f9e249188a3b8005733cf40defab1d86b4750cd6ec2fd0",
    "sha256:3cc880076321c7a5e53323f71d491019f5dbc77c2cda1c9646bde6552b41a1bb",
    "sha256:60e644e5b4a2e40102ea8c08503a99f4186d709b5e21420fb126b6f79185e912",
    "sha256:a64e3cf3a0dea43af928d17c407ddce12a09422a7e28c09a521576feee8771fe",
    "sha256:b564db3600afe2361c0c0a652fb649798ac3e3cb09488a6e539910723a7eb044",
    "sha256:b181d57d8070fcd47a74c1236241b31503be7545d343929cf8fdafdf10bdc933",
    "sha256:aa5916b66195dbb492d4136e5cf4a7ab5a76f80297fbc0651e6cacb7749f07bc",
    "sha256:7c087c82778976e1c17e8a4690b93601e532a355b1b2861141e17ed038e6a0c2",
    "sha256:c516ba2122be517e0482d3a3176dece36783c07ad0320fefa0da571d7b2b3607",
    "sha256:f9b47742633802667f72abeb3e40d1302dded32b178db5a4fd3830cb3fd2fd35",
    "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef"
  ]
}
{
  "id": "sha256:0badce97e1fdc44b4fc1f8fd75da363cb186bc0177f5515917665c41129f53e3",
  "tags": [
    "docker.elastic.co/beats-ci/elastic-agent-cloud:8.16.0-SNAPSHOT"
  ],
  "arch": "amd64",
  "layers": [
    "sha256:2edb767fffebfc8a70f5707270f5149869375b068775e577392935da7b1479fb",
    "sha256:76569ac0371094c9e125284a0b0064714e35301769f61e6438a53005ba51c14f",
    "sha256:4a9b777b7899a0dede9acc199920cc79187340499c425a310efc5a3ed76b9a93",
    "sha256:de395ebe3180e995a0ce4042d7d57189db3cc073eecfb19f724536ec82a7ceda",
    "sha256:cacfb25128769999206886731660d153d65e179ca0cdbc48ed46343130d70d87",
    "sha256:145615672d99eee3c4141006aed32aced5026bf28138c4e4f1a08d702c37a9a5",
    "sha256:c26cc68bb5a38e9bb7de5867417e744e12f57809d0490703c69811c2f798fc43",
    "sha256:5a448e62d052daa6ea985cdde07958e08b03f16e2d183def1111fff16156e2b3",
    "sha256:4a14d733387aa41288ef0088344a64431a34dd7f61afb52746b6a3d0376b97c8",
    "sha256:cd5070c8fefb35da7128adb49880234939bdc4cbd9edd8e9ab151230b5d52c8b",
    "sha256:450c527682673789b9c452f14be8e27ab38a19a93328bf63e085f5d818be350c",
    "sha256:fba67135c63cd1ebf9d0306af475c16e5c73f34cd47c26b17ce612a688e30d19",
    "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef",
    "sha256:14292d6d4baaed40be5300e922ca9ede36eda50c6eb163c2c1036fa4bb67984d"
  ]
}
{
  "id": "sha256:4ce09f39a9f66b87db3894f5719c0fd3b1b870d9297565eb548b934dbd13f184",
  "tags": [
    "docker.elastic.co/beats/elastic-agent-ubi:8.16.0-SNAPSHOT"
  ],
  "arch": "amd64",
  "layers": [
    "sha256:53544948f51f66ab8080a28acc12474abac3103c6194ca36777aa60762ede49c",
    "sha256:08fac24b901a21e09cea780072d324d4f59b25c59df57dc751693fc45d2013c6",
    "sha256:03519818d277314069da56002bd7e9659b8d9e12bb344805046b0c79196d243c",
    "sha256:e9c2e4dffda76a83486323690097aae91b9157ea9d9ae52a4a50e742f232e5c5",
    "sha256:7d9a73ac223fbb81cf7d805e1186699955baae50f45f7755b6565218ec908d9a",
    "sha256:134ca32a71fc8779643553bec547fb3aebe7b38b7141ad423f63edfd8edd7e8f",
    "sha256:f83a9cb758884fcfb3b4fc22a5757e44cf4cbc2924d48baa568a99b2e742d602",
    "sha256:8734dc44c0afe9b751162681999e6a4ab7b7390d94f5a101a70caf067192345f",
    "sha256:1b1ecb12e774737dc5a3e9e5b8bf8e42f6eb52b18d47e4c8b26ec4816fdde4dc",
    "sha256:304cb5fd0b2055ac8478fac7d19f5c5f4d780ec40554be513bd8a47ff6a9b6c7",
    "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef"
  ]
}
{
  "id": "sha256:b1a78ff97f30a78f53e789d3255a27563b4b50b199d0a94a33ee00c2ea6cd525",
  "tags": [
    "docker.elastic.co/beats/elastic-agent-wolfi:8.16.0-SNAPSHOT"
  ],
  "arch": "amd64",
  "layers": [
    "sha256:2edb767fffebfc8a70f5707270f5149869375b068775e577392935da7b1479fb",
    "sha256:76569ac0371094c9e125284a0b0064714e35301769f61e6438a53005ba51c14f",
    "sha256:4a9b777b7899a0dede9acc199920cc79187340499c425a310efc5a3ed76b9a93",
    "sha256:de395ebe3180e995a0ce4042d7d57189db3cc073eecfb19f724536ec82a7ceda",
    "sha256:cacfb25128769999206886731660d153d65e179ca0cdbc48ed46343130d70d87",
    "sha256:145615672d99eee3c4141006aed32aced5026bf28138c4e4f1a08d702c37a9a5",
    "sha256:c26cc68bb5a38e9bb7de5867417e744e12f57809d0490703c69811c2f798fc43",
    "sha256:5a448e62d052daa6ea985cdde07958e08b03f16e2d183def1111fff16156e2b3",
    "sha256:1b1ecb12e774737dc5a3e9e5b8bf8e42f6eb52b18d47e4c8b26ec4816fdde4dc",
    "sha256:304cb5fd0b2055ac8478fac7d19f5c5f4d780ec40554be513bd8a47ff6a9b6c7",
    "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef"
  ]
}
{
  "id": "sha256:18bab22a67babed8fdc70cc0a18546b2536ac99f20732f2db941d96a7f27a1c6",
  "tags": [
    "docker.elastic.co/beats/elastic-agent:8.16.0-SNAPSHOT"
  ],
  "arch": "amd64",
  "layers": [
    "sha256:3ec3ded77c0ce89e931f92aed086b2a2c774a6fbd51617853decc8afa4e1087a",
    "sha256:2bc7e199fd03b7fa1a5ccdd3b40620f4cad3c31cddd4ba35418b749aa4b56a7a",
    "sha256:034383ef36a09b960ecccdf6447f0ef91865b9c143b7cb813970c7cff6f2d486",
    "sha256:c7e981d93236a530ed1f85b1a6a48106ada33ecd5e00ccf441e73d276aed1e0e",
    "sha256:3a3aee489da2e892d6f9e249188a3b8005733cf40defab1d86b4750cd6ec2fd0",
    "sha256:3cc880076321c7a5e53323f71d491019f5dbc77c2cda1c9646bde6552b41a1bb",
    "sha256:60e644e5b4a2e40102ea8c08503a99f4186d709b5e21420fb126b6f79185e912",
    "sha256:a64e3cf3a0dea43af928d17c407ddce12a09422a7e28c09a521576feee8771fe",
    "sha256:b564db3600afe2361c0c0a652fb649798ac3e3cb09488a6e539910723a7eb044",
    "sha256:b181d57d8070fcd47a74c1236241b31503be7545d343929cf8fdafdf10bdc933",
    "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef"
  ]
}

Note: the base layer is always different except elastic-agent-cloud:8.16.0 and elastic-agent-wolfi:8.16.0 which share the same base image – Wolfi.

ARM64

SNAPSHOT=true PLATFORMS=linux/arm64 PACKAGES=docker mage package

Before this change:

REPOSITORY                                       TAG               IMAGE ID       CREATED         SIZE
docker.elastic.co/beats-ci/elastic-agent-cloud   8.16.0-SNAPSHOT   700d4fa3c833   4 minutes ago   939MB
docker.elastic.co/beats/elastic-agent            8.16.0-SNAPSHOT   d35803d66a50   4 minutes ago   587MB
docker.elastic.co/beats/elastic-agent-complete   8.16.0-SNAPSHOT   8579bbe2702f   3 minutes ago   2.09GB
docker.elastic.co/beats/elastic-agent-ubi        8.16.0-SNAPSHOT   3b420a665959   4 minutes ago   651MB

After this change:

REPOSITORY                                       TAG               IMAGE ID       CREATED         SIZE
docker.elastic.co/beats-ci/elastic-agent-cloud   8.16.0-SNAPSHOT   dd883460e711   5 minutes ago   857MB
docker.elastic.co/beats/elastic-agent            8.16.0-SNAPSHOT   388643bca650   5 minutes ago   587MB
docker.elastic.co/beats/elastic-agent-complete   8.16.0-SNAPSHOT   c121e9f0b3fe   3 minutes ago   2.09GB
docker.elastic.co/beats/elastic-agent-ubi        8.16.0-SNAPSHOT   c76846bb58a6   5 minutes ago   584MB
docker.elastic.co/beats/elastic-agent-wolfi      8.16.0-SNAPSHOT   8b6e0408b6a9   5 minutes ago   506MB

Run this command to verify the correct architecture and layers

for id in $(docker image ls --format json | jq .ID -r); do docker inspect $id | jq '{"id":.[].Id, "tags":.[].RepoTags, "arch":.[].Architecture, "layers":.[].RootFS.Layers}'; done;

Output for ARM64

{
  "id": "sha256:c121e9f0b3fea483adda5584871a6c31fb7c4a59fa3d2ab98255dd4f99c44d27",
  "tags": [
    "docker.elastic.co/beats/elastic-agent-complete:8.16.0-SNAPSHOT"
  ],
  "arch": "arm64",
  "layers": [
    "sha256:a8c68591d421fc2d4bdda704f67a796edf5ff880c59358d75107eb5261821650",
    "sha256:21229740e72f63b67049ff375ff86856df76851a41b961ccc0258b657ffebda2",
    "sha256:6840669b7d1a9f2f9c3df0543a9e4631bbba4ba142bfb8f761004a291b63beba",
    "sha256:b5cd85927d89525d1b54f53ab9258e1ea8b30ed34f5da82dcb0b18cc201b8fcb",
    "sha256:b4a623370ab35d201645f15379c4f1b92f124f66e466b2603b068a05db16d3d9",
    "sha256:dc936f5e71f48a6e8e57a4328afaf8e37eb41ffdfb5516423281d0869f5df5b1",
    "sha256:71c4a2ae2df81aa3dc4cd6d6bfa92540effe2393baa42db484601d33b1bf047d",
    "sha256:d5bb4fe5276cdf6557706692c76b6051b14946b426eeb2be1b21b89f849146bf",
    "sha256:e6b6f0f2a25ab40dd864b67516870f3a766178e30abbef3ece8bc23cb833981d",
    "sha256:a21254dc1a33f8724f4e6142b4c41712c5582494a81b9d7321f6ab98756e967f",
    "sha256:5fa9d5af66ec9f9a7bf5afc9982f75ec4fb10b42cc1d05564ca9c4bfda905e8d",
    "sha256:ad755fd23e96c489fef28eb0d75f46edcc0c078594f2c97404775e40484af575",
    "sha256:bdb359ba5f4e2d70d51bf50873f382c52cc973bc6b2a704c863e8c9d2f94751e",
    "sha256:edb11face2b4381516f07e35d7a9b707d4b2052dc6d717972165290321752381",
    "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef"
  ]
}
{
  "id": "sha256:dd883460e7114da32e9f5493dca264160375fb927cb95cf551d64a365fb59608",
  "tags": [
    "docker.elastic.co/beats-ci/elastic-agent-cloud:8.16.0-SNAPSHOT"
  ],
  "arch": "arm64",
  "layers": [
    "sha256:5dac778dd0e9fee84eb001bdee474ff61de7bb829bba016e245b9f5ca872c316",
    "sha256:403f8fb20960b5233268fb9a88804a7309c408108ea375732eb6446271a2fcf2",
    "sha256:21aea1db0998d0e4b32065cddc24250b110343cdb1e0fc26dc392fad1f749656",
    "sha256:d63995dd206cb5a32e74ba032b1d63c83095b8c990690bd6d5f305757d150865",
    "sha256:ae5e6fff71558c5d0e39ab3597d75efb36c3c992e65dd7988aa3e3017ae824c4",
    "sha256:56b8e2d5440ac7f14e2e3cd98f43e42b176fc8cef83503f9902fe4c7027b3338",
    "sha256:823f5659f29b456726d7583c75d3bdf6ed3c86e5e962ac94b8c905052dad14ff",
    "sha256:95e4de2bd43f201fb5783dccf2d50529c6459db00762abba4d2257c90bb5df72",
    "sha256:68c9b420458c7ed43377cde1f3d6198c5febb0e2ddc295e06620c4fb1b3cbfe2",
    "sha256:1c1100d00f0a9c3da1831f65ba11cfa18395608efc39805819c89f0396c51dfb",
    "sha256:26e65983188b6cea86e04051db52bb905f48a1253d164c2a6854b3350d934c27",
    "sha256:578c74ee9711a36c7e5187eb1a47a32472b253239b41f7d0914b1b4f1e798b10",
    "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef",
    "sha256:fd64f6588c4d61caffed04eefd78a04e0aa0caabfa567e65fde6d615d1f6760b"
  ]
}
{
  "id": "sha256:c76846bb58a6f9d832f649e6ddd263d72ae63b7bbc1fc27d1fe9370d122ac8b3",
  "tags": [
    "docker.elastic.co/beats/elastic-agent-ubi:8.16.0-SNAPSHOT"
  ],
  "arch": "arm64",
  "layers": [
    "sha256:53ce7ccd46624c30c3cd21bfbd9589c37e2ce99e7c724595ca4a17e65f1da763",
    "sha256:0902763b71b13c7e5013246515ddfdd761ddf4f2ade8d020d74a1864391cb5ca",
    "sha256:4384f36086948324e63d5e96f3787395d5ee8adc2c6bd81e5f8812bcfe0ef656",
    "sha256:60820fe03ecc3ca16269c97403c77db81d6234799dcba06c6a108e624f1d9f77",
    "sha256:e58b83c62f02b792127741abce7c3b811e0468d409c103e81df43ae6e7670b2f",
    "sha256:c22267259a1f8412cca21e46d14001b3feeefd4578596868277565aca388ad86",
    "sha256:fe0ca0592c517007820d50131070b37ea385a829284bd9e7e4243d6f6b2059be",
    "sha256:d5bb4fe5276cdf6557706692c76b6051b14946b426eeb2be1b21b89f849146bf",
    "sha256:0c7cfd329bf2561dffe3ce26f21e444d614556112bf490781f5fedd4b50afc4c",
    "sha256:8338344e41f8a66c2f0f84728e2dc8d358d23300dba919ba412c9727daadc56c",
    "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef"
  ]
}
{
  "id": "sha256:388643bca650d38118b00ab21ab5fc4f1ee80333d97065de16ff2e2d7736915d",
  "tags": [
    "docker.elastic.co/beats/elastic-agent:8.16.0-SNAPSHOT"
  ],
  "arch": "arm64",
  "layers": [
    "sha256:a8c68591d421fc2d4bdda704f67a796edf5ff880c59358d75107eb5261821650",
    "sha256:21229740e72f63b67049ff375ff86856df76851a41b961ccc0258b657ffebda2",
    "sha256:6840669b7d1a9f2f9c3df0543a9e4631bbba4ba142bfb8f761004a291b63beba",
    "sha256:b5cd85927d89525d1b54f53ab9258e1ea8b30ed34f5da82dcb0b18cc201b8fcb",
    "sha256:b4a623370ab35d201645f15379c4f1b92f124f66e466b2603b068a05db16d3d9",
    "sha256:dc936f5e71f48a6e8e57a4328afaf8e37eb41ffdfb5516423281d0869f5df5b1",
    "sha256:71c4a2ae2df81aa3dc4cd6d6bfa92540effe2393baa42db484601d33b1bf047d",
    "sha256:d5bb4fe5276cdf6557706692c76b6051b14946b426eeb2be1b21b89f849146bf",
    "sha256:e6b6f0f2a25ab40dd864b67516870f3a766178e30abbef3ece8bc23cb833981d",
    "sha256:a21254dc1a33f8724f4e6142b4c41712c5582494a81b9d7321f6ab98756e967f",
    "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef"
  ]
}
{
  "id": "sha256:8b6e0408b6a9904f44fef9ebb655218c2151e8930e2359310313058da51ccd71",
  "tags": [
    "docker.elastic.co/beats/elastic-agent-wolfi:8.16.0-SNAPSHOT"
  ],
  "arch": "arm64",
  "layers": [
    "sha256:5dac778dd0e9fee84eb001bdee474ff61de7bb829bba016e245b9f5ca872c316",
    "sha256:403f8fb20960b5233268fb9a88804a7309c408108ea375732eb6446271a2fcf2",
    "sha256:21aea1db0998d0e4b32065cddc24250b110343cdb1e0fc26dc392fad1f749656",
    "sha256:d63995dd206cb5a32e74ba032b1d63c83095b8c990690bd6d5f305757d150865",
    "sha256:ae5e6fff71558c5d0e39ab3597d75efb36c3c992e65dd7988aa3e3017ae824c4",
    "sha256:0bf79c41e96eadd70505f9bb0b3946573fd445242dd7e2de3f887e2cd6570d41",
    "sha256:9aa24d8e0a2fe6e1ac1b0ab22a0c7710c7cfb12993fca5ce137f97d3777b68c1",
    "sha256:3ce01354e678c9d44bba2eb95b512d724e2dc376eff30314f99ebb45867bcf0d",
    "sha256:0c7cfd329bf2561dffe3ce26f21e444d614556112bf490781f5fedd4b50afc4c",
    "sha256:8338344e41f8a66c2f0f84728e2dc8d358d23300dba919ba412c9727daadc56c",
    "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef"
  ]
}

Note: the base layer is always different except elastic-agent-cloud:8.16.0 and elastic-agent-wolfi:8.16.0 which share the same base image – Wolfi.

Size change

The different size of the x86 docker.elastic.co/beats/elastic-agent image between main and this branch is due to the fixed bug (ignored architecture flag). Before this change, one could not build an amd64 image on an arm64 machine. Now, the architecture is correct and the size of the base layer is different.

Elastic Agent functionality in Wolfi

I created a 8.16.0-SNAPSHOT stack in the cloud
Enrolled the agent using this command:

docker run -ti \
-e FLEET_ENROLL='1' \
-e FLEET_INSECURE='false' \
-e FLEET_URL='https://<host>.fleet.us-west2.gcp.elastic-cloud.com:443' \
-e FLEET_ENROLLMENT_TOKEN='<token>' \
-e KIBANA_HOST='http://kibana:5601' \
-e KIBANA_FLEET_USERNAME='elastic' \
-e KIBANA_FLEET_PASSWORD='changeme' \
-e ELASTIC_NETINFO='false' \
docker.elastic.co/beats/elastic-agent:8.16.0-SNAPSHOT

note: fill the <host> and <value> placeholders

Saw successful enrollment and data in Discover

I repeated the steps for docker.elastic.co/beats/elastic-agent:8.16.0-SNAPSHOT and docker.elastic.co/beats-ci/elastic-agent-cloud:8.16.0-SNAPSHOT images to check for regressions. Both of them enrolled and sent data successfully.
I also double-checked that the capabilities is still there:

docker run -u root -it --entrypoint /bin/ash docker.elastic.co/beats/elastic-agent-wolfi:8.16.0-SNAPSHOT
apk add libcap-utils
getcap data/elastic-agent-68eb65/components/agentbeat

The output was:

data/elastic-agent-68eb65/components/agentbeat cap_setuid,cap_net_raw=p

Kubernetes integration

Created a deployment
Created a policy with the Kubernetes integration
Applied the generated manifest using the custom -wolfi image
Made sure of its successful enrollment and incoming data

Elastic Cloud

I followed this guide to test the new Wolfi-based cloud image https://github.com/elastic/elastic-agent?tab=readme-ov-file#testing-on-elastic-cloud

cmacknz · 2024-07-04T18:58:12Z

Once you get this working, post a comparison of the container sizes to the most recently released agent container. Benign looking changes have doubled the size of the container in the past.

* New images with `-wolfi` suffix are created * The `-cloud` image is now based on Wolfi * Refactored the packaging spec for better readability * Fixed ignored architecture flags when building Docker images. Building ARM64 on AMD64 and AMD64 or ARM64 is now possible

elasticmachine · 2024-07-10T12:43:41Z

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

dev-tools/packaging/templates/docker/Dockerfile.elastic-agent.tmpl

cmacknz · 2024-07-10T19:35:56Z

Are you able to collect container logs on Kubernetes with this container using the default k8s manifests?

The default manifests run the container as root. Does this work?

elastic-agent/deploy/kubernetes/elastic-agent-standalone/elastic-agent-standalone-daemonset.yaml

Lines 73 to 74 in 341a24e

    
           securityContext: 
        
             runAsUser: 0

What happens if you remove that and run as the elastic-agent user?

rdner · 2024-07-16T14:54:31Z

@cmacknz if I remove the lines you referred to the agent crashes on startup with:

Error: preparing STATE_PATH(/usr/share/elastic-agent/state) failed: mkdir /usr/share/elastic-agent/state/data: permission denied

Any idea how to fix it?

rdner · 2024-07-16T15:24:04Z

@cmacknz looks like it requires more changes to the manifest to make it work https://stackoverflow.com/a/57917406

However, I tried to set this for the pod:

   securityContext:
         fsGroup: 1000
         runAsUser: 1000
         runAsGroup: 1000

and still have this error on agent's startup.

rdner · 2024-07-16T15:37:51Z

I created a follow up issue #5141

cmacknz · 2024-07-16T15:41:18Z

We need to add automated tests for this container, this can probably be achieved on k8s using #5013. This can also be a follow up issue. Please create one so we can track this, and I'll approve this.

rdner · 2024-07-16T15:46:52Z

@cmacknz here is the follow up issue #5142

cmacknz

Thanks!

rdner · 2024-07-16T17:14:54Z

When testing on Elastic Cloud, the integration server failed to start, I'm investigating.

rdner · 2024-07-18T12:58:43Z

I followed this guide https://github.com/elastic/elastic-agent?tab=readme-ov-file#testing-on-elastic-cloud to test my custom image on Elastic Cloud.

I found that the Integration Server component didn't start using the new Wolfi-based image.

The cloud injects scripts into our image from here https://github.com/elastic/cloud-assets/tree/master/stackpack/apm and these scripts require Bash https://github.com/elastic/cloud-assets/blob/6c0db22c475b9b2347d5b1cbe484f7c604468501/stackpack/apm/apm.sh#L1

So, I had to install Bash in Wolfi to maintain compatibility with the scripts 01b1ef2

This would cost us around 17MB in the image size which I don't consider to be critical.

I re-tested everything with the fix and everything works normally: the integration server starts, I can enroll a Wolfi-based agent into it.

elastic-sonarqube · 2024-07-18T13:35:35Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

rdner added enhancement New feature or request Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team Team:Elastic-Agent Label for the Agent team backport-skip Packaging labels Jul 4, 2024

rdner self-assigned this Jul 4, 2024

rdner force-pushed the docker-wolfi branch 7 times, most recently from 2b22147 to b532368 Compare July 5, 2024 19:42

cmacknz mentioned this pull request Jul 9, 2024

Extend agent container initialisation logic #4925

Merged

5 tasks

rdner force-pushed the docker-wolfi branch from d19c7bc to aa3b884 Compare July 10, 2024 11:04

rdner force-pushed the docker-wolfi branch from aa3b884 to 6d5fc5f Compare July 10, 2024 12:33

rdner requested a review from cmacknz July 10, 2024 12:43

rdner marked this pull request as ready for review July 10, 2024 12:43

rdner requested a review from a team as a code owner July 10, 2024 12:43

rdner requested a review from blakerouse July 10, 2024 12:43

elastic deleted a comment from mergify bot Jul 10, 2024

blakerouse reviewed Jul 10, 2024

View reviewed changes

dev-tools/packaging/templates/docker/Dockerfile.elastic-agent.tmpl Show resolved Hide resolved

dev-tools/packaging/templates/docker/Dockerfile.elastic-agent.tmpl Show resolved Hide resolved

rdner mentioned this pull request Jul 16, 2024

Make sure we can use the Wolfi image on Kubernetes without root permissions #5141

Closed

rdner mentioned this pull request Jul 16, 2024

Automate testing of the Wolfi container in Kubernetes #5142

Closed

rdner enabled auto-merge (squash) July 16, 2024 15:48

rdner disabled auto-merge July 16, 2024 15:50

cmacknz approved these changes Jul 16, 2024

View reviewed changes

Install Bash in Wolfi, since it's required by the cloud image

01b1ef2

rdner enabled auto-merge (squash) July 18, 2024 12:59

rdner merged commit aa6bd63 into elastic:main Jul 18, 2024
13 checks passed

rdner deleted the docker-wolfi branch July 18, 2024 15:05

rdner mentioned this pull request Jul 26, 2024

Switch the base Docker image to Wolfi elastic/fleet-server#3760

Merged

rdner mentioned this pull request Aug 14, 2024

Make Wolfi a part of Docker packaging elastic/beats#40524

Merged

mrodm mentioned this pull request Aug 27, 2024

Add support to use wolfi Elastic Agent image starting in 8.16.0-SNAPSHOT elastic/elastic-package#2038

Merged

rdner mentioned this pull request Sep 9, 2024

Add elastic-agent-wolfi-complete Docker image #5475

Merged

mergify bot mentioned this pull request Sep 11, 2024

[8.x](backport #5475) Add elastic-agent-wolfi-complete Docker image #5516

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make Wolfi a part of Docker packaging #5062

Make Wolfi a part of Docker packaging #5062

rdner commented Jul 4, 2024 •

edited

Loading

cmacknz commented Jul 4, 2024

elasticmachine commented Jul 10, 2024

cmacknz commented Jul 10, 2024

rdner commented Jul 16, 2024

rdner commented Jul 16, 2024

rdner commented Jul 16, 2024

cmacknz commented Jul 16, 2024

rdner commented Jul 16, 2024

cmacknz left a comment

rdner commented Jul 16, 2024

rdner commented Jul 18, 2024 •

edited

Loading

elastic-sonarqube bot commented Jul 18, 2024

Make Wolfi a part of Docker packaging #5062

Make Wolfi a part of Docker packaging #5062

Conversation

rdner commented Jul 4, 2024 • edited Loading

What does this PR do?

Why is it important?

Checklist

How to test this PR locally

Images

x86

ARM64

Size change

Elastic Agent functionality in Wolfi

Kubernetes integration

Elastic Cloud

cmacknz commented Jul 4, 2024

elasticmachine commented Jul 10, 2024

cmacknz commented Jul 10, 2024

rdner commented Jul 16, 2024

rdner commented Jul 16, 2024

rdner commented Jul 16, 2024

cmacknz commented Jul 16, 2024

rdner commented Jul 16, 2024

cmacknz left a comment

Choose a reason for hiding this comment

rdner commented Jul 16, 2024

rdner commented Jul 18, 2024 • edited Loading

elastic-sonarqube bot commented Jul 18, 2024

Quality Gate passed

rdner commented Jul 4, 2024 •

edited

Loading

rdner commented Jul 18, 2024 •

edited

Loading