docker: switch to a bespoke test container #5683

gdams · 2024-10-10T11:03:42Z

Also moved LIB_DIR and SYSTEM_LIB_DIR to be inside the current workspace for docker builds due to permissions errors

Currently using ghcr.io/adoptium/test-containers:ubuntu2204 which is a lightweight image built loosely off the dockerStatic base images. PR incoming to the infra repo to regularly build this and we can add more base images where appropriate.

Grinder to show this working: https://ci.adoptium.net/view/Test_grinder/job/Grinder/11160/

smlambert · 2024-10-10T15:54:50Z

Removing ADDITIONAL_LABEL values from Grinder run and filling in CLOUD_PROVIDER=azure, since that is how the tests are expected to be launched in general pipeline code:

Run some additional grinders, varying test group and JDK version:

sanity.system, JDK21:

https://ci.adoptium.net/view/Test_grinder/job/Grinder/11162 (sent to idle static node) - runs serially and completes in 3 hrs 51 min
https://ci.adoptium.net/view/Test_grinder/job/Grinder/11168 (PARALLEL=Dynamic, TEST_TIME=2) - spins up 8 nodes and completes in 30 min

special.functional, JDK11:

https://ci.adoptium.net/view/Test_grinder/job/Grinder/11169/ (PARALLEL=Dynamic, NUM_MACHINES=5) - some child jobs running in static docker containers, 1 child job on dynamic agent fails with:

17:05:11  Running on test-linux-x64-b97844 in /home/adoptopenjdk/workspace/Grinder_testList_3
[Pipeline] {
...
17:06:14  Running tests...
[Pipeline] echo
17:06:14  ITERATION: 1/1
[Pipeline] wrap
17:06:14  $ Xvfb -displayfd 2 -screen 0 1024x768x24 -fbdir /home/adoptopenjdk/workspace/Grinder_testList_3/.xvfb-10-..fbdir4319951883012337092
...
17:06:18  Exception: java.io.IOException: Cannot run program "Xvfb": error=2, No such file or directory
...

external/jacoco JDK17, https://ci.adoptium.net/view/Test_grinder/job/Grinder/11172/ (this test requires sw.tool.docker label)

12:42:52       [exec] The test in the build_image() function is jacoco
12:42:52       [exec] #####################################################
12:42:52       [exec] INFO:  docker build  --no-cache -t adoptopenjdk-jacoco-test:17-jdk-ubuntu-hotspot-full -f /home/adoptopenjdk/workspace/Grinder/jvmtest/external/jacoco/dockerfile/17/jdk/ubuntu/Dockerfile.hotspot.full /home/adoptopenjdk/workspace/Grinder/jvmtest/external/
12:42:52       [exec] #####################################################
12:42:52       [exec] /home/adoptopenjdk/workspace/Grinder/aqa-tests/TKG/../../jvmtest/external/build_image.sh: line 75: docker: command not found
12:42:52  
12:42:52  BUILD FAILED

gdams · 2024-10-11T09:52:37Z

Exception: java.io.IOException: Cannot run program "Xvfb": error=2, No such file or directory

@ShelleyLambert looking at that job it didn't run in a docker container? I think it's because we need to add the ci.agent.dynamic label to that job?

gdams · 2024-10-11T09:57:11Z

Rebuilding https://ci.adoptium.net/view/Test_grinder/job/Grinder/11171/ (PARALLEL=Dynamic, NUM_MACHINES=5)

smlambert · 2024-10-11T16:30:42Z

@ShelleyLambert looking at that job it didn't run in a docker container? I think it's because we need to add the ci.agent.dynamic label to that job?

~~Yes, I am well-aware and we should take that machine offline and raise an infra issue.~~

Looking more closely, it looks like it found a static machine, but still ran on a dynamic machine (test-linux-x64-b97844), console output from https://ci.adoptium.net/view/Test_grinder/job/Grinder_testList_3/10/console

17:05:10  Found a total of 30 nodes with the 'ci.role.test&&hw.arch.x86&&sw.os.linux' label
[Pipeline] echo
17:05:10  Found an idle node: test-docker-debian12-x64-4. The program will not start dynamic vm.
[Pipeline] }
[Pipeline] // node
[Pipeline] node
17:05:11  Running on [test-linux-x64-b97844](https://ci.adoptium.net/computer/test%2Dlinux%2Dx64%2Db97844/) in /home/adoptopenjdk/workspace/Grinder_testList_3
[Pipeline] {
[Pipeline] retry
[Pipeline] {
[Pipeline] timeout
17:05:11  Timeout set to expire in 1 hr 0 min

In 'real' pipeline runs, the test pipeline code will first try to send to idle static machines and spin up dynamic ones as needed after that.

As per the current design, we will not be passing in ci.agent.dynamic explicitly when we trigger test pipelines (only updating to set CLOUD_PROVIDER=azure), so I wanted to check that it works as designed.

gdams · 2024-10-11T19:47:14Z

Right okay, so if you don't expect the dynamic label to be used I'll have to tweak the existing code slightly, I'll have a playb

gdams · 2024-10-11T19:49:24Z

@smlambert I've updated the code so it won't explicitly require someone to pass the dynamic label to the job anymore. As long as the cloud is set as Azure it will default to using a container image

smlambert · 2024-10-11T23:17:32Z

@gdams - you do not need to make the change on L461. I was explaining how the current logic already works, not asking for you to change your PR, which looks fine as it is, I am just running many additional tests to verify that each test group works on these agents (please see what we do on L351).

gdams · 2024-10-12T09:11:53Z

@gdams - you do not need to make the change on L461. I was explaining how the current logic already works, not asking for you to change your PR, which looks fine as it is, I am just running many additional tests to verify that each test group works on these agents (please see what we do on L351).

reverted PTAL

smlambert · 2024-11-12T18:02:54Z

Couple extra Grinder runs to verify:
without CLOUD_PROVIDER: https://ci.adoptium.net/view/Test_grinder/job/Grinder/11320/
with CLOUD_PROVIDER=azure https://ci.adoptium.net/view/Test_grinder/job/Grinder/11321/

smlambert · 2024-11-12T18:04:52Z

https://ci.adoptium.net/view/Test_grinder/job/Grinder/11321/ fails with:

13:03:02  + ./get.sh -s /home/jenkins/workspace/Grinder/aqa-tests/.. -p x86-64_linux -r nightly -j 21 -i hotspot --clone_openj9 false --tkg_repo https://github.com/AdoptOpenJDK/TKG.git --tkg_branch master
13:03:02  TESTDIR: /home/jenkins/workspace/Grinder/aqa-tests
13:03:02  get jdk binary...
13:03:02  _ENCODE_FILE_NEW=UNTAGGED curl -OLJSks  https://api.adoptium.net/v3/binary/latest/21/ea/linux/x64/jdk/hotspot/normal/adoptium?project=jdk
13:03:05  _ENCODE_FILE_NEW=UNTAGGED curl -OLJSks  https://api.adoptium.net/v3/binary/latest/21/ea/linux/x64/sbom/hotspot/normal/adoptium?project=jdk
13:03:06  _ENCODE_FILE_NEW=UNTAGGED curl -OLJSks  https://api.adoptium.net/v3/binary/latest/21/ea/linux/x64/testimage/hotspot/normal/adoptium?project=jdk
13:03:09  Uncompressing file: OpenJDK21U-jdk_x64_linux_hotspot_21.0.6_2-ea.tar.gz ...
13:03:13  Uncompressing file: OpenJDK21U-testimage_x64_linux_hotspot_21.0.6_2-ea.tar.gz ...
13:03:16  Run /home/jenkins/workspace/Grinder/jdkbinary/j2sdk-image/bin/java -version
13:03:16  =JAVA VERSION OUTPUT BEGIN=
13:03:16  /home/jenkins/workspace/Grinder/jdkbinary/j2sdk-image/bin/java: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /home/jenkins/workspace/Grinder/jdkbinary/j2sdk-image/bin/../lib/libjli.so)
[Pipeline] }

This failure is unrelated to this PR (related to: #5754)

smlambert · 2024-11-13T17:02:44Z

I will launch a couple of Grinders to 'exhaust' our list of static nodes, and see the use of dynamic agents shortly.

https://ci.adoptium.net/view/Test_grinder/job/Grinder/11477

karianna · 2024-11-14T07:10:27Z

buildenv/jenkins/openjdk_tests

-            docker.image('adoptopenjdk/centos7_build_image').pull()
-            docker.image('adoptopenjdk/centos7_build_image').inside {
+            // Set dockerimage for azure agent. Fyre has stencil to setup the right environment
+            docker.image('ghcr.io/adoptium/test-containers:ubuntu2204').pull()


Should we extract that URL or the ubuntu version?

Referencing a set of containers from https://github.com/orgs/adoptium/packages/container/package/test-containers is good from my point of view (they have the test prereqs), and we can build out several to give us a variety of coverage.

docker: switch to a bespoke test container

bd8dc33

gdams mentioned this pull request Oct 10, 2024

add test docker image to ghcr for dynamic VM testing adoptium/infrastructure#3775

Open

6 tasks

smlambert mentioned this pull request Oct 10, 2024

Update Test pipeline code to support Azure Dynamic agents #5684

Open

karianna requested a review from smlambert October 14, 2024 04:40

karianna approved these changes Oct 14, 2024

View reviewed changes

improve label detection

6a1e047

gdams force-pushed the docker branch from 5f260b9 to 6a1e047 Compare October 16, 2024 15:02

improve label detection

10b6061

gdams force-pushed the docker branch from 1b52742 to 10b6061 Compare October 16, 2024 15:07

karianna approved these changes Nov 14, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docker: switch to a bespoke test container #5683

docker: switch to a bespoke test container #5683

gdams commented Oct 10, 2024 •

edited

Loading

smlambert commented Oct 10, 2024 •

edited

Loading

gdams commented Oct 11, 2024

gdams commented Oct 11, 2024

smlambert commented Oct 11, 2024 •

edited

Loading

gdams commented Oct 11, 2024

gdams commented Oct 11, 2024

smlambert commented Oct 11, 2024

gdams commented Oct 12, 2024

smlambert commented Nov 12, 2024

smlambert commented Nov 12, 2024 •

edited

Loading

smlambert commented Nov 13, 2024 •

edited

Loading

karianna Nov 14, 2024

smlambert Nov 15, 2024 •

edited

Loading

docker: switch to a bespoke test container #5683

Are you sure you want to change the base?

docker: switch to a bespoke test container #5683

Conversation

gdams commented Oct 10, 2024 • edited Loading

smlambert commented Oct 10, 2024 • edited Loading

gdams commented Oct 11, 2024

gdams commented Oct 11, 2024

smlambert commented Oct 11, 2024 • edited Loading

gdams commented Oct 11, 2024

gdams commented Oct 11, 2024

smlambert commented Oct 11, 2024

gdams commented Oct 12, 2024

smlambert commented Nov 12, 2024

smlambert commented Nov 12, 2024 • edited Loading

smlambert commented Nov 13, 2024 • edited Loading

karianna Nov 14, 2024

Choose a reason for hiding this comment

smlambert Nov 15, 2024 • edited Loading

Choose a reason for hiding this comment

gdams commented Oct 10, 2024 •

edited

Loading

smlambert commented Oct 10, 2024 •

edited

Loading

smlambert commented Oct 11, 2024 •

edited

Loading

smlambert commented Nov 12, 2024 •

edited

Loading

smlambert commented Nov 13, 2024 •

edited

Loading

smlambert Nov 15, 2024 •

edited

Loading