[CI] :test:fixtures:s3-fixture:composeUp fails due to HTTP connection issue

[adnapibar]

Describe the bug
On CI gradle checks on PRs, :test:fixtures:s3-fixture:composeUp fails due to an HTTP time out error. This is a recurring issue.

To Reproduce

See https://ci.opensearch.org/logs/ci/workflow/OpenSearch_CI/PR_Checks/Gradle_Check/gradle_check_1658.log

Expected behavior

The tests shouldn't be timing out.

Additional context

We can increase the time out to 120 seconds using the env var COMPOSE_HTTP_TIMEOUT on the Jenkins CI agents.

> Task :test:fixtures:s3-fixture:composeUp

ERROR: for 56aa93ed60e3e11b06e2067e06b0dd8c_s3-fixture__s3-fixture-other_1  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)

ERROR: for 56aa93ed60e3e11b06e2067e06b0dd8c_s3-fixture__s3-fixture_1  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)

ERROR: for 56aa93ed60e3e11b06e2067e06b0dd8c_s3-fixture__s3-fixture-with-session-token_1  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
Creating 56aa93ed60e3e11b06e2067e06b0dd8c_s3-fixture__s3-fixture-with-ec2_1              ... done
Creating 56aa93ed60e3e11b06e2067e06b0dd8c_s3-fixture__s3-fixture-repositories-metering_1 ... done
Creating 56aa93ed60e3e11b06e2067e06b0dd8c_s3-fixture__s3-fixture-with-ecs_1              ... done

ERROR: for s3-fixture-other  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)

ERROR: for s3-fixture  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)

ERROR: for s3-fixture-with-session-token  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
An HTTP request took too long to complete. Retry with --verbose to obtain debug information.
If you encounter this issue regularly because of slow network conditions, consider setting COMPOSE_HTTP_TIMEOUT to a higher value (current value: 60).

[andrross]

Same issue as: #1566

[andrross]

Is it possible to set this timeout value in a gradle or docker setting so that it doesn't have to be a custom configuration in the CI agent environment?

[reta]

@andrross @adnapibar I will take a look at this one :)

[reta]

@peterzhuamazon @dblock may I ask you for help here please, I see that build log includes docker ps (or alike) as part of the build:

[Gradle_Check] $ /bin/bash /tmp/CITOOL11853126605119070871.sh
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
...

Could you please also include these 2 commands into build steps as well:

docker -v
docker-compose -v

Would be great to understand if the issue is related to the particular Docker / Docker Compose versions (fe I could not reproduce it on my envs), thank you.

[tlfeng]

Another occurrence in https://ci.opensearch.org/logs/ci/workflow/OpenSearch_CI/PR_Checks/Gradle_Check/gradle_check_1760.log, found in PR #1837

[reta]

Some thoughts on the issue, the Docker Compose plugin does not support COMPOSE_HTTP_TIMEOUT as of now:

• Add support for COMPOSE_HTTP_TIMEOUT avast/gradle-docker-compose-plugin#265

However, increasing the timeout may not be the best option. Right now the build throttles the number of concurrent docker-compose up invocations to 1. However, the number of the Rest Integration tests (the ones using docker-compose) is not throttled, the hypothesis is that by limiting the amount of the concurrent Rest Integration tests execution should shave the load off the Docker and (docker-compose should not be timing out)