Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

environments-preview feature not exchanging DOCKER_HOST #18915

Open
Tracked by #17355
juftin opened this issue May 5, 2023 · 2 comments
Open
Tracked by #17355

environments-preview feature not exchanging DOCKER_HOST #18915

juftin opened this issue May 5, 2023 · 2 comments
Labels
backend: Docker Docker backend-related issues backend: Environments {local,docker,remote}._environment-related issues bug

Comments

@juftin
Copy link

juftin commented May 5, 2023

Describe the bug

Enabling environments breaks the docker build process in CI/CD and complains that the DOCKER_HOST is in the wrong place.


We have a docker_image that depends on a pex_binary

BUILD
pex_binary(
    name = "bin",
    dependencies = [
        ":lib",
    ],
    execution_mode = "venv",
    include_tools = True,
    layout = "packed",
)

docker_image(
    name = "docker",
    dependencies = [
        ":docker_resources",
        ":bin",
    ],
    image_tags = ["{build_args.PANTS_DISTRIBUTION_VERSION}"],
)

Due to some wheel compatibility issues, we cannot build this pex_binary (and subsequent docker_image) on our Macs, it only works up in CI/CD (self-hosted GitHub Actions runners) where the machine is linux_x86.

To resolve this issue for ourselves locally, we enabled the environments feature

BUILD
local_environment(
  name="local_linux_x86",
  description="Localhost x86 Linux Environment",
  compatible_platforms=["linux_x86_64"],
  fallback_environment="docker_x86",
)

docker_environment(
  name="local_docker_x86",
  description="Dockerized x86 Linux Environment",
  platform="linux_x86_64",
  image="python:3.8.16",
)

And we attached the environment to the pex_binary

BUILD
pex_binary(
    name = "bin",
    dependencies = [
        ":lib",
    ],
    execution_mode = "venv",
    include_tools = True,
    layout = "packed",
    environment = "local_linux_x86",
)

The idea here is that the pex_binary needs to be built on a linux_x86_64 platform and when that platform isn't available locally it should just use the docker_environment.

This solution is working for us locally, but when we run it up in CI/CD it no longer works. Here is the issue that we're seeing in CI/CD:

Log Output
15:54:05.51 [DEBUG] Starting: Scheduling: Building docker image ***.dkr.ecr.us-east-1.amazonaws.com/moz-batch-job:16.0.0-beta.1
15:54:05.54 [DEBUG] Starting: acquire_command_runner_slot
15:54:19.81 [DEBUG] Completed: setup_sandbox
15:54:19.86 [DEBUG] spawned local process as Some(1133) for Process { argv: ["/usr/bin/docker", "build", "--pull=False", "--tag", "***.dkr.ecr.us-east-1.amazonaws.com/moz-helper-services:latest", "--tag", "***.dkr.ecr.us-east-1.amazonaws.com/moz-helper-services:16.0.0-beta.1", "--build-arg", "PANTS_DISTRIBUTION_VERSION", "--file", "services/moz-helper-services/Dockerfile", "."], env: {"PANTS_DISTRIBUTION_VERSION": "16.0.0-beta.1", "PATH": "/tmp/pants-sandbox-q6Zu6z/.shims/bin", "__UPSTREAM_IMAGE_IDS": ""}, working_directory: None, input_digests: InputDigests { complete: DirectoryDigest { digest: Digest { hash: Fingerprint<2c4455e78a6e491e19a07eca09e00a2fc9d026b6e428026680cb253cbbf4d541>, size_bytes: 264 }, tree: "Some(..)" }, nailgun: DirectoryDigest { digest: Digest { hash: Fingerprint<e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855>, size_bytes: 0 }, tree: "Some(..)" }, input_files: DirectoryDigest { digest: Digest { hash: Fingerprint<79b1e31b408edaf0ecb646a2deaf492f0d62436d46c7e020659821ef142b3191>, size_bytes: 184 }, tree: "Some(..)" }, immutable_inputs: {RelativePath(".shims"): DirectoryDigest { digest: Digest { hash: Fingerprint<e6c1927a092ff6beaa1fa4f372248c975cb51869b1c28245cb0281ebc6816ecb>, size_bytes: 78 }, tree: "Some(..)" }}, use_nailgun: {} }, output_files: {}, output_directories: {}, timeout: None, execution_slot_variable: None, concurrency_available: 0, description: "Building docker image ***.dkr.ecr.us-east-1.amazonaws.com/moz-helper-services:latest +1 additional tag.", level: Info, append_only_caches: {}, jdk_home: None, platform: Linux_x86_64, cache_scope: PerSession, execution_strategy: Local, remote_cache_speculation_delay: 0ns }
15:54:19.89 [INFO] Completed: Building docker image ***.dkr.ecr.us-east-1.amazonaws.com/moz-helper-services:latest +1 additional tag.
15:54:19.89 [DEBUG] Completed: Scheduling: Building docker image ***.dkr.ecr.us-east-1.amazonaws.com/moz-helper-services:latest +1 additional tag.
15:54:19.89 [DEBUG] Completed: acquire_command_runner_slot
15:54:19.89 [DEBUG] Running Building docker image ***.dkr.ecr.us-east-1.amazonaws.com/moz-result-processor:16.0.0-beta.1 under semaphore with concurrency id: 1, and concurrency: 1
15:54:19.89 [INFO] Starting: Building docker image ***.dkr.ecr.us-east-1.amazonaws.com/moz-result-processor:16.0.0-beta.1
15:54:19.89 [DEBUG] Starting: setup_sandbox
15:54:19.89 [WARN] Docker build failed for `docker_image` services/moz-helper-services:docker. The services/moz-helper-services/Dockerfile has `COPY` instructions for source files that may not have been found in the Docker build context.

However there are possible matches. Please review the following list of suggested renames:

  * services.moz-helper-services/bin.pex => services/moz-helper-services


15:54:19.90 [DEBUG] Completed: `publish` goal
15:54:19.90 [DEBUG] computed 1 nodes in 44.638043 seconds. there are 13299 total nodes.
15:54:19.90 [ERROR] 1 Exception encountered:

Engine traceback:
  in select
    ..
  in pants.core.goals.publish.run_publish
    `publish` goal
  in pants.core.goals.publish.package_for_publish
    ..
  in pants.core.goals.package.environment_aware_package
    ..
  in pants.backend.docker.goals.package_image.build_docker_image
    ..

Traceback (most recent call last):
  File "/home/runner/.cache/pants/setup/bootstrap-Linux-x86_64/2.15.1rc2_py38/lib/python3.8/site-packages/pants/engine/internals/selectors.py", line 593, in native_engine_generator_send
    res = func.send(arg)
  File "/home/runner/.cache/pants/setup/bootstrap-Linux-x86_64/2.15.1rc2_py38/lib/python3.8/site-packages/pants/backend/docker/goals/package_image.py", line 309, in build_docker_image
    raise ProcessExecutionFailure(
pants.engine.process.ProcessExecutionFailure: Process 'Building docker image ***.dkr.ecr.us-east-1.amazonaws.com/moz-helper-services:latest +1 additional tag.' failed with exit code 1.
stdout:

stderr:
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

As far as I can tell, enabling the environments feature confuses pants about where the DOCKER_HOST should be (our GitHub runners have DOCKER_HOST set to unix:///run/docker/docker.sock instead of the default unix:///var/run/docker.sock). We pass this environment variable down in our pants.toml file though:

pants.toml
[docker]
env_vars = [
  "DOCKER_CONFIG=%(homedir)s/.docker",
  "DOCKER_DEFAULT_PLATFORM=linux/amd64",
  "HOME",
  "USER",
  "PATH",
  # used by action-runner-controller dind
  "DOCKER_CERT_PATH",
  "DOCKER_HOST",
  "DOCKER_TLS_VERIFY",
]
tools = [
  "dirname",
  "readlink",
  "python3",
  # These may be necessary if using Pyenv-installed Python.
  "cut",
  "sed",
  "bash",
  "sh",
]
default_repository = "{directory}"
build_args = ["PANTS_DISTRIBUTION_VERSION"]

For solutions to this we've tried upgrading to pants==2.15.1rc2, removing the local_environment completely and just using the docker_environment, and also using pants.ci.toml to override which environment to use, and even hardcoding the DOCKER_HOST env var inside of pants.ci.toml.

All of this worked within CI/CD when we specify no environments at all, but doesn't work locally. However it works locally when we do specify an environment, but doesn't work in CI/CD.

Pants version
2.15.0 / 2.15.1rc2

OS
Linux (CI/CD)

Additional info
Add any other information about the problem here, such as attachments or links to gists, if relevant.

@juftin juftin added the bug label May 5, 2023
@huonw huonw added the backend: Docker Docker backend-related issues label May 16, 2023
@riisi
Copy link
Contributor

riisi commented Apr 4, 2024

I think I've found a workaround for this by adding the DOCKER_HOST to the local_environment docker_env_vars. E.g.:

local_environment(
  name="local_linux_x86",
  description="Localhost x86 Linux Environment",
  compatible_platforms=["linux_x86_64"],
  fallback_environment="docker_x86",
  docker_env_vars=["DOCKER_HOST"]
)

I can't reproduce this locally running Pants from sources (on a mac).

It seems like the local_environment config is effectively unsetting DOCKER_HOST for the subsequent docker image build (which is not configured to use the environment). I had a quick look at the source code, but it wasn't clear to me where this would be happening.

@riisi riisi added backend: Environments {local,docker,remote}._environment-related issues and removed backend: Docker Docker backend-related issues labels Apr 12, 2024
@riisi riisi added the backend: Docker Docker backend-related issues label Sep 25, 2024
@riisi
Copy link
Contributor

riisi commented Sep 25, 2024

Re-adding the Docker label for visibility, since it affects that backend.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backend: Docker Docker backend-related issues backend: Environments {local,docker,remote}._environment-related issues bug
Projects
None yet
Development

No branches or pull requests

3 participants