Skip to content

Commit

Permalink
airbyte-ci: custom tag on connector build (#32819)
Browse files Browse the repository at this point in the history
  • Loading branch information
@alafanechere
alafanechere authored Nov 29, 2023
1 parent 965ed21 commit fb4cf98
Show file tree
Hide file tree
Showing 6 changed files with 48 additions and 31 deletions.
34 changes: 20 additions & 14 deletions airbyte-ci/connectors/pipelines/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -124,17 +124,17 @@ At this point you can run `airbyte-ci` commands.

#### Options

| Option | Default value | Mapped environment variable | Description |
| ------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------- | ----------------------------- | ------------------------------------------------------------------------------------------- |
| `--enable-dagger-run/--disable-dagger-run` | `--enable-dagger-run`` | | Disables the Dagger terminal UI. | | | | |
| `--is-local/--is-ci` | `--is-local` | | Determines the environment in which the CLI runs: local environment or CI environment. |
| `--git-branch` | The checked out git branch name | `CI_GIT_BRANCH` | The git branch on which the pipelines will run. |
| `--git-revision` | The current branch head | `CI_GIT_REVISION` | The commit hash on which the pipelines will run. |
| `--diffed-branch` | `origin/master` | | Branch to which the git diff will happen to detect new or modified files. |
| `--gha-workflow-run-id` | | | GHA CI only - The run id of the GitHub action workflow |
| `--ci-context` | `manual` | | The current CI context: `manual` for manual run, `pull_request`, `nightly_builds`, `master` |
| `--pipeline-start-timestamp` | Current epoch time | `CI_PIPELINE_START_TIMESTAMP` | Start time of the pipeline as epoch time. Used for pipeline run duration computation. |
| `--show-dagger-logs/--hide-dagger-logs` | `--hide-dagger-logs` | | Flag to show or hide the dagger logs. |
| Option | Default value | Mapped environment variable | Description |
| ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------- | ------------------------------------------------------------------------------------------- |
| `--enable-dagger-run/--disable-dagger-run` | `--enable-dagger-run`` | | Disables the Dagger terminal UI. | | | | | | |
| `--is-local/--is-ci` | `--is-local` | | Determines the environment in which the CLI runs: local environment or CI environment. |
| `--git-branch` | The checked out git branch name | `CI_GIT_BRANCH` | The git branch on which the pipelines will run. |
| `--git-revision` | The current branch head | `CI_GIT_REVISION` | The commit hash on which the pipelines will run. |
| `--diffed-branch` | `origin/master` | | Branch to which the git diff will happen to detect new or modified files. |
| `--gha-workflow-run-id` | | | GHA CI only - The run id of the GitHub action workflow |
| `--ci-context` | `manual` | | The current CI context: `manual` for manual run, `pull_request`, `nightly_builds`, `master` |
| `--pipeline-start-timestamp` | Current epoch time | `CI_PIPELINE_START_TIMESTAMP` | Start time of the pipeline as epoch time. Used for pipeline run duration computation. |
| `--show-dagger-logs/--hide-dagger-logs` | `--hide-dagger-logs` | | Flag to show or hide the dagger logs. |

### <a id="connectors-command-subgroup"></a>`connectors` command subgroup

Expand Down Expand Up @@ -254,6 +254,9 @@ It's mainly purposed for local use.
Build a single connector:
`airbyte-ci connectors --name=source-pokeapi build`

Build a single connector with a custom image tag:
`airbyte-ci connectors --name=source-pokeapi build --tag=my-custom-tag`

Build a single connector for multiple architectures:
`airbyte-ci connectors --name=source-pokeapi build --architecture=linux/amd64 --architecture=linux/arm64`

Expand Down Expand Up @@ -303,6 +306,8 @@ flowchart TD
| Option | Multiple | Default value | Description |
| --------------------- | -------- | -------------- | ----------------------------------------------------------------- |
| `--architecture`/`-a` | True | Local platform | Defines for which architecture the connector image will be built. |
| `--tag` | False | `dev` | Image tag for the built image. |


### <a id="connectors-publish-command"></a>`connectors publish` command
Run a publish pipeline for one or multiple connectors.
Expand Down Expand Up @@ -442,13 +447,14 @@ This command runs the Python tests for a airbyte-ci poetry package.
## Changelog
| Version | PR | Description |
| ------- | ---------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- |
| 2.10.0 | [#32819](https://github.com/airbytehq/airbyte/pull/32819) | Add `--tag` option to connector build. |
| 2.9.0 | [#32816](https://github.com/airbytehq/airbyte/pull/32816) | Add `--architecture` option to connector build. |
| 2.8.0 | [#31930](https://github.com/airbytehq/airbyte/pull/31930) | Move pipx install to `airbyte-ci-dev`, and add auto-update feature targeting binary |
| 2.7.3 | [#32847](https://github.com/airbytehq/airbyte/pull/32847) | Improve --modified behaviour for pull requests. |
| 2.7.3 | [#32847](https://github.com/airbytehq/airbyte/pull/32847) | Improve --modified behaviour for pull requests. |
| 2.7.2 | [#32839](https://github.com/airbytehq/airbyte/pull/32839) | Revert changes in v2.7.1. |
| 2.7.1 | [#32806](https://github.com/airbytehq/airbyte/pull/32806) | Improve --modified behaviour for pull requests. |
| 2.7.1 | [#32806](https://github.com/airbytehq/airbyte/pull/32806) | Improve --modified behaviour for pull requests. |
| 2.7.0 | [#31930](https://github.com/airbytehq/airbyte/pull/31930) | Merge airbyte-ci-internal into airbyte-ci |
| 2.6.0 | [#31831](https://github.com/airbytehq/airbyte/pull/31831) | Add `airbyte-ci format` commands, remove connector-specific formatting check |
| 2.6.0 | [#31831](https://github.com/airbytehq/airbyte/pull/31831) | Add `airbyte-ci format` commands, remove connector-specific formatting check |
| 2.5.9 | [#32427](https://github.com/airbytehq/airbyte/pull/32427) | Re-enable caching for source-postgres |
| 2.5.8 | [#32402](https://github.com/airbytehq/airbyte/pull/32402) | Set Dagger Cloud token for airbyters only |
| 2.5.7 | [#31628](https://github.com/airbytehq/airbyte/pull/31628) | Add ClickPipelineContext class |
Expand Down
10 changes: 9 additions & 1 deletion airbyte-ci/connectors/pipelines/pipelines/airbyte_ci/connectors/build_image/commands.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -31,8 +31,15 @@
default=[LOCAL_BUILD_PLATFORM],
type=click.Choice(BUILD_PLATFORMS, case_sensitive=True),
)
@click.option(
"-t",
"--tag",
help="The tag to use for the built image.",
default="dev",
type=str,
)
@click.pass_context
async def build(ctx: click.Context, use_host_gradle_dist_tar: bool, build_architectures: List[str]) -> bool:
async def build(ctx: click.Context, use_host_gradle_dist_tar: bool, build_architectures: List[str], tag: str) -> bool:
"""Runs a build pipeline for the selected connectors."""
build_platforms = [dagger.Platform(architecture) for architecture in build_architectures]
main_logger.info(f"Building connectors for {build_platforms}, use --architecture to change this.")
Expand Down Expand Up @@ -69,6 +76,7 @@ async def build(ctx: click.Context, use_host_gradle_dist_tar: bool, build_archit
ctx.obj["concurrency"],
ctx.obj["dagger_logs_path"],
ctx.obj["execute_timeout"],
tag,
)

return True
7 changes: 5 additions & 2 deletions ...yte-ci/connectors/pipelines/pipelines/airbyte_ci/connectors/build_image/steps/__init__.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -32,12 +32,15 @@ async def run_connector_build(context: ConnectorContext) -> StepResult:
return await LANGUAGE_BUILD_CONNECTOR_MAPPING[context.connector.language](context)


async def run_connector_build_pipeline(context: ConnectorContext, semaphore: anyio.Semaphore) -> ConnectorReport:
async def run_connector_build_pipeline(
context: ConnectorContext, semaphore: anyio.Semaphore, image_tag: str
) -> ConnectorReport:
"""Run a build pipeline for a single connector.
Args:
context (ConnectorContext): The initialized connector context.
semaphore (anyio.Semaphore): The semaphore to use to limit the number of concurrent builds.
image_tag (str): The tag to use for the built image.
Returns:
ConnectorReport: The reports holding builds results.
"""
Expand All @@ -48,7 +51,7 @@ async def run_connector_build_pipeline(context: ConnectorContext, semaphore: any
per_platform_built_containers = build_result.output_artifact
step_results.append(build_result)
if context.is_local and build_result.status is StepStatus.SUCCESS:
load_image_result = await LoadContainerToLocalDockerHost(context, per_platform_built_containers).run()
load_image_result = await LoadContainerToLocalDockerHost(context, per_platform_built_containers, image_tag).run()
step_results.append(load_image_result)
context.report = ConnectorReport(context, step_results, name="BUILD RESULTS")
return context.report
17 changes: 8 additions & 9 deletions airbyte-ci/connectors/pipelines/pipelines/airbyte_ci/connectors/build_image/steps/common.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

import json
from abc import ABC
from typing import List, Tuple
from typing import List, Optional, Tuple

import docker
from dagger import Container, ExecError, Platform, QueryError
Expand Down Expand Up @@ -56,15 +56,14 @@ async def _build_connector(self, platform: Platform, *args) -> Container:


class LoadContainerToLocalDockerHost(Step):
IMAGE_TAG = "dev"

def __init__(self, context: ConnectorContext, containers: dict[Platform, Container]) -> None:
def __init__(self, context: ConnectorContext, containers: dict[Platform, Container], image_tag: Optional[str] = "dev") -> None:
super().__init__(context)
self.image_tag = image_tag
self.containers = containers

@property
def title(self):
return f"Load {self.image_name}:{self.IMAGE_TAG} to the local docker host."
return f"Load {self.image_name}:{self.image_tag} to the local docker host."

@property
def image_name(self) -> Tuple:
Expand All @@ -77,21 +76,21 @@ async def _run(self) -> StepResult:
return StepResult(
self,
StepStatus.FAILURE,
stderr=f"Failed to export the connector image {self.image_name}:{self.IMAGE_TAG} to a tarball.",
stderr=f"Failed to export the connector image {self.image_name}:{self.image_tag} to a tarball.",
)
try:
client = docker.from_env()
response = client.api.import_image_from_file(str(exported_tar_path), repository=self.image_name, tag=self.IMAGE_TAG)
response = client.api.import_image_from_file(str(exported_tar_path), repository=self.image_name, tag=self.image_tag)
try:
image_sha = json.loads(response)["status"]
except (json.JSONDecodeError, KeyError):
return StepResult(
self,
StepStatus.FAILURE,
stderr=f"Failed to import the connector image {self.image_name}:{self.IMAGE_TAG} to your Docker host: {response}",
stderr=f"Failed to import the connector image {self.image_name}:{self.image_tag} to your Docker host: {response}",
)
return StepResult(
self, StepStatus.SUCCESS, stdout=f"Loaded image {self.image_name}:{self.IMAGE_TAG} to your Docker host ({image_sha})."
self, StepStatus.SUCCESS, stdout=f"Loaded image {self.image_name}:{self.image_tag} to your Docker host ({image_sha})."
)
except docker.errors.DockerException as e:
return StepResult(self, StepStatus.FAILURE, stderr=f"Something went wrong while interacting with the local docker client: {e}")
2 changes: 1 addition & 1 deletion airbyte-ci/connectors/pipelines/pyproject.toml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ build-backend = "poetry.core.masonry.api"

[tool.poetry]
name = "pipelines"
version = "2.9.0"
version = "2.10.0"
description = "Packaged maintained by the connector operations team to perform CI for connectors' pipelines"
authors = ["Airbyte <contact@airbyte.io>"]

Expand Down
9 changes: 5 additions & 4 deletions airbyte-ci/connectors/pipelines/tests/test_build_image/test_steps/test_common.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -54,16 +54,17 @@ def bad_docker_host(self):

async def test_run(self, test_context, step):
"""Test that the step runs successfully and that the image is loaded in the local docker host."""
assert step.image_tag == "dev"
docker_client = docker.from_env()
step.IMAGE_TAG = "test-load-container"
step.image_tag = "test-load-container"
try:
docker_client.images.remove(f"{test_context.connector.metadata['dockerRepository']}:{step.IMAGE_TAG}")
docker_client.images.remove(f"{test_context.connector.metadata['dockerRepository']}:{step.image_tag}")
except docker.errors.ImageNotFound:
pass
result = await step.run()
assert result.status is StepStatus.SUCCESS
docker_client.images.get(f"{test_context.connector.metadata['dockerRepository']}:{step.IMAGE_TAG}")
docker_client.images.remove(f"{test_context.connector.metadata['dockerRepository']}:{step.IMAGE_TAG}")
docker_client.images.get(f"{test_context.connector.metadata['dockerRepository']}:{step.image_tag}")
docker_client.images.remove(f"{test_context.connector.metadata['dockerRepository']}:{step.image_tag}")

async def test_run_export_failure(self, step, mocker):
"""Test that the step fails if the export of the container fails."""
Expand Down

0 comments on commit fb4cf98

Please sign in to comment.