Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

airbyte-ci: use binary in CI #32497

Merged
merged 2 commits into from
Nov 15, 2023

Conversation

alafanechere
Copy link
Contributor

@alafanechere alafanechere commented Nov 14, 2023

What

Use our airbyte-ci linux binary in the CI

How

  • Remove Install Python 3.10 and Install ci-connector-ops package steps from .github/actions/run-dagger-pipeline/action.yml
  • Add a Install airbyte-ci binary step to .github/actions/run-dagger-pipeline/action.yml which curls the binary and moves it to /usr/local/bin
  • Expose a airbyte_ci_binary_url option at the action level, defaults to latest: can be useful to pin airbyte-ci version in the CI or tryout pre-releases
  • Fix: hide dagger logs from public GHA logs by setting --disable-dagger-run flag

TODO

🚨 User Impact 🚨

  • Faster CI run: Python / Pipx install / airbyte-ci pix install is not required anymore: CI is ~1mn faster
  • No more "please pull master to get the latest airbyte-ci bug fixes": workflows will download the latest airbyte-ci version on each run
  • No more "We reached GHA rate limit on Python install": Python is not installed on each workflow run

Copy link

vercel bot commented Nov 14, 2023

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment
Name Status Preview Comments Updated (UTC)
airbyte-docs ⬜️ Ignored (Inspect) Visit Preview Nov 15, 2023 0:08am

Copy link
Contributor Author

Current dependencies on/for this PR:

This stack of pull requests is managed by Graphite.

@alafanechere alafanechere force-pushed the augustin/11-14-airbyte-ci_use_binary_in_CI branch 2 times, most recently from db2dcd1 to c25b46d Compare November 14, 2023 12:15
@alafanechere alafanechere marked this pull request as ready for review November 14, 2023 12:19
@alafanechere alafanechere requested a review from a team November 14, 2023 12:20
.github/actions/run-dagger-pipeline/action.yml Outdated Show resolved Hide resolved
- name: Run airbyte-ci
shell: bash
run: |
export _EXPERIMENTAL_DAGGER_RUNNER_HOST="unix:///var/run/buildkit/buildkitd.sock"
airbyte-ci --is-ci --gha-workflow-run-id=${{ github.run_id }} ${{ inputs.subcommand }} ${{ inputs.options }}
airbyte-ci --disable-dagger-run --is-ci --gha-workflow-run-id=${{ github.run_id }} ${{ inputs.subcommand }} ${{ inputs.options }}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what does --disable-dagger-run do?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is what enables the terminal UI when running locally.
We don't want this in CI because it might expose sensitive information / clutter the GHA logs.
It's called disable dagger run because getting the terminal UI is achieved by running dagger run airbyte-ci (using Dagger's CLI)

@CLAassistant
Copy link

CLAassistant commented Nov 15, 2023

CLA assistant check
All committers have signed the CLA.

@alafanechere alafanechere force-pushed the augustin/11-14-airbyte-ci_use_binary_in_CI branch from 4ba9772 to f703af6 Compare November 15, 2023 12:07
@octavia-squidington-iii octavia-squidington-iii removed CDK Connector Development Kit area/documentation Improvements or additions to documentation area/connectors Connector related issues labels Nov 15, 2023
@alafanechere alafanechere enabled auto-merge (squash) November 15, 2023 12:11
@alafanechere alafanechere merged commit 7ae7e42 into master Nov 15, 2023
16 checks passed
@alafanechere alafanechere deleted the augustin/11-14-airbyte-ci_use_binary_in_CI branch November 15, 2023 12:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment