Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CI (Buildkite): upload musl64 nightly binaries to S3 #42495

Closed
wants to merge 1 commit into from
Closed

CI (Buildkite): upload musl64 nightly binaries to S3 #42495

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .buildkite/pipelines/main/0_webui.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,5 +19,7 @@ steps:
# Our signed pipelines must have a `signature` or `signature_file` parameter that
# verifies the treehash of the pipeline itself and the inputs listed in `inputs`
signed_pipelines:
- pipeline: .buildkite/pipelines/main/misc/signed_pipeline_test.yml
- pipeline: .buildkite/pipelines/main/misc/signed_pipeline_test.yml
signature_file: .buildkite/pipelines/main/misc/signed_pipeline_test.yml.signature
- pipeline: .buildkite/pipelines/main/upload_s3/musl.yml
signature_file: .buildkite/pipelines/main/upload_s3/musl.yml.signature
Comment on lines +24 to +25
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of adding this to 0_webui.yml, which is somewhat disruptive, as you have seen, we should instead launch a signed_pipelines.yml which serves as a launchpad for our signed pipelines, just like how we have a launch_unsigned_builders.yml. This minimizes the amount of changes to 0_webui.yml that we have to perform.

When we do need to change 0_webui.yml, what we should do is test it out with the experimental pipeline, then add all the files we need, let it sit on master for a week, then add it to the 0_webui.yml file. This will give our changes a chance to propagate to most of the PRs that are being opened at the time.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So is the idea that 0_webui.yml (and the contents of the web UI) will look like this:

# This file represents what is put into the webUI.
# It is purely for keeping track of the changes we make to the webUI configuration; modifying this file has no effect.
# We use the `cryptic` buildkite plugin to provide secrets management, which requires some integration into the WebUI's steps.
agents:
  queue: "julia"
  sandbox.jl: "true"
steps:
  - label: ":unlock: Unlock secrets, launch pipelines"
    plugins:
      - staticfloat/cryptic:
          # Our list of pipelines that should be launched (but don't require a signature)
          # These pipelines can be modified by any contributor and CI will still run.
          # Build secrets will not be available in these pipelines (or their children)
          # but some of our signed pipelines can wait upon the completion of these unsigned
          # pipelines.
          unsigned_pipelines:
            - .buildkite/pipelines/main/launch_unsigned_builders.yml

          # Our signed pipelines must have a `signature` or `signature_file` parameter that
          # verifies the treehash of the pipeline itself and the inputs listed in `inputs`
          signed_pipelines:
            - pipeline:       .buildkite/pipelines/main/signed_builders.yml
              signature_file: .buildkite/pipelines/main/signed_builders.yml.signature

And then the .buildkite/pipelines/main/signed_pipelines.yml file looks like this:

agents:
  queue: "julia"
  sandbox.jl: "true"
steps:
  - label: "Signed pipelines"
    plugins:
      - staticfloat/cryptic:
          # Our signed pipelines must have a `signature` or `signature_file` parameter that
          # verifies the treehash of the pipeline itself and the inputs listed in `inputs`
          signed_pipelines:
            - pipeline:       .buildkite/pipelines/main/misc/signed_pipeline_test.yml
              signature_file: .buildkite/pipelines/main/misc/signed_pipeline_test.yml.signature
            - pipeline:       .buildkite/pipelines/main/upload_s3/musl.yml
              signature_file: .buildkite/pipelines/main/upload_s3/musl.yml.signature

Is this correct?

43 changes: 43 additions & 0 deletions .buildkite/pipelines/main/upload_s3/musl.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
agents:
queue: "julia"
# Only run on `sandbox.jl` machines (not `docker`-isolated ones) since we need nestable sandboxing
sandbox.jl: "true"
os: "linux"

steps:
- label: "upload_s3_musl64"
key: upload_s3_musl64
depends_on: package_musl64
if: |
(pipeline.slug == "julia-master") && (build.branch == "master") && (build.pull_request.id == null)
commands: |
echo "--- Print the full and short commit hashes"
SHORT_COMMIT_LENGTH=10
SHORT_COMMIT=`echo $${BUILDKITE_COMMIT:?} | cut -c1-$${SHORT_COMMIT_LENGTH:?}`
ARTIFACT_FILENAME="julia-$${SHORT_COMMIT:?}-musl64.tar.gz"

echo "--- Download build artifacts"
rm -rf $${ARTIFACT_FILENAME:?}
buildkite-agent artifact download $${ARTIFACT_FILENAME:?} .

echo "--- Upload build artifacts to S3"
# Upload to the following URL:
# https://julialangnightlies-s3.julialang.org/bin/linux/x64/julia-latest-musl64.tar.gz
export NEW_ARTIFACT_FILENAME="julia-latest-musl64.tar.gz"
export BUILDKITE_ARTIFACT_UPLOAD_DESTINATION="s3://julialangnightlies-s3/bin/linux/x64"
export BUILDKITE_S3_DEFAULT_REGION="us-east-1"
if [[ "$${ARTIFACT_FILENAME:?}" != "$${NEW_ARTIFACT_FILENAME:?}" ]]; then
mv $${ARTIFACT_FILENAME:?} $${NEW_ARTIFACT_FILENAME:?}
fi

# From the Buildkite documentation:
# > By default the agent will create objects with public-read permissions, so that
# > clicking on an artifact link in the Buildkite web interface can go directly to the
# > S3 object to be viewed in the browser.

buildkite-agent artifact upload $${ARTIFACT_FILENAME:?}
timeout_in_minutes: 60

# We must accept the signed job id secret in order to propagate secrets
env:
BUILDKITE_PLUGIN_CRYPTIC_BASE64_SIGNED_JOB_ID_SECRET: ${BUILDKITE_PLUGIN_CRYPTIC_BASE64_SIGNED_JOB_ID_SECRET?}
1 change: 1 addition & 0 deletions .buildkite/pipelines/main/upload_s3/musl.yml.signature
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Salted__j� �iE�.���� Z9���3r~G׾�Mւa�?��L,���^o�%���Z����8�2zX�h�p�J�j��7��
Expand Down