Workflow updates and improvements #21

c-ryan-k · 2023-08-30T20:15:46Z

This PR makes the following enhancements to our workflow setup:

Added a README for workflows to help view and understand reusable workflow dependencies
Added an [auto] discriminator to workflows that are not triggered manually
Rearranged Python 3.8 tests to lint before unit tests for faster linting failures
Moved upload_wheel and update_private_index tasks to separate reusable workflows
Release workflow changes:
- SBOM now uses the actual extension version
- Added an input param to choose post-build deployment method
  - none finishes after successful build and test
  - upload-wheel will upload the wheel to storage
  - prod-release will draft a github release, upload the wheel to storage, and update the extension index
- Added a single approval job before deployments occur to avoid too many approval checks
  Note: update_private_index.yml still requires an additional approval so that we can update / edit the github release and smoke test / verify the uploaded wheel before deploying the index update

Individual workflow notes:

Upload wheel will currently fail gracefully (maybe it should fail with an error) but write output to the job summary if a wheel already exists in the storage account with the same name.
Similarly, private index updating will not occur unless:
- upload_wheel successfully uploaded a wheel (no duplicates, no errors)
- The wheel version does not already exist in the index

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.

Thank you for contributing to Azure Edge tooling!

This checklist is used to make sure that common guidelines for a pull request are followed.

General Guidelines

Intent for Production

It is expected that pull requests made to default or core branches such as dev or main are of production grade. Corollary to this, any merged contributions to these branches may be deployed in a public release at any given time. By checking this box, you agree and commit to the expected production quality of code.

Basic expectations

If introducing new functionality or modified behavior, are they backed by unit and/or integration tests?
In the same context as above are command names and their parameter definitions accurate? Do help docs have sufficient content?
Have all the relevant unit and integration tests pass? i.e. pytest <project root> -vv. Please provide evidence in the form of a screenshot showing a succesful run of tests locally OR a link to a test pipeline that has been run against the change-set.
Have linter checks passed using the .pylintrc and .flake8 rules? Look at the CI scripts for example usage.
Have extraneous print or debug statements, commented out code-blocks or code-statements (if any) been removed from the surface area of changes?
Have you made an entry in HISTORY.rst which concisely explains your user-facing feature or change?

Azure Edge CLI maintainers reserve the right to enforce any of the outlined expectations.

A PR is considered ready for review when all basic expectations have been met (or do not apply).

Uses latest version of SBOM tool Use real wheel version as parameter

Split up wheel upload and index update tasks Updated wheel upload task to check for existing wheels in container Added release build parameter to upload wheel

Github does not allow even reusable workflows to be in a child folder Modified workflow names to tag worfklows that can't be triggered as [auto] Fixed up workflow errors / inconsistencies

Approval step can run on ubuntu image for efficiency Modified run name to describe deployment method wheel upload does not rely on github release

…o index update

.github/workflows/README.md

digimaun · 2023-08-30T20:33:19Z

.github/workflows/update_private_index.yml

@@ -8,6 +8,7 @@ env:
  sas_token: ${{ secrets.sas_token }}
 jobs:
  update_index:
+    # prompt for explicit approval as this will update the index for customers
    environment: production


Does this mean double approval when release_strategy == prod-release?

Technically yes - my thinking is that the pipeline will kick off the github release and upload the wheel to the extension index, but then pause for another approval before the index update.

Github workflows don't allow you to access artifacts from the UI until a workflow is 100% completed, so pausing to check the wheel in the storage account or the GH release asset is our last chance to smoke test the actual built artifact before pushing it to the extension index (for now).

digimaun · 2023-08-30T20:35:08Z

.github/workflows/upload_wheel.yml

+        description: 'Whether or not the wheel was uploaded from this workflow'
+        value: ${{ jobs.upload_wheel.outputs.upload_success }}
+env:
+  storage_container_url: https://azedgecli.blob.core.windows.net/drop


we should parameterize storage_container_url

As in a runtime parameter, or a stored secret / environment variable?

If the second, it could be possible for us to create a secondary "environment" for the repo, and each "environment" could have separate SAS tokens / permissions for approvals / URLs for storage containers and indexes etc

right now, the latter option sounds good

.github/workflows/upload_wheel.yml

digimaun · 2023-08-30T21:43:04Z

Is this resolved in this PR

c-ryan-k · 2023-08-30T22:48:44Z

Bumped the SBOM tool version in case that helps resolve the build warning, but I might need to do some SBOM debugging / comparison to further narrow down the issue if it persists.

Also noting for future iterations we should try and more efficiently utilize deployment environments to keep configs/variables/paths/etc. tidy.

c-ryan-k added 16 commits August 25, 2023 13:43

Moved non-manual trigger actions into actions folder

342137e

SBOM updates

d0f4f2f

Uses latest version of SBOM tool Use real wheel version as parameter

Moved security checks into actions folder

86703e2

Release workflow updates

b8690ef

Split up wheel upload and index update tasks Updated wheel upload task to check for existing wheels in container Added release build parameter to upload wheel

Moved workflows back to main workflow directory

4bf4487

Github does not allow even reusable workflows to be in a child folder Modified workflow names to tag worfklows that can't be triggered as [auto] Fixed up workflow errors / inconsistencies

Refactored workflows for release scenario choices

fa2304d

Workflow cleanup and readme update

aa7a772

Moving approval stage to one jobthat will kick off deployment steps

7df1dfc

Moving lint checks before unit tests in py38 env

8139a1f

[Revert] testing workflow deployments in test storage container

345f4a6

Final tweaks

676455f

Approval step can run on ubuntu image for efficiency Modified run name to describe deployment method wheel upload does not rely on github release

Testing upload wheel duplication logic

325d4a1

fixup to previous commit

92b77a2

Fixup output variable of upload_wheel

4f9378a

Fix upload_wheel output (for real) and add explicit second approval t…

25d92b9

…o index update

Revert to prod storage account container

bb064ac

c-ryan-k requested a review from digimaun as a code owner August 30, 2023 20:15

digimaun reviewed Aug 30, 2023

View reviewed changes

.github/workflows/README.md Outdated Show resolved Hide resolved

digimaun reviewed Aug 30, 2023

View reviewed changes

.github/workflows/upload_wheel.yml Outdated Show resolved Hide resolved

Fix readme, remove unused var

38dd9ca

digimaun approved these changes Aug 30, 2023

View reviewed changes

c-ryan-k added 2 commits August 30, 2023 15:43

Bump SBOM version to 1.5.2

0499a7c

Fixed DiagnosticService resource check logic

1eb266e

c-ryan-k merged commit b928baf into dev Aug 30, 2023

digimaun deleted the feature/workflow_updates branch September 20, 2023 17:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Workflow updates and improvements #21

Workflow updates and improvements #21

c-ryan-k commented Aug 30, 2023

digimaun Aug 30, 2023

c-ryan-k Aug 30, 2023

digimaun Aug 30, 2023

c-ryan-k Aug 30, 2023

digimaun Aug 30, 2023

digimaun commented Aug 30, 2023

c-ryan-k commented Aug 30, 2023 •

edited

Loading

Workflow updates and improvements #21

Workflow updates and improvements #21

Conversation

c-ryan-k commented Aug 30, 2023

General Guidelines

digimaun Aug 30, 2023

Choose a reason for hiding this comment

c-ryan-k Aug 30, 2023

Choose a reason for hiding this comment

digimaun Aug 30, 2023

Choose a reason for hiding this comment

c-ryan-k Aug 30, 2023

Choose a reason for hiding this comment

digimaun Aug 30, 2023

Choose a reason for hiding this comment

digimaun commented Aug 30, 2023

c-ryan-k commented Aug 30, 2023 • edited Loading

c-ryan-k commented Aug 30, 2023 •

edited

Loading