Centralize More Universal CI Actions #8

rmartin16 · 2023-02-04T23:50:49Z

New CI Workflows

`pre-commit-run`

Runs pre-commit checks for an arbitrary repo.
pre-commit-source defaults to .[dev]; use pre-commit if the repo doesn't define a pinned version.
Example usage:

jobs:
  pre-commit:
    name: Pre-commit checks
    uses: beeware/.github/.github/workflows/pre-commit-run.yml@main
    with:
      pre-commit-source: './core[dev]'

`towncrier-run`

Runs tox -e towncrier-check for an arbitrary repo.
tox-source defaults to .[dev]; use tox if the repo doesn't define a pinned version.
Example usage:

jobs:
  towncrier:
    name: Check towncrier
    uses: beeware/.github/.github/workflows/towncrier-run.yml@main
    with:
      tox-source: './core[dev]'

`package-python`

Creates a Python package via tox -e package and uploads as an artifact named 'packages-<repo name>'.
Example usage:

jobs:
  package:
    name: Python Package
    uses: beeware/.github/.github/workflows/package-python.yml@main
    with:
      tox-source: './core[dev]'
      build-subdirectory: ${{ matrix.subdir }}
      distribution-path: '*/dist/*'
    strategy:
      matrix:
        subdir:
        - 'android'
        - 'cocoa'
        ...

`verify-app-build`

Builds an app for a target platform, format, and framework on a particular OS using the invoking repo.
runner-os and framework are required at minimum.
If target-platform and target-format are not explicitly define, then all apps compatible with the runner-os and framework are built.
- Example usage:

jobs:
  verify-apps:
    name: Build apps
    uses: beeware/.github/.github/workflows/verify-app-build.yml@main
    with:
      python-version: ${{ matrix.python-version }}
      python-source: ${{ matrix.python-source }}
      runner-os: windows-latest
      framework: ${{ matrix.framework }}
      target-platform: windows
      target-format: VisualStudio
    strategy:
      fail-fast: false
      matrix:
        python-version: [ '3.8', '3.9', '3.10', '3.11' ]
        python-source: [ 'github', 'winstore' ]
        framework: ['toga', 'pyside2', 'pyside6', 'ppb']

`install-briefcase`

In several CI runs, we need to install Briefcase; however, the appropriate version to install is dependent on the context.
When CI runs on the main branch in a repo, Briefcase should be installed from its main branch.
- Example: https://github.com/rmartin16/briefcase-windows-app-template-test-1/actions/runs/4086765608/jobs/7049375420
When CI runs for an arbitrary branch, Briefcase should be installed using branch as version.
- Example for v0.3.11 branch: https://github.com/rmartin16/briefcase-windows-app-template-test-1/actions/runs/4087055913/jobs/7049392731
- Example for v10.0.0 branch: https://github.com/rmartin16/briefcase-windows-app-template-test-1/actions/runs/4087072170/jobs/7049409704
When CI runs for a tag, Briefcase should be installed using the tag version.
- Example for v40.10.6 tag: https://github.com/rmartin16/briefcase-windows-app-template-test-1/actions/runs/4087695409/jobs/7049424029
- Example for v0.3.12 tag: https://github.com/rmartin16/briefcase-windows-app-template-test-1/actions/runs/4087711584/jobs/7049438896
When CI runs for a PR, Briefcase should be installed using PR's target branch version.
- For instance, when a PR for a template is targeting the v0.3.12 branch, the v0.3.12 version of Briefcase should be installed.
- Example for main branch: https://github.com/rmartin16/briefcase-windows-app-template-test-1/actions/runs/4086946506/jobs/7049469167
  - Of note, we are really close to the maximum windows path length (hence the build failures) 😟
- Example for v0.3.11 branch: https://github.com/rmartin16/briefcase-windows-app-template-test-1/actions/runs/4086976986/jobs/7049493706
- Example for v10.0.0 branch: https://github.com/rmartin16/briefcase-windows-app-template-test-1/actions/runs/4087022947/jobs/7049517402
  - Briefcase's main branch is used when the version is unknown.
Example usage:

jobs:
  job-name:
  runs-on: ubuntu-latest
  steps:
    - name: Install Briefcase
      uses: beeware/.github/.github/actions/install-briefcase@main

`ci` worklfow for `.github`

Exercises each of the workflows in the context of where they will be used.
In practice, this should help detect issues with changes to these workflows before those issues surface in the downstream repos.
This also means additional repos should be added here when such repos implement these workflows.

Implementation PRs

PR Checklist:

All new features have been tested
All new features have been documented
I have read the CONTRIBUTING.md file
I will abide by the code of conduct

rmartin16 · 2023-02-06T23:29:35Z

@freakboy3742, this should be ready for review now. I've got it working everywhere I think we need to use it.

Also, these scripts kinda grew organically as I was finding requirements for each repo....so fresh eyes may see the forest i can't see through all the trees right now.

freakboy3742

Woof... that's a lot of changes and dependent PRs :-)

I've flagged a few questions and comments inline; many of them are clarifications for my own benefit to make sure I fully understand what is going on; but broadly speaking, this is going in the right direction (and should make our lives a whole lot easier... once we land the changes 😅).

I haven't fully reviewed all the downstream repos yet - I've been working on the theory that we need to get this one right first, then get the other repos to be consistent with whatever lands. So - my comments here are mostly about the API surface and operation, guided by what I can see as the pre-existing CI usage, and intended usage once this PR is in place.

That said - a couple of things that I noticed:

In the downstream repos (esp. Briefcase and Toga), I can see you've updated the CI target, but not the package target; that will be pulling artefacts from the CI build, so it will be impacted by the name change on the package workflow.
I can't see anywhere in the Briefcase config that hits the Windows store python builds - am I missing something?

.github/workflows/pre-commit-run.yml

freakboy3742 · 2023-02-07T01:06:04Z

.github/workflows/towncrier-run.yml

+      with:
+        python-version: ${{ inputs.python-version }}
+        cache: 'pip'
+        cache-dependency-path: '**/setup.cfg'


I wonder if we might want to add pyproject.toml and setup.py to this list as well. We're still using setup.py in toga; I imagine that a move to full pyproject.toml format in the future is inevitable (either as part of a setuptools upgrade, or a move to hatch), so including those files would seem like a good protective mechanism.

Added **/pyproject.toml as a path.

freakboy3742 · 2023-02-07T01:06:22Z

.github/workflows/package-python.yml

+        with:
+          python-version: ${{ inputs.python-version }}
+          cache: 'pip'
+          cache-dependency-path: '**/setup.cfg'


As above - do we want to add pyproject.toml and setup.py to this list?

So....this starts to get a little tricky. The setup-python action hard errors if the files specified here don't exist. I'll give it a go and see if all the repos have these files.

(On a related note, this is why some of the other uses of setup-python don't/can't use this built in caching.)

I can save you the trouble; pyproject should exist everywhere (and if it doesn't, that's a bug); setup.py currently does, but should be slowly disappearing. (I've deleted it in the deb PR branch, for example).

Oh - and the template repositories won't have any of these files - but then, we won't be building Python packages for those repos.

The tests finished after I added pyproject.toml alongside setup.cfg.

freakboy3742 · 2023-02-07T01:07:37Z

.github/workflows/package-python.yml

@@ -0,0 +1,80 @@
+name: Create Python Package


The "package-python" name is a little misleading... at the very least, it's "Python-package"; but if it's a beeware project, it's going to be Python regardless.

That's definitely fair.

With that, GitHub currently has a limitation requiring all workflows to exist in the .github/workflows/ directory. So, while we may not end up with too many workflows, I was looking for a way for workflows related to the same thing/tool/concept to sort well alphabetically. That's why I started the workflows for pre-commit with pre-commit and same with towncrier. This leads, though, to a bit of awkward grammar with the action word coming last.

With that in mind, perhaps package-beeware-create.yml best follows those rules (although equally awkward)....and perhaps where a workflow like package-beeware-deploy.yml could theoretically exist alongside it...

Landed on python-package-create.yml.

freakboy3742 · 2023-02-07T01:09:40Z

.github/actions/install-briefcase/action.yml

+
+inputs:
+  briefcase-url:
+    description: 'URL to Briefcase git repository.'


We've been trying to converge on " rather than ' in config files (for broad consistency with black); is there a particular reason you've reverted a bunch of those changes and moved to '?

Mostly because " and ' have different functional meanings in YAML....while in Python they are functionally equivalent.

Double quoted strings in YAML pass through a level of escape parsing that single quotes strings do not. So, really, I'm trying to emphasize that I literally want the value between the two single quotes without any interpretation.

That said, none of this is particularly consistent. For instance, all the places that GitHub is substituting values for ${{ github.event_name }} (and the like), practically none of those are quoted at all.

So, unless this is compelling, I'm fine converging on double quotes. I also just realized that I was confusing TOML with YAML; TOML is a bit more restrictive about double quotes...

.github/actions/install-briefcase/action.yml

freakboy3742 · 2023-02-07T01:27:00Z

.github/workflows/verify-app-build.yml

+      run: python -m pip install dist/briefcase-*.whl
+
+    - name: Install Briefcase
+      if: endsWith(inputs.repository, 'briefcase') != true


Is this not just !endWith...?

definitely is....my brain didn't want to believe ! was supported or something...

Updated.

interestingly....GitHub allows endsWith(inputs.repository, 'briefcase') != true but requires the brackets for ${{ !endsWith(inputs.repository, 'briefcase') }}....hmm...

.github/workflows/verify-app-build.yml

.github/actions/install-briefcase/action.yml

rmartin16 · 2023-02-07T04:34:03Z

I haven't fully reviewed all the downstream repos yet - I've been working on the theory that we need to get this one right first, then get the other repos to be consistent with whatever lands. So - my comments here are mostly about the API surface and operation, guided by what I can see as the pre-existing CI usage, and intended usage once this PR is in place.

This is definitely the approach I intend.

That said - a couple of things that I noticed:

In the downstream repos (esp. Briefcase and Toga), I can see you've updated the CI target, but not the package target; that will be pulling artefacts from the CI build, so it will be impacted by the name change on the package workflow.

I think you're talking about, e.g. the release.yml in Briefcase where it download the uploaded package. That's definitely a good point. This is one of my weaker areas in understanding about the CI processes.

Reviewing release.yml, it runs ci.yml and then creates a release with the produced artifact. Based on that, I think I'll update Briefcase's ci.yml to propagate the artifact-name output from python-package-create.yml so release.yml can use it.

I can't see anywhere in the Briefcase config that hits the Windows store python builds - am I missing something?

There's still the extra include to run Briefcase unit tests against one version of Windows Store Python:

        # Run tests against the latest Windows Store Python
        - platform: "windows"
          python-version: "winstore"
          experimental: false

rmartin16 · 2023-02-07T23:15:40Z

I removed the testing code last night that was allowing app-build-verify.yml to work from the branch for this PR. So, I'll need to wait until this is merged to finish updating the dependent PRs.

    - name: Checkout beeware/.github
      uses: actions/checkout@v3.3.0
      with:
        repository: "beeware/.github"
        path: "beeware-.github"

freakboy3742

I picked up a couple of minor whitespace issues which I've pushed; but otherwise, I think this is good to go.

rmartin16 · 2023-02-08T00:00:19Z

ohh. interesting....well, I've added .pre-commit-config.yaml to the other PRs....probably should have done that here as well.

freakboy3742 · 2023-02-08T00:01:23Z

I haven't merged yet - so if there's something else you want to squeeze in here, I can hold off...

rmartin16 · 2023-02-08T00:17:30Z

All right. Added the same basic .pre-commit-config.yaml I put in the other repos.

rmartin16 force-pushed the shared-ci branch 30 times, most recently from 8f40a7e to 9ea0e60 Compare February 5, 2023 18:15

Introduce reusable workflows for common CI tasks

dffa5ce

rmartin16 force-pushed the shared-ci branch from 05120fd to dffa5ce Compare February 6, 2023 19:28

rmartin16 marked this pull request as ready for review February 6, 2023 23:26

freakboy3742 requested changes Feb 7, 2023

View reviewed changes

rmartin16 force-pushed the shared-ci branch 3 times, most recently from 97893cc to 4b8d858 Compare February 7, 2023 04:16

Use double quotes in CI yaml and cache pip with pyproject.toml

b2b17e4

rmartin16 force-pushed the shared-ci branch from 4b8d858 to b2b17e4 Compare February 7, 2023 15:19

rmartin16 requested a review from freakboy3742 February 7, 2023 15:47

Minor whitespace tweaks.

d334bc1

freakboy3742 approved these changes Feb 7, 2023

View reviewed changes

rmartin16 added 2 commits February 7, 2023 19:06

Add basic pre-commit configuration

bcf23bf

Run pre-commit during CI

42dfffa

rmartin16 force-pushed the shared-ci branch from fcecb50 to 42dfffa Compare February 8, 2023 00:13

Cosmetic target renames, plus ensure an up-to-date pip.

989e339

freakboy3742 merged commit 83927d1 into beeware:main Feb 8, 2023

rmartin16 deleted the shared-ci branch February 8, 2023 01:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Centralize More Universal CI Actions #8

Centralize More Universal CI Actions #8

rmartin16 commented Feb 4, 2023 •

edited

Loading

rmartin16 commented Feb 6, 2023 •

edited

Loading

freakboy3742 left a comment

freakboy3742 Feb 7, 2023

rmartin16 Feb 7, 2023

freakboy3742 Feb 7, 2023

rmartin16 Feb 7, 2023 •

edited

Loading

freakboy3742 Feb 7, 2023

freakboy3742 Feb 7, 2023

rmartin16 Feb 7, 2023

freakboy3742 Feb 7, 2023

rmartin16 Feb 7, 2023 •

edited

Loading

rmartin16 Feb 7, 2023

freakboy3742 Feb 7, 2023

rmartin16 Feb 7, 2023

freakboy3742 Feb 7, 2023

rmartin16 Feb 7, 2023

rmartin16 Feb 7, 2023 •

edited

Loading

rmartin16 commented Feb 7, 2023

rmartin16 commented Feb 7, 2023 •

edited

Loading

freakboy3742 left a comment

rmartin16 commented Feb 8, 2023

freakboy3742 commented Feb 8, 2023

rmartin16 commented Feb 8, 2023

Centralize More Universal CI Actions #8

Centralize More Universal CI Actions #8

Conversation

rmartin16 commented Feb 4, 2023 • edited Loading

New CI Workflows

pre-commit-run

towncrier-run

package-python

verify-app-build

install-briefcase

ci worklfow for .github

Implementation PRs

PR Checklist:

rmartin16 commented Feb 6, 2023 • edited Loading

freakboy3742 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rmartin16 Feb 7, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rmartin16 Feb 7, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rmartin16 Feb 7, 2023 • edited Loading

Choose a reason for hiding this comment

rmartin16 commented Feb 7, 2023

rmartin16 commented Feb 7, 2023 • edited Loading

freakboy3742 left a comment

Choose a reason for hiding this comment

rmartin16 commented Feb 8, 2023

freakboy3742 commented Feb 8, 2023

rmartin16 commented Feb 8, 2023

rmartin16 commented Feb 4, 2023 •

edited

Loading

`pre-commit-run`

`towncrier-run`

`package-python`

`verify-app-build`

`install-briefcase`

`ci` worklfow for `.github`

rmartin16 commented Feb 6, 2023 •

edited

Loading

rmartin16 Feb 7, 2023 •

edited

Loading

rmartin16 Feb 7, 2023 •

edited

Loading

rmartin16 Feb 7, 2023 •

edited

Loading

rmartin16 commented Feb 7, 2023 •

edited

Loading