Add GitHub action for publishing artifacts to PyPI

[andersy005]

This addresses #5232 (comment), and will (hopefully) reduce the time it takes to cut a new release.

• Passes pre-commit run --all-files

[max-sixty]

Excellent! Thanks a lot @andersy005 .

We could change the "how to release" steps too, in another PR is fine (or I can do it).

[andersy005]

We could change the "how to release" steps too, in another PR is fine (or I can do it).

+1 for updating the how to release doc in another PR... I should point out that there are steps that this action doesn't address. For instance, step 2 and step 16

2. Confirm there are no commits on stable that are not yet merged
   (ref):

   git merge upstream/stable
   

3. Update the stable branch (used by ReadTheDocs) and switch back to master:

    git switch stable
    git rebase master
    git push --force upstream stable
    git switch master

How should we address these steps as part of the semi-automated release? Ccing @pydata/xarray in case they want to chime in

[max-sixty]

I think both of those should be script-able as actions:

git merge upstream/stable

This could run on each commit to stable, opening a PR to master if required, I think

Update the stable branch (used by ReadTheDocs) and switch back to master:

This could run on each tag, like the push to PyPI.

There's one action of "update the release tag in RTD" which may still be manual

(though no need to do them now, I generally have tried to make the process 25% easier each release, I think these incremental PRs are ideal)

[andersy005]

I think both of those should be script-able as actions:

git merge upstream/stable

This could run on each commit to stable, opening a PR to master if required, I think

Update the stable branch (used by ReadTheDocs) and switch back to master:

This could run on each tag, like the push to PyPI.

There's one action of "update the release tag in RTD" which may still be manual

(though no need to do them now, I generally have tried to make the process 25% easier each release, I think these incremental PRs are ideal)

👍🏽 for addressing these in separate PRs

[dopplershift]

@andersy005 I'm curious, why do you go with multiple jobs within the workflow, and using artifacts to transfer state between them, rather than multiple steps in a single job?

[andersy005]

@andersy005 I'm curious, why do you go with multiple jobs within the workflow, and using artifacts to transfer state between them, rather than multiple steps in a single job?

I have a tendency to split a workflow into multiple jobs because it makes reasoning about the workflow easy (at least for me :)) . However, I think using a single job here would reduce overhead since the logic isn't complex to warrant a need for multiple jobs...

[keewis]

if it's possible, it might still make sense to create and test the wheels for uploaded v0.x.y tags in a job and only upload to PyPI if a release was created through the webpage (which would save us from re-creating a release if the build failed).

[alexamici]

@andersy005 are you OK to try and merge this "as is" sometime today and then help us making a couple of tests on test.pypi.prg before doing the real release tomorrow?

(worst case: we release the old way)

[keewis]

from #5232:

@kmuehlbauer

Why not always upload pushes to master to test.pypi.org?

@andersy005, what do you think? This should be pretty easy if we have separate jobs for building / testing / uploads (where the upload to pypi is skipped unless a release was created)

[kmuehlbauer]

This should be pretty easy if we have separate jobs for building / testing / uploads (where the upload to pypi is skipped unless a release was created)

I've doing this for a while now in one of my projects. Not sure if it will be used much, but users could easily install versions from test.pypi.org and check it. I'm not saying that pip-installing from github is too complex. Another aspect is, that you could even install the package again after upload to check that everything works. But that may be overkill.

[andersy005]

Here's the workflow visualization graph. Let me know if the current job dependency is okay...

Also, someone with admin permissions on PyPI should make sure to get the necessary tokens from PyPI and TestPyPI and set them on this repo.

[andersy005]

This will be triggered on every push event on the master branch and release tag... Should we constrain this to be triggered on push events on the master branch only?

[keewis]

I think we should include tags but not releases (which are generated via the web page). That way, we can check the wheels on TestPyPI before creating the release and uploading to PyPI.

[andersy005]

This should be resolved by the latest commit...

[keewis]

should make sure to get the necessary tokens from PyPI and TestPyPI

I grabbed the xarray name on TestPyPI so TESTPYPI_TOKEN is already set

[alexamici]

Any consensus on whether this is ready to merge? We should start the release process shortly (once the US wakes up).

[keewis]

I'd say let's resolve #5244 (comment) and #5244 (comment) first

[keewis]

great, looks good. I'd say let's merge and check whether the packages are uploaded correctly to TestPyPI (PYPI_TOKEN will have to be set by either @max-sixty, @dcherian, @jhamman, or @shoyer).

Edit: @andersy005, feel free to merge once you think it is ready

[keewis]

I usually use created here, but this probably doesn't make much of a difference since we don't use "draft releases" (and I can't seem to find a page which explains the difference between the "activity types")

[andersy005]

Not sure if you've seen this page already. There's this note:

Note: The prereleased type will not trigger for pre-releases published from draft releases, but the published type will trigger. If you want a workflow to run when stable and pre-releases publish, subscribe to published instead of released and prereleased.

[keewis]

it seems PyPI does not allow packages with versions like 0.17.1.dev108+gf5e4fd5f. I'm not sure if it's possible to have setuptools-scm create a different version. Should we run this only when a tag is created?