Add github action for packaging and publishing python packages to PyPI (#1592)

[vlcek]

Hello,

this PR relates to #1592. I tried to create a gits of the packaging and publish workflows to PyPI so people can more easily use Nickel withing Python code.

What I've managed to do here:

• multiplatform support
  • Linux via manylinux, both aarch64 and x86_64 and glibc and musl support
  • Windows
  • MacOS - both aarch64 and x86_64
• uploading to PyPI via trusted publishers
• support for creating and uploading build attestations via the official pypa GH action
• basic support for "test uploads" to TestPyPI (pypi deployment for testing package uploads)

Unfortunately, for sake of testing I had to rename pyckel package to py-nickel (and nickel as the python module), because the name on PyPI is already taken. The py-nickel is just a name I picked to progress further. If you like this approach, I can easily change the name to whatever you like.

I found testing github workflows in a forked repository rather hard and my inexperience with GitHub did not helped either so please take this as a draft :) I highly suspect that the workflow will need some adjustments to fit your release process, however, I am not sure if I can help you with this any further :)

Setting up repository on PyPI is quite easy, here is the official guide.
Basically, once the pypi repository is created, you just set up the "trusted publisher" on github and there is no need to set up any credentials - everything is handled via OIDC and trust setup between PyPI and GitHub.

You can check/test the PyPI uploads here https://test.pypi.org/project/py-nickel/

You can see the list of uploaded artifacts here https://test.pypi.org/project/py-nickel/1.9.0/#files
Attestation https://test.pypi.org/project/py-nickel/1.9.0/#py_nickel-1.9.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (some files don't have attestations, that's only because of my experiments with uploads via maturin).

Example of a slightly different workflow version here - I needed different triggers to check if the builds and publishing work.

[yannham]

Hi,

Thanks a lot for tackling this! This is great. We'll give it a look soon. Just to make sure I understand, the package that you published on TestPyPI has been done through this workflow, right?

I don't have a strong opinion about the name - I find pyckel nice but it's fine if we pick something else. However, although it's been taken on TestPyPI, it's not taken on the actual PyPI, if I understand correctly. Do you think we could keep it as pyckel once all the tests are satisfactory, or it'd be a pain to change back and forth? I don't know too much about Python convention, so if there's a official or unofficial naming convention for language bindings in PyPI, we could also just use that, to make it more discoverable.

[github-actions]

Bencher Report

Branch	add_python_packages_to_github_actions
Testbed	ubuntu-latest

Click to view all benchmark results

Benchmark	Latency	microseconds (µs)
fibonacci 10	📈 view plot 🚷 view threshold	498.87
foldl arrays 50	📈 view plot 🚷 view threshold	1,753.70
foldl arrays 500	📈 view plot 🚷 view threshold	6,804.20
foldr strings 50	📈 view plot 🚷 view threshold	7,148.10
foldr strings 500	📈 view plot 🚷 view threshold	61,711.00
generate normal 250	📈 view plot 🚷 view threshold	41,976.00
generate normal 50	📈 view plot 🚷 view threshold	2,054.10
generate normal unchecked 1000	📈 view plot 🚷 view threshold	3,332.00
generate normal unchecked 200	📈 view plot 🚷 view threshold	770.23
pidigits 100	📈 view plot 🚷 view threshold	3,221.10
pipe normal 20	📈 view plot 🚷 view threshold	1,527.10
pipe normal 200	📈 view plot 🚷 view threshold	10,025.00
product 30	📈 view plot 🚷 view threshold	853.20
scalar 10	📈 view plot 🚷 view threshold	1,556.60
sum 30	📈 view plot 🚷 view threshold	846.50

🐰 View full continuous benchmarking report in Bencher

[vlcek]

Just to make sure I understand, the package that you published on TestPyPI has been done through this workflow, right?

Yes, I wanted to make sure the whole workflow works as expected. You can check, for example, this job to see some details https://github.com/vlcek/nickel/actions/runs/12383038938/job/34565207353
It took me few iterations to get it working with attestations so that's the reason why you can see some differences between uploads for different platforms.

However, although it's been taken on TestPyPI, it's not taken on the actual PyPI, if I understand correctly.

Yes, I made change just as a workaround so I can test the recommended way of package publishing on TestPyPI.

From what I understand, the packages/projects on TestPiPY are regularly wiped and somewhere in some FAQ I read you can ask administrators to "hand it over" if you want to claim the name.

Just a word of caution. pyckel is quite similar to pickle which is part of the stdlib and it's a serialization library.
There is also pyckle on PyPI and that quite confused me when I started with this PR. At first I though pyckel was some play on words using the pickle library.
I am not native English speaker so maybe that's part of the problem (it took me a while before I noticed different letter order), however, maybe it will cause similar confusion to other people looking for Nickel python package.

Renaming packages later is... problematic and perhaps a different name would make python bindings on PyPI more discoverable. I really like Nickel and I hope it will gain more adoption and "infiltrating" Python ecosystem seems to me like a great help :)
Nonetheless, the choice is of course yours :) I don't mind changing the names as necessary to make official packages on PyPI a reality.

I don't know too much about Python convention, so if there's a official or unofficial naming convention for language bindings in PyPI, we could also just use that, to make it more discoverable.

I am no expert either. From what I read there is no strict convention. From what I saw, packages containing python bindings usually have py- prefix or -py postfix to the project name and tend to skip the prefix/postfix when importing the module (the same thing I've done).
Some Rust projects use -rs postfix.

[yannham]

I think the fact that Nickel is implemented in Rust is a bit of an implementation detail for Pythonistas (I don't think it deserves to be in the name). Several config languages just have the language as package name (Dhall or Jsonnet for example), but nickel is unfortunately also already taken. We could use nickel-lang, but the wrapper crate in this repo might become confusing. So maybe py-nickel is the best pick, indeed.

[suimong]

I think it's my earlier attempt on this (shortly after posting the issue) that occupied the pyckel name, then my focus turned away from Python, leaving a mess behind...Sorry about that, will clean it up shortly. Btw I also like pyckel

[suimong]

I deleted the pyckel project from TestPyPI. The name should be available now.

[yannham]

Thanks, I'm not very knowledgeable in Python packaging, but this looks good plus you've tested it before.

[yannham]

IIRC, this means: if the first condition evaluates to true, then && evaluates to its second condition, and || to its first condition. Otherwise, && evaluates to false and || to its second condition.

I'm not a GitHub action expert, and given that workflow expressions are limited (well, maybe they should use Nickel :p), that might be the idiomatic way to do things but I find this a bit confusing.

[vlcek]

I am also not a github action expert, this is something I've managed to find after googling for quite some time.
Github actions are based on typescript plugins and their behavior ... finicky :)
Using Nickel would be great:D

Nonetheless, this is something I've found that should be working, however, I could not test it properly since testing github actions in forks is severely limited by github.

[yannham]

Regarding the name, while I like pyckel as well, I also agree that it can be confusing with Pkl and other things. In the end a boring but discoverable name is probably better than a clever but confusing one, so I would vote to keep the naming as it is currently in the PR. @vlcek any reason to keep this PR as a draft anymore, or anything else left to do? I would be inclined to merge it as it is

[yannham]

I took the liberty to rebase, as there was a trivial conflict.

[vlcek]

Sorry for going dark on you for the last few weeks.

The reason I marked this as a Draft is that I wasn't really able to test all the github action release flows in this repository. Basically, github limits the git forks ability to trigger actions in a other way that editing them to use "push" actions.
This meas I had to test the core functionality using the "push" actions (e.g. triggering the action on git push) and later I've adjusted it so it is more in line to what I saw in the rest of the github release actions in this repository.

I hope that everything went alright during the merging and everything works as expected :)