document release process #875
Conversation
houqp
requested review from
Dandandan,
alamb,
andygrove,
nevi-me and
jorgecarleitao
|
The current release process is definitely more complicated than I would like it to be. One thing that we could simplify is to automate the crates.io and PyPI publish with Github action driven by git tag pushes. |
|
|
||
| ### Publish on PyPI | ||
|
|
||
| TODO |
There was a problem hiding this comment.
@jorgecarleitao Are you using maturin or twine to do the PyPI release?
There was a problem hiding this comment.
the actual job is still present in the repo here. It uses a github action (based on twine I believe).
There was a problem hiding this comment.
driven by tags. AFAIK currently we can't release binaries without first voting them (as they constitute artifacts beyond source code). This means that we also need to sign and push them to the apache server, i.e. we need to at least download them from github, sign them, upload them, and them upload them to pypi?
In summary, it is difficult to automatize. A way to go here would be for Apache to offer a shared key pair that we could place in github secrets that would be used to sign artifacts. However, this requires restrictions on who can push tags to the repos to avoid anyone from releasing.
Another issue is that the binary contains software that is not necessarily Apache 2.0 licensed, as the binary is compiled using all dependencies of the crates (e.g. Tokio is only MIT). I think that the same applies for Ballista docker images (not the Dockerfile), since the image now contains software beyond our direct licensing control (thinking about this notice in apache/arrow repo)
(admittedly, I did not think about this when donated python-datafusion)
There was a problem hiding this comment.
Ha, ok, let me think more about how to best handle pypi release tomorrow. I totally missed the binary release part as well :(
There was a problem hiding this comment.
we need to at least download them from github, sign them, upload them, and them upload them to pypi?
I believe this is not too bad to get started for now. Including the wheel binaries as part of the datafusion release artifacts for voting won't further complicate the current release process. The only difference is we will need to update create-tarball.sh to download the artifact from github and sign these wheels together with the datafusion source tarball.
I will also look into whether it's possible to automate the signing with Github Action by provisioning a Github Action key. Today we are already giving committer access to create signed release tarballs. I believe only committer and PMC member have access to create tags today? If so, tag push based release signing should work.
Another issue is that the binary contains software that is not necessarily Apache 2.0 licensed
I will prepare a NOTICE file similar to what arrow has tomorrow for the python binding.
|
I will fill in the python section after the upcoming 5.1.0 release since it will take us couple iteration to get the automation ready. |
|
Merging this one in to clear the PR queue (and we can update the docs for python in a follow on PR perhaps) |
Rendered README: https://github.com/houqp/arrow-datafusion/blob/qp_release_doc/dev/release/README.md
Which issue does this PR close?
Related to #771.
Rationale for this change
Formally document the release process so others can also follow it.
Update email template in create_tarball automation.
What changes are included in this PR?
Release process doc with minor change in automation script.
Are there any user-facing changes?
No