update the release-process document #157

cloud-fan · 2018-11-02T14:34:08Z

Our document of doing release is out-dated. This PR updates the release doc to match the process of Spark 2.4.0 release.

cutting RC has been fully automated. People should always use the docker script, instead of setting up the environment manually. Then we don't need to update document everytime an environment change is needed.
when finalizing the release, suggest people to retain the generated docs of latest RC, and copy it to spark-website, instead of re-generating it.

cloud-fan · 2018-11-02T14:35:41Z

release-process.md

 Since it requires further manual steps, please also contact the <a href="mailto:private@spark.apache.org">PMC</a>.


-<h4> Remove RC artifacts from repositories</h4>


I moved the cleanup process to the end, since the docs are still needed when updating spark-website

cloud-fan · 2018-11-02T14:36:26Z

cc @vanzin @srowen @holdenk @felixcheung @jerryshao @gatorsmile

vanzin · 2018-11-02T15:34:57Z

release-process.md

-
-# Copy the new documentation to Apache
+# copy the docs of the voted RC to spark-website
+$ svn co "https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc5-docs" spark-docs


Since you may have this info locally, how many files are in the current docs build?

Last time I tried to checkout a huge directory from svn, the ASF infra guys blocked our network from talking to the ASF servers. So I'd avoid this if that directory is kinda large.

I don't remember if I actually did that or just thought about it, but one idea was to leave the docs build locally after the rc is generated, instead of cleaning it up when the docker script is gone. That would allow you to just upload your local copy to spark-website.

For 2.4.0 it's 3572 files, 145 MB.

During a release, we upload these doc files many times(for each RC), and download them once. Will this be a problem?

BTW during RC voting, if one people want to validate the doc, he has to check it out locally and start a server to look at the web pages. If the ASF infra blocks it, it will be hard for people to evaluate an RC.

That's a lot of files. I think the thing the ASF doesn't like is a burst of requests, and this might trigger it. Uploading is not a problem, since it's authenticated. Downloading is not, so it could be a DoS, which is why they do the blocking.

if one people want to validate the doc, he has to check it out locally

Actually they can see it directly on the ASF server without downloading it locally. Using a browser and manually looking at the logs automatically "throttles" the number of requests, so it shouldn't trigger any warnings to the infra guys.

I think it's safer to recommend keeping the docs for the last RC locally in case it passes, and only checkout from svn if necessary.

felixcheung · 2018-11-03T05:49:56Z

site/release-process.html

+<p>After the vote passes and you moved the approved RC to the release repository, you should delete
+the RC directories from the staging repository. For example:</p>
+
+<pre><code>svn rm https://dist.apache.org/repos/dist/dev/spark/v2.3.1-rc1-bin/ \


the version string in the same should be consistent through out I think? shouldn't this be v2.4.0-rc5?

jerryshao · 2018-11-05T02:22:51Z

release-process.md

+svn ci --username $ASF_USERNAME --password "$ASF_PASSWORD" -m"Update KEYS"
+```
+
+<h3>Installing docker</h3>


Maybe we should add an another paragraph to do the release without docker. It is pretty similar to docker, just running the script do-release.sh is enough. The only difference is that user should manually install all the dependencies as listed in docker file.

The thing I try to avoid is, maintain a document about how to set up the environment/dependencies. We should control it fully through the scripts.

I think we should not recommend people to do release without docker. If someone does want to run do-release.sh directly, he should read the docker file and other related scripts.

jerryshao · 2018-11-05T07:10:57Z

release-process.md


+<h3>Preparing gpg key</h3>
+
+You can skip this section if you have been a release manager before.


"if you have been a release manager before" => "if you've already uploaded your key". No need to be a release manager before :).

jerryshao · 2018-11-05T07:15:34Z

release-process.md

+# set JIRA_USERNAME, JIRA_PASSWORD, and GITHUB_API_TOKEN
+$ export JIRA_USERNAME=blabla
+$ export JIRA_PASSWORD=blabla
+$ export GITHUB_API_TOKEN=blabla


Do we require this?

I think so. It needs to access github data to get a people's full name.

I see. Got it.

jerryshao · 2018-11-05T07:17:03Z

release-process.md

+1. Create a git tag for the release candidate.
+1. Package the release binaries & sources, and upload them to a staging SVN repo.
+1. Create the release docs, and upload them to a staging SVN repo.
+1. Publish a snapshot to the Apache release repo.


Sorry what's the meaning of this step, I don't remember I have a such step.

- Publish a snapshot to the Apache release repo dev/create-release/release-build.sh publish-release``

I do think this is a bit confusing for new RM. Perhaps Apache release repo should be Apache staging Maven repo

Ohh, you mean pushing artifacts to apache staging maven repo? This seems confusing.

This is how we describe it before: https://github.com/apache/spark-website/pull/157/files#diff-12257e3523e2ce22fccd80687597e7cdL94

+1 that "Apache staging Maven repo" is cleaner, I'll update soon

vanzin · 2018-11-06T19:00:44Z

LGTM

holdenk · 2018-11-07T01:59:24Z

Thank for updating the release process docs!

cloud-fan · 2018-11-07T02:09:32Z

I'm having a problem to merge this PR. The merge script needs to know the original apache git repo, but do we have one for spark-website?

srowen · 2018-11-07T02:10:20Z

Setup should be the same as for the spark repo @cloud-fan ; here's mine for example:

$ git remote -v
apache	https://git-wip-us.apache.org/repos/asf/spark-website.git (fetch)
apache	https://git-wip-us.apache.org/repos/asf/spark-website.git (push)
apache-github	git://github.com/apache/spark-website (fetch)
apache-github	git://github.com/apache/spark-website (push)
origin	https://github.com/srowen/spark-website.git (fetch)
origin	https://github.com/srowen/spark-website.git (push)
upstream	https://github.com/apache/spark-website.git (fetch)
upstream	https://github.com/apache/spark-website.git (push)

cloud-fan · 2018-11-07T02:18:15Z

thanks @srowen , it works!

felixcheung · 2018-11-07T17:16:05Z

FYI, there is a merge script: https://github.com/apache/spark-website/blob/asf-site/merge_pr.py
EDIT: sorry, I might have misunderstood. is it the remote repo needs to be added?

cloud-fan · 2018-11-08T04:17:32Z

yes we need to add the remote repo, which I didn't know before

update the release-process document

705e910

cloud-fan commented Nov 2, 2018

View reviewed changes

srowen approved these changes Nov 2, 2018

View reviewed changes

vanzin reviewed Nov 2, 2018

View reviewed changes

more details

49c6009

felixcheung reviewed Nov 3, 2018

View reviewed changes

jerryshao reviewed Nov 5, 2018

View reviewed changes

address comment

31c06fa

jerryshao reviewed Nov 5, 2018

View reviewed changes

cloud-fan added 2 commits November 5, 2018 15:58

address comments

566d6b3

fix a typo

9c82a0d

cloud-fan mentioned this pull request Nov 5, 2018

Apache Spark 2.4.0 Release: Announcements & Release Notes #158

Merged

HyukjinKwon approved these changes Nov 7, 2018

View reviewed changes

asfgit closed this in 44d2552 Nov 7, 2018

cloud-fan deleted the do-release branch December 25, 2025 16:39

		Since it requires further manual steps, please also contact the <a href="mailto:private@spark.apache.org">PMC</a>.


		<h4> Remove RC artifacts from repositories</h4>


		<h3>Preparing gpg key</h3>

		You can skip this section if you have been a release manager before.

update the release-process document #157

update the release-process document #157

Uh oh!

Conversation

cloud-fan commented Nov 2, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cloud-fan commented Nov 2, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jerryshao Nov 5, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vanzin commented Nov 6, 2018

Uh oh!

holdenk commented Nov 7, 2018

Uh oh!

cloud-fan commented Nov 7, 2018

Uh oh!

srowen commented Nov 7, 2018

Uh oh!

cloud-fan commented Nov 7, 2018

Uh oh!

felixcheung commented Nov 7, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cloud-fan commented Nov 8, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

cloud-fan commented Nov 2, 2018 •

edited

Loading

jerryshao Nov 5, 2018 •

edited

Loading

felixcheung commented Nov 7, 2018 •

edited

Loading