Skip to content

Remove use of s3 cloudfront mirror for downloads; other minor fixes #72

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
srowen wants to merge 1 commit into from
Closed

Remove use of s3 cloudfront mirror for downloads; other minor fixes #72

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 3 additions & 6 deletions downloads.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,12 +22,9 @@ $(document).ready(function() {
2. Choose a package type:
<select id="sparkPackageSelect" onChange="javascript:onPackageSelect();"></select><br>

3. Choose a download type:
<select id="sparkDownloadSelect" onChange="javascript:onDownloadSelect()"></select><br>
3. Download Spark: <span id="spanDownloadLink"></span>

4. Download Spark: <span id="spanDownloadLink"></span>

5. Verify this release using the <span id="sparkDownloadVerify"></span> and [project release KEYS](https://www.apache.org/dist/spark/KEYS).
4. Verify this release using the <span id="sparkDownloadVerify"></span> and [project release KEYS](https://www.apache.org/dist/spark/KEYS).

_Note: Starting version 2.0, Spark is built with Scala 2.11 by default.
Scala 2.10 users should download the Spark source package and build
Expand Down Expand Up @@ -62,7 +59,7 @@ If you are interested in working with the newest under-development code or contr
# Master development branch
git clone git://github.com/apache/spark.git

# 2.1 maintenance branch with stability fixes on top of Spark 2.2.0
# Maintenance branch with stability fixes on top of Spark 2.2.0
git clone git://github.com/apache/spark.git -b branch-2.2

Once you've downloaded Spark, you can find instructions for installing and building it on the <a href="{{site.baseurl}}/documentation.html">documentation page</a>.
Expand Down
81 changes: 18 additions & 63 deletions js/downloads.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ var sources = {pretty: "Source Code", tag: "sources"};
var hadoopFree = {pretty: "Pre-build with user-provided Apache Hadoop", tag: "without-hadoop"};
var hadoop1 = {pretty: "Pre-built for Apache Hadoop 1.X", tag: "hadoop1"};
var cdh4 = {pretty: "Pre-built for CDH 4", tag: "cdh4"};
var hadoop2 = {pretty: "Pre-built for Apache Hadoop 2.2", tag: "hadoop2"};
//var hadoop2 = {pretty: "Pre-built for Apache Hadoop 2.2", tag: "hadoop2"};
var hadoop2p3 = {pretty: "Pre-built for Apache Hadoop 2.3", tag: "hadoop2.3"};
var hadoop2p4 = {pretty: "Pre-built for Apache Hadoop 2.4", tag: "hadoop2.4"};
var hadoop2p6 = {pretty: "Pre-built for Apache Hadoop 2.6", tag: "hadoop2.6"};
Expand All @@ -24,44 +24,23 @@ var packagesV7 = [hadoop2p7, hadoop2p6, hadoop2p4, hadoop2p3, hadoopFree, source
// 2.2.0+
var packagesV8 = [hadoop2p7, hadoop2p6, hadoopFree, sources]

// addRelease("2.0.0-preview", new Date("05/24/2016"), sources.concat(packagesV7), true, false);

//addRelease("2.2.0", new Date("x/x/2017"), packagesV8, true);

addRelease("2.2.0", new Date("07/11/2017"), packagesV8, true);
addRelease("2.1.2", new Date("10/09/2017"), packagesV7, true);
addRelease("2.1.1", new Date("05/02/2017"), packagesV7, true);
addRelease("2.1.0", new Date("12/28/2016"), packagesV7, true);
addRelease("2.0.2", new Date("11/14/2016"), packagesV7, true);
addRelease("2.0.1", new Date("10/03/2016"), packagesV7, true);
addRelease("2.0.0", new Date("07/26/2016"), packagesV7, true);
//addRelease("2.0.0-preview", new Date("05/24/2016"), sources.concat(packagesV7), true, false);
addRelease("1.6.3", new Date("11/07/2016"), packagesV6, true);
addRelease("1.6.2", new Date("06/25/2016"), packagesV6, true);
addRelease("1.6.1", new Date("03/09/2016"), packagesV6, true);
addRelease("1.6.0", new Date("01/04/2016"), packagesV6, true);
addRelease("1.5.2", new Date("11/09/2015"), packagesV6, true);
addRelease("1.5.1", new Date("10/02/2015"), packagesV6, true);
addRelease("1.5.0", new Date("9/09/2015"), packagesV6, true);
addRelease("1.4.1", new Date("7/15/2015"), packagesV6, true);
addRelease("1.4.0", new Date("6/11/2015"), packagesV6, true);
// addRelease("1.3.1", new Date("4/17/2015"), packagesV5, true, true);
// addRelease("1.3.0", new Date("3/13/2015"), packagesV4, true, true);
// addRelease("1.2.2", new Date("4/17/2015"), packagesV4, true, true);
// addRelease("1.2.1", new Date("2/9/2015"), packagesV4, true, true);
// addRelease("1.2.0", new Date("12/18/2014"), packagesV4, true, true);
// addRelease("1.1.1", new Date("11/26/2014"), packagesV4, true, true);
// addRelease("1.1.0", new Date("9/11/2014"), packagesV4, true, true);
// addRelease("1.0.2", new Date("8/5/2014"), packagesV3, true, true);
// addRelease("1.0.1", new Date("7/11/2014"), packagesV3, false, true);
// addRelease("1.0.0", new Date("5/30/2014"), packagesV2, false, true);
// addRelease("0.9.2", new Date("7/23/2014"), packagesV2, true, true);
// addRelease("0.9.1", new Date("4/9/2014"), packagesV2, false, true);
// addRelease("0.9.0-incubating", new Date("2/2/2014"), packagesV2, false, true);
// addRelease("0.8.1-incubating", new Date("12/19/2013"), packagesV2, true, true);
// addRelease("0.8.0-incubating", new Date("9/25/2013"), packagesV1, true, true);
// addRelease("0.7.3", new Date("7/16/2013"), packagesV1, true, true);
// addRelease("0.7.2", new Date("2/6/2013"), packagesV1, false, true);
// addRelease("0.7.0", new Date("2/27/2013"), [sources], false, true);
//addRelease("1.5.2", new Date("11/09/2015"), packagesV6, true);
//addRelease("1.5.1", new Date("10/02/2015"), packagesV6, true);
//addRelease("1.5.0", new Date("9/09/2015"), packagesV6, true);
//addRelease("1.4.1", new Date("7/15/2015"), packagesV6, true);
//addRelease("1.4.0", new Date("6/11/2015"), packagesV6, true);

function append(el, contents) {
el.innerHTML += contents;
Expand Down Expand Up @@ -137,69 +116,45 @@ function onVersionSelect() {
append(packageSelect, option);
}

var href = "https://www.apache.org/dist/spark/spark-" + version + "/";
var href = "https://archive.apache.org/dist/spark/spark-" + version + "/";
var link = "<a href=\"" + href + "\">" + versionShort(version) + " signatures and checksums</a>";
append(verifyLink, link);

// Populate releases
onPackageSelect();
updateDownloadLink();
}

function onPackageSelect() {
var downloadSelect = document.getElementById("sparkDownloadSelect");
var packageSelect = document.getElementById("sparkPackageSelect");

empty(downloadSelect);

var pkg = getSelectedValue(packageSelect);

append(downloadSelect, "<option value='direct'>Direct Download</option>");
append(downloadSelect, "<option value='apache'>Select Apache Mirror</option>");
updateDownloadLink();
}

function onDownloadSelect() {
updateDownloadLink();
}

function updateDownloadLink() {
var versionSelect = document.getElementById("sparkVersionSelect");
var packageSelect = document.getElementById("sparkPackageSelect");
var downloadSelect = document.getElementById("sparkDownloadSelect");
var downloadLink = document.getElementById("spanDownloadLink");

empty(downloadLink);

var version = getSelectedValue(versionSelect);
var pkg = getSelectedValue(packageSelect);
var download = getSelectedValue(downloadSelect);


var artifactName = "spark-$ver-bin-$pkg.tgz"
.replace(/\$ver/g, version)
.replace(/\$pkg/g, pkg)
.replace(/-bin-sources/, ""); // special case for source packages

var link = "https://d3kbcqa49mib13.cloudfront.net/$artifact";
if (download == "apache") {
if (version < "1.6.3" ||
(version >= "2.0.0" && version <= "2.0.1") ||
(version >= "2.1.0" && version <= "2.1.0")) {
link = "https://archive.apache.org/dist/spark/spark-$ver/$artifact";
} else {
link = "https://www.apache.org/dyn/closer.lua/spark/spark-$ver/$artifact";
}
var link = "";
if (version < "1.6.3" ||
(version >= "2.0.0" && version <= "2.0.1") ||
(version >= "2.1.0" && version <= "2.1.1")) {
link = "https://archive.apache.org/dist/spark/spark-$ver/$artifact";
} else {
link = "https://www.apache.org/dyn/closer.lua/spark/spark-$ver/$artifact";
}
link = link
.replace(/\$ver/, version)
.replace(/\$artifact/, artifactName);
var text = link.split("/").reverse()[0];

var onClick = "trackOutboundLink(this, 'Release Download Links', " +
"'$download_$artifact'); return false;"
.replace(/\$download/, download)
.replace(/\$artifact/, artifactName);
var onClick =
"trackOutboundLink(this, 'Release Download Links', 'apache_$artifact'); return false;"
.replace(/\$artifact/, artifactName);

var contents = "<a href=\"" + link + "\" onClick=\"" + onClick + "\">" + text + "</a>";
append(downloadLink, contents);
Expand Down
6 changes: 1 addition & 5 deletions site/downloads.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -212,10 +212,6 @@ <h2 id="download-apache-spark">Download Apache Spark&#8482;</h2>
<p>Choose a package type:
<select id="sparkPackageSelect" onchange="javascript:onPackageSelect();"></select><br /></p>
</li>
<li>
<p>Choose a download type:
<select id="sparkDownloadSelect" onchange="javascript:onDownloadSelect()"></select><br /></p>
</li>
<li>
<p>Download Spark: <span id="spanDownloadLink"></span></p>
</li>
Expand Down Expand Up @@ -258,7 +254,7 @@ <h3 id="spark-source-code-management">Spark Source Code Management</h3>
<pre><code># Master development branch
git clone git://github.com/apache/spark.git

# 2.1 maintenance branch with stability fixes on top of Spark 2.2.0
# Maintenance branch with stability fixes on top of Spark 2.2.0
git clone git://github.com/apache/spark.git -b branch-2.2
</code></pre>

Expand Down
81 changes: 18 additions & 63 deletions site/js/downloads.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ var sources = {pretty: "Source Code", tag: "sources"};
var hadoopFree = {pretty: "Pre-build with user-provided Apache Hadoop", tag: "without-hadoop"};
var hadoop1 = {pretty: "Pre-built for Apache Hadoop 1.X", tag: "hadoop1"};
var cdh4 = {pretty: "Pre-built for CDH 4", tag: "cdh4"};
var hadoop2 = {pretty: "Pre-built for Apache Hadoop 2.2", tag: "hadoop2"};
//var hadoop2 = {pretty: "Pre-built for Apache Hadoop 2.2", tag: "hadoop2"};
var hadoop2p3 = {pretty: "Pre-built for Apache Hadoop 2.3", tag: "hadoop2.3"};
var hadoop2p4 = {pretty: "Pre-built for Apache Hadoop 2.4", tag: "hadoop2.4"};
var hadoop2p6 = {pretty: "Pre-built for Apache Hadoop 2.6", tag: "hadoop2.6"};
Expand All @@ -24,44 +24,23 @@ var packagesV7 = [hadoop2p7, hadoop2p6, hadoop2p4, hadoop2p3, hadoopFree, source
// 2.2.0+
var packagesV8 = [hadoop2p7, hadoop2p6, hadoopFree, sources]

// addRelease("2.0.0-preview", new Date("05/24/2016"), sources.concat(packagesV7), true, false);

//addRelease("2.2.0", new Date("x/x/2017"), packagesV8, true);

addRelease("2.2.0", new Date("07/11/2017"), packagesV8, true);
addRelease("2.1.2", new Date("10/09/2017"), packagesV7, true);
addRelease("2.1.1", new Date("05/02/2017"), packagesV7, true);
addRelease("2.1.0", new Date("12/28/2016"), packagesV7, true);
addRelease("2.0.2", new Date("11/14/2016"), packagesV7, true);
addRelease("2.0.1", new Date("10/03/2016"), packagesV7, true);
addRelease("2.0.0", new Date("07/26/2016"), packagesV7, true);
//addRelease("2.0.0-preview", new Date("05/24/2016"), sources.concat(packagesV7), true, false);
addRelease("1.6.3", new Date("11/07/2016"), packagesV6, true);
addRelease("1.6.2", new Date("06/25/2016"), packagesV6, true);
addRelease("1.6.1", new Date("03/09/2016"), packagesV6, true);
addRelease("1.6.0", new Date("01/04/2016"), packagesV6, true);
addRelease("1.5.2", new Date("11/09/2015"), packagesV6, true);
addRelease("1.5.1", new Date("10/02/2015"), packagesV6, true);
addRelease("1.5.0", new Date("9/09/2015"), packagesV6, true);
addRelease("1.4.1", new Date("7/15/2015"), packagesV6, true);
addRelease("1.4.0", new Date("6/11/2015"), packagesV6, true);
// addRelease("1.3.1", new Date("4/17/2015"), packagesV5, true, true);
// addRelease("1.3.0", new Date("3/13/2015"), packagesV4, true, true);
// addRelease("1.2.2", new Date("4/17/2015"), packagesV4, true, true);
// addRelease("1.2.1", new Date("2/9/2015"), packagesV4, true, true);
// addRelease("1.2.0", new Date("12/18/2014"), packagesV4, true, true);
// addRelease("1.1.1", new Date("11/26/2014"), packagesV4, true, true);
// addRelease("1.1.0", new Date("9/11/2014"), packagesV4, true, true);
// addRelease("1.0.2", new Date("8/5/2014"), packagesV3, true, true);
// addRelease("1.0.1", new Date("7/11/2014"), packagesV3, false, true);
// addRelease("1.0.0", new Date("5/30/2014"), packagesV2, false, true);
// addRelease("0.9.2", new Date("7/23/2014"), packagesV2, true, true);
// addRelease("0.9.1", new Date("4/9/2014"), packagesV2, false, true);
// addRelease("0.9.0-incubating", new Date("2/2/2014"), packagesV2, false, true);
// addRelease("0.8.1-incubating", new Date("12/19/2013"), packagesV2, true, true);
// addRelease("0.8.0-incubating", new Date("9/25/2013"), packagesV1, true, true);
// addRelease("0.7.3", new Date("7/16/2013"), packagesV1, true, true);
// addRelease("0.7.2", new Date("2/6/2013"), packagesV1, false, true);
// addRelease("0.7.0", new Date("2/27/2013"), [sources], false, true);
//addRelease("1.5.2", new Date("11/09/2015"), packagesV6, true);
//addRelease("1.5.1", new Date("10/02/2015"), packagesV6, true);
//addRelease("1.5.0", new Date("9/09/2015"), packagesV6, true);
//addRelease("1.4.1", new Date("7/15/2015"), packagesV6, true);
//addRelease("1.4.0", new Date("6/11/2015"), packagesV6, true);

function append(el, contents) {
el.innerHTML += contents;
Expand Down Expand Up @@ -137,69 +116,45 @@ function onVersionSelect() {
append(packageSelect, option);
}

var href = "https://www.apache.org/dist/spark/spark-" + version + "/";
var href = "https://archive.apache.org/dist/spark/spark-" + version + "/";
var link = "<a href=\"" + href + "\">" + versionShort(version) + " signatures and checksums</a>";
append(verifyLink, link);

// Populate releases
onPackageSelect();
updateDownloadLink();
}

function onPackageSelect() {
var downloadSelect = document.getElementById("sparkDownloadSelect");
var packageSelect = document.getElementById("sparkPackageSelect");

empty(downloadSelect);

var pkg = getSelectedValue(packageSelect);

append(downloadSelect, "<option value='direct'>Direct Download</option>");
append(downloadSelect, "<option value='apache'>Select Apache Mirror</option>");
updateDownloadLink();
}

function onDownloadSelect() {
updateDownloadLink();
}

function updateDownloadLink() {
var versionSelect = document.getElementById("sparkVersionSelect");
var packageSelect = document.getElementById("sparkPackageSelect");
var downloadSelect = document.getElementById("sparkDownloadSelect");
var downloadLink = document.getElementById("spanDownloadLink");

empty(downloadLink);

var version = getSelectedValue(versionSelect);
var pkg = getSelectedValue(packageSelect);
var download = getSelectedValue(downloadSelect);


var artifactName = "spark-$ver-bin-$pkg.tgz"
.replace(/\$ver/g, version)
.replace(/\$pkg/g, pkg)
.replace(/-bin-sources/, ""); // special case for source packages

var link = "https://d3kbcqa49mib13.cloudfront.net/$artifact";
if (download == "apache") {
if (version < "1.6.3" ||
(version >= "2.0.0" && version <= "2.0.1") ||
(version >= "2.1.0" && version <= "2.1.0")) {
link = "https://archive.apache.org/dist/spark/spark-$ver/$artifact";
} else {
link = "https://www.apache.org/dyn/closer.lua/spark/spark-$ver/$artifact";
}
var link = "";
if (version < "1.6.3" ||
(version >= "2.0.0" && version <= "2.0.1") ||
(version >= "2.1.0" && version <= "2.1.1")) {
link = "https://archive.apache.org/dist/spark/spark-$ver/$artifact";
} else {
link = "https://www.apache.org/dyn/closer.lua/spark/spark-$ver/$artifact";
}
link = link
.replace(/\$ver/, version)
.replace(/\$artifact/, artifactName);
var text = link.split("/").reverse()[0];

var onClick = "trackOutboundLink(this, 'Release Download Links', " +
"'$download_$artifact'); return false;"
.replace(/\$download/, download)
.replace(/\$artifact/, artifactName);
var onClick =
"trackOutboundLink(this, 'Release Download Links', 'apache_$artifact'); return false;"
.replace(/\$artifact/, artifactName);

var contents = "<a href=\"" + link + "\" onClick=\"" + onClick + "\">" + text + "</a>";
append(downloadLink, contents);
Expand Down
28 changes: 20 additions & 8 deletions site/release-process.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -416,15 +416,27 @@ <h4>Update the Spark Website</h4>
$ ln -s 1.1.1 latest
</code></pre>

<p>Next, update the rest of the Spark website. See how the previous releases are documented.
In particular, have a look at the changes to the <code>*.md</code> files in this commit (all the HTML
file changes are generated by <code>jekyll</code>).</p>
<p>Next, update the rest of the Spark website. See how the previous releases are documented
(all the HTML file changes are generated by <code>jekyll</code>).
In particular, update <code>documentation.md</code> to add link to <code>docs</code> for the previous release. Add
the new release to <code>js/downloads.js</code>. Check <code>security.md</code> for anything to update.</p>

<pre><code>$ git add 1.1.1
$ git commit -m "Add docs for Spark 1.1.1"
</code></pre>

<p>Then, create the release notes. The contributors list can be automatically generated through
<p>Then, create the release notes. Go to the
<a href="https://issues.apache.org/jira/projects/SPARK?selectedItem=com.atlassian.jira.jira-projects-plugin:release-page">release page in JIRA</a>,
pick the release version from the list, then click on &#8220;Release Notes&#8221;. Copy this URL and then make a short URL on
<a href="https://s.apache.org/">s.apache.org</a>, sign in to your Apache account, and pick the ID as something like
<code>spark-2.1.2</code>. Create a new release post under <code>releases/_posts</code> to include this short URL.</p>

<p>Then run <code>jekyll build</code> to update the <code>site</code> directory.</p>

<p>On a related note, make sure the version is marked as released on JIRA. Go find the release page as above, eg.,
<code>https://issues.apache.org/jira/projects/SPARK/versions/12340295</code>, and click the &#8220;Release&#8221; button on the right and enter the release date.</p>

<p>(Generally, this is only for major and minor, but not patch releases) The contributors list can be automatically generated through
<a href="https://github.com/apache/spark/blob/branch-1.1/dev/create-release/generate-contributors.py">this script</a>.
It accepts the tag that corresponds to the current release and another tag that
corresponds to the previous (not including maintenance release). For instance, if you are
Expand Down Expand Up @@ -465,13 +477,13 @@ <h4>Update the Spark Website</h4>
$ git log v1.1.1 --grep "$expr" --shortstat --oneline | grep -B 1 -e "[3-9][0-9][0-9] insert" -e "[1-9][1-9][1-9][1-9] insert" | grep SPARK &gt; large-patches.txt
</code></pre>

<p>Then, update the downloads page, and then the main page with a news item.</p>

<h4>Create an Announcement</h4>

<p>Once everything is working (website docs, website changes) create an announcement on the website
and then send an e-mail to the mailing list. Enjoy an adult beverage of your choice, and
congratulations on making a Spark release.</p>
and then send an e-mail to the mailing list. To create an announcement, create a post under
<code>news/_posts</code> and then run <code>jekyll build</code>.</p>

<p>Enjoy an adult beverage of your choice, and congratulations on making a Spark release.</p>

</div>
</div>
Expand Down