[INFRA-2615] [INFRA-1021] More flexible tiering (#376)

Co-authored-by: Daniel Beck <daniel-beck@users.noreply.github.com>

README.adoc

@@ -126,11 +126,11 @@ These warnings are defined in the file `resources/warnings.json`.
 
 Build (`mvn clean verify`) the generator and then invoke it as follows:
 
-    java -jar target/update-center2-*-SNAPSHOT-bin/update-center2-*-SNAPSHOT.jar --id default ...
+    java -Dfile.encoding=UTF-8 -jar target/update-center2-*-SNAPSHOT-bin/update-center2-*-SNAPSHOT.jar --id default ...
 
 The tool also supports batch mode execution, generating multiple update sites with a single invocation:
 
-    java -jar target/update-center2-*-SNAPSHOT-bin/update-center2-*-SNAPSHOT.jar --arguments-file <filename.txt>
+    java -Dfile.encoding=UTF-8 -jar target/update-center2-*-SNAPSHOT-bin/update-center2-*-SNAPSHOT.jar --arguments-file <filename.txt>
 
 `filename.txt` is a text file with a list of arguments on each line.
 Lines that start with `#` are comments and ignored.
@@ -157,7 +157,7 @@ Example:
 
 For a full list of arguments, invoke the tool as follows:
 
-    java -jar target/update-center2-*-SNAPSHOT-bin/update-center2-*-SNAPSHOT.jar --help
+    java -Dfile.encoding=UTF-8 -jar target/update-center2-*-SNAPSHOT-bin/update-center2-*-SNAPSHOT.jar --help
 
 NOTE: `--help` isn't a real argument, but usage instructions are printed when an invalid argument is provided.
 
@@ -170,8 +170,15 @@ The following steps are therefore useful when trying to generate output correspo
 1. Implement changes in `src/main/`.
 2. Run `./site/generate.sh` until the Java tool is actually launched, then abort. This requires some environment variables to be defined.
 3. Edit `tmp/args.lst`, changing or removing the `--key`, `--certificate`, and `--root-certificate` arguments as necessary.
-4. Run `+java -jar target/update-center2-*-SNAPSHOT-bin/update-center2-*-SNAPSHOT.jar --arguments-file tmp/args.lst+`
+4. Run `+java -Dfile.encoding=UTF-8 -jar target/update-center2-*-SNAPSHOT-bin/update-center2-*-SNAPSHOT.jar --arguments-file tmp/args.lst+`
 
+Alternatively, the closest you can get to real executions in local development:
+
+1. Implement changes in `src/main/`.
+2. Deploy a snapshot using `mvn deploy`. Requires an account in the Jenkins project, see _Deploying changes_ below.
+3. Edit `site/generate.sh` to reference the specific snapshot you deployed (including timestamp) where it is downloaded using `wget`, see previous build output.
+4. Optionally, to speed things up, edit `site/generate.sh` and remove the arguments `--downloads-directory "$DOWNLOAD_ROOT_DIR"` from some of the invocations.
+5. Run `./site/generate.sh <www-dir> <downloads-dir>`. The first argument is the output directory for metadata, the second argument is the output directory for downloads and unused unless the previous step 4 was skipped.
 
 === Running within an IDE
 
@@ -198,6 +205,16 @@ A new release (or at minimum a snapshot deployment) is needed, which is then ref
 NOTE: As of May 2020, everyone can deploy snapshots to Artifactory, so permissions issues shouldn't hinder development.
 
 
+=== Working with htaccess/mod_rewrite rules
+
+The wrapper script `site/generate.sh` calls the script `site/generate-htaccess.sh` with chosen arguments.
+The latter script will generate the `.htaccess` file mostly containing mod_rewrite rules to redirect requests to appropriate tiered update sites.
+To learn more about tiers, see link:site/LAYOUT.md[LAYOUT.md].
+
+To test changes to `site/generate-htaccess.sh`, run `site/test/test.sh`.
+It executes `site/generate-htaccess.sh` and places it inside an Apache HTTPD Docker container and tests whether redirect rules are correctly applied.
+
+
 === Working with certificates
 
 To sign JSON output files, create a development certificate:

site/LAYOUT.md

@@ -196,3 +196,33 @@ The top-level `.htaccess` file is created by `generate-htaccess.sh` and implemen
 * Redirects from top-level release history and plugin versions JSON files to the real files in `current`
 * Redirects for any other `.json` / `.json.html` files (i.e. tool downloader metadata in `updates/`) to the mirrors network. <!-- does this even work / matter? -->
 * Redirects for `.war` and `.hpi` files in the `download/` directory tree to the mirrors network.
+
+
+## Known consumers
+
+The following is a list of known references to specific files or directories in the Jenkins project.
+It is intended to ensure that Jenkins to update-center2 will not break other parts of project infrastructure.
+
+This list is current as of June 2020, but may not be complete.
+
+### jenkins-infra/jenkins.io
+
+* `/latestCore.txt` in [fetch-external-resources](https://github.com/jenkins-infra/jenkins.io/blob/6eddbd1e2891a39a88f04387d2aea9f23bb2bdf1/scripts/fetch-external-resources#L18)
+* `/stable/latestCore.txt` in [fetch-external-resources](https://github.com/jenkins-infra/jenkins.io/blob/6eddbd1e2891a39a88f04387d2aea9f23bb2bdf1/scripts/fetch-external-resources#L24)
+* `/update-center.actual.json` in [fetch-external-resources](https://github.com/jenkins-infra/jenkins.io/blob/6eddbd1e2891a39a88f04387d2aea9f23bb2bdf1/scripts/fetch-external-resources#L48)
+* `/release-history.json` in [Makefile](https://github.com/jenkins-infra/jenkins.io/blob/6eddbd1e2891a39a88f04387d2aea9f23bb2bdf1/Makefile#L46)
+
+### jenkins-infra/plugin-site-api
+
+* `/current/plugin-documentation-urls.json` in [WikiPluginDataParser.java](https://github.com/jenkins-infra/plugin-site-api/blob/290e6117eec06a70e2652e6270f26c3ab6e7058e/src/main/java/io/jenkins/plugins/generate/parsers/WikiPluginDataParser.java#L30)
+
+### jenkinsci/plugin-installation-manager-tool
+
+* `/update-center.actual.json` in [Settings.java](https://github.com/jenkinsci/plugin-installation-manager-tool/blob/master/plugin-management-library/src/main/java/io/jenkins/tools/pluginmanager/config/Settings.java#L13)
+* `/experimental/update-center.actual.json` in [Settings.java](https://github.com/jenkinsci/plugin-installation-manager-tool/blob/master/plugin-management-library/src/main/java/io/jenkins/tools/pluginmanager/config/Settings.java#L15)
+* `/current/plugin-versions.json` in [Settings.java](https://github.com/jenkinsci/plugin-installation-manager-tool/blob/master/plugin-management-library/src/main/java/io/jenkins/tools/pluginmanager/config/Settings.java#L18)
+
+### jenkinsci/docker
+
+* LTS tiered update sites per https://github.com/jenkinsci/docker/issues/954
+* Tiered `latest/` directories per https://github.com/jenkinsci/docker/issues/953

site/README.md

@@ -1,67 +1,53 @@
 # OSS Update Center Site Architecture
 This script generate the code and data behind https://updates.jenkins-ci.org/
 
-The service this website provides is as follows.
+The service this website provides is as follows:
 
  1. Jenkins will hit well-known URLs hard-coded into Jenkins binaries with
     the Jenkins version number attached as a query string, like
-    http://updates.jenkins-ci.org/update-center.json.html?version=1.345
+    https://updates.jenkins.io/update-center.json?version=1.345
 
  1. We use the version number to redirect the traffic to the right update site,
     among all the ones that we generate.
 
- 1. HTTP traffic will be directed to mirrors through http://mirror.jenkins-ci.org/
-    HTTPS traffic will be served locally because we don't have any HTTPS mirrors.
-    In this way, Jenkins running in HTTPS will not see the insecure content warning
-    in the browser.
 
 ## Multiple update sites for different version ranges
 
-Update center metadata can contain only one version per a plugin.
-Because newer versions of the same plugin may depend on newer version of Jenkins,
-if we just serve one update center for every Jenkins out there, older versions of
-Jenkins will see plugin versions that do not work with them, making it impossible to install
-the said plugin. This is unfortunate because some younger versions of the plugin might have
-worked with that Jenkins core. This creates a disincentive for plugin developers
-to move to the new base version, and slows down the pace in which it adopts new core features.
-
-So in this "v3" UC, we generate several update centers targeted for different
-version ranges. Consider the following example:
-
-
+### Dynamic update site tiers
 
-              1.532      1.554        1.565
-     -----------+-----------+-----------+--------------->  1.590
-                 \           \           \
-                  \           \           \
-                   --------->  --------->  ----------->
+Update center metadata can contain only one version per a plugin.
+Because newer versions of the same plugin may depend on newer version of Jenkins, if we just serve one update center for every Jenkins out there, older versions of Jenkins will see plugin versions that do not work with them, making it impossible to install the said plugin.
+This is unfortunate because some younger versions of the plugin might have worked with that Jenkins core.
+This creates a disincentive for plugin developers to move to the new base version, and slows down the pace in which it adopts new core features.
 
+So we generate several update centers targeted for different version ranges.
 
-The version range is defined based on LTS branch points. We pick up 3 or 4 recent
-LTS branch points to create 6 or 8 ranges.
+To accommodate all recent Jenkins releases, we first inspect all plugin releases for their Jenkins core dependencies.
+We then generate tiered update sites for all releases identified this way that are more recent than a cutoff (~3 months).
+Directories containing these tiered update sites have the prefix `dynamic-`.
 
- * version in the range (,1.532] will see the UC that advertises 1.590 as the core release,
-   and they will only see plugins that are compatible with 1.532 or earlier.
+mod_rewrite rules in an `.htaccess` file are then used to redirect requests from Jenkins versions to the next lower update site.
+It will serve the newest release of each plugin that is compatible with the specified Jenkins version.
+See [generate-htaccess.sh](generate-htaccess.sh) for how these rules are generated.
 
- * next, version range in (1.532, 1.532.*], which is basically 1.532.x LTS, will see
-   the UC that advertises 1.532.x as the core release, and only see plugins that are
-   compatible with 1.532.* or earlier.
+### Static update site tiers
 
- * next, version in the range (1.532, 1.554] will see 1.590 as the core, and
-   only see plugins compatible with 1.554 or earlier.
+Before June 2020, update site tiers were fixed:
+Update sites were generated for the five most recent LTS baselines, and each update site would offer plugins compatible with the latest release of each LTS line.
 
- * next, version range is (1,554, 1.554.*] and you get the idea
+These update site tiers are *deprecated*: They are currently still being generated, but `.htaccess` no longer reference them.
+See e.g. https://github.com/jenkinsci/docker/issues/954
 
- * finally, the catch-all update center advertises 1.590 core release and latest plugins
-   regardless of their required core versions. This applies to (1.565,*]
 
-## Redirection logic
+## Generating update sites
 
 [generate.sh](generate.sh) is run by [a CI job](https://trusted.ci.jenkins.io/job/update_center/)
 and generates all the different sites as static files, and deploys the directory into Apache.
 
 A part of this is [.htaccess](static/.htaccess) that uses `mod_rewrite` to
 redirect inbound requests to the right version specific website.
 
+
 ## Layout
-See [a separate doc](LAYOUT.md) for the layout of the generated update site.
+
+See [a separate doc](LAYOUT.md) for the layout of the generated update site.

site/generate-htaccess.sh

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-USAGE="Usage: $0 [<LTS baseline> ...]
+USAGE="Usage: $0 [<release> ...]
 "
 
 [[ $# -gt 0 ]] || { echo "${USAGE}Expected at least one argument." >&2 ; exit 1 ; }
@@ -12,66 +12,109 @@ set -o nounset
 cat <<EOF
 # THIS FILE IS GENERATED
 #
-# See: <https://github.com/jenkinsci/backend-update-center2/blob/master/site/generate-htaccess.sh>
+# See: <https://github.com/jenkins-infra/update-center2/blob/master/site/generate-htaccess.sh>
 RewriteEngine on
 
-# If we have a match that looks like an LTS version, e.g. 1.554.1, redirect to stable-1.554
-RewriteCond %{QUERY_STRING} ^.*version=([0-9]*\.[0-9]*)\..*$ [NC]
-RewriteCond %{DOCUMENT_ROOT}/stable\-%1%{REQUEST_URI} -f
-
-RewriteRule ^(update\-center.*\.(json|html)+|plugin\-documentation\-urls\.json|latestCore\.txt) /stable\-%1%{REQUEST_URI}? [NC,L,R=301]
-
 EOF
 
 echo "# Version-specific rulesets generated by generate.sh"
+n=$#
+versions=( "$@" )
+newestStable=
+oldestStable=
+oldestWeekly=
+
+for (( i = n-1 ; i >= 0 ; i-- )) ; do
+  version="${versions[i]}"
+  IFS=. read -ra versionPieces <<< "$version"
+
+  major=${versionPieces[0]}
+  minor=${versionPieces[1]}
+  patch=
+  if [[ ${#versionPieces[@]} -gt 2 ]] ; then
+    patch=${versionPieces[2]}
+  fi
+
+  if [[ "$version" =~ ^2[.][0-9]+[.][0-9]$ ]] ; then
+    # This is an LTS version
+    if [[ -z "$newestStable" ]] ; then
+      newestStable="$version"
+    fi
 
-for ltsv in $@ ; do
-    v=${ltsv%.1} # support args both as '1.234' and '1.234.1'.
-    lastLTS=$v
+    cat <<EOF
 
+# If major > ${major} or major = ${major} and minor >= ${minor} or major = ${major} and minor = ${minor} and patch >= ${patch}, use this LTS update site
+RewriteCond %{QUERY_STRING} ^.*version=(\d)\.(\d+)\.(\d+)$ [NC]
+RewriteCond %1 >${major}
+RewriteRule ^(update\-center.*\.(json|html)+) /dynamic-stable-${major}\.${minor}\.${patch}%{REQUEST_URI}? [NC,L,R=301]
+RewriteCond %{QUERY_STRING} ^.*version=(\d)\.(\d+)\.(\d+)$ [NC]
+RewriteCond %1 =${major}
+RewriteCond %2 >=${minor}
+RewriteRule ^(update\-center.*\.(json|html)+) /dynamic-stable-${major}\.${minor}\.${patch}%{REQUEST_URI}? [NC,L,R=301]
+RewriteCond %{QUERY_STRING} ^.*version=(\d)\.(\d+)\.(\d+)$ [NC]
+RewriteCond %1 =${major}
+RewriteCond %2 =${minor}
+RewriteCond %3 >=${minor}
+RewriteRule ^(update\-center.*\.(json|html)+) /dynamic-stable-${major}\.${minor}\.${patch}%{REQUEST_URI}? [NC,L,R=301]
+EOF
+    oldestStable="$version"
+  else
+    # This is a weekly version
     # Split our version up into an array for rewriting
     # 1.651 becomes (1 651)
-    versionPieces=(${v//./ })
-    major=${versionPieces[0]}
-    minor=${versionPieces[1]}
+    oldestWeekly="$version"
     cat <<EOF
-RewriteCond %{QUERY_STRING} ^.*version=${major}\.(\d+)$ [NC]
-RewriteCond %1 <=${minor}
-RewriteRule ^(update\-center.*\.(json|html)+|plugin\-documentation\-urls\.json|latestCore\.txt) /${major}\.${minor}%{REQUEST_URI}? [NC,L,R=301]
 
+# If major > ${major} or major = ${major} and minor >= ${minor}, use this weekly update site
+RewriteCond %{QUERY_STRING} ^.*version=(\d)\.(\d+)$ [NC]
+RewriteCond %1 >${major}
+RewriteRule ^(update\-center.*\.(json|html)+) /dynamic-${major}\.${minor}%{REQUEST_URI}? [NC,L,R=301]
+RewriteCond %{QUERY_STRING} ^.*version=(\d)\.(\d+)$ [NC]
+RewriteCond %1 =${major}
+RewriteCond %2 >${minor}
+RewriteRule ^(update\-center.*\.(json|html)+) /dynamic-${major}\.${minor}%{REQUEST_URI}? [NC,L,R=301]
 EOF
 
+  fi
 done
 
-
-lts=$1
-versionPieces=(${lts//./ })
-major=${versionPieces[0]}
-minor=${versionPieces[1]}
-echo "# First LTS update site (stable-$major.$minor) gets all older releases"
 cat <<EOF
+
+
+# First LTS update site (stable-$oldestStable) gets all older LTS releases
+
 RewriteCond %{QUERY_STRING} ^.*version=\d\.(\d+)\.\d+$ [NC]
-RewriteRule ^(update\-center.*\.(json|html)+|plugin\-documentation\-urls\.json|latestCore\.txt) /stable-${major}\.${minor}%{REQUEST_URI}? [NC,L,R=301]
+RewriteRule ^(update\-center.*\.(json|html)+) /dynamic-stable-${oldestStable}%{REQUEST_URI}? [NC,L,R=301]
+
+RewriteCond %{QUERY_STRING} ^.*version=\d\.(\d+)+$ [NC]
+RewriteRule ^(update\-center.*\.(json|html)+) /dynamic-${oldestWeekly}%{REQUEST_URI}? [NC,L,R=301]
 
 EOF
 
 
-echo "# Add a RewriteRule for the last LTS we have, which should always rewrite to /stable"
+echo "# Add a RewriteRule for /stable which will always rewrite to the last LTS site we have"
 cat <<EOF
-RewriteRule ^stable/(.+) "/stable-${lastLTS}/\$1" [NC,L,R=301]
+RewriteRule ^stable/(.+) "/dynamic-stable-${newestStable}/\$1" [NC,L,R=301]
 
 EOF
 
 
 
 # Further static rules
 cat <<EOF
+
+
+# These are static rules
+
 # If that all failed, but we have an update center, let's go to current
-RewriteRule ^(update\-center.*\.(json|html)+|plugin\-documentation\-urls\.json|latestCore\.txt) /current%{REQUEST_URI}? [NC,L,R=301]
+RewriteRule ^(update\-center.*\.(json|html)+|latestCore\.txt) /current%{REQUEST_URI}? [NC,L,R=301]
 
 # Ensure /release-history.json goes to the right place
 RewriteRule ^release\-history\.json+ /current%{REQUEST_URI}? [NC,L,R=301]
 
+# Ensure /plugin-documentation-urls.json goes to the right place
+RewriteRule ^plugin\-documentation\-urls\.json+ /current%{REQUEST_URI}? [NC,L,R=301]
+
 # Ensure /plugin-versions.json goes to the right place
 RewriteRule ^plugin\-versions\.json+ /current%{REQUEST_URI}? [NC,L,R=301]
 
@@ -87,7 +130,7 @@ RewriteCond %{HTTPS} !=on
 RewriteRule (.*\.json(\.html)?)$ http://mirrors.jenkins-ci.org/updates/\$1
 
 
-
+# TODO this might be unnecessary?
 # download/* directories contain virtual URL spaces for redirecting download traffic to mirrors.
 RedirectMatch 302 /download/war/([0-9]*\.[0-9]*\.[0-9]*/jenkins)\.war$ http://mirrors.jenkins-ci.org/war-stable/\$1.war
 RedirectMatch 302 /download/war/(.*)\.war$ http://mirrors.jenkins-ci.org/war/\$1.war