Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[INFRA-2615] [INFRA-1021] More flexible tiering #376

Merged
merged 9 commits into from
Jun 25, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 21 additions & 4 deletions README.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -126,11 +126,11 @@ These warnings are defined in the file `resources/warnings.json`.

Build (`mvn clean verify`) the generator and then invoke it as follows:

java -jar target/update-center2-*-SNAPSHOT-bin/update-center2-*-SNAPSHOT.jar --id default ...
java -Dfile.encoding=UTF-8 -jar target/update-center2-*-SNAPSHOT-bin/update-center2-*-SNAPSHOT.jar --id default ...

The tool also supports batch mode execution, generating multiple update sites with a single invocation:

java -jar target/update-center2-*-SNAPSHOT-bin/update-center2-*-SNAPSHOT.jar --arguments-file <filename.txt>
java -Dfile.encoding=UTF-8 -jar target/update-center2-*-SNAPSHOT-bin/update-center2-*-SNAPSHOT.jar --arguments-file <filename.txt>

`filename.txt` is a text file with a list of arguments on each line.
Lines that start with `#` are comments and ignored.
Expand All @@ -157,7 +157,7 @@ Example:

For a full list of arguments, invoke the tool as follows:

java -jar target/update-center2-*-SNAPSHOT-bin/update-center2-*-SNAPSHOT.jar --help
java -Dfile.encoding=UTF-8 -jar target/update-center2-*-SNAPSHOT-bin/update-center2-*-SNAPSHOT.jar --help

NOTE: `--help` isn't a real argument, but usage instructions are printed when an invalid argument is provided.

Expand All @@ -170,8 +170,15 @@ The following steps are therefore useful when trying to generate output correspo
1. Implement changes in `src/main/`.
2. Run `./site/generate.sh` until the Java tool is actually launched, then abort. This requires some environment variables to be defined.
3. Edit `tmp/args.lst`, changing or removing the `--key`, `--certificate`, and `--root-certificate` arguments as necessary.
4. Run `+java -jar target/update-center2-*-SNAPSHOT-bin/update-center2-*-SNAPSHOT.jar --arguments-file tmp/args.lst+`
4. Run `+java -Dfile.encoding=UTF-8 -jar target/update-center2-*-SNAPSHOT-bin/update-center2-*-SNAPSHOT.jar --arguments-file tmp/args.lst+`

Alternatively, the closest you can get to real executions in local development:

1. Implement changes in `src/main/`.
2. Deploy a snapshot using `mvn deploy`. Requires an account in the Jenkins project, see _Deploying changes_ below.
3. Edit `site/generate.sh` to reference the specific snapshot you deployed (including timestamp) where it is downloaded using `wget`, see previous build output.
4. Optionally, to speed things up, edit `site/generate.sh` and remove the arguments `--downloads-directory "$DOWNLOAD_ROOT_DIR"` from some of the invocations.
5. Run `./site/generate.sh <www-dir> <downloads-dir>`. The first argument is the output directory for metadata, the second argument is the output directory for downloads and unused unless the previous step 4 was skipped.

=== Running within an IDE

Expand All @@ -198,6 +205,16 @@ A new release (or at minimum a snapshot deployment) is needed, which is then ref
NOTE: As of May 2020, everyone can deploy snapshots to Artifactory, so permissions issues shouldn't hinder development.


=== Working with htaccess/mod_rewrite rules

The wrapper script `site/generate.sh` calls the script `site/generate-htaccess.sh` with chosen arguments.
The latter script will generate the `.htaccess` file mostly containing mod_rewrite rules to redirect requests to appropriate tiered update sites.
To learn more about tiers, see link:site/LAYOUT.md[LAYOUT.md].

To test changes to `site/generate-htaccess.sh`, run `site/test/test.sh`.
It executes `site/generate-htaccess.sh` and places it inside an Apache HTTPD Docker container and tests whether redirect rules are correctly applied.


=== Working with certificates

To sign JSON output files, create a development certificate:
Expand Down
30 changes: 30 additions & 0 deletions site/LAYOUT.md
Original file line number Diff line number Diff line change
Expand Up @@ -196,3 +196,33 @@ The top-level `.htaccess` file is created by `generate-htaccess.sh` and implemen
* Redirects from top-level release history and plugin versions JSON files to the real files in `current`
* Redirects for any other `.json` / `.json.html` files (i.e. tool downloader metadata in `updates/`) to the mirrors network. <!-- does this even work / matter? -->
* Redirects for `.war` and `.hpi` files in the `download/` directory tree to the mirrors network.


## Known consumers

The following is a list of known references to specific files or directories in the Jenkins project.
It is intended to ensure that Jenkins to update-center2 will not break other parts of project infrastructure.

This list is current as of June 2020, but may not be complete.

### jenkins-infra/jenkins.io

* `/latestCore.txt` in [fetch-external-resources](https://github.com/jenkins-infra/jenkins.io/blob/6eddbd1e2891a39a88f04387d2aea9f23bb2bdf1/scripts/fetch-external-resources#L18)
* `/stable/latestCore.txt` in [fetch-external-resources](https://github.com/jenkins-infra/jenkins.io/blob/6eddbd1e2891a39a88f04387d2aea9f23bb2bdf1/scripts/fetch-external-resources#L24)
* `/update-center.actual.json` in [fetch-external-resources](https://github.com/jenkins-infra/jenkins.io/blob/6eddbd1e2891a39a88f04387d2aea9f23bb2bdf1/scripts/fetch-external-resources#L48)
* `/release-history.json` in [Makefile](https://github.com/jenkins-infra/jenkins.io/blob/6eddbd1e2891a39a88f04387d2aea9f23bb2bdf1/Makefile#L46)

### jenkins-infra/plugin-site-api

* `/current/plugin-documentation-urls.json` in [WikiPluginDataParser.java](https://github.com/jenkins-infra/plugin-site-api/blob/290e6117eec06a70e2652e6270f26c3ab6e7058e/src/main/java/io/jenkins/plugins/generate/parsers/WikiPluginDataParser.java#L30)

### jenkinsci/plugin-installation-manager-tool

* `/update-center.actual.json` in [Settings.java](https://github.com/jenkinsci/plugin-installation-manager-tool/blob/master/plugin-management-library/src/main/java/io/jenkins/tools/pluginmanager/config/Settings.java#L13)
* `/experimental/update-center.actual.json` in [Settings.java](https://github.com/jenkinsci/plugin-installation-manager-tool/blob/master/plugin-management-library/src/main/java/io/jenkins/tools/pluginmanager/config/Settings.java#L15)
* `/current/plugin-versions.json` in [Settings.java](https://github.com/jenkinsci/plugin-installation-manager-tool/blob/master/plugin-management-library/src/main/java/io/jenkins/tools/pluginmanager/config/Settings.java#L18)

### jenkinsci/docker

* LTS tiered update sites per https://github.com/jenkinsci/docker/issues/954
* Tiered `latest/` directories per https://github.com/jenkinsci/docker/issues/953
60 changes: 23 additions & 37 deletions site/README.md
Original file line number Diff line number Diff line change
@@ -1,67 +1,53 @@
# OSS Update Center Site Architecture
This script generate the code and data behind https://updates.jenkins-ci.org/

The service this website provides is as follows.
The service this website provides is as follows:

1. Jenkins will hit well-known URLs hard-coded into Jenkins binaries with
the Jenkins version number attached as a query string, like
http://updates.jenkins-ci.org/update-center.json.html?version=1.345
https://updates.jenkins.io/update-center.json?version=1.345

1. We use the version number to redirect the traffic to the right update site,
among all the ones that we generate.

1. HTTP traffic will be directed to mirrors through http://mirror.jenkins-ci.org/
HTTPS traffic will be served locally because we don't have any HTTPS mirrors.
In this way, Jenkins running in HTTPS will not see the insecure content warning
in the browser.

## Multiple update sites for different version ranges

Update center metadata can contain only one version per a plugin.
Because newer versions of the same plugin may depend on newer version of Jenkins,
if we just serve one update center for every Jenkins out there, older versions of
Jenkins will see plugin versions that do not work with them, making it impossible to install
the said plugin. This is unfortunate because some younger versions of the plugin might have
worked with that Jenkins core. This creates a disincentive for plugin developers
to move to the new base version, and slows down the pace in which it adopts new core features.

So in this "v3" UC, we generate several update centers targeted for different
version ranges. Consider the following example:


### Dynamic update site tiers

1.532 1.554 1.565
-----------+-----------+-----------+---------------> 1.590
\ \ \
\ \ \
---------> ---------> ----------->
Update center metadata can contain only one version per a plugin.
Because newer versions of the same plugin may depend on newer version of Jenkins, if we just serve one update center for every Jenkins out there, older versions of Jenkins will see plugin versions that do not work with them, making it impossible to install the said plugin.
This is unfortunate because some younger versions of the plugin might have worked with that Jenkins core.
This creates a disincentive for plugin developers to move to the new base version, and slows down the pace in which it adopts new core features.

So we generate several update centers targeted for different version ranges.

The version range is defined based on LTS branch points. We pick up 3 or 4 recent
LTS branch points to create 6 or 8 ranges.
To accommodate all recent Jenkins releases, we first inspect all plugin releases for their Jenkins core dependencies.
We then generate tiered update sites for all releases identified this way that are more recent than a cutoff (~3 months).
Directories containing these tiered update sites have the prefix `dynamic-`.

* version in the range (,1.532] will see the UC that advertises 1.590 as the core release,
and they will only see plugins that are compatible with 1.532 or earlier.
mod_rewrite rules in an `.htaccess` file are then used to redirect requests from Jenkins versions to the next lower update site.
It will serve the newest release of each plugin that is compatible with the specified Jenkins version.
See [generate-htaccess.sh](generate-htaccess.sh) for how these rules are generated.

* next, version range in (1.532, 1.532.*], which is basically 1.532.x LTS, will see
the UC that advertises 1.532.x as the core release, and only see plugins that are
compatible with 1.532.* or earlier.
### Static update site tiers

* next, version in the range (1.532, 1.554] will see 1.590 as the core, and
only see plugins compatible with 1.554 or earlier.
Before June 2020, update site tiers were fixed:
Update sites were generated for the five most recent LTS baselines, and each update site would offer plugins compatible with the latest release of each LTS line.

* next, version range is (1,554, 1.554.*] and you get the idea
These update site tiers are *deprecated*: They are currently still being generated, but `.htaccess` no longer reference them.
See e.g. https://github.com/jenkinsci/docker/issues/954

* finally, the catch-all update center advertises 1.590 core release and latest plugins
regardless of their required core versions. This applies to (1.565,*]

## Redirection logic
## Generating update sites

[generate.sh](generate.sh) is run by [a CI job](https://trusted.ci.jenkins.io/job/update_center/)
and generates all the different sites as static files, and deploys the directory into Apache.

A part of this is [.htaccess](static/.htaccess) that uses `mod_rewrite` to
redirect inbound requests to the right version specific website.


## Layout
See [a separate doc](LAYOUT.md) for the layout of the generated update site.

See [a separate doc](LAYOUT.md) for the layout of the generated update site.
99 changes: 71 additions & 28 deletions site/generate-htaccess.sh
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
#!/bin/bash

USAGE="Usage: $0 [<LTS baseline> ...]
USAGE="Usage: $0 [<release> ...]
"

[[ $# -gt 0 ]] || { echo "${USAGE}Expected at least one argument." >&2 ; exit 1 ; }
Expand All @@ -12,66 +12,109 @@ set -o nounset
cat <<EOF
# THIS FILE IS GENERATED
#
# See: <https://github.com/jenkinsci/backend-update-center2/blob/master/site/generate-htaccess.sh>
# See: <https://github.com/jenkins-infra/update-center2/blob/master/site/generate-htaccess.sh>
RewriteEngine on

# If we have a match that looks like an LTS version, e.g. 1.554.1, redirect to stable-1.554
RewriteCond %{QUERY_STRING} ^.*version=([0-9]*\.[0-9]*)\..*$ [NC]
RewriteCond %{DOCUMENT_ROOT}/stable\-%1%{REQUEST_URI} -f

RewriteRule ^(update\-center.*\.(json|html)+|plugin\-documentation\-urls\.json|latestCore\.txt) /stable\-%1%{REQUEST_URI}? [NC,L,R=301]

EOF

echo "# Version-specific rulesets generated by generate.sh"
n=$#
versions=( "$@" )
newestStable=
oldestStable=
oldestWeekly=

for (( i = n-1 ; i >= 0 ; i-- )) ; do
version="${versions[i]}"
IFS=. read -ra versionPieces <<< "$version"

major=${versionPieces[0]}
minor=${versionPieces[1]}
patch=
if [[ ${#versionPieces[@]} -gt 2 ]] ; then
patch=${versionPieces[2]}
fi

if [[ "$version" =~ ^2[.][0-9]+[.][0-9]$ ]] ; then
# This is an LTS version
if [[ -z "$newestStable" ]] ; then
newestStable="$version"
fi

for ltsv in $@ ; do
v=${ltsv%.1} # support args both as '1.234' and '1.234.1'.
lastLTS=$v
cat <<EOF

# If major > ${major} or major = ${major} and minor >= ${minor} or major = ${major} and minor = ${minor} and patch >= ${patch}, use this LTS update site
RewriteCond %{QUERY_STRING} ^.*version=(\d)\.(\d+)\.(\d+)$ [NC]
RewriteCond %1 >${major}
RewriteRule ^(update\-center.*\.(json|html)+) /dynamic-stable-${major}\.${minor}\.${patch}%{REQUEST_URI}? [NC,L,R=301]
RewriteCond %{QUERY_STRING} ^.*version=(\d)\.(\d+)\.(\d+)$ [NC]
RewriteCond %1 =${major}
RewriteCond %2 >=${minor}
RewriteRule ^(update\-center.*\.(json|html)+) /dynamic-stable-${major}\.${minor}\.${patch}%{REQUEST_URI}? [NC,L,R=301]
RewriteCond %{QUERY_STRING} ^.*version=(\d)\.(\d+)\.(\d+)$ [NC]
RewriteCond %1 =${major}
RewriteCond %2 =${minor}
RewriteCond %3 >=${minor}
RewriteRule ^(update\-center.*\.(json|html)+) /dynamic-stable-${major}\.${minor}\.${patch}%{REQUEST_URI}? [NC,L,R=301]
EOF
oldestStable="$version"
else
# This is a weekly version
# Split our version up into an array for rewriting
# 1.651 becomes (1 651)
versionPieces=(${v//./ })
major=${versionPieces[0]}
minor=${versionPieces[1]}
oldestWeekly="$version"
cat <<EOF
RewriteCond %{QUERY_STRING} ^.*version=${major}\.(\d+)$ [NC]
RewriteCond %1 <=${minor}
RewriteRule ^(update\-center.*\.(json|html)+|plugin\-documentation\-urls\.json|latestCore\.txt) /${major}\.${minor}%{REQUEST_URI}? [NC,L,R=301]

# If major > ${major} or major = ${major} and minor >= ${minor}, use this weekly update site
RewriteCond %{QUERY_STRING} ^.*version=(\d)\.(\d+)$ [NC]
RewriteCond %1 >${major}
RewriteRule ^(update\-center.*\.(json|html)+) /dynamic-${major}\.${minor}%{REQUEST_URI}? [NC,L,R=301]
RewriteCond %{QUERY_STRING} ^.*version=(\d)\.(\d+)$ [NC]
RewriteCond %1 =${major}
RewriteCond %2 >${minor}
RewriteRule ^(update\-center.*\.(json|html)+) /dynamic-${major}\.${minor}%{REQUEST_URI}? [NC,L,R=301]
EOF

fi
done


lts=$1
versionPieces=(${lts//./ })
major=${versionPieces[0]}
minor=${versionPieces[1]}
echo "# First LTS update site (stable-$major.$minor) gets all older releases"
cat <<EOF


# First LTS update site (stable-$oldestStable) gets all older LTS releases

RewriteCond %{QUERY_STRING} ^.*version=\d\.(\d+)\.\d+$ [NC]
RewriteRule ^(update\-center.*\.(json|html)+|plugin\-documentation\-urls\.json|latestCore\.txt) /stable-${major}\.${minor}%{REQUEST_URI}? [NC,L,R=301]
RewriteRule ^(update\-center.*\.(json|html)+) /dynamic-stable-${oldestStable}%{REQUEST_URI}? [NC,L,R=301]

RewriteCond %{QUERY_STRING} ^.*version=\d\.(\d+)+$ [NC]
RewriteRule ^(update\-center.*\.(json|html)+) /dynamic-${oldestWeekly}%{REQUEST_URI}? [NC,L,R=301]

EOF


echo "# Add a RewriteRule for the last LTS we have, which should always rewrite to /stable"
echo "# Add a RewriteRule for /stable which will always rewrite to the last LTS site we have"
cat <<EOF
RewriteRule ^stable/(.+) "/stable-${lastLTS}/\$1" [NC,L,R=301]
RewriteRule ^stable/(.+) "/dynamic-stable-${newestStable}/\$1" [NC,L,R=301]

EOF



# Further static rules
cat <<EOF


# These are static rules

# If that all failed, but we have an update center, let's go to current
RewriteRule ^(update\-center.*\.(json|html)+|plugin\-documentation\-urls\.json|latestCore\.txt) /current%{REQUEST_URI}? [NC,L,R=301]
RewriteRule ^(update\-center.*\.(json|html)+|latestCore\.txt) /current%{REQUEST_URI}? [NC,L,R=301]

# Ensure /release-history.json goes to the right place
RewriteRule ^release\-history\.json+ /current%{REQUEST_URI}? [NC,L,R=301]

# Ensure /plugin-documentation-urls.json goes to the right place
RewriteRule ^plugin\-documentation\-urls\.json+ /current%{REQUEST_URI}? [NC,L,R=301]

# Ensure /plugin-versions.json goes to the right place
RewriteRule ^plugin\-versions\.json+ /current%{REQUEST_URI}? [NC,L,R=301]

Expand All @@ -87,7 +130,7 @@ RewriteCond %{HTTPS} !=on
RewriteRule (.*\.json(\.html)?)$ http://mirrors.jenkins-ci.org/updates/\$1



# TODO this might be unnecessary?
# download/* directories contain virtual URL spaces for redirecting download traffic to mirrors.
RedirectMatch 302 /download/war/([0-9]*\.[0-9]*\.[0-9]*/jenkins)\.war$ http://mirrors.jenkins-ci.org/war-stable/\$1.war
RedirectMatch 302 /download/war/(.*)\.war$ http://mirrors.jenkins-ci.org/war/\$1.war
Expand Down
Loading