Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[INFRA-2615] [INFRA-1021] More flexible tiering #376

Merged
merged 9 commits into from
Jun 25, 2020

Conversation

daniel-beck
Copy link
Contributor

@daniel-beck daniel-beck commented May 16, 2020

Mostly INFRA-2615, but fixed INFRA-1021 in the process.

Announcement/discussion: https://groups.google.com/d/msg/jenkinsci-dev/aNWx61URGdk/qgT8wxojDAAJ

How this works:
We add a mode to update-center2 that inspects all plugin releases for their core dependency, and every core release used as any plugin's core dependency that's newer than ~13 months (400 days) will be added to this mode's output file. Then, the wrapper bash scripts use that as input to generate .htaccess redirect rules and the arguments file for the "real" invocation of update-center2.

Output file example from today, mid May 2020 ("tiers.json"):

{
    "stableCores": [
        "2.164.2",
        "2.164.3",
        "2.176.1",
        "2.176.2",
        "2.176.3",
        "2.176.4",
        "2.190.1",
        "2.190.3",
        "2.204.1",
        "2.204.2",
        "2.204.4",
        "2.204.6",
        "2.222.1"
    ],
    "weeklyCores": [
        "2.172",
        "2.173",
        "2.176",
        "2.177",
        "2.181",
        "2.184",
        "2.185",
        "2.191",
        "2.195",
        "2.199",
        "2.204",
        "2.205",
        "2.212",
        "2.217",
        "2.222",
        "2.223"
    ]
}

Both 2.172 and 2.164.2 were released April 10, 2019. This is the newest release that is both used as a baseline and is older than 400 days, ensuring we support whatever release was current 400 days ago.

This file is not part of actual output, but just an intermediate step. Since .htaccess generation happens outside update-center2, we cannot do everything in one invocation of update-center2, unless we integrate .htaccess into it.

This is intended to replace querying the Artifactory API for the 5 newest "LTS.1" releases. For now, we do both. Version redirects will only redirect to the dynamic tiers though, while the fixed tiers from before only exist if they're hardcoded in clients.

@@ -259,7 +272,7 @@ private String updateCenterPostMessageHtml(String updateCenterJson) {

private static void writeToFile(String string, final File file) throws IOException {
File parentFile = file.getParentFile();
if (!parentFile.isDirectory() && !parentFile.mkdirs()) {
if (parentFile != null && !parentFile.isDirectory() && !parentFile.mkdirs()) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixes an NPE when there's no parent file.

for (VersionNumber dependencyVersion : coreDependencyVersions) {
final JenkinsWar war = allJenkinsWarsByVersionNumber.get(dependencyVersion);
if (war == null) {
LOGGER.log(Level.INFO, "Did not find declared core dependency version among all core releases: " + dependencyVersion.toString());
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Someone released a plugin based on an incremental.

@jglick
Copy link
Contributor

jglick commented May 18, 2020

every core release used as any plugin's core dependency that's newer than ~13 months (400 days)

OK, this makes more sense than what you wrote on the dev list

every release ever used as a plugin's core dependency will be included

Given the new policy, in your opinion should https://github.com/jenkinsci/bom/#lts-lines be updated accordingly and bom-2.nnn.x published as far back as the oldest entry in stableCores? Not that there is any necessary relationship; we could decide to publish, or not publish, older bom lines according to other criteria.

@daniel-beck
Copy link
Contributor Author

what you wrote on the dev list

FTR https://groups.google.com/d/msg/jenkinsci-dev/aNWx61URGdk/qgT8wxojDAAJ

Yes, in hindsight that was misleading; there is a cutoff based on core release age (but could be any criteria).

@daniel-beck
Copy link
Contributor Author

Given the new policy

Note that nothing changes in terms of "longevity" of older releases. You can see on https://updates.jenkins-ci.org/ that we currently serve the latest five LTS baselines, which is roughly the same as this does. In fact, my email to the dev list explains that this is in no way intended to add support for older releases beyond what already existed for years – it's just getting rid of the fixed tiering that could cause problems even for users on recent releases.

@daniel-beck daniel-beck added the on-hold This PR is on hold, typically because of a dependency to another change or event. label May 21, 2020
@daniel-beck
Copy link
Contributor Author

On-holding this as merging it would break the Docker image because of jenkinsci/docker#954 😭

@daniel-beck daniel-beck added the enhancement This is an enhancement for the tool or wrapper scripts, typically adding features. label May 24, 2020
@daniel-beck daniel-beck removed the on-hold This PR is on hold, typically because of a dependency to another change or event. label Jun 18, 2020
@daniel-beck
Copy link
Contributor Author

Updated this PR to keep generating the fixed tiered update sites in addition to the new dynamic tiers.

$ ls -l www2/
total 8
drwxr-xr-x   8 danielbeck  staff  256 Jun 18 15:19 2.176
drwxr-xr-x   8 danielbeck  staff  256 Jun 18 15:19 2.190
drwxr-xr-x   8 danielbeck  staff  256 Jun 18 15:19 2.204
drwxr-xr-x   8 danielbeck  staff  256 Jun 18 15:19 2.222
drwxr-xr-x   8 danielbeck  staff  256 Jun 18 15:19 2.235
drwxr-xr-x  12 danielbeck  staff  384 Jun 18 15:19 current
drwxr-xr-x   4 danielbeck  staff  128 Jun 18 15:19 download
drwxr-xr-x   8 danielbeck  staff  256 Jun 18 15:19 dynamic-2.176
drwxr-xr-x   8 danielbeck  staff  256 Jun 18 15:19 dynamic-2.177
drwxr-xr-x   8 danielbeck  staff  256 Jun 18 15:19 dynamic-2.181
drwxr-xr-x   8 danielbeck  staff  256 Jun 18 15:19 dynamic-2.184
drwxr-xr-x   8 danielbeck  staff  256 Jun 18 15:19 dynamic-2.185
drwxr-xr-x   8 danielbeck  staff  256 Jun 18 15:19 dynamic-2.191
drwxr-xr-x   8 danielbeck  staff  256 Jun 18 15:19 dynamic-2.195
drwxr-xr-x   8 danielbeck  staff  256 Jun 18 15:19 dynamic-2.199
drwxr-xr-x   8 danielbeck  staff  256 Jun 18 15:19 dynamic-2.204
drwxr-xr-x   8 danielbeck  staff  256 Jun 18 15:19 dynamic-2.205
drwxr-xr-x   8 danielbeck  staff  256 Jun 18 15:19 dynamic-2.212
drwxr-xr-x   8 danielbeck  staff  256 Jun 18 15:19 dynamic-2.217
drwxr-xr-x   8 danielbeck  staff  256 Jun 18 15:19 dynamic-2.222
drwxr-xr-x   8 danielbeck  staff  256 Jun 18 15:19 dynamic-2.223
drwxr-xr-x   8 danielbeck  staff  256 Jun 18 15:19 dynamic-2.238
drwxr-xr-x   8 danielbeck  staff  256 Jun 18 15:19 dynamic-2.240
drwxr-xr-x   8 danielbeck  staff  256 Jun 18 15:19 dynamic-stable-2.164.3
drwxr-xr-x   8 danielbeck  staff  256 Jun 18 15:19 dynamic-stable-2.176.1
drwxr-xr-x   8 danielbeck  staff  256 Jun 18 15:19 dynamic-stable-2.176.2
drwxr-xr-x   8 danielbeck  staff  256 Jun 18 15:19 dynamic-stable-2.176.3
drwxr-xr-x   8 danielbeck  staff  256 Jun 18 15:19 dynamic-stable-2.176.4
drwxr-xr-x   8 danielbeck  staff  256 Jun 18 15:19 dynamic-stable-2.190.1
drwxr-xr-x   8 danielbeck  staff  256 Jun 18 15:19 dynamic-stable-2.190.3
drwxr-xr-x   8 danielbeck  staff  256 Jun 18 15:19 dynamic-stable-2.204.1
drwxr-xr-x   8 danielbeck  staff  256 Jun 18 15:19 dynamic-stable-2.204.2
drwxr-xr-x   8 danielbeck  staff  256 Jun 18 15:19 dynamic-stable-2.204.4
drwxr-xr-x   8 danielbeck  staff  256 Jun 18 15:19 dynamic-stable-2.204.6
drwxr-xr-x   8 danielbeck  staff  256 Jun 18 15:19 dynamic-stable-2.222.1
drwxr-xr-x   8 danielbeck  staff  256 Jun 18 15:19 dynamic-stable-2.222.3
drwxr-xr-x   8 danielbeck  staff  256 Jun 18 15:19 dynamic-stable-2.222.4
drwxr-xr-x   7 danielbeck  staff  224 Jun 18 15:19 experimental
lrwxr-xr-x   1 danielbeck  staff   14 Jun 18 15:19 latest -> current/latest
lrwxr-xr-x   1 danielbeck  staff   22 Jun 18 15:19 latestCore.txt -> current/latestCore.txt
lrwxr-xr-x   1 danielbeck  staff   38 Jun 18 15:19 plugin-documentation-urls.json -> current/plugin-documentation-urls.json
lrwxr-xr-x   1 danielbeck  staff   28 Jun 18 15:19 plugin-versions.json -> current/plugin-versions.json
-rw-r--r--   1 danielbeck  staff  297 May 26 21:07 readme.html
lrwxr-xr-x   1 danielbeck  staff   28 Jun 18 15:19 release-history.json -> current/release-history.json
lrwxr-xr-x   1 danielbeck  staff   12 Jun 18 15:19 stable -> stable-2.235
drwxr-xr-x   8 danielbeck  staff  256 Jun 18 15:19 stable-2.176
drwxr-xr-x   8 danielbeck  staff  256 Jun 18 15:19 stable-2.190
drwxr-xr-x   8 danielbeck  staff  256 Jun 18 15:19 stable-2.204
drwxr-xr-x   8 danielbeck  staff  256 Jun 18 15:19 stable-2.222
drwxr-xr-x   8 danielbeck  staff  256 Jun 18 15:19 stable-2.235
lrwxr-xr-x   1 danielbeck  staff   33 Jun 18 15:19 update-center.actual.json -> current/update-center.actual.json
lrwxr-xr-x   1 danielbeck  staff   26 Jun 18 15:19 update-center.json -> current/update-center.json
lrwxr-xr-x   1 danielbeck  staff   31 Jun 18 15:19 update-center.json.html -> current/update-center.json.html

@daniel-beck daniel-beck merged commit 5526acc into jenkins-infra:master Jun 25, 2020
jglick added a commit to jenkinsci/bom that referenced this pull request Jun 25, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement This is an enhancement for the tool or wrapper scripts, typically adding features.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants