Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implements automated release to SDKMAN #241

Merged
merged 5 commits into from
Apr 28, 2022

Conversation

Karm
Copy link
Collaborator

@Karm Karm commented Mar 17, 2022

$ ./mandrel-release.java sdkman -m /home/karm/workspaceRH/graal/graal -s Final -f Karm/graal
[jbang] Building jar...
[INFO] Current version is 21.3.1.1
[INFO] New version will be 21.3.1.2
[INFO] Releasing: mandrel-java11-linux-amd64-21.3.1.1-Final.tar.gz
[INFO] Released Response Code : 201
[INFO] Released Response Body : {"message":"Released: java 21.3.1.1.r11-mandrel for LINUX_64","status":201}
[INFO] Releasing: mandrel-java11-linux-aarch64-21.3.1.1-Final.tar.gz
[INFO] Released Response Code : 201
[INFO] Released Response Body : {"message":"Released: java 21.3.1.1.r11-mandrel for LINUX_ARM64","status":201}
[INFO] Releasing: mandrel-java11-windows-amd64-21.3.1.1-Final.zip
[INFO] Released Response Code : 201
[INFO] Released Response Body : {"message":"Released: java 21.3.1.1.r11-mandrel for WINDOWS_64","status":201}
[INFO] Releasing: mandrel-java17-linux-amd64-21.3.1.1-Final.tar.gz
[INFO] Released Response Code : 201
[INFO] Released Response Body : {"message":"Released: java 21.3.1.1.r17-mandrel for LINUX_64","status":201}
[INFO] Releasing: mandrel-java17-linux-aarch64-21.3.1.1-Final.tar.gz
[INFO] Released Response Code : 201
[INFO] Released Response Body : {"message":"Released: java 21.3.1.1.r17-mandrel for LINUX_ARM64","status":201}
[INFO] Releasing: mandrel-java17-windows-amd64-21.3.1.1-Final.zip
[INFO] Released Response Code : 201
[INFO] Released Response Body : {"message":"Released: java 21.3.1.1.r17-mandrel for WINDOWS_64","status":201}
[INFO] Promoting version 21.3.1.1.r11
[INFO] Promoted Response Code : 202
[INFO] Promoted Response Body : {"message":"Defaulted: java 21.3.1.1.r11-mandrel","status":202}
[INFO] Annoncing version 21.3.1.1.r11
[INFO] Annonce Response Code : 200
[INFO] Annonce Response Body : {"status":200,"id":"623342f06344590007b7da9c","message":"java 21.3.1.1.r11-mandrel available on SDKMAN! https://github.com/graalvm/mandrel/releases/"}
[INFO] Promoting version 21.3.1.1.r17
[INFO] Promoted Response Code : 202
[INFO] Promoted Response Body : {"message":"Defaulted: java 21.3.1.1.r17-mandrel","status":202}
[INFO] Annoncing version 21.3.1.1.r17
[INFO] Annonce Response Code : 200
[INFO] Annonce Response Body : {"status":200,"id":"623342f16344590007b7da9d","message":"java 21.3.1.1.r17-mandrel available on SDKMAN! https://github.com/graalvm/mandrel/releases/"}

```
$ ./mandrel-release.java sdkman -m /home/karm/workspaceRH/graal/graal -s Final -f Karm/graal
[jbang] Building jar...
[INFO] Current version is 21.3.1.1
[INFO] New version will be 21.3.1.2
[INFO] Releasing: mandrel-java11-linux-amd64-21.3.1.1-Final.tar.gz
[INFO] Released Response Code : 201
[INFO] Released Response Body : {"message":"Released: java 21.3.1.1.r11-mandrel for LINUX_64","status":201}
[INFO] Releasing: mandrel-java11-linux-aarch64-21.3.1.1-Final.tar.gz
[INFO] Released Response Code : 201
[INFO] Released Response Body : {"message":"Released: java 21.3.1.1.r11-mandrel for LINUX_ARM64","status":201}
[INFO] Releasing: mandrel-java11-windows-amd64-21.3.1.1-Final.zip
[INFO] Released Response Code : 201
[INFO] Released Response Body : {"message":"Released: java 21.3.1.1.r11-mandrel for WINDOWS_64","status":201}
[INFO] Releasing: mandrel-java17-linux-amd64-21.3.1.1-Final.tar.gz
[INFO] Released Response Code : 201
[INFO] Released Response Body : {"message":"Released: java 21.3.1.1.r17-mandrel for LINUX_64","status":201}
[INFO] Releasing: mandrel-java17-linux-aarch64-21.3.1.1-Final.tar.gz
[INFO] Released Response Code : 201
[INFO] Released Response Body : {"message":"Released: java 21.3.1.1.r17-mandrel for LINUX_ARM64","status":201}
[INFO] Releasing: mandrel-java17-windows-amd64-21.3.1.1-Final.zip
[INFO] Released Response Code : 201
[INFO] Released Response Body : {"message":"Released: java 21.3.1.1.r17-mandrel for WINDOWS_64","status":201}
[INFO] Promoting version 21.3.1.1.r11
[INFO] Promoted Response Code : 202
[INFO] Promoted Response Body : {"message":"Defaulted: java 21.3.1.1.r11-mandrel","status":202}
[INFO] Annoncing version 21.3.1.1.r11
[INFO] Annonce Response Code : 200
[INFO] Annonce Response Body : {"status":200,"id":"623342f06344590007b7da9c","message":"java 21.3.1.1.r11-mandrel available on SDKMAN! https://github.com/graalvm/mandrel/releases/"}
[INFO] Promoting version 21.3.1.1.r17
[INFO] Promoted Response Code : 202
[INFO] Promoted Response Body : {"message":"Defaulted: java 21.3.1.1.r17-mandrel","status":202}
[INFO] Annoncing version 21.3.1.1.r17
[INFO] Annonce Response Code : 200
[INFO] Annonce Response Body : {"status":200,"id":"623342f16344590007b7da9d","message":"java 21.3.1.1.r17-mandrel available on SDKMAN! https://github.com/graalvm/mandrel/releases/"}
```
@Karm Karm requested a review from zakkak March 17, 2022 14:22
@Karm Karm self-assigned this Mar 17, 2022
@Karm
Copy link
Collaborator Author

Karm commented Mar 17, 2022

@zakkak IMHO implemented. The release is visible on sdkman, but, I cannot verify it atm:

$ sdk install java 21.3.1.1.r17-mandrel

Downloading: java 21.3.1.1.r17-mandrel

In progress...

-#O=-#     #        #                                                                                                                                                                                             
Download has failed, aborting!

Can not install java 21.3.1.1.r17-mandrel at this time...

The reason it GitHub is mostly down: https://www.githubstatus.com/

So let's try to check it later....

@Karm
Copy link
Collaborator Author

Karm commented Mar 17, 2022

Hmm, not sure what is wrong.

$ sdk install java

Stop! java is not available. Possible causes:
 * is an invalid version
 * java binaries are incompatible with your platform
 * java has not been released yet

Tip: see all available versions for your platform:

  $ sdk list java

And we are also used in the docs as the default Java. Probably because we are the latest release...?
image

Older ones work just fine:

$ sdk install java 21.3.1.0.r17-mandrel

Downloading: java 21.3.1.0.r17-mandrel

In progress...

############################################################################################################################################################################################################ 100.0%############################################################################################################################################################################################################ 100.0%

Repackaging Java 21.3.1.0.r17-mandrel...

Done repackaging...

Installing: java 21.3.1.0.r17-mandrel
Done installing!


Setting java 21.3.1.0.r17-mandrel as default.

The one published by this automation doesn't:

$ sdk install java 21.3.1.1.r17-mandrel

Downloading: java 21.3.1.1.r17-mandrel

In progress...

-#O=-#     #        #                                                                                                                                                                                             
Download has failed, aborting!

Can not install java 21.3.1.1.r17-mandrel at this time...

I am not sure what could be wrong. It's seems to be really fool proof, e.g. my tested payload:

{
    "candidate": "java",
    "version": "21.3.1.1.r17",
    "vendor": "mandrel",
    "url": "https://github.com/graalvm/mandrel/releases/download/mandrel-21.3.1.1-Final/mandrel-java17-linux-amd64-21.3.1.1-Final.tar.gz",
    "platform": "LINUX_64",
    "checksums": {
        "SHA-1": "adf1387196504dcfcc8a8080657f281d866d336d",
        "SHA-256": "768b608203cbde91d571ff03f9a9ce768a2bed6b62cffccaff2f19a0a7897edd"
    }
}


/**
* One needs vendor credentials. Decipher those in:
* gpg --decrypt sdkman-vendor.json.asc > ~/.sdkman/vendor.json
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would possibly be even better if we could get the script to do the decrypt on the fly.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will this be run on one of your local machines? The only reason I sent the credentials encrypted was that we don't like sending credentials in an email.

On the other hand, if this is an automated release script, please consider storing the key and token pair in GitHub secrets instead.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will this be run on one of your local machines?

Yes, this will be run locally. So it's more about whether we would need to have the credentials stored in an unencrypted file (most probably on an encrypt disk though).

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is a local workstation with an exclusive physical access and an encrypted drive. The file enjoys the same level of security, including filesystem permissions, as e.g. ~/.aws/credentials or, for the matter of fact, ~/.ssh/id_rsa.

@zakkak
Copy link
Collaborator

zakkak commented Mar 18, 2022

Setting sdkman_debug_mode to true in ~./sdkman/etc/config gives the following output:

$ sdk install java 21.3.1.1.r17-mandrel                                   
No update at this time. Using existing cache: activemq,ant,asciidoctorj,ballerina,bpipe,btrace,concurnas,connor,cuba,cxf,doctoolchain,flink,gaiden,gradle,gradleprofiler,grails,groovy,groovyserv,hadoop,http4k,infrastructor,java,jbake,jbang,jmc,jmeter,joern,jreleaser,karaf,kcctl,ki,kobweb,kotlin,kscript,layrry,leiningen,maven,mcs,micronaut,mulefd,mvnd,mybatis,pierrot,pomchecker,quarkus,sbt,scala,scalacli,schemacrawler,spark,springboot,sshoogr,taxi,test,tomcat,vertx,visualvm,webtau,znai
Not refreshing version cache now...
Validate java 21.3.1.1.r17-mandrel for linuxx64: valid
Validation URL: https://api.sdkman.io/2/candidates/validate/java/21.3.1.1.r17-mandrel/linuxx64
Get pre-installation hook: https://api.sdkman.io/2/hooks/pre/java/21.3.1.1.r17-mandrel/linuxx64
Copy remote pre-installation hook: /home/zakkak/.sdkman/tmp/hook_pre_java_21.3.1.1.r17-mandrel.sh
No Linux pre-install hook found for Java 21.3.1.1.r17-mandrel.
Completed pre-installation hook...

Downloading: java 21.3.1.1.r17-mandrel

In progress...

*   Trying 45.55.42.78:443...
* Connected to api.sdkman.io (45.55.42.78) port 443 (#0)
* ALPN, offering h2                                                                                                                                                                          
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
*  CApath: none
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [25 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [5660 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=*.sdkman.io
*  start date: May 30 00:00:00 2020 GMT
*  expire date: Apr 28 23:59:59 2022 GMT
*  subjectAltName: host "api.sdkman.io" matched cert's "*.sdkman.io"
*  issuer: C=GB; ST=Greater Manchester; L=Salford; O=Sectigo Limited; CN=Sectigo RSA Domain Validation Secure Server CA
*  SSL certificate verify ok.
} [5 bytes data]                                                                                                                                                                             
> GET /2/broker/download/java/21.3.1.1.r17-mandrel/linuxx64 HTTP/1.1
> Host: api.sdkman.io
> User-Agent: curl/7.79.1
> Accept: */*
> 
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [265 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):                                                                                                                                        
{ [265 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
* Mark bundle as not supporting multiuse
< HTTP/1.1 500 Internal Server Error
< Server: nginx/1.19.1
< Date: Fri, 18 Mar 2022 08:25:05 GMT
< Content-Length: 0
< Connection: keep-alive
< 
* Connection #0 to host api.sdkman.io left intact

Downloaded binary to: /home/zakkak/.sdkman/tmp/java-21.3.1.1.r17-mandrel.bin (HTTP headers written to: /home/zakkak/.sdkman/var/metadata/java-21.3.1.1.r17-mandrel.headers)
Get post-installation hook: https://api.sdkman.io/2/hooks/post/java/21.3.1.1.r17-mandrel/linuxx64
Copy remote post-installation hook: /home/zakkak/.sdkman/tmp/hook_post_java_21.3.1.1.r17-mandrel.sh
A Linux post-install hook was found for Java 21.3.1.1.r17-mandrel.
Download has failed, aborting!

Can not install java 21.3.1.1.r17-mandrel at this time...

Notice the < HTTP/1.1 500 Internal Server Error.

My suspicion is that you "published" the new version while github was down and that somehow messed up the sdkman database update for the new version.

Copy link
Collaborator

@zakkak zakkak left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for taking care of this @Karm !

Please find my suggestions/comments inline.

release-scripts/mandrel-release.java Outdated Show resolved Hide resolved
release-scripts/mandrel-release.java Outdated Show resolved Hide resolved
release-scripts/mandrel-release.java Outdated Show resolved Hide resolved
release-scripts/mandrel-release.java Outdated Show resolved Hide resolved
release-scripts/mandrel-release.java Outdated Show resolved Hide resolved
release-scripts/mandrel-release.java Outdated Show resolved Hide resolved
release-scripts/mandrel-release.java Outdated Show resolved Hide resolved
@Karm
Copy link
Collaborator Author

Karm commented Mar 18, 2022

@zakkak debug & dryRun:

$ ./mandrel-release.java sdkman -m /home/karm/workspaceRH/graal/graal -s Final -f Karm/graal --verbose -D --sdkman-announce
[jbang] Building jar...
[INFO] Current version is 21.3.1.1
[INFO] New version will be 21.3.1.2
[INFO] Releasing: mandrel-java11-linux-amd64-21.3.1.1-Final.tar.gz
[DEBUG] {
    "candidate": "java",
    "version": "21.3.1.1.r11",
    "vendor": "mandrel",
    "url": "https://github.com/graalvm/mandrel/releases/download/mandrel-21.3.1.1-Final/mandrel-java11-linux-amd64-21.3.1.1-Final.tar.gz",
    "platform": "LINUX_64",
    "checksums": {
        "SHA-1": "ef1396f20e393d69924a60a83d0ca3b3e52d2211",
        "SHA-256": "4afccbc431c62f119c91a7bb077f885949f405e0a21be344edeb77dfeb0f5118"
    }
}
[INFO] Releasing: mandrel-java11-linux-aarch64-21.3.1.1-Final.tar.gz
[DEBUG] {
    "candidate": "java",
    "version": "21.3.1.1.r11",
    "vendor": "mandrel",
    "url": "https://github.com/graalvm/mandrel/releases/download/mandrel-21.3.1.1-Final/mandrel-java11-linux-aarch64-21.3.1.1-Final.tar.gz",
    "platform": "LINUX_ARM64",
    "checksums": {
        "SHA-1": "3eec38c1948e682af4a75ec7f7052076383f9b55",
        "SHA-256": "c5f75f0aa88dd60c7e0c8918f77f6b078dc2454545819d74ec085e0424cadf10"
    }
}
[INFO] Releasing: mandrel-java11-windows-amd64-21.3.1.1-Final.zip
[DEBUG] {
    "candidate": "java",
    "version": "21.3.1.1.r11",
    "vendor": "mandrel",
    "url": "https://github.com/graalvm/mandrel/releases/download/mandrel-21.3.1.1-Final/mandrel-java11-windows-amd64-21.3.1.1-Final.zip",
    "platform": "WINDOWS_64",
    "checksums": {
        "SHA-1": "a9c930ac92d875f72b72339bda54adcb33cefb2b",
        "SHA-256": "dae4910c9e804dfd44d158bc9e078a241f2f6f565de26bc32fc24fb5667b3f31"
    }
}
[INFO] Releasing: mandrel-java17-linux-amd64-21.3.1.1-Final.tar.gz
[DEBUG] {
    "candidate": "java",
    "version": "21.3.1.1.r17",
    "vendor": "mandrel",
    "url": "https://github.com/graalvm/mandrel/releases/download/mandrel-21.3.1.1-Final/mandrel-java17-linux-amd64-21.3.1.1-Final.tar.gz",
    "platform": "LINUX_64",
    "checksums": {
        "SHA-1": "adf1387196504dcfcc8a8080657f281d866d336d",
        "SHA-256": "768b608203cbde91d571ff03f9a9ce768a2bed6b62cffccaff2f19a0a7897edd"
    }
}
[INFO] Releasing: mandrel-java17-linux-aarch64-21.3.1.1-Final.tar.gz
[DEBUG] {
    "candidate": "java",
    "version": "21.3.1.1.r17",
    "vendor": "mandrel",
    "url": "https://github.com/graalvm/mandrel/releases/download/mandrel-21.3.1.1-Final/mandrel-java17-linux-aarch64-21.3.1.1-Final.tar.gz",
    "platform": "LINUX_ARM64",
    "checksums": {
        "SHA-1": "bfa341f17155940c586fb6e32c759ab07c96d7a1",
        "SHA-256": "83bc940a5bd1f8a9236c19dc4746be065a4c08cb095d8798fc715ea873d74e35"
    }
}
[INFO] Releasing: mandrel-java17-windows-amd64-21.3.1.1-Final.zip
[DEBUG] {
    "candidate": "java",
    "version": "21.3.1.1.r17",
    "vendor": "mandrel",
    "url": "https://github.com/graalvm/mandrel/releases/download/mandrel-21.3.1.1-Final/mandrel-java17-windows-amd64-21.3.1.1-Final.zip",
    "platform": "WINDOWS_64",
    "checksums": {
        "SHA-1": "1837b4f05d550a7e32330867d3b247198465ac35",
        "SHA-256": "45a0b454484af6bd4103ea033f781277093d2cd0d4b215ddd198ce71d9018218"
    }
}
[INFO] Promoting version 21.3.1.1.r11
[DEBUG] {
    "candidate": "java",
    "version": "21.3.1.1.r11-mandrel",
    "vendor": "mandrel"
}
[INFO] Announcing version 21.3.1.1.r11
[DEBUG] {
    "candidate": "java",
    "version": "21.3.1.1.r11-mandrel",
    "url": "https://github.com/graalvm/mandrel/releases/"
}
[INFO] Promoting version 21.3.1.1.r17
[DEBUG] {
    "candidate": "java",
    "version": "21.3.1.1.r17-mandrel",
    "vendor": "mandrel"
}
[INFO] Announcing version 21.3.1.1.r17
[DEBUG] {
    "candidate": "java",
    "version": "21.3.1.1.r17-mandrel",
    "url": "https://github.com/graalvm/mandrel/releases/"
}

@Karm
Copy link
Collaborator Author

Karm commented Mar 20, 2022

TL;DR from our conversation with Marco:

  • remove announce, it won't be used
  • remove promoting candidates; the semantic of the expression is different from what I though it to be
  • add visible=false to hide older, surpassed versions of our Mandrel JDK, as in this doc

@aalmiray
Copy link

FWIW JReleaser can perform most of thew heavy lifting you're doing with the mandrel-release.java script

https://jreleaser.org

@Karm
Copy link
Collaborator Author

Karm commented Mar 21, 2022

Hello @marc0der,

Taking your recommendation about "visible": false to heart, I did what seems to be aligned with this doc.
There seems to be something odd though. Is the vendor string expected as a separate filed (contrary to the aforementioned -zulu example...)? Thx for a hint.

$ ./mandrel-release.java sdkman --sdkman-version-invisible=21.3.1.0.r11  --verbose
[jbang] Building jar...
[DEBUG] {
  "candidate": "java",
  "version": "21.3.1.0.r11-mandrel",
  "platform": "LINUX_64",
  "visible": false
}
[INFO] Response Code : 400
[INFO] Response Body : {"message":"Version length exceeds 17 chars: 21.3.1.0.r11-mandrel","status":400}
[DEBUG] {
  "candidate": "java",
  "version": "21.3.1.0.r11-mandrel",
  "platform": "LINUX_ARM64",
  "visible": false
}
[INFO] Response Code : 400
[INFO] Response Body : {"message":"Version length exceeds 17 chars: 21.3.1.0.r11-mandrel","status":400}
[DEBUG] {
  "candidate": "java",
  "version": "21.3.1.0.r11-mandrel",
  "platform": "WINDOWS_64",
  "visible": false
}
[INFO] Response Code : 400
[INFO] Response Body : {"message":"Version length exceeds 17 chars: 21.3.1.0.r11-mandrel","status":400}

@zakkak
I removed

  • announce part of the script
  • "promoting" part of the script
  • changed the script to use GitHub releases instead of a local directory when processing artifacts

@Karm
Copy link
Collaborator Author

Karm commented Mar 21, 2022

@marc0der ...or we can drop the topic and wait for DISCO 😃

@marc0der
Copy link

marc0der commented Mar 23, 2022

Well, DISCO integration is already fully operational (we performed the JDK 18 release with this new mechanism yesterday). I'd suggest just rolling with that instead. @eddumelendez can pick it up if you like. I believe that all he needs to do is to add a few lines of config for this to start working.

@Karm Karm requested a review from zakkak April 27, 2022 17:36
@Karm
Copy link
Collaborator Author

Karm commented Apr 28, 2022

Merging as per our conversation on SDKMAN Slack.

@Karm Karm merged commit dd3a47b into graalvm:master Apr 28, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants