provide no-jdk distribution #99
Comments
|
[Triage] The build process can support this, it would need to be integrated and tested so it could be released. |
|
out of interest: where's the code which packages the JDK with the ZIP file? is this here or in the opensearch repo? i couldn't find it in either with a cursory glance (plus i'm not certain if this is even something which could easily be contributed by somebody who doesn't have access to the complete build infrastructure as, after all, the no-jdk dist will have to be published and linked to in various places?) |
|
It's in OpenSearch, search for I think the first step for this one is to produce the no-jdk distribution as part of the daily build. Probably the easiest is to always make a JDK and no-JDK build when calling |
|
i browsed around a bit and found the following relevant things:
what i haven't seen yet at first glance is where |
|
what is missing here? from what i can see there are various based on that i'd somehow expect that now only (AWS-internal?) tooling is missing which grabs this during the release process and publishes it everywhere? |
|
@rursprung I think the next step is to add it to https://github.com/opensearch-project/opensearch-build/blob/main/jenkins/opensearch/distribution-build.jenkinsfile, so it starts appearing in the build output These currently run in a private Jenkins instance, but are this close to being run in public https://build.ci.opensearch.org/, see opensearch-project/opensearch-ci#2. |
thanks, that's good news! however, i'm not sure how much sense it'll make if i'll try to contribute to this as i can't test whether my changes to the script would work. |
|
@peterzhuamazon Could you help @rursprung make sense of this stuff please? This is all visible and testable, and even documented, but a bit dense ;) let's try to update the documentation as we go |
|
[Copy from another conversation]: Hi All, This is a rather complicated task. Just from the top of my head you need to introduce a new param in python side to allow jdk/non-jdk input. If we introduce non-jdk this needs to be discussed extensively as well as on an issue. Adding @bbarani into the discussion on this. Thanks, |
|
Thanks a lot for your feedback @peterzhuamazon! That does sound more complicated than it sounded initially! I ran a quick test (on the latest main branch and on Windows) and it does seem that the standard assemble call already builds both the non-jdk as well as the bundled package (both Note that I raised the ticket explicitly for OpenSearch, not OpenSearch Dashboards – we’re currently not using that, so it’s of no relevance for me. Though I can see that the same discussion can be had there, I think it should be split away (and be driven by somebody involved in / using OpenSearch Dashboards) and could potentially also result in a different decision (AFAIK Node.JS versions are more finicky than JDK versions?). |
|
I think the next step is to add a step to https://github.com/opensearch-project/opensearch-build/blob/main/jenkins/opensearch/distribution-build.jenkinsfile that would call these gradle tasks that exist. |
|
@rursprung all the non-jdk task I have seen so far: task logs |
|
Notable: Basically covered all the distributions we are looking at. |
this is where i got stuck in may (see our comments from may 5 & 6) - i haven't seen how that build script decides what to actually build, i.e. i haven't understood what i'd have to add/change to get a no-jdk build. if you could point me in the right direction there (and tell me how to test it) that'd be great! |
|
So from the top: This says take a manifest, e.g. https://github.com/opensearch-project/opensearch-build/blob/main/manifests/3.0.0/opensearch-3.0.0.yml and build it with Alternatively, and possibly easier, build.sh for opensearch could always produce both a JDK and no-JDK distribution, so you would add a second line below |
i've now added it on the same line even (makes the build much faster as it avoids running the other tasks twice): ./gradlew :distribution:$TYPE:$TARGET:assemble :distribution:$TYPE:no-jdk-$TARGET:assemble -Dbuild.snapshot=$SNAPSHOT -Dbuild.version_qualifier=$QUALIFIERhowever, this does require that every target called this way also has a corresponding no-jdk target.
is there a specific reason why these do not exist or could/should they be added? i'm also a bit surprised that for some targets there's a sub-folder with an empty suggested next steps (if you agree with them):
does the |
I don't know of any specific reason, simply unused.
That's because it is unused, so no "good" reason.
Sounds good!
We have been moving build.sh's into each repo so that it branches with the repo, and you can do the same for OpenSearch repo. The default in this repo can go away if all built branches in opensearch-project/OpenSearch have script/build.sh. |
|
i've done the first and third step in the PRs opensearch-project/OpenSearch#4722 and opensearch-project/OpenSearch#4723 (the latter isn't actually relevant for this topic here, just a cleanup).
where exactly should the file go? directly as |
Yes.
They can stay here until there's a build.sh in all active branches of OpenSearch. We should backport build.sh all over the place, and then delete here. |
is there a full list of those? my presumption would be that it's the following:
and maybe also the following?
though with pure semver i don't see why they'd still be maintained (or why they exist as branches at all), as e.g. (random example) 2.4.0 with a bugfix would be a perfectly valid upgrade path for a user on 2.1.0, so no 2.1.1 would be needed. but that'd then be a discussion for elsewhere 😄
does it afterwards automatically look for the correct script in the other repo or are more changes needed here (or elsewhere) than just deleting the files? |
|
@rursprung That list is here: https://github.com/opensearch-project/opensearch-build/blob/main/jenkins/check-for-build.jenkinsfile#L17 Yes. No other changes needed. The logic is documented in https://github.com/opensearch-project/opensearch-build/tree/main/src/build_workflow#custom-build-scripts and implemented in https://github.com/opensearch-project/opensearch-build/blob/main/src/paths/script_finder.py. |
|
@peterzhuamazon Am I correct for the list of active branches? I don't see 1.3.x being built in that list, but I know we just released 1.3.6. Do you typically add that on demand? Should @rursprung backport build.sh changes to 1.x and 1.3 in opensearch? |
with commit e357246 the `build.sh` has been moved to this repository and with commit dfa1118 the missing no-jdk builds have been added. with these preliminary works done we can now start building the no-jdk distributions as part of the release build so that they can then be published. this is part of opensearch-project/opensearch-build#99. Signed-off-by: Ralph Ursprung <Ralph.Ursprung@avaloq.com>
with commit e357246 the `build.sh` has been moved to this repository and with commit dfa1118 the missing no-jdk builds have been added. with these preliminary works done we can now start building the no-jdk distributions as part of the release build so that they can then be published. this is part of opensearch-project/opensearch-build#99. Signed-off-by: Ralph Ursprung <Ralph.Ursprung@avaloq.com> Signed-off-by: Ralph Ursprung <Ralph.Ursprung@avaloq.com> Co-authored-by: Daniel (dB.) Doubrovkine <dblock@amazon.com>
…ct#4902) with commit e357246 the `build.sh` has been moved to this repository and with commit dfa1118 the missing no-jdk builds have been added. with these preliminary works done we can now start building the no-jdk distributions as part of the release build so that they can then be published. this is part of opensearch-project/opensearch-build#99. Signed-off-by: Ralph Ursprung <Ralph.Ursprung@avaloq.com> Signed-off-by: Ralph Ursprung <Ralph.Ursprung@avaloq.com> Co-authored-by: Daniel (dB.) Doubrovkine <dblock@amazon.com> (cherry picked from commit 8c9ca4e)
with commit e357246 the `build.sh` has been moved to this repository and with commit dfa1118 the missing no-jdk builds have been added. with these preliminary works done we can now start building the no-jdk distributions as part of the release build so that they can then be published. this is part of opensearch-project/opensearch-build#99. Signed-off-by: Ralph Ursprung <Ralph.Ursprung@avaloq.com> Signed-off-by: Ralph Ursprung <Ralph.Ursprung@avaloq.com> Co-authored-by: Daniel (dB.) Doubrovkine <dblock@amazon.com> (cherry picked from commit 8c9ca4e)
`build.sh` has been copied over to all active branches(*) of the OpenSearch repo and can now consequently be deleted here. this means, that from now on, the scripts from the respository will be picked up. for more information on the order of evaluation of the custom build scripts, see the [documentation][]. this gives the advantage to have branch-specific new logic in place there. this has already been used to add the no-jdk targets to the builds on main and 2.x. (*): the following branches have been considered as active: - main - 2.x - 2.3 - 1.x - 1.3 this is part of opensearch-project#99 [documentation]: https://github.com/opensearch-project/opensearch-build/tree/main/src/build_workflow#custom-build-scripts Signed-off-by: Ralph Ursprung <Ralph.Ursprung@avaloq.com>
`build.sh` has been copied over to all active branches(*) of the OpenSearch repo and can now consequently be deleted here. this means, that from now on, the scripts from the respository will be picked up. for more information on the order of evaluation of the custom build scripts, see the [documentation][]. this gives the advantage to have branch-specific new logic in place there. this has already been used to add the no-jdk targets to the builds on main and 2.x. two tests were based on the assumption that there would be a `build.sh` for OpenSearch in this repository, however these are generic tests and just used OpenSearch as an example. they have been switched to be based on OpenSearch-Dashboards instead which - for the time being - still has its own `build.sh` here. (*): the following branches have been considered as active: - main - 2.x - 2.3 - 1.x - 1.3 this is part of opensearch-project#99 [documentation]: https://github.com/opensearch-project/opensearch-build/tree/main/src/build_workflow#custom-build-scripts Signed-off-by: Ralph Ursprung <Ralph.Ursprung@avaloq.com>
`build.sh` has been copied over to all active branches(*) of the OpenSearch repo and can now consequently be deleted here. this means, that from now on, the scripts from the respository will be picked up. for more information on the order of evaluation of the custom build scripts, see the [documentation][]. this gives the advantage to have branch-specific new logic in place there. this has already been used to add the no-jdk targets to the builds on main and 2.x. two tests were based on the assumption that there would be a `build.sh` for OpenSearch in this repository, however these are generic tests and just used OpenSearch as an example. they have been switched to be based on OpenSearch-Dashboards instead which - for the time being - still has its own `build.sh` here. (*): the following branches have been considered as active: - main - 2.x - 2.3 - 1.x - 1.3 this is part of opensearch-project#99 [documentation]: https://github.com/opensearch-project/opensearch-build/tree/main/src/build_workflow#custom-build-scripts Signed-off-by: Ralph Ursprung <Ralph.Ursprung@avaloq.com>
`build.sh` has been copied over to all active branches(*) of the OpenSearch repo and can now consequently be deleted here. this means, that from now on, the scripts from the respository will be picked up. for more information on the order of evaluation of the custom build scripts, see the [documentation][]. this gives the advantage to have branch-specific new logic in place there. this has already been used to add the no-jdk targets to the builds on main and 2.x. two tests were based on the assumption that there would be a `build.sh` for OpenSearch in this repository, however these are generic tests and just used OpenSearch as an example. they have been switched to be based on OpenSearch-Dashboards instead which - for the time being - still has its own `build.sh` here. (*): the following branches have been considered as active: - main - 2.x - 2.3 - 1.x - 1.3 this is part of opensearch-project#99 [documentation]: https://github.com/opensearch-project/opensearch-build/tree/main/src/build_workflow#custom-build-scripts Signed-off-by: Ralph Ursprung <Ralph.Ursprung@avaloq.com>
`build.sh` has been copied over to all active branches(*) of the OpenSearch repo and can now consequently be deleted here. this means, that from now on, the scripts from the respository will be picked up. for more information on the order of evaluation of the custom build scripts, see the [documentation][]. this gives the advantage to have branch-specific new logic in place there. this has already been used to add the no-jdk targets to the builds on main and 2.x. two tests were based on the assumption that there would be a `build.sh` for OpenSearch in this repository, however these are generic tests and just used OpenSearch as an example. they have been switched to be based on OpenSearch-Dashboards instead which - for the time being - still has its own `build.sh` here. (*): the following branches have been considered as active: - main - 2.x - 2.3 - 1.x - 1.3 this is part of opensearch-project#99 [documentation]: https://github.com/opensearch-project/opensearch-build/tree/main/src/build_workflow#custom-build-scripts Signed-off-by: Ralph Ursprung <Ralph.Ursprung@avaloq.com>
|
@peterzhuamazon (or anyone else who knows this): is there anything else left to be done so that the files created by the build process are also published to the other question is: where should these files be linked to? i'd suggest to just add them in the |
Based on the email exchange let's start some discussion on this. Thanks. |
…ct#4902) with commit e357246 the `build.sh` has been moved to this repository and with commit dfa1118 the missing no-jdk builds have been added. with these preliminary works done we can now start building the no-jdk distributions as part of the release build so that they can then be published. this is part of opensearch-project/opensearch-build#99. Signed-off-by: Ralph Ursprung <Ralph.Ursprung@avaloq.com> Signed-off-by: Ralph Ursprung <Ralph.Ursprung@avaloq.com> Co-authored-by: Daniel (dB.) Doubrovkine <dblock@amazon.com>
Is your feature request related to a problem? Please describe.
we want to be able to control the JDK being used (we have our own base docker images with a JDK configured the way we want (correct vendor, up-to-date security fixes, etc.) and thus don't need a JDK to be shipped with OpenSearch.
Describe the solution you'd like
please provide a no-jdk distribution (at least of the current "minimal" distribution) which - as the name implies - does not come with a pre-packaged JDK.
Describe alternatives you've considered
downloading the existing archive and manually removing the JDK would be a huge overhead and would also make the docker image larger (if we do this in the docker image; we could of course do it in a pre-step but that'd make things even more complicated) and it would be a non-standard setup which is harder for others to replicate.
Additional context
this has been discussed in the forum: No-jdk distribution of opensearch?
The text was updated successfully, but these errors were encountered: