[Proposal] Branching strategy for build

[gaiksaya]

Is your feature request related to a problem? Please describe

As we move towards next major release, more breaking changes will be introduced in the repo with respect to distribution.
This is fine for major versions, however as of today our build workflows checkout main branch of this repository (hard coded) for any upcoming release.

For example, if we are releasing 1.3.2 (upcoming) and 2.0.0-rc1 changes going actively in main, there might be breaking changes introduced unintentionally when it comes to distribution consumed by our customers. We do not have proper checks in place to detect this today.

The enhancements and features of next major release should not impact the minor releases of previous version.

Describe the solution you'd like

Follow standard branching strategy like other components, where 1.3.2 should be released using 1.3 branch.
This would involve decoupling of build workflows with main branch.

Acceptance Criteria

1. Collect requirements, write one pager with all the information
2. We should be able to release next minor and patch version based on previous major version with minimal efforts and no major changes going in
3. Create sub-tasks following this META.

Describe alternatives you've considered

No response

Additional context

No response

[bbarani]

[Tirage] We will groom this issue and add appropriate acceptance criteria before moving to backlog.

[peterzhuamazon]

Update 2022/05/10:

I have discussed with @dblock on branching with build repo and here are the thoughts.

Two Options:

1. Branching within opensearch-build.

• Pros:
  • Easy to do
  • Everything in one place
• Cons:
  • Jenkins requires a branch being defines during workflow / pipeline creation, and we are hardcoding to main as of now. This is not easy to change and would only add up more complexity in the process. Branching in build repo mean everytime there is a different version, admin would manually go to the workflow and switch branch.
  • We can also have 1 version per workflow, but that would add up number of workflows to maintain
  • Extra effort to maintain multiple copies of Jenkinsfile and libs while in reality it is not gonna change across versions

2. Stay on main for opensearch-build and move distribution/plugin specific scripts/configs to new repo opensearch-dist and branch on the dist repo, add dist repo as a component on manifest file.

• Pros:
  • Build scripts and automation sit in build repo, while dist/plugin specific files stays in dist repo
  • Once setup, we just need to add dist repo as a component in manifest, and easily switch b/w multiple branches
  • Each manifest will have an entry for dist repo, and effectively lock versions through tags after releases
• Cons:
  • Maintain additional repo
  • Need efforts to migrate scripts and change logic on many places, as everything was assumed to be in one place
  • Structure change on the build repo on its entirety

@bbarani @dblock please let me know about your thoughts.

Thanks.

[dblock]

I don't agree with the pros of (1), especially Easy to do - it's only easy to do the branching part, but it creates all the problems in the cons that would need to be solved, which is not easy at all.

I think(2) is very clean and easy to understand! All the cons are "it's a bit of work". I recommend (2).

[bbarani]

@peterzhuamazon I still dont understand the exact scope of Option 2. Will the new opensearch-dist repo hold all the distribution / plugin specific scripts? The plugin teams will need to raise PR for making manifest related changes to opensearch-dist repo? I am trying to understand the boundary between opensearch-build vs opensearch-dist repo. We also need to focus on moving the component scripts to component repo itself from our build repo.

[prudhvigodithi]

From what I understood, we are already recommending the plugin teams to maintain plugin specific scripts inside their own repo, as this will allow more flexibility is building plugin specific tasks, now if we go with option 2 and if plugin's team maintains their own scripts and configuration, will this new repo opensearch-dist be redundant?

There is another option we can consider.

• Upon Each release just like for plugin repos, cut a tag to opensearch-build repo as well.

• Organize Jenkins in views as per release version

• We can add all the required distribution build jobs under each view and add suffix with release version
  example
  <JOB_NAME>-1.3.2 , these jobs will point to 1.3.2 release tag in opensearch-build repo, which will have manifests that point to 1.3.2 release tag in plugin repos.

Note: We can also organize views as 1.3.x, 2.0.x and so on and point the jobs to latest 1.3 release tag and 2.0 release tag.

• Now if we have to build any released version, we can directly go to specific view and trigger the specific job.

• Once we are confident a version reaches EOL, we can safely delete the view and jobs inside the view.

With this we dont need to touch the actual core logic of our current build system and need not maintain any additional repos.
We can automate this workflow that would create views, jobs inside those views and jobs referring to release tag.

is there any other dependency that I have missed in our build-system (in actual core logic) that has hardcoded values to look for opensearch-build main branch ?
If no and only in jenkinsfile and under jobs configs, this approach could solve our problem, thoughts ?
@dblock @bbarani @peterzhuamazon

[peternied]

The scope is hard to understand as the there are many different components with different purposes in the opensearch-build branch, please add a diagram/table of what exist and how its coupled to different systems, then how this will be updated with the selected proposed 'branching' changes. With this table it should be easier to identify what should be grouped together vs apart.

Here are what could be a list of these components: pipeline jenkins scripts, opensearch manifest files, data-prepper manifests, default build scripts, custom build scripts, docker build scripts, other build processes (rpm?)

[peternied]

There is an question of 'effort' with this change in process, I would recommend revisiting the last N changes into opensearch-build and then determine if they would go into a certain branch and how much work to manage them would be entailed (forward/back porting?) This would be useful in informing different segregation strategies.

[prudhvigodithi]

Build Repo Summary

components: The existing components within the opensearch-build repo
central-code(yes/no) : Denotes if the component is directly dependent of build ecosystem.
required-branching(yes/no): Denotes if the component should be part of repo’s branching strategy.
Git URL: component git code location.

components	central-code	required-branching	Git URL	should this be part of build repo?	Additional Feedback/Concerns
Jenkinsfiles	yes	yes	https://github.com/opensearch-project/opensearch-build/tree/main/jenkins	yes	Jenkinsfiles are not only used for distribution builds, but also used for other build automation workflows. Jenkinsfile that support distribution builds should be branched and other's used for automation can be part of central-code which need not be branched.
Input Manifest files	yes	no	https://github.com/opensearch-project/opensearch-build/tree/main/manifests	no	The Input Manifest files, does not require branching, but in solutions proposed above they has to be organized as per <MAJOR> release version.Example: with solution [2] 2.0.0 manifies should be part of 2.x branch.
Cloudfront deployment	no	no	https://github.com/opensearch-project/opensearch-build/tree/main/deployment	no	This cloudFront deployment should not be part of this build repo and not required to be branched as its only for one time installation.
data-prepper	no	no	https://github.com/opensearch-project/opensearch-build/blob/main/jenkins/data-prepper/release-data-prepper-all-artifacts.jenkinsfile	no	Data prepper should not be part of build-repo as its a sperate OpenSearch tool used for a specific purpose. This does not particilate in any workflow that is part of opensearch-build.
Dockerfiles	no	no	https://github.com/opensearch-project/opensearch-build/tree/main/docker	yes	Dockerfiles do not directly depend on the build workflows, but are used to containerize the OpenSearch applications. So Dockerfiles can still reside in opensearch-build but not required to be branched.
vulnerability-scan	no	yes	https://github.com/opensearch-project/opensearch-build/tree/main/tools/vulnerability-scan	no
Plugin scripts	no	yes	https://github.com/opensearch-project/opensearch-build/tree/main/scripts	no	Plugin scripts should be part of each plugin repo which adds more flexibility to run plugin specifc tasks.
Jenkins shared libs	yes	yes	https://github.com/opensearch-project/opensearch-build/tree/main/vars	yes	The Jenkins shared libraries are used by jenkinfiles, not all Jenkins shared libraries required to be branched, but should be developed in way it supports jenkinsfiles used for past and future releases
Build workflow	yes	yes	https://github.com/opensearch-project/opensearch-build/tree/main/src/build_workflow	yes
Assemble workflow	yes	yes	https://github.com/opensearch-project/opensearch-build/tree/main/src/assemble_workflow	yes
Test workflow	yes	yes	https://github.com/opensearch-project/opensearch-build/tree/main/src/test_workflow	yes	Test Workflow should be considered branching, with different release versions, the test workflow could be changed and cannot rely on same test workflow for each version.GH issue: #2117
signing workflow	no	yes	https://github.com/opensearch-project/opensearch-build/tree/main/src/sign_workflow	no	The entire signing workflow can be considered moving out of opensearch-build repo. Right now opensearch-signer-client is used which is already fetched from external repo. This signing workflow should be externalized and fetched during runtime to sign and publish artifcats.
Checkout workflow	no	yes	https://github.com/opensearch-project/opensearch-build/tree/main/src/checkout_workflow	yes
Sanity check workflow	no	yes	https://github.com/opensearch-project/opensearch-build/tree/main/src/ci_workflow	yes
Manifest workflow	no	yes	https://github.com/opensearch-project/opensearch-build/tree/main/src/manifests_workflow	yes	This can reside in opensearch-build repo, but does not directly dependant on build ecosystem, this creates input manifests.

[prudhvigodithi]

Proposed Solution:

Assumptions:

All the distribution build jobs only depend upon Jenkins job configuration pipeline from SCM repository URL and the repository URL’s are not hardcoded inside Jenkinsfiles.

Solution Details:

opensearch-build repo will actively maintain main and branches with <MAJOR_VERION>.x
Ex: main, 1.x, 2.x, 3.x
Note: The starting point (source of truth) of <MAJOR_VERION>.x branch will be main

Scenario: 1

If a specific PR is an enhancement that can be used across the build-system, the PR should be merged to main and then backport to desired <MAJOR_VERION>.x branch.
Example: If a new security enhancement is proposed and let’s say the current target release is 1.3.3, the proposed enhancement should be pushed to main and backported to 1.x.

Scenario: 2

If a specific PR is limited to only specific version, then it’s allowed to directly merge to respective <MAJOR_VERION>.x branch.
Example: let’s say the target release is 2.1.0 and a pipeline fix has to be made only for 2.x releases, the PR has to be merged to 2.x branch.

Workflow:

• Organize Jenkins views as per major release version.

• Add all the required distribution build jobs to the view (choosing the view as per major release version).
  Example: The distribution jobs inside 1.x view will point to 1.x branch from opensearch-build repo, which will have manifests that point to upcoming 1.x release branch from plugin repos. These jobs can now support the upcoming 1.x release and can be used to build a patch/hotfix release.

NOTE: From @peterzhuamazon suggestion, distribution jobs can be modified to have an input <MAJOR_VERION>.x that pull the required release branch, instead of maintaining multiple jenkins workflows and views.

• Upon Each release just like for plugin repos, cut a release version tag and branch if required to opensearch-build repo using the respective <MAJOR_VERION>.x.

• Once we are confident a version reaches EOL, we can safely archive/delete the Jenkins view and jobs inside the view.

• Pros:

  • No modification on existing build-scripts.
  • This strategy can be used to isolate the build-code per release.
  • Freezing the code using the tag strategy inclines with existing plugin repos.
  • Changes/Fixes can only be pushed to required <MAJOR_VERION> release branch.
  • Backport mechanism, to combine a new enhancement from main to <MAJOR_VERION> release branch.

• Cons:

  • Maintenance of multiple pipelines, views in Jenkins.
  • Need right development strategy maintain multiple <MAJOR_VERION>.x branches
  • Required development of workflows that will be inclined to this solution.

[prudhvigodithi]

Hey @dblock @peternied @bbarani @peterzhuamazon please add your thoughts to my above two comments.
Thank you

[prudhvigodithi]

NOTE: From @peterzhuamazon suggestion, distribution jobs can be modified to have an input <MAJOR_VERION>.x that pull the required release branch, instead of maintaining multiple jenkins workflows and views.

@peterzhuamazon in this case if we have to go with single job that can have <MAJOR_VERION>.x as input, we should be changing the build, assemble and test workflow logic to use the respective workflow code from<MAJOR_VERION>.x branch, please correct me if i'm wrong.

[peternied]

@prudhvigodithi Thanks for the detailed breakout of the different areas of the repository and what you'd like to move.

The Input Manifest files, does not require branching, but in solutions proposed above they has to be organized as per release version.Example: with solution [2] 2.0.0 manifies should be part of 2.x branch.

The current organization of manifests has them in folders that are named, branching these files seems redundant? While these can move to another repo, the branch 'name' is effective a part of the naming scheme.

Once we are confident a version reaches EOL, we can safely archive/delete the Jenkins view and jobs inside the view.

Why delete anything? We can never know if something is really unused or not, and deletion is not reversible. Deprecation/relegation to a static state is fine, but I would avoid any actual deletion activities.

---

Could you restate what problem you are trying to solve in the Proposed Solution? Its hard to evaluate the scenarios without knowing what needs to be overcomed?

[dblock]

Which product's versioning scheme is going to be used for branching and those Jenkins tabs? For example, is 2.0 the tooling for DataPrepper 2.0 or OpenSearch 2.0? I am afraid that after we create the proposed branching scheme we'll be back to square 1 where Data Prepper 2.0 wants newest tooling from OpenSearch 3.0 and we've just made things very confusing again.

[prudhvigodithi]

The current organization of manifests has them in folders that are named, branching these files seems redundant? While these can move to another repo, the branch 'name' is effective a part of the naming scheme.

Hey @peternied, thanks for your thought, we dont need to maintain a separate repo, in above solution since we maintain main, 1.x, 2.x, 3.x and so on , we can check in the respective version manifests to specific release branch, Example all 1.3.3, 1.3.4 will be part of 1.x branch, 2.0.0, 2.0.1 will be part of 2.x branch.

Why delete anything? We can never know if something is really unused or not, and deletion is not reversible. Deprecation/relegation to a static state is fine, but I would avoid any actual deletion activities.

So my thought was, the release branches (1.x, 2.x, 3.x) still exists, but from jenkins we archive/delete views and jobs, these jobs can always be recreated as jenkinsfiles are part of the respective release branches, this way jenkins is not cluttered with EOL version jobs and views, we can always keep them forever, open for suggestions.

Could you restate what problem you are trying to solve in the Proposed Solution? Its hard to evaluate the scenarios without knowing what needs to be overcomed?

@peternied more than a problem I'm trying to propose a process of branching to the build repo, this will eventually solve problems of code being not compatible with release versions (current or future). Its not ideal to have entire development and release workflow on a single source of truth main branch. Also having a proper branching strategy will help organize a series of planned, structured releases.

[prudhvigodithi]

Which product's versioning scheme is going to be used for branching and those Jenkins tabs? For example, is 2.0 the tooling for DataPrepper 2.0 or OpenSearch 2.0? I am afraid that after we create the proposed branching scheme we'll be back to square 1 where Data Prepper 2.0 wants newest tooling from OpenSearch 3.0 and we've just made things very confusing again.

@dblock considering 2.0, 2.x release branch, 2.x jenkins views will have all build jobs and code, that can build both DataPrepper 2.0 or OpenSearch 2.0.
DataPrepper 3.0, OpenSearch 3.0, all build jobs and code will be part of 3.x and jenkins 3.x views.
New enhancements (New tooling), features will be merged to main and backported to desired release branch, considering the advance tooling, we can still backport them to 2.x, where DataPrepper 2.0 jobs depend on
How does that sound to you? @dblock

[dblock]

I think this isn't going to work. Let's say there's 1 year of development between OpenSearch 2.0 and DataPrepper 2.0. When you introduce a feature on main to build DataPrepper 2.0 it may be incompatible with building OpenSearch 2.0, so you cannot back-port it to the 2.x branch, because it will break a very old OpenSearch 2.0 build. My recommendation is to move any code that needs to branch with a product (e.g. docker scripts) to the products' repos, branch and version tooling in opensearch-build independently, move any Jenkinsfile's for each product into those product repos and refer to the specific versions of tooling from those Jenkinsfile's.

[prudhvigodithi]

My thought is If there's 1 year of development difference between OpenSearch 2.0 and DataPrepper 2.0 and both being build from main branch, then we are in same situation right ? because both use the same main branch and Ideally this is even same with Current OpenSearch Core repo as well it could break with backport, but considering the way of development it should be accommodated with both OpenSearch 2.0 and DataPrepper 2.0 (then only we consider it to backport). If required in these cases we can still maintain component specific scripts, thats too possible, Imagine we end up in this same situation (If there's 1 year of development difference between OpenSearch 2.0 and DataPrepper 2.0) with single source of truth main ? it would be more cluttered.

To separate as products' repos is good thought. We can even consider exploring a tooling framework that can be used and imported with versions.
Example entire 2.0.0 release will be importing tooling:2.0 that has everything related to build, test and assemble.
so we can periodically update the tooling framework version with fixes and enhancements as per release.
@dblock

[dblock]

I agree that the current situation of everything on main isn't great. My opinion is that the branching strategy proposed creates many main's, so it's net worse. Thanks for a good discussion, thought, I am just trying to help, do what you think will work best!

[prudhvigodithi]

Hey going forward with solution posted above, closing this issue and will open a new META issue with proper breakdown of tasks and once done will link back here.
@bbarani
thank you