[Retrospective] Release 1.2.0

[peternied]

The OpenSearch and OpenSearch Dashboards Release 1.2.0 is currently in progress. To make sure that we do not lose issues that arose during the process please add them here. This issue will be updated after the release is finished.

How to use this issue?

Please add comments to this issue, they can be small or large in scope. Honest feedback is important to improve our processes, suggestions are also welcomed but not required.

What will happen to this issue post release?

There will be a discussion(s) about how the release went and how the next release can be improved. Then this ticket will be updated with the notes of that discussion along side action items.

[peternied]

Component tickets off of the 1.2.0 release issue don't have dates on them. It makes it hard to know what the critical dates are from a component. Originally this was by design to prevent dates from drifting away from the agreed central data source of the primary release ticket.

Possible solutions:

1. Have an automated task than checks for updates on the primary release ticket and trickles down date updates to component tasks.
2. See about getting an iframe/ or external loaded data source into the issue
3. Super hacky idea - create an issue that represents a milestone and update its title with a date which is loaded dynamically

[ylwu-amzn]

Checkstyle removed from OpenSearch opensearch-project/OpenSearch#1370, that caused AD plugin 1.2 build failed, took team hours to figure out why. We should notify all plugin teams about such changes in future.

[peternied]

Automate snapshot builds were being created and used by the distribution downloader tool and they were out of date #953. A few issues this exposed:

• We did not know that part of the build infrastructure broke
• We did not realize that we were testing again old bits until really close inspection was conduced, decompiling binaries
• We needed to re-run a number of test cases and this involved a manual reach out to all components
• We updated the release date on the project board and that felt a little too easy / hard to manage

[peternied]

Seeing random test failures, might be associated with out of data code that was copypasted
opensearch-project/common-utils#97

[peternied]

Random integration test failures seem to happen frequently, the starting point of a test failure investigation is 'is this a random failure' indication low trust in the test systems.

[dblock]

There was some confusion of what should and what should not be included in the manifest, see #974.

[dblock]

• Let's add to the template "All issues labelled 1.2.0 are closed" with a link to https://github.com/issues?q=is%3Aopen+is%3Aissue+user%3Aopensearch-project++label%3Av1.2.0+
• Do plugins need to do a GitHub release?

[kavilla]

For improvement, plugin version bumping and adding to manifest as soon as possible would be ideal. Logically once a 1.1 branch was created and a 1.1.0.0 tag was created, the main branch version should have been bumped then we can start building releases as soon as possible for testing and validation. That way we can know way ahead of time of incompatibility with plugins.

[peternied]

Reflecting on OpenSearch Release 1.2.0, 1.2.1, 1.2.2, and 1.2.3.

There have been a massive number of contributors authoring code, reviewing pull requests, contribution to issues – thank you all.

Larger areas of consideration

• We changed communication to leverage publicly visible GitHub issues such as [RELEASE] Release version 1.2.0 #567. This is part our ongoing to allow for contribution. This large change created churn as for teams experiences these processes for the first time.
• Performance test results initially indicated a regression and after investigation were determined to be inconclusive. This delayed the release and created uncertainty within the features added in 1.2 We need investment in these systems to build confidence in the results.
• The process for patch releases was created for 1.2.1 there were setbacks when it was discovered that already built components could not be reused. This increased the amount of work needed to support the patch release from only two directly impacted teams to all components.
• Log4j CVE patch released were made available quickly and there were back-to-back releases, illustrating the need to keep a release ready posture for other critical changes.
• Automation of the build process has made headway with docker images available as builds are release but integration tests for the distribution, performance tests, artifact promotion have largely been done manually.

Action Items

• Updates for release documentation around 1.2 #760 @peternied
• Document the end-to-end OpenSearch release process #1258 @abhinavGupta16
• [META] Automate performance testing #126 @chenkle
• [META] Run integ tests with every build of OpenSearch and OpenSearch Dashboards and publish test results #818 @OhTyler
• [opensearch-plugin] Cannot install old patch version of plugins on newer opensearch builds  OpenSearch#1707 @tlfeng
• OpenSearch: Automate increment to next development iteration #1375 @dblock
• [Meta] Automate the OpenSearch release process #1361 @bbarani

With this we are closing out the retro items for 1.2.0-1.2.3 - its 2022 🎆 and I can not wait for the next set of releases!