Release notes automation #4196

SachinSahu431 · 2023-11-02T18:20:25Z

Description

Python script to automate the process of collecting all plugins' components release notes and creating a consolidated release note.

Issues Resolved

Closes #4009

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Sachin Sahu <sachinksahu.431@gmail.com>

Add markdown parser 'mistune' in Pipfile Python script generates 'release-notes availability' table and URL file for verification Signed-off-by: Sachin Sahu <sachinksahu.431@gmail.com>

Signed-off-by: Sachin Sahu <sachinksahu.431@gmail.com>

Add custom heading mapping Fix code format Signed-off-by: Sachin Sahu <sachinksahu.431@gmail.com>

- Handle empty headings - Support multiple formats - Fix naming of 'Opensearch.job Scheduler' Signed-off-by: Sachin Sahu <sachinksahu.431@gmail.com>

peterzhuamazon · 2023-11-07T19:42:01Z

Reviewed in officehour, suggest @SachinSahu431 to sort the repos based on repo name before adding entries.
Once tests are added we can start reviewing this before adding github actions.

Thanks.

Signed-off-by: Sachin Sahu <sachinksahu.431@gmail.com>

Resolve merge conflict Code cleanup Signed-off-by: Sachin Sahu <sachinksahu.431@gmail.com>

Signed-off-by: Sachin Sahu <sachinksahu.431@gmail.com>

Signed-off-by: Sachin Sahu <75629410+SachinSahu431@users.noreply.github.com>

peterzhuamazon

Thanks @SachinSahu431 for the initial draft, ping @gaiksaya for the co-review of this PR.

Two questions:

Could you only update new package in pipfile.lock instead of touching others? I remember able to do that in the past but I dont recall the commands I use. Just to make sure we are not upgrading the existing stable versions of other packages.
You have a lot of .md and .txt in the automation dir, are they needed or just for testing purposes?
Considering adding some sample commands and descriptions to README file within the new dir you created, see build_workflow for example but this can be done later.
Can this be part of the existing release_notes_workflow dir?

Thanks.

Signed-off-by: Sachin Sahu <sachinksahu.431@gmail.com>

Directory Restructure Fix Type Checker errors Signed-off-by: Sachin Sahu <sachinksahu.431@gmail.com>

Signed-off-by: Sachin Sahu <sachinksahu.431@gmail.com>

SachinSahu431 · 2023-11-10T14:13:25Z

@peterzhuamazon Thank you for the review!

I've undone the modifications in Pipfile.lock and updated it for new package with pipenv lock --keep-outdated.
Affirming that the .md and .txt files are specifically used to validate the results of the ongoing Python script; they will be removed before the merge.
The directory has been reorganized, and the README.md file has been updated.

Thanks again!

gaiksaya

Thanks for the contribution @SachinSahu431 . This will definitely save us a lot of manual effort.
Added some nits and questions. Few suggestions:

Please replace all print statements with logging.info as best practice.
Remove all temp / test files
Fix test cases and add new ones for new methods.

src/release_notes_workflow/README.md

src/run_releasenotes_check.py

gaiksaya · 2023-11-17T18:21:29Z

Hi @SachinSahu431

Please feel free to convert this PR ready for review when you are ready. Looks like tests needs to be fixed and also some comments need to be addressed.
Thanks!

Add 'results' to gitignore Fix nits Add manifest file for tests Signed-off-by: Sachin Sahu <sachinksahu.431@gmail.com>

codecov · 2023-11-18T12:17:51Z

Codecov Report

Attention: 132 lines in your changes are missing coverage. Please review.

Comparison is base (a97cf7d) 93.17% compared to head (150c5d4) 91.24%.
Report is 18 commits behind head on main.

Files	Patch %	Lines
src/run_releasenotes_check.py	5.60%	118 Missing ⚠️
src/release_notes_workflow/release_notes.py	52.94%	8 Missing ⚠️
.../release_notes_workflow/release_notes_component.py	70.00%	6 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4196      +/-   ##
==========================================
- Coverage   93.17%   91.24%   -1.94%     
==========================================
  Files         188      188              
  Lines        5920     6063     +143     
==========================================
+ Hits         5516     5532      +16     
- Misses        404      531     +127

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

SachinSahu431 · 2023-11-18T12:23:03Z

Hi @gaiksaya!

I have added tests for the new compile option. Explicit tests for the check option are not required as it is a subset of the compile option. Also, since the consolidation script ignores OpenSearch component, I've created a new manifest file with component named Opensearch-test in manifests/templates/opensearch/test for testing purposes.

manifests/templates/opensearch/test/manifest.yml

Signed-off-by: Sachin Sahu <sachinksahu.431@gmail.com>

bbarani · 2023-12-01T23:26:13Z

@gaiksaya @peterzhuamazon Can we move ahead with this PR?

gaiksaya

Looks like code coverage as commented on the lines missing the test cases. Is it possible for you to add those?

If not, you can take that up as a follow up PR.

gaiksaya · 2023-12-04T17:53:55Z

.gitignore

@@ -37,3 +37,4 @@ out.txt
 vars/

 test-report.yml
+src/release_notes_workflow/results/*


Is this still required?

@gaiksaya Yes, this is to ignore the folder containing temporary release notes created by script.

We can work on missing test cases in a new PR.

gaiksaya · 2024-03-21T01:02:26Z

Hey @SachinSahu431

Looks like a bug is introduced by this PR #4549
Please feel free to pick up the issue if you are interested.

Thanks!

SachinSahu431 added 2 commits November 2, 2023 13:09

Add compile option

1515a1a

Signed-off-by: Sachin Sahu <sachinksahu.431@gmail.com>

Add Python script to compile release notes

bbf3ebe

Add markdown parser 'mistune' in Pipfile Python script generates 'release-notes availability' table and URL file for verification Signed-off-by: Sachin Sahu <sachinksahu.431@gmail.com>

github-actions bot added the first-time-contributor label Nov 2, 2023

SachinSahu431 added 4 commits November 3, 2023 02:01

Add --output argument support

9061023

Signed-off-by: Sachin Sahu <sachinksahu.431@gmail.com>

Fix Python code format

5ac3224

Signed-off-by: Sachin Sahu <sachinksahu.431@gmail.com>

Handle unknown categories

c9c46bb

Add custom heading mapping Fix code format Signed-off-by: Sachin Sahu <sachinksahu.431@gmail.com>

Add non-compliant section

072898e

- Handle empty headings - Support multiple formats - Fix naming of 'Opensearch.job Scheduler' Signed-off-by: Sachin Sahu <sachinksahu.431@gmail.com>

SachinSahu431 and others added 5 commits November 8, 2023 17:55

sort repo names

e327d33

Signed-off-by: Sachin Sahu <sachinksahu.431@gmail.com>

Fix name 'opensearch-sql' and resolve merge conflict

01d0e5b

Signed-off-by: Sachin Sahu <sachinksahu.431@gmail.com>

Fix name 'opensearch-sql'

bc0bf6e

Resolve merge conflict Code cleanup Signed-off-by: Sachin Sahu <sachinksahu.431@gmail.com>

Resolve merge conflict

daa1df0

Signed-off-by: Sachin Sahu <sachinksahu.431@gmail.com>

Merge branch 'main' into release-notes-automation

78115e2

Signed-off-by: Sachin Sahu <75629410+SachinSahu431@users.noreply.github.com>

peterzhuamazon reviewed Nov 9, 2023

View reviewed changes

SachinSahu431 added 3 commits November 10, 2023 09:11

Revert changes in Pipfile.lock

f7af7c5

Signed-off-by: Sachin Sahu <sachinksahu.431@gmail.com>

Update Pipfile.lock

2da5dce

Directory Restructure Fix Type Checker errors Signed-off-by: Sachin Sahu <sachinksahu.431@gmail.com>

Update README.md for release_notes_workflow

c371fc4

Signed-off-by: Sachin Sahu <sachinksahu.431@gmail.com>

gaiksaya reviewed Nov 14, 2023

View reviewed changes

Add tests for compile option

8f074f3

Add 'results' to gitignore Fix nits Add manifest file for tests Signed-off-by: Sachin Sahu <sachinksahu.431@gmail.com>

SachinSahu431 requested a review from gaiksaya November 18, 2023 15:53

SachinSahu431 marked this pull request as ready for review November 19, 2023 18:09

SachinSahu431 requested review from dblock, bbarani, rishabh6788, zelinh and jordarlu as code owners November 19, 2023 18:09

SachinSahu431 requested review from prudhvigodithi, Divyaasm and tianleh as code owners November 19, 2023 18:09

gaiksaya reviewed Nov 20, 2023

View reviewed changes

manifests/templates/opensearch/test/manifest.yml Outdated Show resolved Hide resolved

Update path for test data

150c5d4

Signed-off-by: Sachin Sahu <sachinksahu.431@gmail.com>

gaiksaya reviewed Dec 4, 2023

View reviewed changes

gaiksaya approved these changes Dec 6, 2023

View reviewed changes

gaiksaya merged commit 5dbcbaf into opensearch-project:main Dec 6, 2023
11 of 12 checks passed

gaiksaya mentioned this pull request Mar 21, 2024

[Bug]: Release notes checker does not check release notes for core engines #4549

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Release notes automation #4196

Release notes automation #4196

SachinSahu431 commented Nov 2, 2023

peterzhuamazon commented Nov 7, 2023

peterzhuamazon left a comment

SachinSahu431 commented Nov 10, 2023

gaiksaya left a comment

gaiksaya commented Nov 17, 2023

codecov bot commented Nov 18, 2023 •

edited

Loading

SachinSahu431 commented Nov 18, 2023

bbarani commented Dec 1, 2023

gaiksaya left a comment

gaiksaya Dec 4, 2023

SachinSahu431 Dec 6, 2023

SachinSahu431 Dec 6, 2023

gaiksaya commented Mar 21, 2024

Release notes automation #4196

Release notes automation #4196

Conversation

SachinSahu431 commented Nov 2, 2023

Description

Issues Resolved

peterzhuamazon commented Nov 7, 2023

peterzhuamazon left a comment

Choose a reason for hiding this comment

SachinSahu431 commented Nov 10, 2023

gaiksaya left a comment

Choose a reason for hiding this comment

gaiksaya commented Nov 17, 2023

codecov bot commented Nov 18, 2023 • edited Loading

Codecov Report

SachinSahu431 commented Nov 18, 2023

bbarani commented Dec 1, 2023

gaiksaya left a comment

Choose a reason for hiding this comment

gaiksaya Dec 4, 2023

Choose a reason for hiding this comment

SachinSahu431 Dec 6, 2023

Choose a reason for hiding this comment

SachinSahu431 Dec 6, 2023

Choose a reason for hiding this comment

gaiksaya commented Mar 21, 2024

codecov bot commented Nov 18, 2023 •

edited

Loading