[Backport 2.x][OSCI][FEAT] Changelog Project - PoC Changelog and rel…

…ease notes automation tool (opensearch-project#6918)

* [OSCI][FEAT] Changelog Project - PoC Changelog and release notes automation tool - OpenSearch Dashboards (opensearch-project#5519)

Refactor and Enhance Workflow Management

- Added and updated changesets for multiple PRs to improve tracking and documentation of changes.
- Removed unnecessary test and dummy files (`test.txt`, various `.yml` fragments) to clean up the repository.
- Refactored workflow scripts to streamline changelog generation and fragment handling, moving temporary files to a designated folder.
- Updated GitHub Actions workflows by changing event triggers from `pull_request` to `pull_request_target` and vice versa to optimize workflow execution.
- Enhanced security and automation by updating token names and adding write permissions to the changeset workflow.
- Deleted obsolete workflow file for creating changeset files, now handled by an automated process.
- Major clean-up of dummy fragment files and unnecessary changelog entries to maintain clarity and relevancy in documentation.
- Implemented minor updates and improvements in codebase, specifically in generating release notes and handling fragments.

---------

Signed-off-by: Johnathon Bowers <johnathonbowers@gmail.com>
Signed-off-by: CMDWillYang <williamyang721@gmail.com>
Signed-off-by: Qiwen Li <qiwen_li@brown.edu>
Signed-off-by: qiwen li <qiwen_li@brown.edu>
Signed-off-by: Samuel Valdes Gutierrez <valdesgutierrez@gmail.com>
Signed-off-by: Ashwin P Chandran <ashwinpc1993@gmail.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Johnathon Bowers <johnathonbowers@gmail.com>
Co-authored-by: CMDWillYang <williamyang721@gmail.com>
Co-authored-by: Qiwen Li <qiwen_li@brown.edu>
Co-authored-by: Ashwin P Chandran <ashwinpc1993@gmail.com>
Co-authored-by: Anan Zhuang <ananzh@amazon.com>
Co-authored-by: Josh Romero <rmerqg@amazon.com>
Co-authored-by: autochangeset[bot] <154024398+autochangeset[bot]@users.noreply.github.com>
Co-authored-by: opensearch-bot[bot] <154024398+opensearch-bot[bot]@users.noreply.github.com>
Co-authored-by: opensearch-bot-dev[bot] <154634848+opensearch-bot-dev[bot]@users.noreply.github.com>
Co-authored-by: Ashwin P Chandran <ashwinpc@amazon.com>
Co-authored-by: Miki <amoo_miki@yahoo.com>
Co-authored-by: Kawika Avilla <kavilla414@gmail.com>

* delete file erroneous restored when handling conflict

Signed-off-by: Kawika Avilla <kavilla414@gmail.com>

---------

Signed-off-by: Johnathon Bowers <johnathonbowers@gmail.com>
Signed-off-by: CMDWillYang <williamyang721@gmail.com>
Signed-off-by: Qiwen Li <qiwen_li@brown.edu>
Signed-off-by: qiwen li <qiwen_li@brown.edu>
Signed-off-by: Samuel Valdes Gutierrez <valdesgutierrez@gmail.com>
Signed-off-by: Ashwin P Chandran <ashwinpc1993@gmail.com>
Signed-off-by: Kawika Avilla <kavilla414@gmail.com>
Co-authored-by: Samuel Valdes Gutierrez <valdesgutierrez@gmail.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Johnathon Bowers <johnathonbowers@gmail.com>
Co-authored-by: CMDWillYang <williamyang721@gmail.com>
Co-authored-by: Qiwen Li <qiwen_li@brown.edu>
Co-authored-by: Ashwin P Chandran <ashwinpc1993@gmail.com>
Co-authored-by: Anan Zhuang <ananzh@amazon.com>
Co-authored-by: Josh Romero <rmerqg@amazon.com>
Co-authored-by: autochangeset[bot] <154024398+autochangeset[bot]@users.noreply.github.com>
Co-authored-by: opensearch-bot[bot] <154024398+opensearch-bot[bot]@users.noreply.github.com>
Co-authored-by: opensearch-bot-dev[bot] <154634848+opensearch-bot-dev[bot]@users.noreply.github.com>
Co-authored-by: Ashwin P Chandran <ashwinpc@amazon.com>
Co-authored-by: Miki <amoo_miki@yahoo.com>

.github/pull_request_template.md

@@ -7,6 +7,18 @@
 <!-- List any issues this PR will resolve. -->
 <!-- Example: closes #1234 -->
 
+## Changelog
+<!--
+Add a short but concise sentence about the impact of this pull request. Prefix an entry with the type of change they correspond to: breaking, chore, deprecate, doc, feat, fix, infra, refactor, test.
+- fix: Update the graph
+- feat: Add a new feature
+
+If this change does not need to added to the changelog, just add a single `skip` line e.g.
+- skip
+
+Descriptions following the prefixes must be 100 characters long or less
+-->
+
 ### Check List
 
 - [ ] All tests pass

.github/workflows/opensearch_changelog_workflow.yml

@@ -0,0 +1,23 @@
+name: OpenSearch Changelog Workflow
+
+on:
+  pull_request_target:
+    types: [opened, reopened, edited]
+
+permissions:
+  contents: read
+  issues: write
+  pull-requests: write
+
+jobs:
+  update-changelog:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Check out repository
+        uses: actions/checkout@v4
+      - name: Parse changelog entries and submit request for changset creation
+        uses: BigSamu/OpenSearch_Changelog_Workflow@1.0.0-alpha1
+        with:
+          token: ${{secrets.GITHUB_TOKEN}}
+          CHANGELOG_PR_BRIDGE_URL_DOMAIN: ${{secrets.CHANGELOG_PR_BRIDGE_URL_DOMAIN}}
+          CHANGELOG_PR_BRIDGE_API_KEY: ${{secrets.CHANGELOG_PR_BRIDGE_API_KEY}}

changelogs/README.md

@@ -0,0 +1,5 @@
+# Changelog and Release Notes
+
+For information regarding the changelog and release notes process, please consult the README in the GitHub Actions repository that this process utilizes. To view this README, follow the link below:
+
+[GitHub Actions Workflow README](https://github.com/BigSamu/OpenSearch_Change_Set_Create_Action/blob/main/README.md)

package.json

@@ -75,7 +75,8 @@
     "docs:acceptApiChanges": "scripts/use_node --max-old-space-size=6144 scripts/check_published_api_changes.js --accept",
     "osd:bootstrap": "scripts/use_node scripts/build_ts_refs && scripts/use_node scripts/register_git_hook",
     "spec_to_console": "scripts/use_node scripts/spec_to_console",
-    "pkg-version": "scripts/use_node -e \"console.log(require('./package.json').version)\""
+    "pkg-version": "scripts/use_node -e \"console.log(require('./package.json').version)\"",
+    "release_note:generate": "scripts/use_node scripts/generate_release_note"
   },
   "repository": {
     "type": "git",

scripts/generate_release_note.js

@@ -0,0 +1,8 @@
+/*
+ * Copyright OpenSearch Contributors
+ * SPDX-License-Identifier: Apache-2.0
+ */
+
+require('../src/setup_node_env');
+require('../src/dev/generate_release_note');
+require('../src/dev/generate_release_note_helper');

src/dev/generate_release_note.ts

@@ -0,0 +1,134 @@
+/*
+ * Copyright OpenSearch Contributors
+ * SPDX-License-Identifier: Apache-2.0
+ */
+
+import { join, resolve } from 'path';
+import { readFileSync, writeFileSync, Dirent, rm, rename, promises as fsPromises } from 'fs';
+import { load as loadYaml } from 'js-yaml';
+import { readdir } from 'fs/promises';
+import { version as pkgVersion } from '../../package.json';
+import {
+  validateFragment,
+  getCurrentDateFormatted,
+  Changelog,
+  SECTION_MAPPING,
+  fragmentDirPath,
+  SectionKey,
+  releaseNotesDirPath,
+  filePath,
+} from './generate_release_note_helper';
+
+// Function to add content after the 'Unreleased' section in the changelog
+function addContentAfterUnreleased(path: string, newContent: string): void {
+  let fileContent = readFileSync(path, 'utf8');
+  const targetString = '## [Unreleased]';
+  const targetIndex = fileContent.indexOf(targetString);
+
+  if (targetIndex !== -1) {
+    const endOfLineIndex = fileContent.indexOf('\n', targetIndex);
+    if (endOfLineIndex !== -1) {
+      fileContent =
+        fileContent.slice(0, endOfLineIndex + 1) +
+        '\n' +
+        newContent +
+        '\n' +
+        fileContent.slice(endOfLineIndex + 1);
+    } else {
+      throw new Error('End of line for "Unreleased" section not found.');
+    }
+  } else {
+    throw new Error("'## [Unreleased]' not found in the file.");
+  }
+
+  writeFileSync(path, fileContent);
+}
+
+async function deleteFragments(fragmentTempDirPath: string) {
+  rm(fragmentTempDirPath, { recursive: true }, (err: any) => {
+    if (err) {
+      throw err;
+    }
+  });
+}
+
+// Read fragment files and populate sections
+async function readFragments() {
+  // Initialize sections
+  const sections: Changelog = (Object.fromEntries(
+    Object.keys(SECTION_MAPPING).map((key) => [key, []])
+  ) as unknown) as Changelog;
+
+  const fragmentPaths = await readdir(fragmentDirPath, { withFileTypes: true });
+  for (const fragmentFilename of fragmentPaths) {
+    // skip non yml or yaml files
+    if (!/\.ya?ml$/i.test(fragmentFilename.name)) {
+      // eslint-disable-next-line no-console
+      console.warn(`Skipping non yml or yaml file ${fragmentFilename.name}`);
+      continue;
+    }
+
+    const fragmentPath = join(fragmentDirPath, fragmentFilename.name);
+    const fragmentContents = readFileSync(fragmentPath, { encoding: 'utf-8' });
+
+    validateFragment(fragmentContents);
+
+    const fragmentYaml = loadYaml(fragmentContents) as Changelog;
+
+    for (const [sectionKey, entries] of Object.entries(fragmentYaml)) {
+      sections[sectionKey as SectionKey].push(...entries);
+    }
+  }
+  return { sections, fragmentPaths };
+}
+
+async function moveFragments(fragmentPaths: Dirent[], fragmentTempDirPath: string): Promise<void> {
+  // Move fragment files to temp fragments folder
+  for (const fragmentFilename of fragmentPaths) {
+    const fragmentPath = resolve(fragmentDirPath, fragmentFilename.name);
+    const fragmentTempPath = resolve(fragmentTempDirPath, fragmentFilename.name);
+    rename(fragmentPath, fragmentTempPath, () => {});
+  }
+}
+
+function generateChangelog(sections: Changelog) {
+  // Generate changelog sections
+  const changelogSections = Object.entries(sections).map(([sectionKey, entries]) => {
+    const sectionName = SECTION_MAPPING[sectionKey as SectionKey];
+    return entries.length === 0
+      ? `### ${sectionName}`
+      : `### ${sectionName}\n\n${entries.map((entry) => ` - ${entry}`).join('\n')}`;
+  });
+
+  // Generate full changelog
+  const currentDate = getCurrentDateFormatted();
+  const changelog = `## [${pkgVersion}-${currentDate}](https://github.com/opensearch-project/OpenSearch-Dashboards/releases/tag/${pkgVersion})\n\n${changelogSections.join(
+    '\n\n'
+  )}`;
+  // Update changelog file
+  addContentAfterUnreleased(filePath, changelog);
+  return changelogSections;
+}
+
+function generateReleaseNote(changelogSections: string[]) {
+  // Generate release note
+  const releaseNoteFilename = `opensearch-dashboards.release-notes-${pkgVersion}.md`;
+  const releaseNoteHeader = `# VERSION ${pkgVersion} Release Note`;
+  const releaseNote = `${releaseNoteHeader}\n\n${changelogSections.join('\n\n')}`;
+  writeFileSync(resolve(releaseNotesDirPath, releaseNoteFilename), releaseNote);
+}
+
+(async () => {
+  const { sections, fragmentPaths } = await readFragments();
+  // create folder for temp fragments
+  const fragmentTempDirPath = await fsPromises.mkdtemp(join(fragmentDirPath, 'tmp_fragments-'));
+  // move fragments to temp fragments folder
+  await moveFragments(fragmentPaths, fragmentTempDirPath);
+
+  const changelogSections = generateChangelog(sections);
+
+  generateReleaseNote(changelogSections);
+
+  // remove temp fragments folder
+  await deleteFragments(fragmentTempDirPath);
+})();

src/dev/generate_release_note_helper.ts

@@ -0,0 +1,59 @@
+/*
+ * Copyright OpenSearch Contributors
+ * SPDX-License-Identifier: Apache-2.0
+ */
+
+import { resolve } from 'path';
+
+export const filePath = resolve(__dirname, '..', '..', 'CHANGELOG.md');
+export const fragmentDirPath = resolve(__dirname, '..', '..', 'changelogs', 'fragments');
+export const releaseNotesDirPath = resolve(__dirname, '..', '..', 'release-notes');
+
+export function getCurrentDateFormatted(): string {
+  return new Date().toISOString().slice(0, 10);
+}
+
+export const SECTION_MAPPING = {
+  breaking: '💥 Breaking Changes',
+  deprecate: 'Deprecations',
+  security: '🛡 Security',
+  feat: '📈 Features/Enhancements',
+  fix: '🐛 Bug Fixes',
+  infra: '🚞 Infrastructure',
+  doc: '📝 Documentation',
+  chore: '🛠 Maintenance',
+  refactor: '🪛 Refactoring',
+  test: '🔩 Tests',
+};
+
+export type SectionKey = keyof typeof SECTION_MAPPING;
+export type Changelog = Record<SectionKey, string[]>;
+
+const MAX_ENTRY_LENGTH = 100;
+// Each entry must start with '-' and a space, followed by a non-empty string, and be no longer that MAX_ENTRY_LENGTH characters
+const entryRegex = new RegExp(`^-.{1,${MAX_ENTRY_LENGTH}}\\(\\[#.+]\\(.+\\)\\)$`);
+
+// validate format of fragment files
+export function validateFragment(content: string) {
+  const sections = content.split(/(?:\r?\n){2,}/);
+
+  // validate each section
+  for (const section of sections) {
+    const lines = section.split('\n');
+    const sectionName = lines[0];
+    const sectionKey = sectionName.slice(0, -1);
+
+    if (!SECTION_MAPPING[sectionKey as SectionKey] || !sectionName.endsWith(':')) {
+      throw new Error(`Unknown section ${sectionKey}.`);
+    }
+    for (const entry of lines.slice(1)) {
+      if (entry === '') {
+        continue;
+      }
+      // if (!entryRegex.test(entry)) {
+      if (!entryRegex.test(entry.trim())) {
+        throw new Error(`Invalid entry ${entry} in section ${sectionKey}.`);
+      }
+    }
+  }
+}