[Bug] Update Prebuilt Detection Rules Release Process

[terrancedejesus]

Issues

• [Meta] Update Prebuilt Detection Rules Release Process #3374

Summary

This pull request fixes the Detection Rules release workflow and related CLI commands used. Below is a list of issues that were addressed and have more detailed information in the related issue.

• Automatic Locked Version Workflow Commit Hash Identification
• Incorrect use of branch version for determining previous release packages from EPR
• Add historical rules not default since the dropped support for 8.7 and previous
• Integrations PR labeling and description updates

Tasks

Preview Give feedback

1. Fix/Update Release Meta Template in TRADE Repository
2. Review release-fleet workflow for updates and bugs
3. Adjust fleet package building to add historical rules by default
4. Adjust packages.yml in 8.10 branch so kibana.version is 8.10.1
5. Update documentation for step to ask for backport branch in integrations if not available
6. Update bump-pkg-versions command to address schema failures

Testing

Testing will be a big part of these updates. The importance of testing it to ensure these important steps work as intended:

• Locked Versions workflow commit is properly obtained and used after checking out protected branches
• When bump-pkg-versions CLI command is used, packages.yml reflects the accurate results; only distinguishable when integrations PR is created as it will reflect in the manifest

Example run: https://github.com/elastic/detection-rules/actions/runs/7791122549

Release Workflow Test:

1. Detection Rules repo > Actions
2. Select release-fleet > Run workflow
3. Change "User workflow from" from main to bug-release-process-updates
4. Leave default options
5. Select "run workflow"
6. Monitor workflow progress
7. Under Build package and create PR to integrations job > Build package and create PR to integrations; ensure that it checks out a version lock commit hash as shown below

9. Note that the Build release package step is expected to fail. This is because we are checking out a commit hash that is PRIOR to this PR. As a result, when the package builds, it will incorrectly add release to the manifest which is fixed here.

[eric-forte-elastic]

NIt:

Since we are now defining registry_data = config['registry_data']

I think we can reuse it in places like the following to make;

docs = IntegrationSecurityDocsMDX(config['registry_data']['version'], Path(f'releases/{config["name"]}-docs'),
                                          True, historical_rules, package, note=update_message

 
 docs = IntegrationSecurityDocsMDX(registry_data['version'], Path(f'releases/{config["name"]}-docs'),
                                          True, historical_rules, package, note=update_message)
 

[terrancedejesus]

Since we are now defining registry_data = config['registry_data']. I think we can reuse it in places like the following to make;

Good point, updated.

[eric-forte-elastic]

Great peer review and demo/testing meeting. LGTM 👍