Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Security Solution] Tests for prebuilt rule upgrade workflow #202078

Open
2 of 8 tasks
Tracked by #174168
banderror opened this issue Nov 27, 2024 · 6 comments · Fixed by #203331 · May be fixed by #205217 or #205339
Open
2 of 8 tasks
Tracked by #174168

[Security Solution] Tests for prebuilt rule upgrade workflow #202078

banderror opened this issue Nov 27, 2024 · 6 comments · Fixed by #203331 · May be fixed by #205217 or #205339
Assignees
@maximpn @jpdjere
Labels
8.18 candidate Feature:Prebuilt Detection Rules Security Solution Prebuilt Detection Rules area Team:Detection Rule Management Security Detection Rule Management Team Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. test test-coverage issues & PRs for improving code test coverage test-plan v8.18.0

Comments

@banderror
Copy link
Contributor

banderror commented Nov 27, 2024 •
edited by maximpn
Loading

Epics: https://github.com/elastic/security-team/issues/1974 (internal), #174168

Summary

Write test plans (one or a few) and create a comprehensive suite of automated tests for the workflow of upgrading prebuilt rules.

Features to cover:

  • Reviewing rules for upgrading @jpdjere
  • Upgrading prebuilt rules one-by-one without preview (Rule Updates table) @jpdjere
  • Upgrading prebuilt rules in bulk (Rule Updates table) @jpdjere
  • Upgrading rule fields - per each rule field - via API and UI @jpdjere
    • name
    • description
    • etc - I feel that we need to cover literally every rule field
  • Upgrading prebuilt rules one-by-one with preview (Rule Upgrade flyout) @maximpn
    • Non-conflicts, solvable conflicts, non-solvable conflicts
    • Diffs and diff options
    • Suggesting merged versions
    • Editing and accepting changes to the Final version
    • Callouts
    • Validation
    • State persistence
    • Upgrading and saving changes
    • Anything else?
  • Specific and edge cases
    • Type changes on upgrade
    • Missing base versions
    • Race conditions and Optimistic Concurrency Control

Please cover both the features under the feature flag turned ON and OFF.

Related tickets

Related functional tickets to cover with tests:

Related bugs to cover with tests:

TODO
@banderror banderror added 8.18 candidate Feature:Prebuilt Detection Rules Security Solution Prebuilt Detection Rules area Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Team:Detection Rule Management Security Detection Rule Management Team Team:Detections and Resp Security Detection Response Team test test-coverage issues & PRs for improving code test coverage test-plan labels Nov 27, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

@elasticmachine
Copy link
Contributor

Pinging @elastic/security-detections-response (Team:Detections and Resp)

@elasticmachine
Copy link
Contributor

Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management)

@banderror banderror mentioned this issue Nov 27, 2024
Open
@banderror banderror changed the title [Security Solution] Tests for prebuilt rule upgrade workflow (DRAFT) [Security Solution] Tests for prebuilt rule upgrade workflow Nov 27, 2024
@banderror
Copy link
Contributor Author

@jpdjere @maximpn @nikitaindik

I'd suggest us to split the work on the upgrade test coverage between @maximpn and @jpdjere. Considering the fact that @maximpn and @nikitaindik are busy with finalizing the ThreeWayDiff UI and fixing bugs in it, maybe @jpdjere could address most of this ticket, especially because he is very familiar with the requirements.

Let's sync up on it when @jpdjere is back next week.

@banderror banderror mentioned this issue Nov 27, 2024
Open
5 tasks
@banderror
Copy link
Contributor Author

cc @pborgonovi

This was referenced Dec 10, 2024
Merged
Closed
@jpdjere jpdjere mentioned this issue Dec 18, 2024
Merged
kibanamachine pushed a commit to kibanamachine/kibana that referenced this issue Dec 19, 2024
@jpdjere
[Security Solution] Test Plan for Rule Updates workflow (elastic#203331)
4951af0 
Partially resolves: elastic#202078

## Summary

This PR refactors the test plans involving the Installation and Upgrade
Workflows:

- breaks down the gigantic `installation_and_upgrade.md` file into three
files:
- `installation.md` covering exclusively prebuilt rule install workflows
    - `upgrade.md` covering exclusively prebuilt rule upgrade workflows
- `package_installation_and_upgrade.md` which covers scenarios for the
installation and upgrades of the Prebuilt Rules package itself.

The *Upgrade of Prebuilt Rules* (`upgrade.md`) was modified as follows:

- expanded to cover scenarios detailed in the linked ticket (`Upgrading
prebuilt rules one-by-one without preview` and `Upgrading prebuilt rules
in bulk` in the Rule Updates table, including cases with and without
conflicts, as well as a mixture of both)
- internally refactored so that sections that will become legacy with
the release of Milestone 3 were pushed to the end and marked as pending
deletion once the feature is live.
- from point above, the test plan now is in a better position to add the
scenarios for the Three-Way-Diff upgrade workflow. cc @maximpn
@nikitaindik

### Checklist

Check the PR satisfies following conditions.

Reviewers should verify this PR satisfies this list as well.
- [ ] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)

---------

Co-authored-by: Maxim Palenov <maxim.palenov@elastic.co>
(cherry picked from commit cd74ecf)
@maximpn
Copy link
Contributor

maximpn commented Dec 19, 2024

The ticket was closed automatically after merging #203331. Reopening it since all scope is not covered yet.

@maximpn maximpn reopened this Dec 19, 2024
JoseLuisGJ pushed a commit to JoseLuisGJ/kibana that referenced this issue Dec 19, 2024
@jpdjere @maximpn @JoseLuisGJ
[Security Solution] Test Plan for Rule Updates workflow (elastic#203331)
4643f7e 
Partially resolves: elastic#202078

## Summary

This PR refactors the test plans involving the Installation and Upgrade
Workflows:

- breaks down the gigantic `installation_and_upgrade.md` file into three
files:
- `installation.md` covering exclusively prebuilt rule install workflows
    - `upgrade.md` covering exclusively prebuilt rule upgrade workflows 
- `package_installation_and_upgrade.md` which covers scenarios for the
installation and upgrades of the Prebuilt Rules package itself.

The *Upgrade of Prebuilt Rules* (`upgrade.md`) was modified as follows:

- expanded to cover scenarios detailed in the linked ticket (`Upgrading
prebuilt rules one-by-one without preview` and `Upgrading prebuilt rules
in bulk` in the Rule Updates table, including cases with and without
conflicts, as well as a mixture of both)
- internally refactored so that sections that will become legacy with
the release of Milestone 3 were pushed to the end and marked as pending
deletion once the feature is live.
- from point above, the test plan now is in a better position to add the
scenarios for the Three-Way-Diff upgrade workflow. cc @maximpn
@nikitaindik


### Checklist

Check the PR satisfies following conditions. 

Reviewers should verify this PR satisfies this list as well.
- [ ] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)

---------

Co-authored-by: Maxim Palenov <maxim.palenov@elastic.co>
@jpdjere jpdjere linked a pull request Dec 31, 2024 that will close this issue
Draft
2 tasks
@jpdjere jpdjere linked a pull request Dec 31, 2024 that will close this issue
Draft
2 tasks
stratoula pushed a commit to stratoula/kibana that referenced this issue Jan 2, 2025
@jpdjere @maximpn @stratoula
[Security Solution] Test Plan for Rule Updates workflow (elastic#203331)
3d370ad 
Partially resolves: elastic#202078

## Summary

This PR refactors the test plans involving the Installation and Upgrade
Workflows:

- breaks down the gigantic `installation_and_upgrade.md` file into three
files:
- `installation.md` covering exclusively prebuilt rule install workflows
    - `upgrade.md` covering exclusively prebuilt rule upgrade workflows 
- `package_installation_and_upgrade.md` which covers scenarios for the
installation and upgrades of the Prebuilt Rules package itself.

The *Upgrade of Prebuilt Rules* (`upgrade.md`) was modified as follows:

- expanded to cover scenarios detailed in the linked ticket (`Upgrading
prebuilt rules one-by-one without preview` and `Upgrading prebuilt rules
in bulk` in the Rule Updates table, including cases with and without
conflicts, as well as a mixture of both)
- internally refactored so that sections that will become legacy with
the release of Milestone 3 were pushed to the end and marked as pending
deletion once the feature is live.
- from point above, the test plan now is in a better position to add the
scenarios for the Three-Way-Diff upgrade workflow. cc @maximpn
@nikitaindik


### Checklist

Check the PR satisfies following conditions. 

Reviewers should verify this PR satisfies this list as well.
- [ ] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)

---------

Co-authored-by: Maxim Palenov <maxim.palenov@elastic.co>
benakansara pushed a commit to benakansara/kibana that referenced this issue Jan 2, 2025
@jpdjere @maximpn @benakansara
[Security Solution] Test Plan for Rule Updates workflow (elastic#203331)
4f7b289 
Partially resolves: elastic#202078

## Summary

This PR refactors the test plans involving the Installation and Upgrade
Workflows:

- breaks down the gigantic `installation_and_upgrade.md` file into three
files:
- `installation.md` covering exclusively prebuilt rule install workflows
    - `upgrade.md` covering exclusively prebuilt rule upgrade workflows 
- `package_installation_and_upgrade.md` which covers scenarios for the
installation and upgrades of the Prebuilt Rules package itself.

The *Upgrade of Prebuilt Rules* (`upgrade.md`) was modified as follows:

- expanded to cover scenarios detailed in the linked ticket (`Upgrading
prebuilt rules one-by-one without preview` and `Upgrading prebuilt rules
in bulk` in the Rule Updates table, including cases with and without
conflicts, as well as a mixture of both)
- internally refactored so that sections that will become legacy with
the release of Milestone 3 were pushed to the end and marked as pending
deletion once the feature is live.
- from point above, the test plan now is in a better position to add the
scenarios for the Three-Way-Diff upgrade workflow. cc @maximpn
@nikitaindik


### Checklist

Check the PR satisfies following conditions. 

Reviewers should verify this PR satisfies this list as well.
- [ ] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)

---------

Co-authored-by: Maxim Palenov <maxim.palenov@elastic.co>
maximpn pushed a commit that referenced this issue Jan 2, 2025
@jpdjere
[Security Solution] Enhance test coverage for Prebuilt Rules upgrade …
529f833 
…workflows (#204677)

Addresses: #202078

> NOTE: This PR depends on the test plans created in #203331. If any feedback leads to changes in those test plan, these PR will be updated.

## Summary

- Enhances tests on the Rule Upgrade Workflow from the Rule Updates table, as described in test plan created in: #203331

## Flaky test runner

https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7611
kibanamachine pushed a commit to kibanamachine/kibana that referenced this issue Jan 2, 2025
@jpdjere
[Security Solution] Enhance test coverage for Prebuilt Rules upgrade …
7267fab 
…workflows (elastic#204677)

Addresses: elastic#202078

> NOTE: This PR depends on the test plans created in elastic#203331. If any feedback leads to changes in those test plan, these PR will be updated.

## Summary

- Enhances tests on the Rule Upgrade Workflow from the Rule Updates table, as described in test plan created in: elastic#203331

## Flaky test runner

https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7611
(cherry picked from commit 529f833)
viduni94 pushed a commit to viduni94/kibana that referenced this issue Jan 2, 2025
@jpdjere @viduni94
[Security Solution] Enhance test coverage for Prebuilt Rules upgrade …
ba062a1 
…workflows (elastic#204677)

Addresses: elastic#202078

> NOTE: This PR depends on the test plans created in elastic#203331. If any feedback leads to changes in those test plan, these PR will be updated.

## Summary

- Enhances tests on the Rule Upgrade Workflow from the Rule Updates table, as described in test plan created in: elastic#203331

## Flaky test runner

https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7611
cqliu1 pushed a commit to cqliu1/kibana that referenced this issue Jan 2, 2025
@jpdjere @cqliu1
[Security Solution] Enhance test coverage for Prebuilt Rules upgrade …
9042da6 
…workflows (elastic#204677)

Addresses: elastic#202078

> NOTE: This PR depends on the test plans created in elastic#203331. If any feedback leads to changes in those test plan, these PR will be updated.

## Summary

- Enhances tests on the Rule Upgrade Workflow from the Rule Updates table, as described in test plan created in: elastic#203331

## Flaky test runner

https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7611
This was referenced Jan 6, 2025
Draft
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@maximpn maximpn

@jpdjere jpdjere

Labels
8.18 candidate Feature:Prebuilt Detection Rules Security Solution Prebuilt Detection Rules area Team:Detection Rule Management Security Detection Rule Management Team Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. test test-coverage issues & PRs for improving code test coverage test-plan v8.18.0
Projects
None yet
Milestone
No milestone
4 participants
@maximpn @jpdjere @banderror @elasticmachine