[Security][Serverless] FTR API Integration tests - Refactoring - Issue fixing #182245

dkirchan · 2024-05-01T13:21:18Z

Summary

This PR is addressing the following issues:

The pipelines defined in .buildkite/pipeline-resource-definitions/security-solution-quality-gate/ folder were skipping intermediate builds. We need to be able to run more than one build in the same time for these pipelines.
As part of the refactoring / optimization of the .buildkite/scripts/pipelines/security_solution_quality_gate/api_integration/api-integration-tests.sh script, it now executes a TS script in order to handle the projects for serverless and execute the yarn script provided.
- As part of this refactoring, the methods and worfklow defined in the x-pack/plugins/security_solution/scripts/run_cypress/parallel_serverless.ts is now followed in order to reduce code duplication and maintenance.
Fixed an issue in x-pack/test/security_solution_api_integration/scripts/index.js. This issue was causing false green test executions in buildkite. The exit code was not actually returned from the child process so the exit code of this script was 0, even though the child process (test execution) was failing giving back an exit code 1.
Parameterized .buildkite/pipelines/security_solution/api_integration.yml to be running the correct test suite (release or periodic) depending on whether the environment variable QUALITY_GATE=1 is passed or not.

The last bullet was misleading the test results interpretation, reading as successful test runtime scripts which had one or more test failures. E.g: Buildkite Test Execution being green with failing tests.

apmmachine · 2024-05-01T13:21:37Z

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

/oblt-deploy : Deploy a Kibana instance using the Observability test environments.
run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2024-05-01T13:24:36Z

Pinging @elastic/security-solution (Team: SecuritySolution)

…-fix'

yctercero

Did cursory review of the code. Diamantis walked me through the relevant changes on Zoom. Action item on our end after this is merged will be to investigate any failing tests owned by our team that have falsely been reporting as passing.

Thank you, Diamantis!

…-fix'

dmlemeshko

on behalf of appex-qa I left a question about project deletion

dmlemeshko · 2024-05-02T15:05:30Z

...ldkite/scripts/pipelines/security_solution_quality_gate/api_integration/api_ftr_execution.ts

+      const statusCode = await executeCommand(command, envVars, workDir);
+
+      // Delete serverless project
+      log.info(`${id} : Deleting project ${PROJECT_NAME}...`);
+      await cloudHandler.deleteSecurityProject(project.id, PROJECT_NAME);
+      process.exit(statusCode);


question: could it happen that the promise is unexpectedly rejected and error will be throw in line 142. In this case project deletion won't be executed.

Actually waitForEsStatusGreen and other async operations that can throw error on timeout might unexpectedly stop the execution and leave project in cloud. Maybe it makes sense to wrap everything between project ready check and tests run completion in try block and do cleanup in finally?

@dmlemeshko Thank you a lot for this. I think that this is a great catch here. I have a handler which automatically deletes all the errant projects after 6 hours in case a project is not deleted properly.

However it is really valuable to be added. I did the change, even though I would refactor it a bit better but for now I feel that we can proceed like this till I structure it more properly. Do you agree?

Totally! Great work.

banderror

Rule Management-related changes LGTM, thank you @dkirchan!

If I'm not mistaken, the tests our team owns shouldn't be affected because we haven't yet enabled any of them in the 2nd quality gate.

banderror · 2024-05-02T20:23:41Z

x-pack/test/security_solution_api_integration/package.json

@@ -223,7 +223,7 @@
    "prebuilt_rules_large_prebuilt_rules_package:server:serverless": "npm run initialize-server:rm prebuilt_rules/large_prebuilt_rules_package serverless",
    "prebuilt_rules_large_prebuilt_rules_package:runner:serverless": "npm run run-tests:rm prebuilt_rules/large_prebuilt_rules_package serverless serverlessEnv",
    "prebuilt_rules_large_prebuilt_rules_package:qa:serverless": "npm run run-tests:rm prebuilt_rules/large_prebuilt_rules_package serverless qaPeriodicEnv",
-    "prebuilt_rules_large_prebuilt_rules_package:qa:serverles:release": "npm run run-tests:rm prebuilt_rules/large_prebuilt_rules_package serverlessQA qaEnv",
+    "prebuilt_rules_large_prebuilt_rules_package:qa:serverless:release": "npm run run-tests:rm prebuilt_rules/large_prebuilt_rules_package serverlessQA qaEnv",


oatkiller

Thanks @dkirchan!

oatkiller · 2024-05-02T19:56:02Z

...ldkite/scripts/pipelines/security_solution_quality_gate/api_integration/api_ftr_execution.ts

+const PROJECT_NAME_PREFIX = 'kibana-ftr-api-integration-security-solution';
+
+// Function to execute a command and return a Promise with the status code
+function executeCommand(command: string, envVars: any, workDir: string): Promise<number> {


👋 Thanks for the PR.

Suggestion: maybe this could be helpful?

const util = require('node:util'); const exec = util.promisify(require('node:child_process').exec); async function lsExample() { const { stdout, stderr } = await exec('ls'); console.log('stdout:', stdout); console.error('stderr:', stderr); } lsExample();

This was copied from the node docs (and the imports are probably different in our environment.) But with a helper like this, we might be able to avoid this custome executeCommand wrapper.

Looks great. I ll close this PR in order to unblock things and I will raise a new one with this change!

oatkiller · 2024-05-02T19:58:40Z

...ldkite/scripts/pipelines/security_solution_quality_gate/api_integration/api_ftr_execution.ts

+export const cli = () => {
+  run(
+    async (context) => {
+      const log = new ToolingLog({


Suggestion: the api-integration-tests.sh file has this at the beginning:

#!/bin/bash if [ -z "$1" ] then echo "No target script from the package.json file, is supplied" exit 1 fi

Perhaps this script could also exit/fail with a message if TARGET_SCRIPT isn't defined.

This is exactly what $1 is. The Target script.

kibana-ci · 2024-05-03T09:13:58Z

💚 Build Succeeded

Buildkite Build
Commit: 636494d

Metrics [docs]

✅ unchanged

History

💔 Build #207690 failed 8374d6d
💛 Build #207574 was flaky b42ae7c
💚 Build #207573 succeeded 395f0af
💚 Build #207417 succeeded 7855a2e
💛 Build #207311 was flaky e8f1ca9

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @dkirchan

## Summary [Try catch statement](https://github.com/elastic/kibana/blob/9473241cf8f02eed535c846546114ee94b1ab5fc/.buildkite/scripts/pipelines/security_solution_quality_gate/api_integration/api_ftr_execution.ts#L144) in the PR for the [API Ftr Integration tests](#182245), introduced a cover above the target fix of the specific PR. The tests were failing however the exit code was still 0 so the failures are hidden. This PR addresses the specific issue.

dkirchan added 8 commits April 30, 2024 17:46

WIP: curl to TS

8169331

Fixed intermediate builds skip for security quality gate in kibana

f59b6ea

Running target script

c1a473b

Fixed product type issue

b727546

Replaced logger with console.log for command execution

b6626e7

Removed sensitive data

9e1070c

Fixed exist status code

b5d25db

Fixed clean code

81cf52b

dkirchan requested review from MadameSheema, patrykkopycinski, oatkiller, maximpn, banderror and a team as code owners May 1, 2024 13:21

Restored previous state of cli.ts

e9d7a5a

dkirchan force-pushed the security-api-integrations branch from 0ebee95 to e9d7a5a Compare May 1, 2024 13:22

Merge branch 'main' into security-api-integrations

2f798f3

dkirchan self-assigned this May 1, 2024

dkirchan requested a review from a team May 1, 2024 13:23

dkirchan changed the title ~~Security api integrations~~ [Security][Serverless] FTR API Integration tests - Refactoring - Issue fixing May 1, 2024

dkirchan added the Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. label May 1, 2024

dkirchan added release_note:skip Skip the PR/issue when compiling release notes v8.15.0 labels May 1, 2024

dkirchan force-pushed the security-api-integrations branch 2 times, most recently from 62c548e to d93cdb4 Compare May 1, 2024 13:50

charlie-pichette approved these changes May 1, 2024

View reviewed changes

Attempt to parameterize quality gate for FTR api integration tests

447b12a

dkirchan force-pushed the security-api-integrations branch from d93cdb4 to 447b12a Compare May 1, 2024 14:26

dkirchan and others added 2 commits May 1, 2024 17:39

Changed the flag from QUALITY_GATE to KIBANA_MKI_QUALITY_GATE

9ad2296

[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -…

e3f6833

…-fix'

yctercero approved these changes May 1, 2024

View reviewed changes

dkirchan and others added 3 commits May 1, 2024 20:05

Fixed lint issue

c20d857

Merge branch 'main' into security-api-integrations

f56543f

[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -…

e8f1ca9

…-fix'

Ikuni17 approved these changes May 1, 2024

View reviewed changes

Merge branch 'main' into security-api-integrations

7855a2e

dmlemeshko reviewed May 2, 2024

View reviewed changes

dkirchan and others added 2 commits May 2, 2024 18:47

Fixed comment for cloud project deletion

395f0af

Merge branch 'main' into security-api-integrations

b42ae7c

dmlemeshko approved these changes May 2, 2024

View reviewed changes

banderror approved these changes May 2, 2024

View reviewed changes

Merge branch 'main' into security-api-integrations

8374d6d

dkirchan enabled auto-merge (squash) May 2, 2024 21:00

oatkiller approved these changes May 2, 2024

View reviewed changes

dkirchan disabled auto-merge May 2, 2024 21:38

Merge branch 'main' into security-api-integrations

636494d

dkirchan merged commit 75c7f11 into main May 3, 2024
35 checks passed

dkirchan deleted the security-api-integrations branch May 3, 2024 11:50

kibanamachine added the backport:skip This commit does not require backporting label May 3, 2024

dkirchan mentioned this pull request May 4, 2024

[Security] Ftr API Integration tests #182626

Merged

mgiota mentioned this pull request May 27, 2024

[Observability solution] [SLO] Run burn rate api tests in serverless & ess using mocha tagging #183113

Closed

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Security][Serverless] FTR API Integration tests - Refactoring - Issue fixing #182245

[Security][Serverless] FTR API Integration tests - Refactoring - Issue fixing #182245

dkirchan commented May 1, 2024 •

edited

Loading

apmmachine commented May 1, 2024

elasticmachine commented May 1, 2024

yctercero left a comment

dmlemeshko left a comment

dmlemeshko May 2, 2024

dmlemeshko May 2, 2024

dkirchan May 2, 2024

dmlemeshko May 2, 2024

banderror left a comment

banderror May 2, 2024

oatkiller left a comment

oatkiller May 2, 2024

dkirchan May 3, 2024

oatkiller May 2, 2024

dkirchan May 2, 2024

kibana-ci commented May 3, 2024

[Security][Serverless] FTR API Integration tests - Refactoring - Issue fixing #182245

[Security][Serverless] FTR API Integration tests - Refactoring - Issue fixing #182245

Conversation

dkirchan commented May 1, 2024 • edited Loading

Summary

apmmachine commented May 1, 2024

🤖 GitHub comments

elasticmachine commented May 1, 2024

yctercero left a comment

Choose a reason for hiding this comment

dmlemeshko left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

banderror left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oatkiller left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kibana-ci commented May 3, 2024

💚 Build Succeeded

Metrics [docs]

History

dkirchan commented May 1, 2024 •

edited

Loading