[Security Solution] fix flaky endpoint ftr tests

[joeypoon]

Summary

Fixes flaky endpoint FTR tests.

• restrict CheckMetadataTransformsTask from running during tests so that it doesn't interfere with expected test transform setups will revisit in separate PR (see comment)
• retry transform start during transform install

Flaky test runs:

• https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/1964 (200x ✅ )
• https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/1971 (200x ✅ )

Fixes: #72874

For maintainers

• This was checked for breaking API changes and was labeled appropriately

[elasticmachine]

Pinging @elastic/security-defend-workflows (Team:Defend Workflows)

[pzl]

looks good, would like to see one thing cleaned up

[pzl]

End goal is good here, we don't need the watchdog running in the test environment (esp. if there are tests verifying watchdog behavior).

But good practice is to keep all knowledge of tests and test environments outside of the "production side" of the code. Probably a few ways to achieve this, but see if you can reorganize so you can control this from a .test.ts file.

Could either try to hook the plugin start earlier/somewhere else and overwrite/inject the checkMetadataTransformTask to be a mock or no-op so this code block still runs and does nothing.

Or if thats not feasible, could try to make a global or imported variable you can replace this check with like

if (plugins.taskManager && RUN_WATCHDOG)

and by default thats just set to RUN_WATCHDOG=true somewhere in production code. And from the test side, give yourself the ability to mock the import or otherwise set to false.

[joeypoon]

great point. since this is blocking all of our FTR tests and the retry logic alone is sufficient for restoring our tests, I'll remove this logic from this PR and work on it as a separate PR.

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[juliaElastic]

Fleet change LGTM

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: 930784e

Metrics [docs]

Unknown metric groups

ESLint disabled line counts

id	before	after	diff
securitySolution	428	430	+2

Total ESLint disabled count

id	before	after	diff
securitySolution	506	508	+2

History

• 💚 Build #110930 succeeded cfaa11ce0f1825247da03d4967a14f8383f43708
• 💔 Build #110894 failed c8dfa91b0e7bff3627f7cba88689bd947b69a6bf
• 💚 Build #110873 succeeded 7900af52a863c84e9eab52c87a3257d2c92a8c63
• 💛 Build #110760 was flaky 48454eb1f84bca12ffc9e04962489c7e474e05ae
• 💔 Build #110729 failed c9c229e0d2c7aee17a0351ee4003ba4d8b1d9789
• 💔 Build #110474 failed 8a94b6561e795d6c6659f7f615644f5a08b7f44a

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream