[Security Solution] Cypress back to live

[MadameSheema]

Summary

In this PR we are bringing back Cypress tests to live.

• We are removing signals indexes for each detection test
• We are reseting kibana index for every test
• We are re-enabling our tests in jenkins
• We are skipping EQL rules since is generating a lot of background errors
• We are skipping Threshold rules since is giving problems and can be the cause of the failings raised lately.

In a different branch I'm working in order to minimise even more the use of es_archive.

@rylnd lots of thanks for making this PR possible ❤️

[MadameSheema]

@elasticmachine merge upstream

[MadameSheema]

@elasticmachine merge upstream

[rylnd]

Added some cursory comments. Going to run this locally a bit and then circle back for approval.

[rylnd]

How are we deciding between before and beforeEach in these tests? Unless there's a huge performance cost my preference is always for beforeEach to minimize shared state between tests.

[MadameSheema]

I understand your point, but there are specific scenarios were there is not going to be shared state of the tests that can impact the behaviour and is going to save us execution time.

[rylnd]

What are the scenarios you're referring to, here? The use of cleanKibana in before only works if each test is allowed to complete its teardown:

• If a test fails after it's created a rule and deleteRule is part of the test itself, then the rule SO will be persisted to the next test
• If a test fails before it's created a rule and deleteRule is part of an after* hook, deleteRule will fail and cause cypress to skip the remaining tests

I personally would rather have a slow test than a flaky one. Performance and reliability are (with the exception of timeouts) mostly independent issues, and reliability is the focus at the moment. IMO we've also got many more options for improving performance/speed of these tests external to the tests themselves (more CI groups, beefier CI machines, etc.) such that I don't think we need to be sacrificing reliability for performance right now.

[rylnd]

If all our tests use cleanKibana in a beforeEach, I don't think we need the same in afterEach.

However, that does bring up a general question about teardown: what is the cypress suite's teardown responsibility as a whole? Is cypress relegated to its own CI group and so we don't have to worry about cleaning up for the "next" suite? Is there a possibility that tests will be added to the cypress group and we do need to start caring about this?

I'm thinking about the situations that have bit us in the past, namely incomplete teardown and shared state, which are strongly related. We should do whatever we can to minimize those and isolate tests.

[rylnd]

Why was this removed? Can we remove the package as well? It seems unused now.

[MadameSheema]

Because we are not using it. I didn't remove the package since now we are using a shared package.json and we are not the only ones using cypress.

[rylnd]

This means that we ignore any uncaught applications, right? This seems like a big change unless I'm misunderstanding. Were we seeing false negatives?

[MadameSheema]

Without that line, cypress fails each time that an error is raised in the console log. Do we want this behaviour to be caught? Because right now there are some tests failing because of that although the UI looks fine.

[rylnd]

Got it. It would be nice to log the error somewhere rather than discard it, but that might be too noisy for CI logs 🤷‍♂️ ? I defer to you.

[rylnd]

I think some wires got crossed on the afterEach thing; I think those should (need to?) go back into after* hooks. Sorry about that!

Also see my idea about meta-programming most of the test setup/teardown for specific use cases; I think that could clean up a lot of complexity here.

[rylnd]

I think these should go back into the afterEach. I had mentioned offline that, due to cypress skipping the suite if the test fails, we should move any test-specific teardown (e.g. deleteRule) into the test itself to avoid unnecessary skips, but I don't think that these unloads fall in that category.

[rylnd]

This is a bit of a tangent, but I just had a couple thoughts:

• Keeping test setup/teardown symmetrical is a PITA
  • reorienting this to do teardown, then setup all within setup would reduce this complexity
    • This would address/sidestep the issue of test failures, which can currently cause teardown not to execute
    • We could lessen developer burden by defining this as a general function(s)

// in test file
describe('my test', () => {
  detectionEngineHooks();
  
  it('does a thing', () => {});
});

// in helper file
const detectionEngineHooks = () => {
  beforeEach(() => {
    cleanKibana(); // deletes `.kibana*`, loads `empty_kibana`
    cleanSignals(); // deletes `.siem-signals*`, hits detections endpoint to create index
    cleanLoad('alerts'); // see below
  });
};

const cleanLoad = (archive) => {
    unload(archive); // might need to try/catch this?
    load(archive);
}

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: 38b05b0

Metrics [docs]

Distributable file count

id	before	after	diff
default	47288	48048	+760

History

• 💔 Build #95077 failed f255c50
• 💔 Build #94975 failed a886917
• 💔 Build #94952 failed 295bc61
• 💚 Build #94868 succeeded 7b519e1
• 💚 Build #94849 succeeded 2ef701a

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[rylnd]

Thank you for taking the time to dig into these! We've got some fun ideas to play with in the near future here, but let's merge this and keep things rolling. Upwards and onwards!