Test deprecated rule modification

[shashank-elastic]

Issues

• [FR] Unit Test for Backporting Deprecated Rule Changes #2532
• https://github.com/elastic/ia-trade-team/issues/90

Summary

• Added a test case based on ideated solutions in the above issues
• To handle usecase when a new rule is deprecated , these are ignored by '--diff-filter=M' where only modified files are considered

Code Changes

• Checkout actions is pointed to latest V4, [code sanity to move to the latest version]
• The fetch-depth:0 is added for the code to be able to point origin/main in diff check. This was a crucial change without which even if we were able to fetch the commit hash of origin/main with method, we received errors like fatal: bad object 856c6c5a1f40916d30be65af5def6e17b1c1a188 when trying to perform diff.

Testing

Testing Complete : Modified deprecated rules are being identified sample_run

[shashank-elastic]

The local testing is failing approproatley

self = <tests.test_all_rules.TestDeprecatedRule testMethod=test_deprecated_rule_modified>

    def test_deprecated_rule_modified(self):
        """Test to ensure deprecated rules are not modified."""
    
        # Iterate over all the files in the deprecated rules directory
        for rule in self.deprecated_rules:
            # Use git diff to check if the file has changed
            result = subprocess.run(['git', 'diff', 'main', '--name-only', rule.path], stdout=subprocess.PIPE)
    
            # If the output is not empty, the file has changed
            if result.stdout:
>               self.fail(f"Deprecated rule {rule.path} has been modified.")
E               AssertionError: Deprecated rule /Users/shashankks/elastic_workspace/detection-rules/rules/_deprecated/command_and_control_connection_attempt_by_non_ssh_root_session.toml has been modified.

tests/test_all_rules.py:1302: AssertionError

Unsure why the Unit Test in workflow is passing, this is being checked.

[shashank-elastic]

The Workflow of Unit Test checkout on this commits

  HEAD is now at 8264255 Merge 157c8b5c8b311f8a96d6b32d1d12363d0e143c33 into 418a95205e551fb77dffcb5216b4d69048ba75c6
/usr/bin/git log -1 --format='%H'
'826425530a7f36261afd928399c56d3d65eceefd'

This has the latest changes for the test and the modified deprecated rules.

The unit test picks up the new test case test_deprecated_rules_modified

But it does not pick up the modified files changes, the argument and rule path is completely as expected. and this that path there should be this changed files as expected.

FAILED tests/test_all_rules.py::TestRuleMetadata::test_deprecated_rules_modified - AssertionError: No deprecated rules have been modified  with command args: ['/usr/bin/git', 'diff', '--diff-filter=M', 'origin/main', '--name-only', PosixPath('/home/runner/work/detection-rules/detection-rules/rules/_deprecated')]```

But locally the test asserts failures as expected.

>           self.fail(f"Deprecated rules {result.stdout} has been modified with command args: {result.args}")
E           AssertionError: Deprecated rules rules/_deprecated/apm_null_user_agent.toml
E           rules/_deprecated/command_and_control_connection_attempt_by_non_ssh_root_session.toml
E            has been modified with command args: ['/usr/bin/git', 'diff', '--diff-filter=M', 'origin/main', '--name-only', PosixPath('/Users/shashankks/elastic_workspace/detection-rules/rules/_deprecated')]

tests/test_all_rules.py:641: AssertionError

The PR will be for review since tests are passing locally, but the failures in the Github workflow is like wierd!

[eric-forte-elastic]

🟢 Manual review, looks good to me! 👍

[traut]

🔥