Add a script for upgrading rules into the new dynamic format

[yelhamer]

At the time of the opening of this PR, the script provides the main logic for parsing the rules. This logic is composed of 3 recursive functions that each handle a specific task, and that call into each other. I settled on this approach after trying other things (such as the strategy mentioned in #1674) and failing/getting stuff too complicated.

There remains the following important tasks, but they should be easy to add for the most part:

• merge the static and dynamic statements generated at the initial node (rn, it returns them separately)
• add all supported dynamic features into the script
• add the logic for saving the generated rules
• add static to dynamic scope translation (right now it assumes the "thread" scope for all)
• sort rules by dependency, and exclude match features that depend on out-of-context rules
• optimize and clean the code

Checklist

• No CHANGELOG update needed

• No new tests needed

• No documentation update needed

[williballenthin]

when you're ready for more review, would you please also provide some example output, such as the git diff for some upgraded rules? this will help us assess the logic of the script.

[mr-tz]

agree that some example output and results would be helpful

[yelhamer]

The script's logic is finished I believe, so feel free to review it. All that remains is to add support for filtering-out match rules that point to static-only rules when appropriate, as well as adding the final list of supported dynamic features (perhaps we want to add some characteristics?)

Here's the script's result on a made up complex rule:

original rule:

rule:
  meta:
    name: test rule
    namespace: test
    authors:
      - test@aol.com
    scope: function
    att&ck:
      - Persistence::Event Triggered Execution::Unix Shell Configuration Modification
        [T1546.004]
    examples:
      - 7351f8a40c5450557b24622417fc478d:0x407D11
  features:
    - and:
      - basic block:
        - or:
          - os: linux
          - match: host-interaction/file-system/write
          - or:
            - mnemonic: call
            - api: someapi_1
          - instruction:
            - mnemonic: write
            - string: somestring_1
      - instruction:
        - api: someapi_2
        - number: 1
      - string: somestring_2

modified rule:

rule:
  meta:
    name: test rule
    namespace: test
    authors:
      - test@aol.com
    scopes:
      static: function
      dynamic: process
    att&ck:
      - Persistence::Event Triggered Execution::Unix Shell Configuration Modification
        [T1546.004]
    examples:
      - 7351f8a40c5450557b24622417fc478d:0x407D11
  features:
    - and:
      - or:
        - basic block:
          - or:
            - os: linux
            - match: host-interaction/file-system/write
            - or:
              - mnemonic: call
              - api: someapi_1
            - instruction:
              - mnemonic: write
              - string: somestring_1
        - thread:
          - or:
            - os: linux
            - match: host-interaction/file-system/write
            - or:
              - api: someapi_1
      - or:
        - instruction:
          - api: someapi_2
          - number: 1
        - call:
          - and:
            - api: someapi_2
            - number: 1
      - string: somestring_2

[mr-tz]

• offset features get removed
  

edit: @yelhamer this seems to still be the case

• same for property (property/read)

[mr-tz]

• comments and quotes get removed
  

• also note the escaping sequences
  

[mr-tz]

• number exported as decimal
  

[mr-tz]

• new line inserted
  

[mr-tz]

• mnemonic/number removed
  

[williballenthin]

good finds @mr-tz

these might be tricky to address, since the yaml parser probably doesn't preserve things like the hex/decimal number format once the data is parsed.

worst case, we could do something like:

1. if its easy to just update scope -> scopes, then do a str.replace(..., ...) on the raw text and emit it, otherwise,
2. print to the user that manual intervention is required, possibly setting scopes.dynamic: TODO (which is invalid and will force us to make the fix)

[mr-tz]

See capa\scripts\capafmt.py for potential hints on how to handle the formatting.

[mr-tz]

• may be a good opportunity to convert all rules to use Unix line endings (\n only)

[yelhamer]

@mr-tz regarding the removal of mnemonic and offset, isn't that desirable? since instructions aren't extracted by sandbox output (to my understanding)

[mr-tz]

For the static rule part mnemonic and offset should still be there.

[yelhamer]

Ah yeah I see. Will work on it...

[yelhamer]

TODO(yelhamer):
once this PR is merged, update #1709 in order to use the default ruleset.

[mr-tz]

Currently, getting capa.rules.InvalidRule: invalid rule: subscope must have exactly one child statement on the rule with name: inject pe.

[yelhamer]

@mr-tz could you please take a look at the rules now for any other issues?

what remains is preserving the double backwards slashes (important), and the comments.

[mr-tz]

See inline comments in mandiant/capa-rules#814
Nice progress, there's some code style issues to address here as well.

[mr-tz]

I think this can be closed?