Update dotnet-main (#979)

* Sync capa rules submodule

* Sync capa-testfiles submodule

* Sync capa rules submodule

* changelog

* *: remove /x32 and /x64 flavors from number and offset features

* *: remove more references to /x32 and /x64

* linter: accept instruction scope

* rules: fix max operand index (4)

* API: better support A/W functions

* vverbose: show lib rule matches

* main: accept multiple paths to rules

* main: fix removal of default rules path

* lint: fix rules path

* changelog

* capa_as_library: fix rules path is list now

* main: better handle multiple rules paths

* main: bail if python 3.6 or below

closes #964

* ida: readme: remove python 3.6 support

* capa2yara: fix rules paths

* render: meta: display rule paths on separate lines

closes #971

* render: verbose: add doc

* verbose: make rule path multiline more concise

* vverbose: don't show examples in output

closes #970

* vverbose: render subscope name, like "basic block:"

closes #963

* build(deps-dev): bump pytest from 7.0.1 to 7.1.1

Bumps [pytest](https://github.com/pytest-dev/pytest) from 7.0.1 to 7.1.1.
- [Release notes](https://github.com/pytest-dev/pytest/releases)
- [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst)
- [Commits](pytest-dev/pytest@7.0.1...7.1.1)

---
updated-dependencies:
- dependency-name: pytest
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* ci: build: update pip and setuptools

* ci: build: bump pyinstall to v4.10

* Sync capa rules submodule

* Dotnet mixed mode detect (#969)

* feat: start dotnet detection (#955)

* feat: start dotnet detection

* Apply suggestions from code review

Co-authored-by: Willi Ballenthin <willi.ballenthin@gmail.com>

* refactor: dn instead of dotnet

* refactor: format branches, extractor reorg

* refactor: format selection and dotnet detect

* feat: get format, arch, os

* refactor: log errors and exceptions

* ci: also test and build for dotnet-main dev

* fix: import path

* fix: circular dep

* fix: remove buf argument
feat: get runtime meta data

* fix: log unsupported runtime error

* fix: type ignore

Co-authored-by: Willi Ballenthin <willi.ballenthin@gmail.com>

* fix: imports and add tests

* feat: detect mixed mode and tests

* feat: start dotnet detection (#955)

* feat: start dotnet detection

* Apply suggestions from code review

Co-authored-by: Willi Ballenthin <willi.ballenthin@gmail.com>

* refactor: dn instead of dotnet

* refactor: format branches, extractor reorg

* refactor: format selection and dotnet detect

* feat: get format, arch, os

* refactor: log errors and exceptions

* ci: also test and build for dotnet-main dev

* fix: import path

* fix: circular dep

* fix: remove buf argument
feat: get runtime meta data

* fix: log unsupported runtime error

* fix: type ignore

Co-authored-by: Willi Ballenthin <willi.ballenthin@gmail.com>

* fix: imports and add tests

Co-authored-by: Willi Ballenthin <willi.ballenthin@gmail.com>

* test: checkout submodules recursively

Co-authored-by: Capa Bot <capa-dev@mandiant.com>
Co-authored-by: Willi Ballenthin <willi.ballenthin@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

.github/workflows/build.yml

@@ -35,8 +35,10 @@ jobs:
           python-version: 3.8
       - if: matrix.os == 'ubuntu-18.04'
         run: sudo apt-get install -y libyaml-dev
+      - name: Upgrade pip, setuptools
+        run: pip install --upgrade pip setuptools
       - name: Install PyInstaller
-        run: pip install 'pyinstaller==4.2'
+        run: pip install 'pyinstaller==4.10'
       - name: Install capa
         run: pip install -e .
       - name: Build standalone executable

.github/workflows/tests.yml

@@ -48,7 +48,7 @@ jobs:
     - name: Checkout capa with submodules
       uses: actions/checkout@v2
       with:
-        submodules: true
+        submodules: recursive
     - name: Set up Python 3.8
       uses: actions/setup-python@v2
       with:
@@ -78,7 +78,7 @@ jobs:
     - name: Checkout capa with submodules
       uses: actions/checkout@v2
       with:
-        submodules: true
+        submodules: recursive
     - name: Set up Python ${{ matrix.python-version }}
       uses: actions/setup-python@v2
       with:

CHANGELOG.md

@@ -11,15 +11,17 @@
 
 ### Breaking Changes
 
-- instruction scope and operand feature are new and are not backwards compatible with older versions of capa
-- Python 3.7 is now the minimum supported Python version #866 @williballenthin
+  - instruction scope and operand feature are new and are not backwards compatible with older versions of capa
+  - remove /x32 and /x64 flavors of number and operand features #932 @williballenthin
+  - the tool now accepts multiple paths to rules, and JSON doc updated accordingly @williballenthin
 
-### New Rules (4)
+### New Rules (5)
 
 - data-manipulation/encryption/aes/manually-build-aes-constants huynh.t.nhan@gmail.com
 - nursery/get-process-image-filename michael.hunhoff@mandiant.com
 - compiler/v/compiled-with-v jakub.jozwiak@mandiant.com
 - compiler/zig/compiled-with-zig jakub.jozwiak@mandiant.com
+- anti-analysis/packer/huan/packed-with-huan jakub.jozwiak@mandiant.com
 -
 
 ### Bug Fixes

capa/features/common.py

@@ -98,33 +98,23 @@ def __nonzero__(self):
 
 
 class Feature(abc.ABC):
-    def __init__(self, value: Union[str, int, bytes], bitness=None, description=None):
+    def __init__(self, value: Union[str, int, bytes], description=None):
         """
         Args:
           value (any): the value of the feature, such as the number or string.
-          bitness (str): one of the VALID_BITNESS values, or None.
-            When None, then the feature applies to any bitness.
-            Modifies the feature name from `feature` to `feature/bitness`, like `offset/x32`.
           description (str): a human-readable description that explains the feature value.
         """
         super(Feature, self).__init__()
-
-        if bitness is not None:
-            if bitness not in VALID_BITNESS:
-                raise ValueError("bitness '%s' must be one of %s" % (bitness, VALID_BITNESS))
-            self.name = self.__class__.__name__.lower() + "/" + bitness
-        else:
-            self.name = self.__class__.__name__.lower()
+        self.name = self.__class__.__name__.lower()
 
         self.value = value
-        self.bitness = bitness
         self.description = description
 
     def __hash__(self):
-        return hash((self.name, self.value, self.bitness))
+        return hash((self.name, self.value))
 
     def __eq__(self, other):
-        return self.name == other.name and self.value == other.value and self.bitness == other.bitness
+        return self.name == other.name and self.value == other.value
 
     def get_value_str(self) -> str:
         """
@@ -153,10 +143,7 @@ def evaluate(self, ctx: Dict["Feature", Set[int]], **kwargs) -> Result:
         return Result(self in ctx, self, [], locations=ctx.get(self, []))
 
     def freeze_serialize(self):
-        if self.bitness is not None:
-            return (self.__class__.__name__, [self.value, {"bitness": self.bitness}])
-        else:
-            return (self.__class__.__name__, [self.value])
+        return (self.__class__.__name__, [self.value])
 
     @classmethod
     def freeze_deserialize(cls, args):
@@ -400,13 +387,6 @@ def freeze_deserialize(cls, args):
         return cls(*[codecs.decode(x, "hex") for x in args])
 
 
-# identifiers for supported bitness names that tweak a feature
-# for example, offset/x32
-BITNESS_X32 = "x32"
-BITNESS_X64 = "x64"
-VALID_BITNESS = (BITNESS_X32, BITNESS_X64)
-
-
 # other candidates here: https://docs.microsoft.com/en-us/windows/win32/debug/pe-format#machine-types
 ARCH_I386 = "i386"
 ARCH_AMD64 = "amd64"

capa/features/extractors/dnfile_.py

@@ -67,6 +67,9 @@ def get_base_address(self) -> int:
         return 0x0
 
     def get_entry_point(self) -> int:
+        # self.pe.net.Flags.CLT_NATIVE_ENTRYPOINT
+        #  True: native EP: Token
+        #  False: managed EP: RVA
         return self.pe.net.struct.EntryPointTokenOrRva
 
     def extract_global_features(self):
@@ -78,6 +81,9 @@ def extract_file_features(self):
     def is_dotnet_file(self) -> bool:
         return bool(self.pe.net)
 
+    def is_mixed_mode(self) -> bool:
+        return not bool(self.pe.net.Flags.CLR_ILONLY)
+
     def get_runtime_version(self) -> Tuple[int, int]:
         return self.pe.net.struct.MajorRuntimeVersion, self.pe.net.struct.MinorRuntimeVersion
 

capa/features/extractors/helpers.py

@@ -29,8 +29,7 @@ def is_aw_function(symbol: str) -> bool:
     if symbol[-1] not in ("A", "W"):
         return False
 
-    # second to last character should be lowercase letter
-    return "a" <= symbol[-2] <= "z" or "0" <= symbol[-2] <= "9"
+    return True
 
 
 def is_ordinal(symbol: str) -> bool:

capa/features/extractors/ida/insn.py

@@ -13,39 +13,13 @@
 import capa.features.extractors.helpers
 import capa.features.extractors.ida.helpers
 from capa.features.insn import API, Number, Offset, Mnemonic, OperandNumber, OperandOffset
-from capa.features.common import (
-    BITNESS_X32,
-    BITNESS_X64,
-    MAX_BYTES_FEATURE_SIZE,
-    THUNK_CHAIN_DEPTH_DELTA,
-    Bytes,
-    String,
-    Characteristic,
-)
+from capa.features.common import MAX_BYTES_FEATURE_SIZE, THUNK_CHAIN_DEPTH_DELTA, Bytes, String, Characteristic
 
 # security cookie checks may perform non-zeroing XORs, these are expected within a certain
 # byte range within the first and returning basic blocks, this helps to reduce FP features
 SECURITY_COOKIE_BYTES_DELTA = 0x40
 
 
-def get_bitness(ctx):
-    """
-    fetch the BITNESS_* constant for the currently open workspace.
-
-    via Tamir Bahar/@tmr232
-    https://reverseengineering.stackexchange.com/a/11398/17194
-    """
-    if "bitness" not in ctx:
-        info = idaapi.get_inf_structure()
-        if info.is_64bit():
-            ctx["bitness"] = BITNESS_X64
-        elif info.is_32bit():
-            ctx["bitness"] = BITNESS_X32
-        else:
-            raise ValueError("unexpected bitness")
-    return ctx["bitness"]
-
-
 def get_imports(ctx):
     if "imports_cache" not in ctx:
         ctx["imports_cache"] = capa.features.extractors.ida.helpers.get_file_imports()
@@ -159,7 +133,6 @@ def extract_insn_number_features(f, bb, insn):
             const = op.addr
 
         yield Number(const), insn.ea
-        yield Number(const, bitness=get_bitness(f.ctx)), insn.ea
         yield OperandNumber(i, const), insn.ea
 
 
@@ -234,7 +207,6 @@ def extract_insn_offset_features(f, bb, insn):
         op_off = capa.features.extractors.helpers.twos_complement(op_off, 32)
 
         yield Offset(op_off), insn.ea
-        yield Offset(op_off, bitness=get_bitness(f.ctx)), insn.ea
         yield OperandOffset(i, op_off), insn.ea
 
 

capa/features/extractors/smda/insn.py

@@ -6,15 +6,7 @@
 
 import capa.features.extractors.helpers
 from capa.features.insn import API, Number, Offset, Mnemonic
-from capa.features.common import (
-    BITNESS_X32,
-    BITNESS_X64,
-    MAX_BYTES_FEATURE_SIZE,
-    THUNK_CHAIN_DEPTH_DELTA,
-    Bytes,
-    String,
-    Characteristic,
-)
+from capa.features.common import MAX_BYTES_FEATURE_SIZE, THUNK_CHAIN_DEPTH_DELTA, Bytes, String, Characteristic
 
 # security cookie checks may perform non-zeroing XORs, these are expected within a certain
 # byte range within the first and returning basic blocks, this helps to reduce FP features
@@ -23,16 +15,6 @@
 PATTERN_SINGLENUM = re.compile(r"[+\-] (?P<num>[0-9])")
 
 
-def get_bitness(smda_report):
-    if smda_report.architecture == "intel":
-        if smda_report.bitness == 32:
-            return BITNESS_X32
-        elif smda_report.bitness == 64:
-            return BITNESS_X64
-    else:
-        raise NotImplementedError
-
-
 def extract_insn_api_features(f, bb, insn):
     """parse API features from the given instruction."""
     if insn.offset in f.apirefs:
@@ -89,7 +71,6 @@ def extract_insn_number_features(f, bb, insn):
             value = int(operand, 16) & ((1 << f.smda_report.bitness) - 1)
 
             yield Number(value), insn.offset
-            yield Number(value, bitness=get_bitness(f.smda_report)), insn.offset
         except:
             continue
 
@@ -232,7 +213,6 @@ def extract_insn_offset_features(f, bb, insn):
             number = int(number_int.group("num"))
             number = -1 * number if number_int.group().startswith("-") else number
         yield Offset(number), insn.offset
-        yield Offset(number, bitness=get_bitness(f.smda_report)), insn.offset
 
 
 def is_security_cookie(f, bb, insn):

capa/features/extractors/viv/insn.py

@@ -18,30 +18,14 @@
 import capa.features.extractors.helpers
 import capa.features.extractors.viv.helpers
 from capa.features.insn import API, Number, Offset, Mnemonic, OperandNumber, OperandOffset
-from capa.features.common import (
-    BITNESS_X32,
-    BITNESS_X64,
-    MAX_BYTES_FEATURE_SIZE,
-    THUNK_CHAIN_DEPTH_DELTA,
-    Bytes,
-    String,
-    Characteristic,
-)
+from capa.features.common import MAX_BYTES_FEATURE_SIZE, THUNK_CHAIN_DEPTH_DELTA, Bytes, String, Characteristic
 from capa.features.extractors.viv.indirect_calls import NotFoundError, resolve_indirect_call
 
 # security cookie checks may perform non-zeroing XORs, these are expected within a certain
 # byte range within the first and returning basic blocks, this helps to reduce FP features
 SECURITY_COOKIE_BYTES_DELTA = 0x40
 
 
-def get_bitness(vw):
-    bitness = vw.getMeta("Architecture")
-    if bitness == "i386":
-        return BITNESS_X32
-    elif bitness == "amd64":
-        return BITNESS_X64
-
-
 def interface_extract_instruction_XXX(f, bb, insn):
     """
     parse features from the given instruction.
@@ -553,7 +537,6 @@ def extract_op_number_features(f, bb, insn, i, oper):
         return
 
     yield Number(v), insn.va
-    yield Number(v, bitness=get_bitness(f.vw)), insn.va
     yield OperandNumber(i, v), insn.va
 
 
@@ -582,7 +565,6 @@ def extract_op_offset_features(f, bb, insn, i, oper):
         v = oper.disp
 
         yield Offset(v), insn.va
-        yield Offset(v, bitness=get_bitness(f.vw)), insn.va
         yield OperandOffset(i, v), insn.va
 
     # like: [esi + ecx + 16384]
@@ -594,7 +576,6 @@ def extract_op_offset_features(f, bb, insn, i, oper):
         v = oper.disp
 
         yield Offset(v), insn.va
-        yield Offset(v, bitness=get_bitness(f.vw)), insn.va
         yield OperandOffset(i, v), insn.va
 
 

capa/features/insn.py

@@ -22,16 +22,16 @@ def __init__(self, name: str, description=None):
 
 
 class Number(Feature):
-    def __init__(self, value: int, bitness=None, description=None):
-        super(Number, self).__init__(value, bitness=bitness, description=description)
+    def __init__(self, value: int, description=None):
+        super(Number, self).__init__(value, description=description)
 
     def get_value_str(self):
         return capa.render.utils.hex(self.value)
 
 
 class Offset(Feature):
-    def __init__(self, value: int, bitness=None, description=None):
-        super(Offset, self).__init__(value, bitness=bitness, description=description)
+    def __init__(self, value: int, description=None):
+        super(Offset, self).__init__(value, description=description)
 
     def get_value_str(self):
         return capa.render.utils.hex(self.value)
@@ -42,7 +42,11 @@ def __init__(self, value: str, description=None):
         super(Mnemonic, self).__init__(value, description=description)
 
 
-MAX_OPERAND_INDEX = 3
+# max number of operands to consider for a given instrucion.
+# since we only support Intel and .NET, we can assume this is 3
+# which covers cases up to e.g. "vinserti128 ymm0,ymm0,ymm5,1"
+MAX_OPERAND_COUNT = 4
+MAX_OPERAND_INDEX = MAX_OPERAND_COUNT - 1
 
 
 class _Operand(Feature, abc.ABC):
@@ -53,7 +57,7 @@ def __init__(self, index: int, value: int, description=None):
         self.index = index
 
     def __hash__(self):
-        return hash((self.name, self.value, self.bitness))
+        return hash((self.name, self.value))
 
     def __eq__(self, other):
         return super().__eq__(other) and self.index == other.index
@@ -64,7 +68,7 @@ def freeze_serialize(self):
 
 class OperandNumber(_Operand):
     # cached names so we don't do extra string formatting every ctor
-    NAMES = ["operand[%d].number" % i for i in range(MAX_OPERAND_INDEX)]
+    NAMES = ["operand[%d].number" % i for i in range(MAX_OPERAND_COUNT)]
 
     # operand[i].number: 0x12
     def __init__(self, index: int, value: int, description=None):
@@ -78,7 +82,7 @@ def get_value_str(self) -> str:
 
 class OperandOffset(_Operand):
     # cached names so we don't do extra string formatting every ctor
-    NAMES = ["operand[%d].offset" % i for i in range(MAX_OPERAND_INDEX)]
+    NAMES = ["operand[%d].offset" % i for i in range(MAX_OPERAND_COUNT)]
 
     # operand[i].offset: 0x12
     def __init__(self, index: int, value: int, description=None):

capa/ida/plugin/README.md

@@ -34,19 +34,18 @@ For more information on the FLARE team's open-source framework, capa, check out
 
 ### Requirements
 
-capa explorer supports Python versions >= 3.6.x and the following IDA Pro versions:
+capa explorer supports Python versions >= 3.7.x and the following IDA Pro versions:
 
 * IDA 7.4
 * IDA 7.5
 * IDA 7.6 (caveat below)
 * IDA 7.7
 
-capa explorer is however limited to the Python versions supported by your IDA installation (which may not include all Python versions >= 3.6.x). Based on our testing the following matrix shows the Python versions supported
+capa explorer is however limited to the Python versions supported by your IDA installation (which may not include all Python versions >= 3.7.x). Based on our testing the following matrix shows the Python versions supported
 by each supported IDA version:
 
 | | IDA 7.4 | IDA 7.5 | IDA 7.6 |
 | --- | --- | --- | --- |
-| Python 3.6.x | Yes | Yes | Yes |
 | Python 3.7.x | Yes | Yes | Yes |
 | Python 3.8.x | Partial (see below) | Yes | Yes |
 | Python 3.9.x | No | Partial (see below) | Yes |