triage binary ninja backend failures

[williballenthin]

from capa-testfiles, these files fail to process today:

failures for binja:                                                                                                                                                         
  - 0761142efbda6c4b1e801223de723578.dll_                                                                                                                                   
  - 1038a23daad86042c66bfe6c9d052d27048de9653bde5750dc0f240c792d9ac8.elf_                                                                                                   
  - 112f9f0e8d349858a80dd8c14190e620.exe_                                                                                                                                   
  - 253309d8b3675d3cc61d4bf23aa15d4b.dll_                                                                                                                                   
  - 2dae11cc5f86f5399b560b8837c26274b7e09431deed669b0844fef44e917915.exe_                                                                                                   
  - 2f7f5fb5de175e770d7eae87666f9831.elf_                                                                                                                                   
  - 3b13b6f1d7cd14dc4a097a12e2e505c0a4cff495262261e2bfc991df238b9b04.dll_                                                                                                   
  - 4122acca2f9ea98fc3f3ad040688e4ce.exe_                                                                                                                                   
  - 44d40faf3f1fe4ed969befab7afcd2f0.exe_                                                                                                                                   
  - 49a34cfbeed733c24392c9217ef46bb6.exe_                                                                                                                                   
  - 54390bda109aab7fc006b8b4ead5b6c2.dll_                                                                                                                                   
  - 55d77ab16377a8a314982f723fcc6fae.exe_                                                                                                                                   
  - 5f66b82558ca92e54e77f216ef4c066c.exe_                                                                                                                                   
  - 5fbbfeed28b258c42e0cfeb16718b31c.exe_                                                                                                                                   
  - 648fc498110b11b4313a47a776e6ba40.exe_                                                                                                                                   
  - 6cc148363200798a12091b97a17181a1.exe_                                                                                                                                   
  - 7f15b1a47bbe031334e23653879e9661f4b8cde80c307548328fdd3aed87ca46.exe_                                                                                                   
  - 91a12a4cf437589ba70b1687f5acad19.exe_                                                                                                                                   
  - 92d8ea10ea30e8b534334a1c9857a455.exe_
  - 94d3c854aadbcfde46b2f82801015c31.exe_
  - 971e599e6e707349eccea2fd4c8e5f67.exe_
  - 9b7ccaa2ae6a5b96e3110ebcbc4311f6.dll_
  - 9ff8e68343cc29c1036650fc153e69f7.exe_
  - a74ee8200aace7d19dee79871bbf2ed3.exe_
  - a933a1a402775cfa94b6bee0963f4b46.dll_
  - b5f0524e69b3a3cf636c7ac366ca57bf5e3a8fdc8a9f01caf196c611a7918a87.elf_
  - c3341b7dfbb9d43bca8c812e07b4299f.exe_
  - e87076a1182ba40758e1d7258442c1ee23bf71ac77fad9f3babc707dce11c144.exe_
  - eaad7dfc78304b977d3844cc63577152.exe_
  - ee3b869b668abec332d07c66d1a39f6dbf3a598cc1325b57a0504f8d24ac2e28.dll_
  - kernel32.dll_
  - mimikatz.exe_

[xusheng6]

I tested 112f9f0e8d349858a80dd8c14190e620.exe_ and it seems to be just slow:

I will look into other samples when possible as well

[xusheng6]

Well I see it crashes now:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/Users/xusheng/capa/capa/main.py", line 1103, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/xusheng/capa/capa/main.py", line 994, in main
    capabilities, counts = find_capabilities(rules, extractor, disable_progress=args.quiet)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/xusheng/capa/capa/capabilities/common.py", line 75, in find_capabilities
    return find_static_capabilities(ruleset, extractor, disable_progress=disable_progress, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/xusheng/capa/capa/capabilities/static.py", line 168, in find_static_capabilities
    function_matches, bb_matches, insn_matches, feature_count = find_code_capabilities(ruleset, extractor, f)
                                                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/xusheng/capa/capa/capabilities/static.py", line 114, in find_code_capabilities
    for bb in extractor.get_basic_blocks(fh):
  File "/Users/xusheng/capa/capa/features/extractors/binja/extractor.py", line 58, in get_basic_blocks
    for mlil_bb in f.mlil.basic_blocks:
                   ^^^^^^
  File "/Applications/Binary Ninja.app/Contents/Resources/python/binaryninja/function.py", line 1039, in mlil
    raise ILException(f"Medium level IL was not loaded for {self!r}")
binaryninja.exceptions.ILException: Medium level IL was not loaded for <func: x86@0x467464>

This is quite similar to #2249

[xusheng6]

With the fix 9dd8b7a, it is no longer crashing due to the IL not being loaded. However, it gets to a different crash:

lot of lines omitted
......
DEBUG    capa.capabilities.static: analyzed function 0x7a83b0 and extracted 13 features, 0 matches in 0.01s                                                                                                                                                                                                                   static.py:179
DEBUG    capa.capabilities.static: analyzed function 0x7a83c0 and extracted 13 features, 0 matches in 0.00s                                                                                                                                                                                                                   static.py:179
DEBUG    capa.capabilities.static: analyzed function 0x7a83d0 and extracted 27 features, 0 matches in 0.01s                                                                                                                                                                                                                   static.py:179
DEBUG    capa.capabilities.static: analyzed function 0x7a8401 and extracted 10 features, 0 matches in 0.00s                                                                                                                                                                                                                   static.py:179
DEBUG    capa.capabilities.common: analyzed file and extracted 96040 features                                                                                                                                                                                                                                                  common.py:35
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/Users/xusheng/capa/capa/main.py", line 1101, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/xusheng/capa/capa/main.py", line 995, in main
    meta.analysis.layout = capa.loader.compute_layout(rules, extractor, capabilities)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/xusheng/capa/capa/loader.py", line 663, in compute_layout
    return compute_static_layout(rules, extractor, capabilities)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/xusheng/capa/capa/loader.py", line 641, in compute_static_layout
    assert addr in functions_by_bb
           ^^^^^^^^^^^^^^^^^^^^^^^
AssertionError

[xusheng6]

@williballenthin I tried a few other files, and it seems capa runs fine on it with binja backend. I am curious when you say "fail to process", does it just crash like for 112f9f0e8d349858a80dd8c14190e620.exe_, or it fails to pass certain unit tests?

[xusheng6]

The "IL cannot be loaded" also happens with b5f0524e69b3a3cf636c7ac366ca57bf5e3a8fdc8a9f01caf196c611a7918a87.elf_ at 0x8091b80

[xusheng6]

Weird enough, if I add --restrict-to-functions 0x8082d40 to restrict the analysis to the offending function, then everything is fine

[xusheng6]

I am triaging these and will update the status here. Please be aware the test is down on top of the recent fixes #2500 .

Sample	Status
0761142efbda6c4b1e801223de723578.dll_	✅
1038a23daad86042c66bfe6c9d052d27048de9653bde5750dc0f240c792d9ac8.elf_	✅
112f9f0e8d349858a80dd8c14190e620.exe_	✅
253309d8b3675d3cc61d4bf23aa15d4b.dll_	✅
2dae11cc5f86f5399b560b8837c26274b7e09431deed669b0844fef44e917915.exe_	✅
2f7f5fb5de175e770d7eae87666f9831.elf_	Crash like #2406 (comment)
3b13b6f1d7cd14dc4a097a12e2e505c0a4cff495262261e2bfc991df238b9b04.dll_	✅
4122acca2f9ea98fc3f3ad040688e4ce.exe_	packed
44d40faf3f1fe4ed969befab7afcd2f0.exe_	✅
49a34cfbeed733c24392c9217ef46bb6.exe_	✅
54390bda109aab7fc006b8b4ead5b6c2.dll_	✅
55d77ab16377a8a314982f723fcc6fae.exe_	AuotIt
5f66b82558ca92e54e77f216ef4c066c.exe_	✅
5fbbfeed28b258c42e0cfeb16718b31c.exe_	✅
648fc498110b11b4313a47a776e6ba40.exe_	Crash like #2406 (comment)
6cc148363200798a12091b97a17181a1.exe_	✅
7f15b1a47bbe031334e23653879e9661f4b8cde80c307548328fdd3aed87ca46.exe_	Crash like #2406 (comment)
91a12a4cf437589ba70b1687f5acad19.exe_	packed
92d8ea10ea30e8b534334a1c9857a455.exe_	AutoHotKey
94d3c854aadbcfde46b2f82801015c31.exe_	hits a binja bug Vector35/binaryninja-api#6191, but capa finishes fine
971e599e6e707349eccea2fd4c8e5f67.exe_	packed
9b7ccaa2ae6a5b96e3110ebcbc4311f6.dll_	Crash like #2406 (comment)
9ff8e68343cc29c1036650fc153e69f7.exe_	✅
a74ee8200aace7d19dee79871bbf2ed3.exe_	✅
a933a1a402775cfa94b6bee0963f4b46.dll_	✅
b5f0524e69b3a3cf636c7ac366ca57bf5e3a8fdc8a9f01caf196c611a7918a87.elf_	Crash like #2406 (comment)
c3341b7dfbb9d43bca8c812e07b4299f.exe_	Crash like #2406 (comment)
e87076a1182ba40758e1d7258442c1ee23bf71ac77fad9f3babc707dce11c144.exe_	✅
eaad7dfc78304b977d3844cc63577152.exe_	✅
ee3b869b668abec332d07c66d1a39f6dbf3a598cc1325b57a0504f8d24ac2e28.dll_	✅
kernel32.dll_	the initial binja analysis runs longer than expected (Vector35/binaryninja-api#6177), but eventually capa is able to complete the analysis
mimikatz.exe_	✅

[williballenthin]

fyi probably some of these are .NET files for which we don't expect Binja to do very much (and a failure is fine).

[xusheng6]

fyi probably some of these are .NET files for which we don't expect Binja to do very much (and a failure is fine).

Thx for letting me know! I am now only looking for crashes, and even if it is .NET files the analysis should not crash

[xusheng6]

I have triaged all of the files and here is a brief recap:

1. Most of the files are just fine with the fix in PR Various binja backend fixes #2500
2. A handful of them now crashes as mentioned in triage binary ninja backend failures #2406 (comment). This is a new crash discovered only after the original crash is fixed. This crash is still related to the IL being unavailable, and it seems to break some assumptions made by capa. Also, interestingly, I tested if I run capa with binja >= dev 4.3.6482, which has the fix to On-Demand Function Analysis is Triggering Time and Update Count Limits Vector35/binaryninja-api#6171, then the crash is gone since the IL will always be available (or not). In other ways, the new crash is caused by the IL initially available, but become unavailable afterwards. Technically, this can be avoided if we tell people to use the dev version, but I want capa be able to run with the stable binja, so I will look into it and see what is happening
3. There are a few packed binaries, as well as AutoIt, AutoHotKey ones that are excluded

In other words, the newly noticed crash in 2) is the only thing remaining for this issue to resolve