run IDA tests headlessly

[williballenthin]

it would be nice to invoke the capa tests under IDA without interacting with the IDA GUI. this would enable things like #1362 (run IDA tests in CI). the test runner inside IDA could emit the test results to a file specified via CLI and a wrapper script that invokes IDA might read and render this back to the user.

[williballenthin]

This technique is described by github here: https://docs.github.com/en/actions/security-guides/encrypted-secrets#storing-large-secrets

[williballenthin]

might be a pain to figure out how to install IDA headlessly in CI

[ggold7046]

Hi @williballenthin , could you please explain the above problem in detail. What is meant by headless IDA ?

[williballenthin]

I mean that we want to run IDA Pro in our CI/CD setup so that we can execute our test cases. Typically this program is run with a GUI so by “headless” I mean we need to control it without a display.

[ggold7046]

Can we use the free IDA version ? I don't have the PRO version though.

[williballenthin]

Unfortunately not because the free version doesn’t support scripting with Python. We’ll need to use one of our paid licenses for this, unless perhaps Hex-Rays donates a server license for us to use in CI/CD.

[williballenthin]

its a little tricky to invoke all the tests today, because they expect to be run against a bunch of different input samples and assert that features are extracted. we'll need to find a way to structure the tests such that they're easy to invoke in CI.

one way might be to only run the test cases that match the currently loaded sample, and skip the remainder. then the CI configuration can invoke the test runner script against each sample referenced in the tests. its a little configuration-heavy, but wouldn't require too many changes and does get the job done.

these changes would also support running Ghidra backend in CI more easily.

[williballenthin]

today we have test_ida_features.py.

this script is named like a pytest file; however, all the test cases inside are skipped because the tests don't actually run in pytest. this is ok. its just documentation.

the file expects to be executed as a script inside of IDA. when the current file MD5 matches the file used by a feature test, then the test is invoked (directly, as a function, not via pytest) and the result printed to stdout.

#1513 refines the test runner so that it can be invoked easily from the command line:

[williballenthin]

our features tests currently reference the following files:

• "294b8db1f2702b60fb2e42fdc50c2cee6a5046112da9a5703a548a4fa50477bc.elf_"
• "64d9f7d96b99467f36e22fada623c3bb.dll_"
• "7351f8a40c5450557b24622417fc478d.elf_"
• "773290480d5445f11d3dc1b800728966.exe_"
• "79abd17391adc6251ecdc58d13d76baf.dll_"
• "946a99f36a46d335dec080d9a4371940.dll_"
• "a198216798ca38f280dc413f8c57f2c2.exe_"
• "al-khaser_x64.exe_"
• "c91887d861d9bd4a5872249b641bc9f9.exe_"
• "kernel32-64.dll_"
• "kernel32.dll_"
• "mimikatz.exe_"
• "Practical Malware Analysis Lab 12-04.exe_"
• "Practical Malware Analysis Lab 16-01.exe_"

which means we can run all the feature tests in IDA like this:

& 'C:\Program Files\IDA Pro 8.2\idat.exe' -S"C:\Exclusions\code\capa\tests\test_ida_features.py --CAPA_AUTOEXIT=true" -A -Lidalog "C:\Exclusions\code\capa\tests\data\294b8db1f2702b60fb2e42fdc50c2cee6a5046112da9a5703a548a4fa50477bc.elf_"
& 'C:\Program Files\IDA Pro 8.2\idat.exe' -S"C:\Exclusions\code\capa\tests\test_ida_features.py --CAPA_AUTOEXIT=true" -A -Lidalog "C:\Exclusions\code\capa\tests\data\64d9f7d96b99467f36e22fada623c3bb.dll_"
& 'C:\Program Files\IDA Pro 8.2\idat.exe' -S"C:\Exclusions\code\capa\tests\test_ida_features.py --CAPA_AUTOEXIT=true" -A -Lidalog "C:\Exclusions\code\capa\tests\data\7351f8a40c5450557b24622417fc478d.elf_"
& 'C:\Program Files\IDA Pro 8.2\idat.exe' -S"C:\Exclusions\code\capa\tests\test_ida_features.py --CAPA_AUTOEXIT=true" -A -Lidalog "C:\Exclusions\code\capa\tests\data\773290480d5445f11d3dc1b800728966.exe_"
& 'C:\Program Files\IDA Pro 8.2\idat.exe' -S"C:\Exclusions\code\capa\tests\test_ida_features.py --CAPA_AUTOEXIT=true" -A -Lidalog "C:\Exclusions\code\capa\tests\data\79abd17391adc6251ecdc58d13d76baf.dll_"
& 'C:\Program Files\IDA Pro 8.2\idat.exe' -S"C:\Exclusions\code\capa\tests\test_ida_features.py --CAPA_AUTOEXIT=true" -A -Lidalog "C:\Exclusions\code\capa\tests\data\946a99f36a46d335dec080d9a4371940.dll_"
& 'C:\Program Files\IDA Pro 8.2\idat.exe' -S"C:\Exclusions\code\capa\tests\test_ida_features.py --CAPA_AUTOEXIT=true" -A -Lidalog "C:\Exclusions\code\capa\tests\data\a198216798ca38f280dc413f8c57f2c2.exe_"
& 'C:\Program Files\IDA Pro 8.2\idat.exe' -S"C:\Exclusions\code\capa\tests\test_ida_features.py --CAPA_AUTOEXIT=true" -A -Lidalog "C:\Exclusions\code\capa\tests\data\al-khaser_x64.exe_"
& 'C:\Program Files\IDA Pro 8.2\idat.exe' -S"C:\Exclusions\code\capa\tests\test_ida_features.py --CAPA_AUTOEXIT=true" -A -Lidalog "C:\Exclusions\code\capa\tests\data\c91887d861d9bd4a5872249b641bc9f9.exe_"
& 'C:\Program Files\IDA Pro 8.2\idat.exe' -S"C:\Exclusions\code\capa\tests\test_ida_features.py --CAPA_AUTOEXIT=true" -A -Lidalog "C:\Exclusions\code\capa\tests\data\kernel32-64.dll_"
& 'C:\Program Files\IDA Pro 8.2\idat.exe' -S"C:\Exclusions\code\capa\tests\test_ida_features.py --CAPA_AUTOEXIT=true" -A -Lidalog "C:\Exclusions\code\capa\tests\data\kernel32.dll_"
& 'C:\Program Files\IDA Pro 8.2\idat.exe' -S"C:\Exclusions\code\capa\tests\test_ida_features.py --CAPA_AUTOEXIT=true" -A -Lidalog "C:\Exclusions\code\capa\tests\data\mimikatz.exe_"
& 'C:\Program Files\IDA Pro 8.2\idat.exe' -S"C:\Exclusions\code\capa\tests\test_ida_features.py --CAPA_AUTOEXIT=true" -A -Lidalog "C:\Exclusions\code\capa\tests\data\Practical Malware Analysis Lab 12-04.exe_"
& 'C:\Program Files\IDA Pro 8.2\idat.exe' -S"C:\Exclusions\code\capa\tests\test_ida_features.py --CAPA_AUTOEXIT=true" -A -Lidalog "C:\Exclusions\code\capa\tests\data\Practical Malware Analysis Lab 16-01.exe_"

[williballenthin]

there are a handful of failing tests @mike-hunhoff

FAIL: test_ida_features/pma12-04-file-characteristic(embedded pe)-True
Traceback (most recent call last):
  File "C:/Exclusions/code/capa/tests/test_ida_features.py", line 62, in test_ida_features
    fixtures.do_test_feature_presence(get_ida_extractor, sample, scope, feature, expected)
  File "C:\Exclusions/code/capa/tests\fixtures.py", line 1018, in do_test_feature_presence
    assert feature.evaluate(features) == expected, msg
AssertionError: characteristic(embedded pe) should be found in file

FAIL: test_ida_features/pma16-01-function=0x404356-os(windows)-True
Traceback (most recent call last):
  File "C:/Exclusions/code/capa/tests/test_ida_features.py", line 62, in test_ida_features
    fixtures.do_test_feature_presence(get_ida_extractor, sample, scope, feature, expected)
  File "C:\Exclusions/code/capa/tests\fixtures.py", line 1013, in do_test_feature_presence
    features = scope(extractor)
  File "C:\Exclusions/code/capa/tests\fixtures.py", line 486, in inner_function
    fh = get_function(extractor, va)
  File "C:\Exclusions/code/capa/tests\fixtures.py", line 395, in get_function
    raise ValueError("function not found")
ValueError: function not found


(...and a few others with the same failure: above function can't be found by IDA)