Extract api names from ELF debug symbols [vivisect]

[yelhamer]

This is a draft PR to add support for utilizing the symbol table's entries to extract the names of statically linked apis. See discussion. also #1445.

I am using the vivisct engine to fetch the symbol names. Please let me know if this is the correct approach, and what are your thoughts on it.

Tests on sample 2bf18d0403677378adad9001b1243211:

before:

after:

Checklist

• No CHANGELOG update needed

• No new tests needed

• No documentation update needed

[github-actions]

Please add bug fixes, new features, breaking changes and anything else you think is worthwhile mentioning to the master (unreleased) section of CHANGELOG.md. If no CHANGELOG update is needed add the following to the PR description: [x] No CHANGELOG update needed

CHANGELOG updated or no update needed, thanks! 😄

[williballenthin]

thanks for asking for input early on. i've provided my vision inline, but i'm open to discussion on any of the topics. don't hesitate to suggest alternatives!

i'd recommend developing a few tests right away so we can agree what features should be extracted - and then we can dig into the implementation more. maybe assert that API features __GI_connect, __libc_connect, and connect are all found at the address of the call instruction (0x40286D in 72f1b91327ffda4cf18a2bf64913b673d39ebbff8cbe50c9cd354b1dcd312bcc).

also, (new feature) we should apply function-name features to the functions themselves. this is one thing we do with the FLIRT matches, so you can say thing like "look for this pattern, unless its found within OpenSSL::create_context" or similar. example here:

capa/capa/features/extractors/viv/file.py

Line 85 in 6ba5b2b

yield FunctionName(name), addr

[yelhamer]

I could not get api matches when using the FunctionName feature — as opposed to its API counterpart — for some reason. I will look into that and add tests on the next couple of commits.

[williballenthin]

Hm, I don’t quite understand. Do you mind explaining what didn’t work a little differently?

[yelhamer]

I apologize for not providing much details @williballenthin. I have further elaborated bellow:

To my understanding, your previous recommendation to apply function-name features to the functions themselves meant using FunctionName() to yield the resolved functions, as opposed to API(). So in my case that means something like this on line 150 of my commit to insn.py:

...
if sym_value == target and sym_info & STT_FUNC != 0:
    yield FunctionName(sym_name), ih.address
...

However, doing this resulted in no capabilities — of symtab origin — being matched by our ruleset, while the usage of API to yield the functions resulted in the intended behavior of all the expected capabilities being extracted:

My questions are:

• Could you please confirm whether I understood your recommendation of using function-name correctly?
• and if I did, could you please direct me at what I should be doing in order to get this working as expected?

Thanks!

[williballenthin]

Thanks for the clarification ! I had not explained clearly enough, so it’s my fault. We should emit function-name *in addition* to API like you were doing before. function-name features are not used to derive API features; they are independent. API features let an author say: find a call to this function and that function together. function-name features let an author say: find this logic when we know the name of the current function isn’t (for example) memcpy. In practice, we might want to use function-name features when we develop the Linux raw syscall rules, because we’ll be able to do things like “look for syscall 3 when the current function isn’t already named `open`”.

[yelhamer]

Thank you for the explanation! I wrongly assumed that function-name did the same job as API in this context, I take responsibility for that.

I have two more questions:

• Regarding tests — should I combine the testing of symtab-based api extraction as well as the possible identification of common gcc wrappers (as discussed here) into a single test file? or should I test the api extraction using symtab information as part of the viv feature extraction tests here?
• In my commit, I define a new function attribute in order to avoid redoing the symbol table extraction process each time the function gets called; however, this is causing the mypy code_style check to fail (error message). What do you think of this idea — usage of function attributes; is it correct? or should I look for an alternative?

Also, please let me know if you think there is anything else that I should be doing differently.

[mr-tz]

Use whichever test you find more logical. Ideally, each functionality is tested separately.

For the optimization the first idea is to implement it similarly to get_imports.

[williballenthin]

good progress, some changes still needed, though.

[williballenthin]

just some error handling remain and we're good to go. nice job @yelhamer

edit: and need to review code style. reach out if you don't have the linters configured and i can explain that.

[yelhamer]

I believe all requested changes have been addressed except for the code style review, I couldn't get the linter working properly and will be asking for explanation on that after the weekend.

Also, please check if the way I am doing error handling is correct. I am assuming that all Exceptions that get raised during SymTab construction are due to the symbol's table being faulty, and I am not handling the case of an empty symtab since that'd just result in an empty symbols' generator (for the for symbol in symtab.get_symbols(): part)

[williballenthin]

isort --profile black --length-sort --line-width 120 --skip-glob "*_pb2.py" . ; \
black -l 120 --extend-exclude ".*_pb2.py" . ; \
ruff check --config .github/ruff.toml . ; \
pycodestyle --exclude="*_pb2.py" --show-source capa/ scripts/ tests/ ; \
mypy --config-file .github/mypy/mypy.ini --check-untyped-defs capa/ scripts/ tests/

via here:

capa/.github/workflows/tests.yml

Lines 36 to 45 in 0cbe461

	- name: Lint with ruff
	run: ruff check --config .github/ruff.toml .
	- name: Lint with isort
	run: isort --profile black --length-sort --line-width 120 --skip-glob "*_pb2.py" -c .
	- name: Lint with black
	run: black -l 120 --extend-exclude ".*_pb2.py" --check .
	- name: Lint with pycodestyle
	run: pycodestyle --exclude="*_pb2.py" --show-source capa/ scripts/ tests/
	- name: Check types with mypy
	run: mypy --config-file .github/mypy/mypy.ini --check-untyped-defs capa/ scripts/ tests/

there's also a helper script here: https://github.com/mandiant/capa/blob/master/scripts/ci.sh
though to be honest, i use the above command and not the script, so im not sure if its out of date.

[williballenthin]

code style passes and tests pass, woohoo!

nice work @yelhamer