Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add machometa unit #36

Merged
merged 14 commits into from
Sep 11, 2023
Merged

Add machometa unit #36

merged 14 commits into from
Sep 11, 2023

Conversation

cxiao
Copy link
Contributor

@cxiao cxiao commented Sep 11, 2023

As per the discussion in #35, this adds a new unit, machometa, which extracts similar metadata from Mach-O files as the existing pemeta unit.

$ machometa -h
usage: machometa [-h] [-L] [-Q] [-0] [-v] [-c] [-H] [-K] [-S] [-V] [-D] [-E] [-I] [-t]

Extract metadata from Mach-O files.

options:
  -c, --custom         Unless enabled, all default categories will be extracted.
  -H, --header         Parse basic data from the Mach-O header.
  -K, --linked-images  Parse all library images linked by the Mach-O.
  -S, --signatures     Parse signature and entitlement information.
  -V, --version        Parse version information from the Mach-O load commands.
  -D, --load-commands  Parse load commands from the Mach-O header.
  -E, --exports        List all exported functions.
  -I, --imports        List all imported functions.
  -t, --tabular        Print information in a table rather than as JSON

generic options:
  -h, --help           Show this help message and exit.
  -L, --lenient        Allow partial results as output.
  -Q, --quiet          Disables all log output.
  -0, --devnull        Do not produce any output.
  -v, --verbose        Specify up to two times to increase log level.
Example output
$ ef a64fa9f1c76457ecc58402142a8728ce34ccba378c17318b3340083eeb7acc67 | machometa
{
    "FileType": "FAT",
    "Slices": [
        {
            "Header": {
                "type": "mach_header_64",
                "magic": 4277009103,
                "cputype": "X86_64",
                "cpusubtype": "ALL",
                "filetype": "DYLIB",
                "loadcount": 14,
                "loadsize": 2312,
                "flags": [
                    "NOUNDEFS",
                    "DYLDLINK",
                    "TWOLEVEL",
                    "NO_REEXPORTED_DYLIBS"
                ],
                "reserved": 0
            },
            "Linked Images": {
                "LOAD_DYLIB": [
                    "/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation",
                    "/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation",
                    "/usr/lib/libobjc.A.dylib",
                    "/usr/lib/libSystem.B.dylib"
                ]
            },
            "Signatures": {
                "Ad-Hoc Signed": false,
                "Signature Identifier": "libffmpeg",
                "Signature": {
                    "Timestamp": "2023-03-13 06:41:00+00:00",
                    "TimestampIssuer": "Developer ID Certification Authority",
                    "Subject": "Developer ID Application: 3CX (33CF4654HL)",
                    "SubjectLocation": "US",
                    "ValidFrom": "2019-04-11 12:03:36+00:00",
                    "ValidUntil": "2024-04-11 12:03:36+00:00",
                    "Issuer": "Developer ID Certification Authority",
                    "Fingerprint": "7df5ed6d71b296ed073a5b3efbcdc4c916ba41be",
                    "Serial": "4b0aaf622b260469"
                },
                "Requirements": "000000010000000300000014fade0c0000000098000000010000000600000002000000096c696266666d706567000000000000060000000f000000060000000e000000010000000a2a864886f76364060206000000000000000000060000000e000000000000000a2a864886f7636406010d0000000000000000000b000000000000000a7375626a6563742e4f550000000000010000000a3333434634363534484c0000",
                "Entitlements": "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<!DOCTYPE plist PUBLIC \"-//Apple//DTD PLIST 1.0//EN\" \"http://www.apple.com/DTDs/PropertyList-1.0.dtd\">\n<plist version=\"1.0\">\n  <dict>\n    <key>com.apple.security.cs.allow-jit</key>\n    <true/>\n    <key>com.apple.security.cs.allow-unsigned-executable-memory</key>\n    <true/>\n    <key>com.apple.security.cs.debugger</key>\n    <true/>\n    <key>com.apple.security.device.audio-input</key>\n    <true/>\n    <key>com.apple.security.device.camera</key>\n    <true/>\n  </dict>\n</plist>\n"
            },
            "UUID": "4c4c445f55553144a1e550e749c861fd",
            "Base Name": "libffmpeg.dylib",
            "Install Name": "@loader_path/libffmpeg.dylib"
        },
        {
            "Header": {
                "type": "mach_header_64",
                "magic": 4277009103,
                "cputype": "ARM64",
                "cpusubtype": "ALL",
                "filetype": "DYLIB",
                "loadcount": 12,
                "loadsize": 1656,
                "flags": [
                    "NOUNDEFS",
                    "DYLDLINK",
                    "TWOLEVEL",
                    "NO_REEXPORTED_DYLIBS"
                ],
                "reserved": 0
            },
            "Linked Images": {
                "LOAD_DYLIB": [
                    "/usr/lib/libSystem.B.dylib"
                ]
            },
            "Signatures": {
                "Ad-Hoc Signed": false,
                "Signature Identifier": "libffmpeg",
                "Signature": {
                    "Timestamp": "2023-03-13 06:41:01+00:00",
                    "TimestampIssuer": "Developer ID Certification Authority",
                    "Subject": "Developer ID Application: 3CX (33CF4654HL)",
                    "SubjectLocation": "US",
                    "ValidFrom": "2019-04-11 12:03:36+00:00",
                    "ValidUntil": "2024-04-11 12:03:36+00:00",
                    "Issuer": "Developer ID Certification Authority",
                    "Fingerprint": "7df5ed6d71b296ed073a5b3efbcdc4c916ba41be",
                    "Serial": "4b0aaf622b260469"
                },
                "Requirements": "000000010000000300000014fade0c0000000098000000010000000600000002000000096c696266666d706567000000000000060000000f000000060000000e000000010000000a2a864886f76364060206000000000000000000060000000e000000000000000a2a864886f7636406010d0000000000000000000b000000000000000a7375626a6563742e4f550000000000010000000a3333434634363534484c0000",
                "Entitlements": "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<!DOCTYPE plist PUBLIC \"-//Apple//DTD PLIST 1.0//EN\" \"http://www.apple.com/DTDs/PropertyList-1.0.dtd\">\n<plist version=\"1.0\">\n  <dict>\n    <key>com.apple.security.cs.allow-jit</key>\n    <true/>\n    <key>com.apple.security.cs.allow-unsigned-executable-memory</key>\n    <true/>\n    <key>com.apple.security.cs.debugger</key>\n    <true/>\n    <key>com.apple.security.device.audio-input</key>\n    <true/>\n    <key>com.apple.security.device.camera</key>\n    <true/>\n  </dict>\n</plist>\n"
            },
            "Version": {
                "BuildVersion": {
                    "Platform": "MACOS",
                    "MinOS": "11.0.0",
                    "SDK": "12.3.0",
                    "Ntools": 1
                }
            },
            "UUID": "4c4c449f55553144a16d1b7f53c45f28",
            "Base Name": "libffmpeg.dylib",
            "Install Name": "@rpath/libffmpeg.dylib"
        }
    ]
}

This also adds the k2l library as a dependency to Refinery.

New unit tests have been added in test/units/formats/macho/test_machometa.py. The following samples, which are used in the unit tests, have been uploaded to Malshare:

SHA-256 Description Reference
6c121f2b2efa6592c2c22b29218157ec9e63f385e7a1d7425857d603ddef8c59 UpdateAgent binary from North Korean supply chain compromise of 3CX software https://objective-see.org/blog/blog_0x74.html (referred to by SHA-1 hash 9e9a5f8d86356796162cee881c843cde9eaedfb3)
3e4bbd21756ae30c24ff7d6942656be024139f8180b7bddd4e5c62a9dfbd8c79 ARM64 MacOS version of LockBit Ransomware https://objective-see.org/blog/blog_0x75.html (referred to by SHA-1 hash 2d15286d25f0e0938823dcd742bc928e78199b3d)
1a9a5c797777f37463b44de2b49a7f95abca786db3977dcdac0f79da739c08ac SysJoker MacOS backdoor https://intezer.com/blog/incident-response/new-backdoor-sysjoker/
a64fa9f1c76457ecc58402142a8728ce34ccba378c17318b3340083eeb7acc67 libffmpeg binary from North Korean supply chain compromise of 3CX software https://objective-see.org/blog/blog_0x73.html (referred to by SHA-1 hash 769383fc65d1386dd141c960c9970114547da0c2
38c9b858c32fcc6b484272a182ae6e7f911dea53a486396037d8f7956d2110be tasker binary component of SilverSparrow https://redcanary.com/blog/clipping-silver-sparrows-wings/ (referred to by MD5 hash b370191228fef82635e39a137be470af)

Please feel free to directly make edits as needed.

)

@classmethod
def parse_pkcs7_signature(cls, data: bytearray) -> dict:
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is almost exactly the same as the function of the same name in the pemeta unit; feel free to refactor and combine these into one function.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will most likely do that, since I hate code duplication to the point of an unhealthy obsession, but this is nothing you need to concern yourself with.

self.log_warn(f"Could not parse the data in CSSLOT_CMS_SIGNATURE as valid PKCS7 data: {pkcs7_parse_error!s}")

if macho_image.codesign_info.req_dat is not None:
# TODO: Parse the requirements blob,
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me know if you prefer to not have TODO comments in the code. Happy to file an issue to keep track of this instead.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not an avid user of TODO comments, but I am not strictly opposed to them either. If it starts bugging me, I'll simply remove it. It's not like anyone reviews my PRs! 🤪

@cxiao cxiao mentioned this pull request Sep 11, 2023
@cxiao
Copy link
Contributor Author

cxiao commented Sep 11, 2023

@huettenhain It looks like unit tests are failing due to a missing Malshare API key. Is this expected for unit tests running via GitHub actions? Should I submit a PR against https://github.com/binref/refinery-test-data with the samples instead?

@huettenhain
Copy link
Member

huettenhain commented Sep 11, 2023

It should work without the Malshare API key, that's just a fallback. However, GitHub has recently been a little flaky and I had tests failing before. I'll figure out what's wrong.

@huettenhain
Copy link
Member

I was a little slow there to understand what's going on. Yes, the samples need to be in that repository, but I can add them myself.

@huettenhain
Copy link
Member

Also: This is probably the most pristine PR I have ever seen. 🙇

@@ -2,6 +2,7 @@
requires = [
"colorama",
"defusedxml",
"k2l",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a minor comment, and you don't have to change this, I will take care of it later: The requirements list here is reserved for "global" requirements, i.e. things that the refinery framework (and libraries) require. Requirements that are "local" to individual units can be specified in that unit itself, an example would be the pyperclip dependency in emit.

@codecov
Copy link

codecov bot commented Sep 11, 2023

Codecov Report

Merging #36 (d4c62fc) into master (24f5ef9) will decrease coverage by 0.15%.
Report is 20 commits behind head on master.
The diff coverage is 86.14%.

@@            Coverage Diff             @@
##           master      #36      +/-   ##
==========================================
- Coverage   83.76%   83.61%   -0.15%     
==========================================
  Files         332      336       +4     
  Lines       25754    26071     +317     
==========================================
+ Hits        21572    21799     +227     
- Misses       4182     4272      +90     
Files Changed Coverage Δ
refinery/units/formats/macho/machometa.py 86.14% <86.14%> (ø)

... and 14 files with indirect coverage changes

@huettenhain huettenhain merged commit f8864b2 into binref:master Sep 11, 2023
9 checks passed
@huettenhain
Copy link
Member

Merged this with rebase; thank you very much for your contribution!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants