[apple-codesign] Error: binary parsing error: Invalid magic number: on Linux

[vdjurovic]

I'm testing apple-codesign on Linux, and encountered this error immediately:

Error: binary parsing error: Invalid magic number: 0x504b0304

I have an app bundle which contains Java application with JRE, native launcher and a bunch of jar files. So, I run the command:

rcodesign sign --pem-source  ~/path/to/self-signed-cert.pem -vv   sipmlejavaFX.app/ 

Relevant log output:

[2022-07-28T10:44:17Z WARN  rcodesign] reading PEM data from /[removed]/self-signed-cert.pem
[2022-07-28T10:44:17Z WARN  rcodesign] registering signing key
[2022-07-28T10:44:17Z WARN  rcodesign] using time-stamp protocol server http://timestamp.apple.com/ts01
[2022-07-28T10:44:17Z WARN  rcodesign] automatically setting team ID from signing certificate: unset
[2022-07-28T10:44:17Z WARN  rcodesign] signing siplejavaFX.app/ to simplejavafx.app/
[2022-07-28T10:44:17Z WARN  rcodesign::signing] signing bundle at siplejavaFX.app/
[2022-07-28T10:44:17Z WARN  rcodesign::bundle_signing] signing 0 nested bundles in the following order:
[2022-07-28T10:44:17Z WARN  rcodesign::bundle_signing] signing bundle at siplejavaFX.app/ into simplejavafx.app/
[2022-07-28T10:44:17Z DEBUG goblin::mach] Ctx: Ctx { container: Big, le: Little }
[2022-07-28T10:44:17Z DEBUG goblin::mach] Mach-o header: Header { magic: 0xfeedfacf, cputype: 16777223, cpusubtype: 0x3, filetype: "EXECUTE", ncmds: 11, sizeofcmds: 2360, flags: 0x0, reserved: 0x0 }
....................
[2022-07-28T10:44:17Z INFO  rcodesign::bundle_signing] main executable targets OS requiring SHA-1 signatures; activating SHA-1 + SHA-256 signing
[2022-07-28T10:44:17Z WARN  rcodesign::bundle_signing] collecting code resources files
[2022-07-28T10:44:17Z INFO  rcodesign::bundle_signing] Contents/Info.plist is the Info.plist file; handling specially
[2022-07-28T10:44:17Z DEBUG rcodesign::code_resources] Info.plist matches exclusion rule ^Info\.plist$
[2022-07-28T10:44:17Z INFO  rcodesign::bundle_signing] copying file siplejavaFX.app/Contents/Info.plist -> simplejavafx.app/Contents/Info.plist
[2022-07-28T10:44:17Z DEBUG rcodesign::code_resources] Info.plist doesn't match any rule
[2022-07-28T10:44:17Z DEBUG rcodesign::code_resources] MacOS/cp/045700bcabd5441a50964944f38babbee8fc2e8159179b8cb5e64d18274e3853.jar matches inclusion rule ^(Frameworks|SharedFrameworks|PlugIns|Plug-ins|XPCServices|Helpers|MacOS|Library/(Automator|Spotlight|LoginItems))/
[2022-07-28T10:44:17Z INFO  rcodesign::code_resources] sealing Mach-O file MacOS/cp/045700bcabd5441a50964944f38babbee8fc2e8159179b8cb5e64d18274e3853.jar
[2022-07-28T10:44:17Z INFO  rcodesign::bundle_signing] signing Mach-O file Contents/MacOS/cp/045700bcabd5441a50964944f38babbee8fc2e8159179b8cb5e64d18274e3853.jar
Error: binary parsing error: Invalid magic number: 0x504b0304

So, it seems rcodesign is trying to sign a jar file and fails with this magic number error. Running hexdump on that jar gives the following output:

0000000 4b50 0403 000a 0800 0008 a916 5350 0000
0000010 0000 0002 0000 0000 0000 0009 0000 454d
0000020 4154 492d 464e 032f 5000 034b 0a04 0000
0000030 0808 1600 50a9 e253 fb21 4469 0000 4200
0000040 0000 1400 0000 4d00 5445 2d41 4e49 2f46
0000050 414d 494e 4546 5453 4d2e f346 cc4d 4ccb
0000060 2d4b d12e 4b0d 2a2d ccce b3cf 3052 33d4
0000070 e5e0 2c72 c92d 4dcf c92c d64c cdf5 294f
0000080 49cd f5d5 cc4b b54d c852 2c4a 4c4b d0ab
0000090 4a4b 4e2c cd75 282d e4a9 e2e5 02e5 5000
00000a0 014b 1402 0a03 0000 0808 1600 50a9 0053
00000b0 0000 0200 0000 0000 0000 0900 0000 0000
00000c0 0000 0000 1000 ed00 0041 0000 4d00 5445
00000d0 2d41 4e49 2f46 4b50 0201 0314 000a 0800
00000e0 0008 a916 5350 21e2 69fb 0044 0000 0042
00000f0 0000 0014 0000 0000 0000 0000 0000 81a4
0000100 0029 0000 454d 4154 492d 464e 4d2f 4e41
0000110 4649 5345 2e54 464d 4b50 0605 0000 0000
0000120 0002 0002 0079 0000 009f 0000 0000     
000012e

The first two bytes look like they are in inverted order to what is reported by rcodesign. So, is it possible that this is some kind of endian-ness problem?
Any idea of what might be the problem?

In addition, when I try to sign only the native launcher, I get the following error:

RUST_BACKTRACE=1 rcodesign sign --pem-source ~/local/work/bitshift/xapps/self-signed-cert.pem --code-signature-flags runtime  simplejavafx 
reading PEM data from /home/vlada/local/work/bitshift/xapps/self-signed-cert.pem
registering signing key
using time-stamp protocol server http://timestamp.apple.com/ts01
automatically setting team ID from signing certificate: unset
signing simplejavafx in place
signing simplejavafx as a Mach-O binary
inferring default signing settings from Mach-O binary
activating SHA-1 digests because minimum OS target 10.9.0 is not >=10.11.4
setting binary identifier to simplejavafx
parsing Mach-O
signing Mach-O binary at index 0
thread 'main' panicked at 'assertion failed: segment.fileoff == 0 || segment.fileoff == cursor.position()', /home/vlada/.cargo/registry/src/github.com-1ecc6299db9ec823/apple-codesign-0.16.0/src/macho_signing.rs:166:9
stack backtrace:
   0: rust_begin_unwind
             at /build/rustc-60tkWq/rustc-1.59.0+dfsg1/library/std/src/panicking.rs:498:5
   1: core::panicking::panic_fmt
             at /build/rustc-60tkWq/rustc-1.59.0+dfsg1/library/core/src/panicking.rs:116:14
   2: core::panicking::panic
             at /build/rustc-60tkWq/rustc-1.59.0+dfsg1/library/core/src/panicking.rs:48:5
   3: rcodesign::macho_signing::create_macho_with_signature
   4: <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::try_fold
   5: <alloc::vec::Vec<T> as alloc::vec::spec_from_iter::SpecFromIter<T,I>>::from_iter
   6: core::iter::adapters::process_results
   7: rcodesign::macho_signing::MachOSigner::write_signed_binary
   8: rcodesign::signing::UnifiedSigner::sign_path_in_place
   9: rcodesign::command_sign
  10: rcodesign::main_impl
  11: rcodesign::main
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

Not sure if the two problems are connected.

[indygreg]

The two problems are likely connected as we seem to be confusing a jar file with a mach-o binary.

If you look at the source code for the sealing Mach-O file log text (https://github.com/indygreg/PyOxidizer/blob/4ab5076a53b7a64adae657f47c6534a0a82ec33b/apple-codesign/src/code_resources.rs#L1252), we see the suspicious comment The assumption that a nested match means Mach-O may not be correct..

So, this comment turned out to likely be accurate. I'll need to teach the code how to handle non Mach-O binaries in nested bundle directories. That entails running Apple's official tooling and looking at their output to see how they handle the files.

I should probably be able to figure this out without steps to reproduce. But if you are able to sign your bundle with Apple's codesign and paste the XML structure from the _CodeSignature/CodeResources file for the JAR file in question, that will likely save me some time. I'm anticipating an entry for this jar in the <key>files2</key> <dict>.

[vdjurovic]

OK, I think I got it working somehow. It seems that my bundle layout was not up to code sign standards. I had entire Java app, including all jars and JRE in Contents/MacOS directory. When I tried to sign it on Mac, got the following error:

simplejavafx.app: code object is not signed at all
simplejavafx.app/Contents/MacOS/jre/man/man1/java.1

It looks like codesign expects only executable code to be in this location. So, I moved everything except launcher to Resources , created symlinks to them and then it got signed.

With this approach, rcodesign was able to sign the bundle as well. The only difference seems to be that Apple tool included symlinks in CodeResource, while rcodesign didn't. I've attached the CodeResources of both tools here, so you can see the difference. I'm not sure if I need to pass some option to rcodesign to include symlinks?
code-resources-apple-codesign.txt
code-resources-rcodesign.txt

Another thing I noticed is that there was a panic at the end of the log output:

2022-07-29T10:49:00Z DEBUG goblin::mach] 10 - LoadCommand { offset: 2336, command: LoadDylib(DylibCommand { cmd: 12, cmdsize: 56, dylib: Dylib { name: 24, timestamp: 0, current_version: 0, compatibility_version: 0 } }) }
thread 'main' panicked at 'assertion failed: segment.fileoff == 0 || segment.fileoff == cursor.position()', /home/vlada/.cargo/registry/src/github.com-1ecc6299db9ec823/apple-codesign-0.16.0/src/macho_signing.rs:166:9
stack backtrace:
   0: rust_begin_unwind
             at /build/rustc-60tkWq/rustc-1.59.0+dfsg1/library/std/src/panicking.rs:498:5
   1: core::panicking::panic_fmt
             at /build/rustc-60tkWq/rustc-1.59.0+dfsg1/library/core/src/panicking.rs:116:14
   2: core::panicking::panic
             at /build/rustc-60tkWq/rustc-1.59.0+dfsg1/library/core/src/panicking.rs:48:5
   3: rcodesign::macho_signing::create_macho_with_signature
   4: <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::try_fold
   5: <alloc::vec::Vec<T> as alloc::vec::spec_from_iter::SpecFromIter<T,I>>::from_iter
   6: core::iter::adapters::process_results
   7: rcodesign::macho_signing::MachOSigner::write_signed_binary
   8: rcodesign::bundle_signing::SingleBundleSigner::write_signed_bundle
   9: rcodesign::bundle_signing::BundleSigner::write_signed_bundle
  10: rcodesign::signing::UnifiedSigner::sign_path_in_place
  11: rcodesign::command_sign
  12: rcodesign::main_impl
  13: rcodesign::main
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

So, I'm not sure if the signing was complete.

[indygreg]

I'm relieved that Apple's official tooling rejected the non-Mach-O binaries! But we should fix rcodesign to not crash when signing an invalid Mach-O.

Regarding symlinks, we should handle those fine. However, they have historically caused complexities. It is possible you found an edge case.

If you attempt to sign a bundle in-place, I could easily see how that might not work with symlinks. Did you try passing multiple arguments to rcodesign so you write the signed bundle to a different output directory? Apple's tooling supports in-place bundle signing, so this is an area where we could potentially improve compatibility.

That being said, the diff might indicate a separate bug where we just don't handle symlinks correctly for some sub-directories. I'll need to reproduce to get to the bottom of it. I think I have enough context in this issue to do that.

Thanks again for the detailed info!

[vdjurovic]

I tried with different output directory, but still getting the same result. Symlinks do not get included.

I've tried running app bundle signed on Linux on Mac, and it didn't complain. So, I suppose signing is valid. The only issue I might see here is:

• I signed the bundle with mine self-signed cert generated on Mac
• ran the bundle on the same Mac (set to allow from AppStore and identified developers)

I'm not sure if this is enough to guarantee that it works OK, but I hope it is.

Another thing I ran into: I run the signing with command like this:

 rcodesign sign --pem-source ~test-signer-2.pem   simplejavafx.app/ simplejavafx-out.app

And I get this error:

error: The following required arguments were not provided:
    <input_path>

USAGE:
    rcodesign sign --pem-source <pem_source>... <input_path>

For more information try --help

This happens for both in-place and external signing. The solution I came up with is to add -v. This fixes the problem:

 rcodesign sign --pem-source ~test-signer-2.pem -v   simplejavafx.app/ simplejavafx-out.app

I suppose this is realted to CLI arguments handling, but I encountered it during signing process. Should I create a separate issue for it?

[staticglobal]

I am encountering a similar issue with symlinks not being included in CodeResources, or copied to destination directory when not signing in-place. In my case, the problematic symlinks are with .dylib files under Frameworks/. One example there is a regular MachO library libssh.4.4.8.dylib and a couple symlinks with various alternate filenames pointing to that library.

Frameworks/libssh.4.8.8.dylib (regular MachO binary file)
Frameworks/libssh.4.dylib -> libssh.4.8.8.dylib
Frameworks/libssh.dylib -> libssh.4.dylib

From the output of rcodesign sign:

[2023-06-07T15:43:21Z DEBUG apple_codesign::code_resources] Frameworks/libssh.4.8.8.dylib matches inclusion rule ^(Frameworks|SharedFrameworks|PlugIns|Plug-ins|XPCServices|Helpers|MacOS|Library/(Automator|Spotlight|LoginItems))/
[2023-06-07T15:43:21Z INFO  apple_codesign::code_resources] sealing Mach-O file Frameworks/libssh.4.8.8.dylib
...
[2023-06-07T15:43:23Z DEBUG apple_codesign::code_resources] Frameworks/libssh.4.dylib matches inclusion rule ^(Frameworks|SharedFrameworks|PlugIns|Plug-ins|XPCServices|Helpers|MacOS|Library/(Automator|Spotlight|LoginItems))/
[2023-06-07T15:43:23Z DEBUG apple_codesign::code_resources] Frameworks/libssh.4.dylib doesn't match any rule
[2023-06-07T15:43:23Z DEBUG apple_codesign::code_resources] Frameworks/libssh.dylib matches inclusion rule ^(Frameworks|SharedFrameworks|PlugIns|Plug-ins|XPCServices|Helpers|MacOS|Library/(Automator|Spotlight|LoginItems))/
[2023-06-07T15:43:23Z DEBUG apple_codesign::code_resources] Frameworks/libssh.dylib doesn't match any rule

Looking at code_resources.rs, I see the matching rule is marked as a nested rule, and inside evaluate_rules() there is this comment where the symlinks are being handled:

                } else if rule.nested && symlink_target.is_some() {
                    // Symlinks in nested bundles can be excluded since they should have
                    // been processed by the nested bundle.
                    Ok(RulesEvaluation::Exclude)

The comment believes that these symlinks are part of a nested bundle, which is not the case. They are part of the main bundle. So we need to call SealSymlink() on these to match the output of Apple tooling. I've tried a few things in hopes that I could submit a PR with a general fix, to no avail. As you say... it's complex. So best I can do right now is post this context and hope the right solution is obvious to someone else :)

[indygreg]

The invalid magic number issue is likely fixed by 1c63bca. I also just rewrote the bundle signing logic in b26376f to hopefully better match Apple's tooling.

I'm going to add a test for behavior of symlinks in a nested directory (like Frameworks/). If behavior matches Apple's tooling, I'll close this issue out.

[indygreg]

I think this is fixed in main. Since this issue tracked multiple distinct problems, please open a new issue if you are encountering any further problem.