Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OpenJDK post-install fails without Xcode on M1 #69100

Closed
4 tasks done
dtrodrigues opened this issue Jan 15, 2021 · 31 comments · Fixed by #69169
Closed
4 tasks done

OpenJDK post-install fails without Xcode on M1 #69100

dtrodrigues opened this issue Jan 15, 2021 · 31 comments · Fixed by #69169
Labels
11-arm64 Big Sur arm64 is specifically affected java Java use is a significant feature of the PR or issue outdated PR was locked due to age stale No recent activity

Comments

@dtrodrigues
Copy link
Member

Bug report

  • ran brew update and can still reproduce the problem?
  • ran brew doctor, fixed all issues and can still reproduce the problem?
  • ran brew gist-logs <formula> (where <formula> is the name of the formula that failed) and included the output link?
  • if brew gist-logs didn't work: ran brew config and brew doctor and included their output with your issue?

What you were trying to do (and why)

Install openjdk on an M1 Mac that doesn't have Xcode installed.

What happened (include command output)

The post-install stage had an error.

$ brew doctor
Your system is ready to brew.
$ brew config
HOMEBREW_VERSION: 2.7.4-32-gbca4804
ORIGIN: https://github.com/Homebrew/brew
HEAD: bca4804a9e48de5319383d3eddadaa7f054c77da
Last commit: 4 hours ago
Core tap ORIGIN: https://github.com/Homebrew/homebrew-core
Core tap HEAD: 74271951793445b680a247300b3c2e621aa4ff80
Core tap last commit: 2 hours ago
Core tap branch: master
HOMEBREW_PREFIX: /opt/homebrew
HOMEBREW_CASK_OPTS: []
HOMEBREW_DEVELOPER: set
HOMEBREW_EDITOR: vim
HOMEBREW_GITHUB_API_TOKEN: set
HOMEBREW_MAKE_JOBS: 8
HOMEBREW_NO_ANALYTICS: set
HOMEBREW_NO_AUTO_UPDATE: set
Homebrew Ruby: 2.6.3 => /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/bin/ruby
CPU: octa-core 64-bit arm_firestorm_icestorm
Clang: 12.0 build 1200
Git: 2.30.0 => /opt/homebrew/bin/git
Curl: 7.64.1 => /usr/bin/curl
macOS: 11.1-arm64
CLT: 12.3.0.0.1.1607026830
Xcode: N/A
Rosetta 2: false
Command output
==> Reinstalling openjdk
==> Pouring openjdk-15.0.1.arm64_big_sur.bottle.tar.gz
Error: Failed applying an ad-hoc signature to /opt/homebrew/Cellar/openjdk/15.0.1/libexec/openjdk.jdk/Contents/MacOS/libjli.dylib
Warning: The post-install step did not complete successfully
You can try again using `brew postinstall openjdk`
==> An exception occurred within a child process:
  Errno::ENOENT: No such file or directory @ rb_file_s_stat - /opt/homebrew/Library/Taps/homebrew/homebrew-core/SharedFrameworks/ContentDeliveryServices.framework/Versions/Current/itms/java/Frameworks/JavaNativeFoundation.framework

/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/fileutils.rb:1305:in stat' /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/fileutils.rb:1305:in lstat'
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/fileutils.rb:1350:in copy' /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/fileutils.rb:478:in block in copy_entry'
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/fileutils.rb:1484:in wrap_traverse' /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/fileutils.rb:475:in copy_entry'
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/fileutils.rb:453:in block in cp_r' /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/fileutils.rb:1557:in block in fu_each_src_dest'
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/fileutils.rb:1573:in fu_each_src_dest0' /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/fileutils.rb:1555:in fu_each_src_dest'
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/fileutils.rb:452:in cp_r' /opt/homebrew/Library/Taps/homebrew/homebrew-core/Formula/openjdk.rb:155:in post_install'
/opt/homebrew/Library/Homebrew/formula.rb:1057:in block (2 levels) in run_post_install' /opt/homebrew/Library/Homebrew/formula.rb:899:in with_logging'
/opt/homebrew/Library/Homebrew/formula.rb:1056:in block in run_post_install' /opt/homebrew/Library/Homebrew/utils.rb:531:in with_env'
/opt/homebrew/Library/Homebrew/formula.rb:1045:in run_post_install' /opt/homebrew/Library/Homebrew/postinstall.rb:22:in

'
==> Caveats
For the system Java wrappers to find this JDK, symlink it with
sudo ln -sfn /opt/homebrew/opt/openjdk/libexec/openjdk.jdk /Library/Java/JavaVirtualMachines/openjdk.jdk
This is a beta version of openjdk for Apple Silicon
(openjdk 16 preview).

openjdk is keg-only, which means it was not symlinked into /opt/homebrew,
because it shadows the macOS java wrapper.

If you need to have openjdk first in your PATH run:
echo 'export PATH="/opt/homebrew/opt/openjdk/bin:$PATH"' >> /Users/dustin/.bash_profile

For compilers to find openjdk you may need to set:
export CPPFLAGS="-I/opt/homebrew/opt/openjdk/include"

==> Summary
🍺 /opt/homebrew/Cellar/openjdk/15.0.1: 607 files, 305.2MB

What you expected to happen

Post-install to not fail or to enforce an Xcode dependency on Apple Silicon.

Step-by-step reproduction instructions (by running brew install commands)

brew install openjdk

@dtrodrigues dtrodrigues changed the title OpenJDK post-install fails without Xcode OpenJDK post-install fails without Xcode on M1 Jan 15, 2021
@fxcoudert
Copy link
Member

@tresf I hadn't thought about it, but the postinstall we've added doesn't really work on systems without Xcode:

  # Calculate Xcode's dual-arch JavaNativeFoundation.framework path
  def framework_path
    File.expand_path("../SharedFrameworks/ContentDeliveryServices.framework/Versions/Current/itms/java/Frameworks",
                     MacOS::Xcode.prefix)
  end

The CLT has a framework by the same name:

/Library/Developer/CommandLineTools/SDKs/MacOSX11.1.sdk/System/Library/Frameworks/JavaNativeFoundation.framework

do you think we could use that? Homebrew requires all users to have the CLT installed, so this is guaranteed to be present.

@fxcoudert
Copy link
Member

Please run brew reinstall openjdk and let us know whether that fixes the issue for you.

@unitof
Copy link
Contributor

unitof commented Jan 17, 2021

Postinstall step no longer fails, but I still get Error: Failed applying an ad-hoc signature to /opt/homebrew/Cellar/openjdk/15.0.1/libexec/openjdk.jdk/Contents/MacOS/libjli.dylib during the Pouring step.

~ ▶ brew reinstall openjdk
==> Downloading https://homebrew.bintray.com/bottles/openjdk-15.0.1.arm64_big_sur.bottle.1.tar.gz
Already downloaded: /Users/jacob/Library/Caches/Homebrew/downloads/71a21626f6690a21ee1764092d91146ae286323e9a02a44d9d3c76aa0c0eb27e--openjdk-15.0.1.arm64_big_sur.bottle.1.tar.gz
==> Reinstalling openjdk 
==> Pouring openjdk-15.0.1.arm64_big_sur.bottle.1.tar.gz
Error: Failed applying an ad-hoc signature to /opt/homebrew/Cellar/openjdk/15.0.1/libexec/openjdk.jdk/Contents/MacOS/libjli.dylib
==> Caveats
For the system Java wrappers to find this JDK, symlink it with
  sudo ln -sfn /opt/homebrew/opt/openjdk/libexec/openjdk.jdk /Library/Java/JavaVirtualMachines/openjdk.jdk
This is a beta version of openjdk for Apple Silicon
(openjdk 16 preview).

openjdk is keg-only, which means it was not symlinked into /opt/homebrew,
because it shadows the macOS `java` wrapper.

If you need to have openjdk first in your PATH run:
  echo 'export PATH="/opt/homebrew/opt/openjdk/bin:$PATH"' >> ~/.zshrc

For compilers to find openjdk you may need to set:
  export CPPFLAGS="-I/opt/homebrew/opt/openjdk/include"

==> Summary
🍺  /opt/homebrew/Cellar/openjdk/15.0.1: 612 files, 305.4MB

And running just postinstall:

~ ▶ brew postinstall openjdk
==> Postinstalling openjdk

@fxcoudert fxcoudert reopened this Jan 17, 2021
@fxcoudert
Copy link
Member

Can you post the output of:

codesign -dvvv /opt/homebrew/Cellar/openjdk/15.0.1/libexec/openjdk.jdk/Contents/MacOS/libjli.dylib

and possibly do the same thing with the pristine libjli.dylib file in the openjdk-15.0.1.arm64_big_sur.bottle.1.tar.gz bottle?

@fxcoudert fxcoudert added 11-arm64 Big Sur arm64 is specifically affected java Java use is a significant feature of the PR or issue labels Jan 17, 2021
@unitof
Copy link
Contributor

unitof commented Jan 17, 2021

Sure thing.

~ ▶ codesign -dvvv /opt/homebrew/Cellar/openjdk/15.0.1/libexec/openjdk.jdk/Contents/MacOS/libjli.dylib
Executable=/opt/homebrew/Cellar/openjdk/15.0.1/libexec/openjdk.jdk/Contents/MacOS/libjli.dylib
Identifier=libjli.dylib
Format=bundle with Mach-O thin (arm64)
CodeDirectory v=20400 size=837 flags=0x20002(adhoc,linker-signed) hashes=23+0 location=embedded
Hash type=sha256 size=32
CandidateCDHash sha256=1bf76f9b7f3218a3ed2c4c302cf50a252a62b106
CandidateCDHashFull sha256=1bf76f9b7f3218a3ed2c4c302cf50a252a62b1062aff5fff20f5aff55560dddd
Hash choices=sha256
CMSDigest=1bf76f9b7f3218a3ed2c4c302cf50a252a62b1062aff5fff20f5aff55560dddd
CMSDigestType=2
CDHash=1bf76f9b7f3218a3ed2c4c302cf50a252a62b106
Signature=adhoc
Info.plist=not bound
TeamIdentifier=not set
Sealed Resources=none
Internal requirements=none

Pristine from https://homebrew.bintray.com/bottles/openjdk-15.0.1.arm64_big_sur.bottle.1.tar.gz, freshly untarred:

~ ▶ codesign -dvvv openjdk/15.0.1/libexec/openjdk.jdk/Contents/MacOS/libjli.dylib
Executable=/Users/jacob/openjdk/15.0.1/libexec/openjdk.jdk/Contents/MacOS/libjli.dylib
Identifier=libjli.dylib
Format=bundle with Mach-O thin (arm64)
CodeDirectory v=20400 size=837 flags=0x20002(adhoc,linker-signed) hashes=23+0 location=embedded
Hash type=sha256 size=32
CandidateCDHash sha256=1bf76f9b7f3218a3ed2c4c302cf50a252a62b106
CandidateCDHashFull sha256=1bf76f9b7f3218a3ed2c4c302cf50a252a62b1062aff5fff20f5aff55560dddd
Hash choices=sha256
CMSDigest=1bf76f9b7f3218a3ed2c4c302cf50a252a62b1062aff5fff20f5aff55560dddd
CMSDigestType=2
CDHash=1bf76f9b7f3218a3ed2c4c302cf50a252a62b106
Signature=adhoc
Info.plist=not bound
TeamIdentifier=not set
Sealed Resources=none
Internal requirements=none

@fxcoudert
Copy link
Member

fxcoudert commented Jan 17, 2021

Can you run this on the pristine file and post the output?

codesign --verify libjli.dylib
codesign --sign - --force --preserve-metadata=entitlements,requirements,flags,runtime libjli.dylib
codesign --verify libjli.dylib

@unitof
Copy link
Contributor

unitof commented Jan 17, 2021

~ ▶ codesign --sign - --force --preserve-metadata=entitlements,requirements,flags,runtime openjdk/15.0.1/libexec/openjdk.jdk/Contents/MacOS/libjli.dylib 
openjdk/15.0.1/libexec/openjdk.jdk/Contents/MacOS/libjli.dylib: replacing existing signature
openjdk/15.0.1/libexec/openjdk.jdk/Contents/MacOS/libjli.dylib: resource fork, Finder information, or similar detritus not allowed

@unitof
Copy link
Contributor

unitof commented Jan 17, 2021

With the verifys and freshly deleted/re-untarred (though seems this is where it's failing):

~ ▶ codesign --verify openjdk/15.0.1/libexec/openjdk.jdk/Contents/MacOS/libjli.dylib
openjdk/15.0.1/libexec/openjdk.jdk/Contents/MacOS/libjli.dylib: invalid signature (code or signature have been modified)
In architecture: arm64

~ ▶ codesign --sign - --force --preserve-metadata=entitlements,requirements,flags,runtime openjdk/15.0.1/libexec/openjdk.jdk/Contents/MacOS/libjli.dylib
openjdk/15.0.1/libexec/openjdk.jdk/Contents/MacOS/libjli.dylib: replacing existing signature
openjdk/15.0.1/libexec/openjdk.jdk/Contents/MacOS/libjli.dylib: resource fork, Finder information, or similar detritus not allowed

~ ▶ codesign --verify openjdk/15.0.1/libexec/openjdk.jdk/Contents/MacOS/libjli.dylib
openjdk/15.0.1/libexec/openjdk.jdk/Contents/MacOS/libjli.dylib: invalid signature (code or signature have been modified)
In architecture: arm64

@fxcoudert
Copy link
Member

hum I wonder what the “detritus” is. Can you try:

codesign --sign - --force --preserve-metadata=entitlements,requirements,flags,runtime  -vvv libjli.dylib

@unitof
Copy link
Contributor

unitof commented Jan 17, 2021

~ ▶ codesign --sign - --force --preserve-metadata=entitlements,requirements,flags,runtime  -vvv openjdk/15.0.1/libexec/openjdk.jdk/Contents/MacOS/libjli.dylib
openjdk/15.0.1/libexec/openjdk.jdk/Contents/MacOS/libjli.dylib: replacing existing signature
openjdk/15.0.1/libexec/openjdk.jdk/Contents/MacOS/libjli.dylib: resource fork, Finder information, or similar detritus not allowed

@carlocab
Copy link
Member

@unitof
Copy link
Contributor

unitof commented Jan 17, 2021

Aha! Just found that page as well, and xattr -lr openjdk/15.0.1/libexec/openjdk.jdk makes codesign work.

Though note the FinderInfo is on the .jdk bundle, nothing on the .dylib file itself.

~ ▶ xattr -lr openjdk/15.0.1/libexec/openjdk.jdk/Contents/MacOS/libjli.dylib
~ ▶ xattr -lr openjdk/15.0.1/libexec/openjdk.jdk
openjdk/15.0.1/libexec/openjdk.jdk: com.apple.FinderInfo:
00000000  00 00 00 00 00 00 00 00 20 00 00 00 00 00 00 00  |........ .......|
00000010  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  |................|
00000020
~ ▶ xattr -cr openjdk/15.0.1/libexec/openjdk.jdk
~ ▶ xattr -lr openjdk/15.0.1/libexec/openjdk.jdk
~ ▶ codesign --sign - --force --preserve-metadata=entitlements,requirements,flags,runtime openjdk/15.0.1/libexec/openjdk.jdk/Contents/MacOS/libjli.dylib
openjdk/15.0.1/libexec/openjdk.jdk/Contents/MacOS/libjli.dylib: replacing existing signature
~ ▶  

@dtrodrigues
Copy link
Member Author

xattr -cr works for me too -- -lr is listing the attributes but -cr clears them, which looks like it allows the signing to work properly.

@fxcoudert
Copy link
Member

OK I think this warrants a report to openjdk, since the FinderInfo resource extended attribute was generated in their build. There is an adoptopenjdk thread adoptium/adoptium-support#97 where @tresf commented, so he probably knows more.

  • Has this been reported to openjdk? (I didn't find anything)
  • Is there a fix we could use?

Depending on that, we can implement a workaround/hack of calling xattr -cr in the formula. But I really thinks this should be fixed upstream.

@tresf
Copy link
Contributor

tresf commented Jan 18, 2021

OK I think this warrants a report to openjdk, since the FinderInfo resource extended attribute was generated in their build. There is an adoptopenjdk thread AdoptOpenJDK/openjdk-support#97 where @tresf commented, so he probably knows more.

  • Has this been reported to openjdk? (I didn't find anything)

I can't either.

Are you sure this is an openjdk bug? I think I'm missing the part which isolates this to openjdk. Note, filing bugs with openjdk is a bit tricky, it's not a public bug report system like most similar projects.

At a very high-level glance, it appears that there are some openjdk tools which attempt to codesign at build time and homebrew's codesigning steps are -- for some files -- redundant with those steps, requiring a forced removal, which comes with it it's own complications due to caching, etc.

If it is a bug with the build system, we should probably write-up an expected vs. actual, otherwise, it's a packaging issue that Homebrew needs to keep in the build system indefinitely (like the framework requirement).

The AdoptOpenJDK bug linked above was ruled as a packaging issue as well, putting the onus of code signature removal/addition onto the packager (in this case, Homebrew). If I'm misunderstanding something, please let me know. I've sponsored several openjdk bug reports so I feel a bit of onus when I'm tagged. 🍻

@fxcoudert
Copy link
Member

The AdoptOpenJDK bug linked above was ruled as a packaging issue as well

I don't understand why, though. The build process produces a signed library with invalid signature, this sounds like a bug in the build process, not a packaging issue.

I'm trying to confirm this in #69279

@tresf
Copy link
Contributor

tresf commented Jan 18, 2021

Hmm... If that's true, then Adopt maintainers would have similar workarounds in their packaging code. That specific bug report was about how hard it is to repackage the already working Adopt installation to be bundled and redistributed by yet another project (e.g. Cyberduck or some Adopt user looking to repackage), so the signature removal is critical, hence companies like BellSoft offering unsigned versions as explicit downloads.

@dtrodrigues
Copy link
Member Author

If I'm understanding it correctly, it's not that it's a signed library with an invalid signature, but a directory of what homebrew is trying to sign has Finder xattrs set.

@fxcoudert
Copy link
Member

has Finder xattrs set

Which makes the signature invalid, as I understand.

@carlocab
Copy link
Member

Does that mean it's invalid? My understanding is that Apple now just doesn't want you to codesign anything that has an extended attribute containing a resource fork or Finder info.

It seems like this would be true whether that thing is signed or not. If this is the case, then it's not really an invalid signature problem.

@fxcoudert
Copy link
Member

@carlocab that's playing on words, really. The resulting binary does not pass codesign -v, and will not run / be loaded on macOS. I call that "invalid" :)

@fxcoudert
Copy link
Member

fxcoudert commented Jan 18, 2021

@tresf Yes, I can confirm it is a build bug. After running make images, with no modification by homebrew whatsoever, the resulting files are invalid:

==> codesign -d -vvv build/macosx-aarch64-server-release/images/jdk-bundle/jdk-16.jdk/Contents/MacOS/libjli.dylib
Executable=/private/tmp/openjdk-20210118-1236-1gwjpmm/jdk-sandbox-a56ddad05cf1808342aeff1b1cd2b0568a6cdc3a/build/macosx-aarch64-server-release/images/jdk-bundle/jdk-16.jdk/Contents/MacOS/libjli.dylib
Identifier=libjli.dylib
Format=bundle with Mach-O thin (arm64)
CodeDirectory v=20400 size=837 flags=0x20002(adhoc,linker-signed) hashes=23+0 location=embedded
Hash type=sha256 size=32
CandidateCDHash sha256=1bf76f9b7f3218a3ed2c4c302cf50a252a62b106
CandidateCDHashFull sha256=1bf76f9b7f3218a3ed2c4c302cf50a252a62b1062aff5fff20f5aff55560dddd
Hash choices=sha256
CMSDigest=1bf76f9b7f3218a3ed2c4c302cf50a252a62b1062aff5fff20f5aff55560dddd
CMSDigestType=2
CDHash=1bf76f9b7f3218a3ed2c4c302cf50a252a62b106
Signature=adhoc
Info.plist=not bound
TeamIdentifier=not set
Sealed Resources=none
Internal requirements=none
==> codesign -v -vvv build/macosx-aarch64-server-release/images/jdk-bundle/jdk-16.jdk/Contents/MacOS/libjli.dylib
build/macosx-aarch64-server-release/images/jdk-bundle/jdk-16.jdk/Contents/MacOS/libjli.dylib: code has no resources but signature indicates they must be present

codesign -v libjli.dylib returns with a nonzero exit code, confirming the signature is invalid in this context.

@xszuflax

This comment has been minimized.

@dtrodrigues
Copy link
Member Author

Has this been reported to AdoptOpenJDK? Until it's resolved, should we try and work around it in Homebrew for CLT-only users or make Xcode an unconditional requirement on ARM instead of build-only?

@tresf
Copy link
Contributor

tresf commented Feb 1, 2021

Has this been reported to AdoptOpenJDK?

Note, Adopt isn't openjdk proper, but rather a downstream JDK binary provider. openjdk's formulae uses a version fetched directly from upstream. At time of writing this, no bug report has been filed with openjdk.

, should we try and work around it in Homebrew for CLT-only users or make Xcode an unconditional requirement on ARM instead of build-only?

Two separate issues... The post-install is just a packaging fix, the issues being discussed now are related to bugs surrounding the upstream build scripts and how they apply code signatures. They may warrant splitting the latter off into a separate bug report.

@xszuflax
Copy link

xszuflax commented Feb 2, 2021

Hello,

I might be missing something but I still get:

Error: Failed applying an ad-hoc signature to /opt/homebrew/Cellar/openjdk/15.0.1/libexec/openjdk.jdk/Contents/MacOS/libjli.dylib

even with XCode installed. Funny enough both homebrew openjdk and openjdk@11 work well and both throw mentioned error.

@BrewTestBot
Copy link
Member

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

@BrewTestBot BrewTestBot added the stale No recent activity label Feb 24, 2021
@dtrodrigues
Copy link
Member Author

Commenting to keep issue open.

@BrewTestBot BrewTestBot removed the stale No recent activity label Feb 24, 2021
@Bo98
Copy link
Member

Bo98 commented Feb 24, 2021

Just to copy what I mentioned elsewhere: I consider this a blocker for making codesign errors fatal in brew (which I hope we can do to avoid people being left with broken essentials like git).

@BrewTestBot
Copy link
Member

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

@BrewTestBot BrewTestBot added the stale No recent activity label Mar 18, 2021
@dtrodrigues
Copy link
Member Author

After #69460 , I'm no longer seeing the codesigning errors

@github-actions github-actions bot added the outdated PR was locked due to age label Apr 18, 2021
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Apr 18, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
11-arm64 Big Sur arm64 is specifically affected java Java use is a significant feature of the PR or issue outdated PR was locked due to age stale No recent activity
Projects
None yet
Development

Successfully merging a pull request may close this issue.

8 participants