warning: macOS truffleruby-head: TS_VERIFY_CTS_set_certs macro redefined in OpenSSL 3.1

[junaruga]

I found one warning in macOS truffleruby-head case at #618 (comment).

In the case, OpenSSL 3.1.1 is used.

https://github.com/ruby/openssl/actions/runs/5580792815/jobs/10198185535#step:8:94

../../../../ext/openssl/openssl_missing.h:195:11: error: 'TS_VERIFY_CTS_set_certs' macro redefined [-Werror,-Wmacro-redefined]
 #  define TS_VERIFY_CTS_set_certs(ctx, crts) ((ctx)->certs=(crts))
          ^
/usr/local/Cellar/openssl@3/3.1.1_1/include/openssl/ts.h:426:11: note: previous definition is here
#  define TS_VERIFY_CTS_set_certs(ctx, cert) TS_VERIFY_CTX_set_certs(ctx,cert)
          ^

In the last success case, the OpenSSL 1.1.1u was used in the macOS truffleruby-head case.
https://github.com/ruby/openssl/actions/runs/5532198527/jobs/10093943683#step:9:17

[junaruga]

It seems this part in OpenSSL 3.1.1.

https://github.com/openssl/openssl/blob/openssl-3.1.1/include/openssl/ts.h#L425-L427

# ifndef OPENSSL_NO_DEPRECATED_3_0
#  define TS_VERIFY_CTS_set_certs(ctx, cert) TS_VERIFY_CTX_set_certs(ctx,cert)
# endif

[rhenium]

From the log:

[...]
checking for SSL_set0_tmp_dh_pkey(NULL, NULL) in openssl/ssl.h... no
checking for ERR_get_error_all(NULL, NULL, NULL, NULL, NULL) in openssl/err.h... no
checking for TS_VERIFY_CTX_set_certs(NULL, NULL) in openssl/ts.h... no
checking for SSL_CTX_load_verify_file(NULL, "") in openssl/ssl.h... no
checking for BN_check_prime(NULL, NULL, NULL) in openssl/bn.h... no
checking for EVP_MD_CTX_get0_md(NULL) in openssl/evp.h... no
checking for EVP_MD_CTX_get_pkey_ctx(NULL) in openssl/evp.h... no
checking for EVP_PKEY_eq(NULL, NULL) in openssl/evp.h... no
checking for EVP_PKEY_dup(NULL) in openssl/evp.h... no
checking for whether -Werror is accepted as CFLAGS... yes
creating extconf.h
creating Makefile
[...]
/Users/runner/.rubies/truffleruby-head/lib/sulong/native/bin/graalvm-native-clang -I. -I/Users/runner/.rubies/truffleruby-head/lib/cext/include -I/Users/runner/.rubies/truffleruby-head/lib/cext/include/ruby/backward -I/Users/runner/.rubies/truffleruby-head/lib/cext/include -I../../../../ext/openssl -I/usr/local/Cellar/openssl@3/3.1.1_1/include -DRUBY_EXTCONF_H=\"extconf.h\"  -D_DARWIN_C_SOURCE -DTRUFFLERUBY_ABI_VERSION=3.1.3.9 -I/usr/local/opt/openssl@1.1/include -fPIC   -Werror=implicit-function-declaration -Wno-int-conversion -Wno-int-to-pointer-cast -Wno-incompatible-pointer-types -Wno-format-invalid-specifier -Wno-format-extra-args -ferror-limit=500  -Werror  -o openssl_missing.o -c ../../../../ext/openssl/openssl_missing.c 

This doesn't seem correct. Both OpenSSL 1.1.1 and 3.1.1 are included.

[junaruga]

This doesn't seem correct. Both OpenSSL 1.1.1 and 3.1.1 are included.

Hmm. Right. I can see both the -I/usr/local/Cellar/openssl@3/3.1.1_1/include and -I/usr/local/opt/openssl@1.1/include in the mentioned gcc command line in the log.

[junaruga]

Hi @eregon, I guess that you are familiar with the truffleruby things. We see the issue that the graalvm-native-clang compiler includes both OpenSSL 1.1 and 3.1.1 header directories on the -I arguments in GitHub Actions macOS truffleruby-head case. And I assume that the situation makes the compiler warning above. Do you know what is the cause of this issue?

[junaruga]

I opened the ticket for ruby-build: rbenv/ruby-build#2220

[junaruga]

I tested by removing OpenSSL 1.1 include directory in the macos-latest truffleruby-head case. But the compiler warning still happens.

diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
index 36609e2..9134c5b 100644
--- a/.github/workflows/test.yml
+++ b/.github/workflows/test.yml
@@ -57,6 +57,13 @@ jobs:
         run: echo "OPENSSL_MODULES=$($env:RI_DEVKIT)\$($env:MSYSTEM_PREFIX)\lib\ossl-modules" >> $env:GITHUB_ENV
         if: runner.os == 'Windows' && matrix.ruby == '3.2'
 
+      # A temporary workaround to avoid a compiler warning.
+      # https://github.com/rbenv/ruby-build/issues/2220
+      - name: remove openssl 1.1 include directory
+        run: |
+          rm -rfv "$(brew --prefix openssl@1.1)/include"
+        if: matrix.os == 'macos-latest' && matrix.ruby == 'truffleruby-head'
+
       - name: compile
         run:  rake compile -- --enable-debug

Here is the log.
https://github.com/junaruga/openssl/actions/runs/5615788064/job/15216818664

[junaruga]

Even when skipping the compiler warning check in the CI case, the compiler error happens in my tests.

https://github.com/junaruga/openssl/actions/runs/5621718703/job/15233027951#step:9:101

 ../../../../ext/openssl/ossl_hmac.c:249:35: error: incomplete definition of type 'struct evp_md_ctx_st'
    pkey = EVP_PKEY_CTX_get0_pkey(EVP_MD_CTX_get_pkey_ctx(ctx));
                                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~

However I found the workaround for this issue. It seems that the problem is openssl@3 brew package's include directory is added as the compiler's -I <include directory> argument. I sent the PR #652.

[eregon]

Hi @eregon, I guess that you are familiar with the truffleruby things. We see the issue that the graalvm-native-clang compiler includes both OpenSSL 1.1 and 3.1.1 header directories on the -I arguments in GitHub Actions macOS truffleruby-head case. And I assume that the situation makes the compiler warning above. Do you know what is the cause of this issue?

I'm confident TruffleRuby doesn't do this. So is this repo/the extconf.rb adding openssl in CPPFLAGS or so?
That's not well supported in general. TruffleRuby was compiled against the system openssl when it's installed. Trying to install a different openssl gem linking to another libssl is unlikely to work.
Notably because when building CRuby/TruffleRuby openssl paths might end up in RbConfig::CONFIG["CPPFLAGS"] etc if --with-openssl-dir was used.

[eregon]

It's related to this code: https://github.com/oracle/truffleruby/blob/16db2e64472e5f4d88176ae122815b6b291c0947/lib/truffle/rbconfig.rb#L97-L107
And https://github.com/oracle/truffleruby/blob/16db2e64472e5f4d88176ae122815b6b291c0947/lib/truffle/truffle/openssl-prefix.rb
As it says, we need C extensions use the same libssl as the one used for the openssl C extension.

Maybe we should prefer libssl 3 to 1.1 in https://github.com/oracle/truffleruby/blob/16db2e64472e5f4d88176ae122815b6b291c0947/lib/truffle/truffle/openssl-prefix.rb
We didn't so far because using libssl 3 was not fully compatible.

Do you know what picks openssl 3 over 1.1 on macOS so that this repo CI's fail?
TruffleRuby would always pick 1.1, so something else must tell it to use 3.

[eregon]

I think the problem is that truffleruby-head is built on macos 11 but in that CI run is used on macos 12. And probably one of them has openssl 1.1 and the other not.