[BUG] OpenSSLTest is not using the OpenSSL Provider

[cwperks]

OpenSSLTest is a subclass of SSLTest and ensures that the OpenSSL Provider from netty tcnative can be used to provide SSL for the cluster. See the output of OpenSSLTest.testHttpsAndNodeSSLKeyPass below and see that the test is actually using the JDK SSL provider:

---------------- Starting JUnit-test: OpenSSLTest testHttpsAndNodeSSLKeyPass ----------------
tcpClusterManagerPorts: [7130]/tcpAllPorts: [7130, 8115, 9070]/httpPorts: [9302, 9470, 9615] for (6024-11023) fork 1
[2022-10-24T13:42:35,917][WARN ][org.opensearch.node.Node] version [3.0.0-SNAPSHOT] is a pre-release version of OpenSearch and is not suitable for production
[2022-10-24T13:42:35,918][WARN ][org.opensearch.security.ssl.util.SSLCertificateHelper] Certificate chain for alias node-0 contains a root certificate
[2022-10-24T13:42:35,968][WARN ][org.opensearch.security.ssl.util.SSLCertificateHelper] Certificate chain for alias node-0 contains a root certificate
[2022-10-24T13:42:35,996][WARN ][org.opensearch.security.OpenSearchSecurityPlugin] OpenSearch Security plugin run in ssl only mode. No authentication or authorization is performed
[2022-10-24T13:42:36,020][WARN ][org.opensearch.gateway.DanglingIndicesState] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
{"client.type":"node","cluster.initial_cluster_manager_nodes":["127.0.0.1:7130"],"cluster.name":"utest_n287_fnull_t1691768839757","cluster.routing.allocation.disk.threshold_enabled":"false","discovery.initial_state_timeout":"8s","discovery.seed_hosts":["127.0.0.1:7130"],"http.compression":"false","http.cors.enabled":"true","http.port":"9302","http.type":"org.opensearch.security.ssl.http.netty.SecuritySSLNettyHttpServerTransport","http.type.default":"netty4","node.attr.shard_indexing_pressure_enabled":"true","node.max_local_storage_nodes":"3","node.name":"node_utest_n287_fnull_t1691768839757_num3","node.roles":["cluster_manager"],"path.data":["/home/runner/work/security/security/build/testrun/test/target/data/utest_n287_fnull_t1691768839757/data"],"path.home":"/home/runner/work/security/security/build/testrun/test/target","path.logs":"/home/runner/work/security/security/build/testrun/test/target/data/utest_n287_fnull_t1691768839757/logs","plugins.security.ssl.http.clientauth_mode":"REQUIRE","plugins.security.ssl.http.enable_openssl_if_available":"true","plugins.security.ssl.http.enabled":"true","plugins.security.ssl.http.keystore_alias":"node-0","plugins.security.ssl.http.keystore_filepath":"/home/runner/work/security/security/build/resources/test/ssl/node-0-keystore.jks","plugins.security.ssl.http.keystore_keypassword":"changeit","plugins.security.ssl.http.truststore_filepath":"/home/runner/work/security/security/build/resources/test/ssl/truststore.jks","plugins.security.ssl.transport.enable_openssl_if_available":"true","plugins.security.ssl.transport.enabled":"true","plugins.security.ssl.transport.enforce_hostname_verification":"false","plugins.security.ssl.transport.keystore_alias":"node-0","plugins.security.ssl.transport.keystore_filepath":"/home/runner/work/security/security/build/resources/test/ssl/node-0-keystore.jks","plugins.security.ssl.transport.keystore_keypassword":"changeit","plugins.security.ssl.transport.resolve_hostname":"false","plugins.security.ssl.transport.truststore_filepath":"/home/runner/work/security/security/build/resources/test/ssl/truststore.jks","plugins.security.ssl_only":"true","transport.tcp.port":"7130","transport.type":"org.opensearch.security.ssl.http.netty.SecuritySSLNettyTransport","transport.type.default":"netty4"}
[2022-10-24T13:42:36,029][WARN ][org.opensearch.node.Node] version [3.0.0-SNAPSHOT] is a pre-release version of OpenSearch and is not suitable for production
[2022-10-24T13:42:36,030][WARN ][org.opensearch.security.ssl.util.SSLCertificateHelper] Certificate chain for alias node-0 contains a root certificate
[2022-10-24T13:42:36,094][WARN ][org.opensearch.security.ssl.util.SSLCertificateHelper] Certificate chain for alias node-0 contains a root certificate
[2022-10-24T13:42:36,121][WARN ][org.opensearch.security.OpenSearchSecurityPlugin] OpenSearch Security plugin run in ssl only mode. No authentication or authorization is performed
[2022-10-24T13:42:36,145][WARN ][org.opensearch.gateway.DanglingIndicesState] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
{"client.type":"node","cluster.initial_cluster_manager_nodes":["127.0.0.1:7130"],"cluster.name":"utest_n287_fnull_t1691768839757","cluster.routing.allocation.disk.threshold_enabled":"false","discovery.initial_state_timeout":"8s","discovery.seed_hosts":["127.0.0.1:7130"],"http.compression":"false","http.cors.enabled":"true","http.port":"9470","http.type":"org.opensearch.security.ssl.http.netty.SecuritySSLNettyHttpServerTransport","http.type.default":"netty4","node.attr.shard_indexing_pressure_enabled":"true","node.max_local_storage_nodes":"3","node.name":"node_utest_n287_fnull_t1691768839757_num2","node.roles":["data"],"path.data":["/home/runner/work/security/security/build/testrun/test/target/data/utest_n287_fnull_t1691768839757/data"],"path.home":"/home/runner/work/security/security/build/testrun/test/target","path.logs":"/home/runner/work/security/security/build/testrun/test/target/data/utest_n287_fnull_t1691768839757/logs","plugins.security.ssl.http.clientauth_mode":"REQUIRE","plugins.security.ssl.http.enable_openssl_if_available":"true","plugins.security.ssl.http.enabled":"true","plugins.security.ssl.http.keystore_alias":"node-0","plugins.security.ssl.http.keystore_filepath":"/home/runner/work/security/security/build/resources/test/ssl/node-0-keystore.jks","plugins.security.ssl.http.keystore_keypassword":"changeit","plugins.security.ssl.http.truststore_filepath":"/home/runner/work/security/security/build/resources/test/ssl/truststore.jks","plugins.security.ssl.transport.enable_openssl_if_available":"true","plugins.security.ssl.transport.enabled":"true","plugins.security.ssl.transport.enforce_hostname_verification":"false","plugins.security.ssl.transport.keystore_alias":"node-0","plugins.security.ssl.transport.keystore_filepath":"/home/runner/work/security/security/build/resources/test/ssl/node-0-keystore.jks","plugins.security.ssl.transport.keystore_keypassword":"changeit","plugins.security.ssl.transport.resolve_hostname":"false","plugins.security.ssl.transport.truststore_filepath":"/home/runner/work/security/security/build/resources/test/ssl/truststore.jks","plugins.security.ssl_only":"true","transport.tcp.port":"8115","transport.type":"org.opensearch.security.ssl.http.netty.SecuritySSLNettyTransport","transport.type.default":"netty4"}
[2022-10-24T13:42:36,154][WARN ][org.opensearch.node.Node] version [3.0.0-SNAPSHOT] is a pre-release version of OpenSearch and is not suitable for production
[2022-10-24T13:42:36,155][WARN ][org.opensearch.security.ssl.util.SSLCertificateHelper] Certificate chain for alias node-0 contains a root certificate
[2022-10-24T13:42:36,221][WARN ][org.opensearch.security.ssl.util.SSLCertificateHelper] Certificate chain for alias node-0 contains a root certificate
[2022-10-24T13:42:36,247][WARN ][org.opensearch.security.OpenSearchSecurityPlugin] OpenSearch Security plugin run in ssl only mode. No authentication or authorization is performed
[2022-10-24T13:42:36,265][WARN ][org.opensearch.gateway.DanglingIndicesState] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
{"client.type":"node","cluster.initial_cluster_manager_nodes":["127.0.0.1:7130"],"cluster.name":"utest_n287_fnull_t1691768839757","cluster.routing.allocation.disk.threshold_enabled":"false","discovery.initial_state_timeout":"8s","discovery.seed_hosts":["127.0.0.1:7130"],"http.compression":"false","http.cors.enabled":"true","http.port":"9615","http.type":"org.opensearch.security.ssl.http.netty.SecuritySSLNettyHttpServerTransport","http.type.default":"netty4","node.attr.shard_indexing_pressure_enabled":"true","node.max_local_storage_nodes":"3","node.name":"node_utest_n287_fnull_t1691768839757_num1","node.roles":["data"],"path.data":["/home/runner/work/security/security/build/testrun/test/target/data/utest_n287_fnull_t1691768839757/data"],"path.home":"/home/runner/work/security/security/build/testrun/test/target","path.logs":"/home/runner/work/security/security/build/testrun/test/target/data/utest_n287_fnull_t1691768839757/logs","plugins.security.ssl.http.clientauth_mode":"REQUIRE","plugins.security.ssl.http.enable_openssl_if_available":"true","plugins.security.ssl.http.enabled":"true","plugins.security.ssl.http.keystore_alias":"node-0","plugins.security.ssl.http.keystore_filepath":"/home/runner/work/security/security/build/resources/test/ssl/node-0-keystore.jks","plugins.security.ssl.http.keystore_keypassword":"changeit","plugins.security.ssl.http.truststore_filepath":"/home/runner/work/security/security/build/resources/test/ssl/truststore.jks","plugins.security.ssl.transport.enable_openssl_if_available":"true","plugins.security.ssl.transport.enabled":"true","plugins.security.ssl.transport.enforce_hostname_verification":"false","plugins.security.ssl.transport.keystore_alias":"node-0","plugins.security.ssl.transport.keystore_filepath":"/home/runner/work/security/security/build/resources/test/ssl/node-0-keystore.jks","plugins.security.ssl.transport.keystore_keypassword":"changeit","plugins.security.ssl.transport.resolve_hostname":"false","plugins.security.ssl.transport.truststore_filepath":"/home/runner/work/security/security/build/resources/test/ssl/truststore.jks","plugins.security.ssl_only":"true","transport.tcp.port":"9070","transport.type":"org.opensearch.security.ssl.http.netty.SecuritySSLNettyTransport","transport.type.default":"netty4"}
{
  "principal" : "CN=node-0.example.com,OU=SSL,O=Test,L=Test,C=DE",
  "peer_certificates" : "3",
  "ssl_protocol" : "TLSv1.2",
  "ssl_cipher" : "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384",
  "ssl_openssl_available" : false,
  "ssl_openssl_version" : -1,
  "ssl_openssl_version_string" : null,
  "ssl_openssl_non_available_cause" : "java.lang.ClassNotFoundException: io.netty.internal.tcnative.SSLContext",
  "ssl_openssl_supports_key_manager_factory" : false,
  "ssl_openssl_supports_hostname_validation" : false,
  "ssl_provider_http" : "JDK",
  "ssl_provider_transport_server" : "JDK",
  "ssl_provider_transport_client" : "JDK"
}

These 2 PRs may be related: #422 and #1649 - since tcnative is not available on the classpath at runtime it will pick the built in JDK provider.

I believe the test is working because this block will return the JDK provider instead of the OpenSSL provider so that cluster is still able to setup SSL: https://github.com/opensearch-project/security/blob/main/src/main/java/org/opensearch/security/ssl/DefaultSecurityKeyStore.java#L161-L169

There is no specific assertion in the test to ensure it was brought up with the OpenSSL provider.

[cwperks]

[Triage] @reta Looks like the OpenSSL feature is not currently working and the tests do not capture this. Would you mind taking a look?