Connecting to Aiven Kafka over TLS Throws an `java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty` Error #1115

vordimous · 2024-06-28T19:48:27Z

Describe the bug
Trying to use zilla with Aiven Kafka I am unable to connect.

To Reproduce
Steps to reproduce the behavior:

Follow the instructions on connecting Zilla to Aiven and the Aiven Quick connect instructions.
try to start zilla and see the error

Zilla Environment:
Describe the Host environment including:

Zilla start command

args: ["start", "-v", "-e", "-Pzilla.engine.worker.capacity=512"]

Environment variables

env:
  ZILLA_INCUBATOR_ENABLED: "true"
  JAVA_OPTIONS: "-Dzilla.binding.kafka.debug=true -Djavax.net.debug=ssl,handshake"

Attach the zilla.yaml config file:

name: http-quickstart
vaults:
  client_vault:
    type: filesystem
    options:
      trust:
        store: /ssl/client.truststore.p12
        type: pkcs12
        password: ${{env.TRUSTSTORE_PASSWORD}}
      keys:
        store: /ssl/client.keystore.p12
        type: pkcs12
        password: ${{env.KEYSTORE_PASSWORD}}
bindings:
  # Proxy service entrypoint
  north_tcp_server:
    type: tcp
    kind: server
    options:
      host: 0.0.0.0
      port:
        - 7114
    routes:
      - when:
          - port: 7114
        exit: north_http_server
    telemetry:
      metrics:
        - stream.*
  north_http_server:
    type: http
    kind: server
    options:
      access-control:
        policy: cross-origin
    routes:
      - when:
          - headers:
              :scheme: http
              :authority: localhost:7114
              :path: /api/*
        exit: north_rest_api_http_kafka_mapping
  # REST proxy endpoints to Kafka a topic
  north_rest_api_http_kafka_mapping:
    type: http-kafka
    kind: proxy
    routes:
      - when:
          - method: GET
            path: /api/items
        exit: north_kafka_cache_client
        with:
          capability: fetch
          topic: items-crud
          merge:
            content-type: application/json
  # Kafka sync layer
  north_kafka_cache_client:
    type: kafka
    kind: cache_client
    exit: south_kafka_cache_server
  south_kafka_cache_server:
    type: kafka
    kind: cache_server
    options:
      bootstrap:
        - items-crud
    exit: south_kafka_client
  south_kafka_client:
    type: kafka
    kind: client
    options:
      servers:
        - ${{env.KAFKA_BOOTSTRAP_SERVER}}
    exit: south_tls_client
  south_tls_client:
    type: tls
    kind: client
    options:
      trust:
        - ca
      keys:
        - service_key
    exit: south_tcp_client
  south_tcp_client:
    type: tcp
    kind: client
telemetry:
  exporters:
    # Enable Standard Out logs
    stdout_logs_exporter:
      type: stdout

Attach the zilla logs:

started
[http-quickstart.south_kafka_cache_server] items-crud DESCRIBE connect
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.913 GMT|ServerHello.java:883|Consuming ServerHello handshake message (
"ServerHello": {
  "server version"      : "TLSv1.2",
  "random"              : "5C0B0FB218C89A8A3C4CE2474236447C87C2ECF94C2FC902566FED0C7A403FC5",
  "session id"          : "4D9A1F4349574653F95591B103916D08B19A6B8C136EECBA2B67AC5C8C3C1BCA",
  "cipher suite"        : "TLS_AES_256_GCM_SHA384(0x1302)",
  "compression methods" : "00",
  "extensions"          : [
    "supported_versions (43)": {
      "selected version": [TLSv1.3]
    },
    "key_share (51)": {
      "server_share": {
        "named group": x25519
        "key_exchange": {
          0000: 2D 0D 0A 05 6E 2F 93 9C   ED 94 55 91 24 C6 43 09  -...n/....U.$.C.
          0010: FC F1 3F 5F 14 50 6F 49   10 5E 19 21 C1 C1 1F 5D  ..?_.PoI.^.!...]
        }
      },
    }
  ]
}
)
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.913 GMT|SSLExtensions.java:204|Consumed extension: supported_versions
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.913 GMT|ServerHello.java:979|Negotiated protocol version: TLSv1.3
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.914 GMT|SSLExtensions.java:175|Ignore unsupported extension: server_name
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.915 GMT|SSLExtensions.java:175|Ignore unsupported extension: max_fragment_length
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.915 GMT|SSLExtensions.java:175|Ignore unsupported extension: status_request
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.915 GMT|SSLExtensions.java:175|Ignore unsupported extension: ec_point_formats
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.915 GMT|SSLExtensions.java:175|Ignore unsupported extension: application_layer_protocol_negotiation
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.915 GMT|SSLExtensions.java:175|Ignore unsupported extension: status_request_v2
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.916 GMT|SSLExtensions.java:175|Ignore unsupported extension: extended_master_secret
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.916 GMT|SSLExtensions.java:175|Ignore unsupported extension: session_ticket
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.916 GMT|SSLExtensions.java:204|Consumed extension: supported_versions
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.917 GMT|SSLExtensions.java:204|Consumed extension: key_share
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.917 GMT|SSLExtensions.java:175|Ignore unsupported extension: renegotiation_info
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.917 GMT|PreSharedKeyExtension.java:922|Handling pre_shared_key absence.
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.917 GMT|SSLExtensions.java:219|Ignore unavailable extension: server_name
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.917 GMT|SSLExtensions.java:219|Ignore unavailable extension: max_fragment_length
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.917 GMT|SSLExtensions.java:219|Ignore unavailable extension: status_request
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.918 GMT|SSLExtensions.java:219|Ignore unavailable extension: ec_point_formats
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.918 GMT|SSLExtensions.java:219|Ignore unavailable extension: application_layer_protocol_negotiation
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.918 GMT|SSLExtensions.java:219|Ignore unavailable extension: status_request_v2
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.918 GMT|SSLExtensions.java:219|Ignore unavailable extension: extended_master_secret
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.918 GMT|SSLExtensions.java:219|Ignore unavailable extension: session_ticket
javax.net.ssl|WARNING|32|engine/task#0|2024-06-28 18:56:19.918 GMT|SSLExtensions.java:227|Ignore impact of unsupported extension: supported_versions
javax.net.ssl|WARNING|32|engine/task#0|2024-06-28 18:56:19.918 GMT|SSLExtensions.java:227|Ignore impact of unsupported extension: key_share
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.918 GMT|SSLExtensions.java:219|Ignore unavailable extension: renegotiation_info
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.918 GMT|SSLExtensions.java:219|Ignore unavailable extension: pre_shared_key
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.923 GMT|SSLCipher.java:1836|KeyLimit read side: algorithm = AES/GCM/NoPadding:KEYUPDATE
countdown value = 137438953472
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.924 GMT|SSLCipher.java:1987|KeyLimit write side: algorithm = AES/GCM/NoPadding:KEYUPDATE
countdown value = 137438953472
javax.net.ssl|DEBUG|B1|engine/data#3|2024-06-28 18:56:19.925 GMT|ChangeCipherSpec.java:244|Consuming ChangeCipherSpec message
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.928 GMT|EncryptedExtensions.java:172|Consuming EncryptedExtensions handshake message (
"EncryptedExtensions": [
  "supported_groups (10)": {
    "named groups": [x25519, secp256r1, secp384r1, secp521r1, x448, ffdhe2048, ffdhe3072, ffdhe4096, ffdhe6144, ffdhe8192]
  }
]
)
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.929 GMT|SSLExtensions.java:185|Ignore unavailable extension: server_name
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.929 GMT|SSLExtensions.java:185|Ignore unavailable extension: max_fragment_length
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.929 GMT|SSLExtensions.java:204|Consumed extension: supported_groups
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.929 GMT|SSLExtensions.java:219|Ignore unavailable extension: server_name
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.929 GMT|SSLExtensions.java:219|Ignore unavailable extension: max_fragment_length
javax.net.ssl|WARNING|32|engine/task#0|2024-06-28 18:56:19.929 GMT|SSLExtensions.java:227|Ignore impact of unsupported extension: supported_groups
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.929 GMT|SSLExtensions.java:219|Ignore unavailable extension: application_layer_protocol_negotiation
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.934 GMT|CertificateMessage.java:1143|Consuming server Certificate handshake message (
"Certificate": {
  "certificate_request_context": "",
  "certificate_list": [  
  {
    "certificate" : {
      "version"            : "v3",
      "serial number"      : "2AA761AA17604BD87F9FB5F30D9D32FE35893F0C",
      "signature algorithm": "SHA384withRSA",
      "issuer"             : "CN=8090602f-c2ac-493e-9a2b-a64639720a5b Project CA",
      "not before"         : "2024-06-27 13:57:41.000 GMT",
      "not  after"         : "2026-09-25 13:57:41.000 GMT",
      "subject"            : "CN=kafka-zilla-quickstart-4, O=8090602f-c2ac-493e-9a2b-a64639720a5b, ST=service",
      "subject public key" : "RSA",
      "extensions"         : [
        {
          ObjectId: 2.5.29.35 Criticality=false
          AuthorityKeyIdentifier [
          KeyIdentifier [
          0000: CD 6F A4 50 AD 16 89 3D   5D 17 0F 4C 06 5B 38 DE  .o.P...=]..L.[8.
          0010: 44 57 9D 8E                                        DW..
          ]
          ]
        },
        {
          ObjectId: 2.5.29.19 Criticality=false
          BasicConstraints:[
            CA:false
            PathLen: undefined
          ]
        },
        {
          ObjectId: 2.5.29.15 Criticality=false
          KeyUsage [
            DigitalSignature
            Key_Encipherment
          ]
        },
        {
          ObjectId: 2.5.29.17 Criticality=false
          SubjectAlternativeName [
            DNSName: 10.162.0.71
            DNSName: public-kafka-zilla-quickstart-zilla-quickstart.l.aivencloud.com
            DNSName: public-n-kafka-zilla-quickstart-4.l.aivencloud.com
            IPAddress: fda7:a938:5bfe:5fa6:0:4c2:5e57:1c42
            IPAddress: 34.47.20.235
            DNSName: kafka-zilla-quickstart-zilla-quickstart.l.aivencloud.com
            IPAddress: 2600:1900:40e0:504:0:47:0:0
            DNSName: 34.47.20.235
            DNSName: n-kafka-zilla-quickstart-4.l.aivencloud.com
            DNSName: fda7:a938:5bfe:5fa6:0:4c2:5e57:1c42
            IPAddress: 10.162.0.71
            DNSName: *.l.aivencloud.com
            DNSName: 2600:1900:40e0:504:0:47::
          ]
        },
        {
          ObjectId: 2.5.29.14 Criticality=false
          SubjectKeyIdentifier [
          KeyIdentifier [
          0000: 0F 28 84 1A C3 4A 42 95   0F 2E 6A 83 C8 79 3E D0  .(...JB...j..y>.
          0010: 61 9C 00 7E                                        a...
          ]
          ]
        }
      ]}
    "extensions": {
      <no extension>
    }
  },
  {
    "certificate" : {
      "version"            : "v3",
      "serial number"      : "0DD54456AF9909A495C584F56169D76C3DC2D319",
      "signature algorithm": "SHA384withRSA",
      "issuer"             : "CN=8090602f-c2ac-493e-9a2b-a64639720a5b Project CA",
      "not before"         : "2024-06-26 21:17:00.000 GMT",
      "not  after"         : "2034-06-24 21:17:00.000 GMT",
      "subject"            : "CN=8090602f-c2ac-493e-9a2b-a64639720a5b Project CA",
      "subject public key" : "RSA",
      "extensions"         : [
        {
          ObjectId: 2.5.29.19 Criticality=false
          BasicConstraints:[
            CA:true
            PathLen:0
          ]
        },
        {
          ObjectId: 2.5.29.15 Criticality=false
          KeyUsage [
            Key_CertSign
            Crl_Sign
          ]
        },
        {
          ObjectId: 2.5.29.14 Criticality=false
          SubjectKeyIdentifier [
          KeyIdentifier [
          0000: CD 6F A4 50 AD 16 89 3D   5D 17 0F 4C 06 5B 38 DE  .o.P...=]..L.[8.
          0010: 44 57 9D 8E                                        DW..
          ]
          ]
        }
      ]}
    "extensions": {
      <no extension>
    }
  },
]
}
)
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.934 GMT|SSLExtensions.java:185|Ignore unavailable extension: status_request
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.934 GMT|SSLExtensions.java:185|Ignore unavailable extension: status_request
javax.net.ssl|ERROR|32|engine/task#0|2024-06-28 18:56:19.936 GMT|TransportContext.java:370|Fatal (INTERNAL_ERROR): Unhandled exception (
"throwable" : {
  java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty
    at java.base/sun.security.validator.PKIXValidator.<init>(PKIXValidator.java:97)
    at java.base/sun.security.validator.Validator.getInstance(Validator.java:173)
    at java.base/sun.security.ssl.X509TrustManagerImpl.getValidator(X509TrustManagerImpl.java:308)
    at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrustedInit(X509TrustManagerImpl.java:183)
    at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:254)
    at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144)
    at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1304)
    at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1203)
    at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1146)
    at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:393)
    at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:476)
    at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1274)
    at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1260)
    at java.base/java.security.AccessController.doPrivileged(AccessController.java:714)
    at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1205)
    at io.aklivity.zilla.runtime.engine@0.9.83/io.aklivity.zilla.runtime.engine.internal.registry.EngineWorker$EngineSignaler.invokeAndSignal(EngineWorker.java:2063)
    at io.aklivity.zilla.runtime.engine@0.9.83/io.aklivity.zilla.runtime.engine.internal.registry.EngineWorker$EngineSignaler.lambda$signalTask$3(EngineWorker.java:1983)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
    at java.base/java.lang.Thread.run(Thread.java:1570)
  Caused by: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty
    at java.base/java.security.cert.PKIXParameters.setTrustAnchors(PKIXParameters.java:200)
    at java.base/java.security.cert.PKIXParameters.<init>(PKIXParameters.java:120)
    at java.base/java.security.cert.PKIXBuilderParameters.<init>(PKIXBuilderParameters.java:104)
    at java.base/sun.security.validator.PKIXValidator.<init>(PKIXValidator.java:94)
    ... 21 more}

)

Kafka Environment:

Provider: Aiven
Version: 3.7
Config: Client cert

Client Environment:

the /opt/bitnami/kafka/bin/kafka-topics.sh --list command works with the below SSL configs:

security.protocol=SSL
ssl.protocol=TLS
ssl.keystore.type=PKCS12
ssl.keystore.location=client.keystore.p12
ssl.keystore.password=pass
ssl.key.password=pass
ssl.truststore.location=client.truststore.jks
ssl.truststore.password=pass
ssl.truststore.type=JKS

Additional context
Aiven Quick connect instructions:
3. Create keystore and truststore:
3.1 Use the openssl utility to create the keystore with the service.key and service.cert files downloaded previously:

openssl pkcs12 -export -inkey service.key -in service.cert -out client.keystore.p12 -name service_key

Enter a password to protect the keystore and the key, when prompted.
3.2 In the folder where the certificates are stored, use the keytool utility to create the truststore with the ca.pem file as input:

keytool -import -file ca.pem -alias CA -keystore client.truststore.jks

Enter a password to protect the trust stores and reply yes to confirm trusting the CA certificate, when prompted.
The result are the keystore named client.keystore.p12 and truststore named client.truststore.jks that can be used for client applications configuration in the next steps.

The text was updated successfully, but these errors were encountered:

ankitk-me · 2024-07-18T09:33:28Z

Update:

I have tested working of Zilla with Aiven Kafka over TLS and it's working as expected. I'm able to produce & consume messages from the Kafka topic (http.kafka). Able to validate the same using java client.

Based on my understanding the issue with shared zilla.yaml config is that vault is not referred at south_tls_client.

Expected config:

south_tls_client:
    type: tls
    kind: client
    vault: client_vault
    options:
      trust:
        - ca
      keys:
        - service_key

vordimous · 2024-07-18T13:49:50Z

@aDaemonThread can we capture this situation and add a log event?

vordimous added the bug Something isn't working label Jun 28, 2024

jfallows assigned ankitk-me Jul 16, 2024

ankitk-me mentioned this issue Jul 18, 2024

tls binding require vault and keys or signers in options #1159

Merged

vordimous mentioned this issue Jul 22, 2024

Add vault ref for mTLS aklivity/zilla-docs#243

Closed

jfallows closed this as completed in #1159 Jul 24, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Connecting to Aiven Kafka over TLS Throws an `java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty` Error #1115

Connecting to Aiven Kafka over TLS Throws an `java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty` Error #1115

vordimous commented Jun 28, 2024 •

edited

Loading

ankitk-me commented Jul 18, 2024

vordimous commented Jul 18, 2024

Connecting to Aiven Kafka over TLS Throws an java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty Error #1115

Connecting to Aiven Kafka over TLS Throws an java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty Error #1115

Comments

vordimous commented Jun 28, 2024 • edited Loading

ankitk-me commented Jul 18, 2024

vordimous commented Jul 18, 2024

Connecting to Aiven Kafka over TLS Throws an `java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty` Error #1115

Connecting to Aiven Kafka over TLS Throws an `java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty` Error #1115

vordimous commented Jun 28, 2024 •

edited

Loading