Authentication failure: (errno: No error)

[galakt]

Description

• After update confluent-kafka-dotnet to v1.2.0 from v1.1.0 (and librdkafka to v1.2.0 from v1.1.0) get Authentication failure: (errno: No error).

• Fail on win10 x64, but works fine on debian

• Different SaslHandshakeRequest versions

Checklist

Please provide the following information:

• librdkafka version (release number or git tag): v1.2.0
• Apache Kafka version: 1.0.1
• librdkafka client configuration: security.protocol=sasl_plaintext, sasl.mechanisms=GSSAPI
• Operating system: windows 10 x64
• Provide logs (with debug=.. as necessary) from librdkafka
• Provide broker log excerpts
• Critical issue

librdkafka log v1.2.0

Selected provider Win32 SSPI for SASL mechanism GSSAPI
librdkafka v1.2.0 (0x10200ff) rdkafka#producer-1 initialized (builtin.features gzip,snappy,ssl,sasl,regex,lz4,sasl_gssapi,sasl_plain,sasl_scram,plugins,zstd,sasl_oauthbearer, SSL ZLIB SNAPPY SASL_SCRAM PLUGINS HDRHISTOGRAM, debug 0x280)
Connected (#1)
Updated enabled protocol features +ApiVersion to ApiVersion
Sent ApiVersionRequest (v0, 25 bytes @ 0, CorrId 1)
Received ApiVersionResponse (v0, 234 bytes, CorrId 1, rtt 7.73ms)
Auth in state APIVERSION_QUERY (handshake supported)
Sent SaslHandshakeRequest (v1, 29 bytes @ 0, CorrId 2)
Received SaslHandshakeResponse (v1, 14 bytes, CorrId 2, rtt 2.42ms)
Broker supported SASL mechanisms: GSSAPI
Auth in state AUTH_HANDSHAKE (handshake supported)
Initializing SASL client: service name kafka, hostname HOSTNAME, mechanisms GSSAPI, provider Win32 SSPI
Acquired Kerberos credentials handle (expiry in 2147483471.-85731329s)
Send SASL Kafka frame to broker (4698 bytes)
Sent SaslAuthenticateRequest (v0, 4723 bytes @ 0, CorrId 3)
Received SaslAuthenticateResponse (v0, 111 bytes, CorrId 3, rtt 13.62ms)
Received SASL frame from broker (103 bytes)
Initialized security context
Send SASL Kafka frame to broker (0 bytes)
Sent SaslAuthenticateRequest (v0, 25 bytes @ 0, CorrId 4)
Received SaslAuthenticateResponse (v0, 58 bytes, CorrId 4, rtt 9.03ms)
Received SASL frame from broker (50 bytes)
Validated server token
Sending response message for user: USER@DOMAIN
Send SASL Kafka frame to broker (71 bytes)
Authenticated
Sent SaslAuthenticateRequest (v0, 96 bytes @ 0, CorrId 5)
Sent MetadataRequest (v2, 25 bytes @ 0, CorrId 6)
Received SaslAuthenticateResponse (v0, 8 bytes, CorrId 5, rtt 5.46ms)
Received SASL frame from broker (0 bytes)
failed: err: Local: Authentication failure: (errno: No error)
SASL authentication error: Decrypt message failed: Invalid token (0x80090308) (after 8ms in state UP)
SASL authentication error: Decrypt message failed: Invalid token (0x80090308) (after 8ms in state UP)
2/2 brokers are down
Metadata request failed: connected: Local: Authentication failure (148783112ms): Permanent
Connected (#2)
Sent ApiVersionRequest (v0, 25 bytes @ 0, CorrId 7)
Received ApiVersionResponse (v0, 234 bytes, CorrId 7, rtt 51.19ms)
Auth in state APIVERSION_QUERY (handshake supported)
Sent SaslHandshakeRequest (v1, 29 bytes @ 0, CorrId 8)
Received SaslHandshakeResponse (v1, 14 bytes, CorrId 8, rtt 0.82ms)
Broker supported SASL mechanisms: GSSAPI
Auth in state AUTH_HANDSHAKE (handshake supported)
Initializing SASL client: service name kafka, hostname HOSTNAME, mechanisms GSSAPI, provider Win32 SSPI
Acquired Kerberos credentials handle (expiry in 2147483471.-85731329s)
Send SASL Kafka frame to broker (4698 bytes)
Sent SaslAuthenticateRequest (v0, 4723 bytes @ 0, CorrId 9)
Received SaslAuthenticateResponse (v0, 111 bytes, CorrId 9, rtt 15.93ms)
Received SASL frame from broker (103 bytes)
Initialized security context
Send SASL Kafka frame to broker (0 bytes)
Sent SaslAuthenticateRequest (v0, 25 bytes @ 0, CorrId 10)
Received SaslAuthenticateResponse (v0, 58 bytes, CorrId 10, rtt 46.22ms)
Received SASL frame from broker (50 bytes)
Validated server token
Sending response message for user: USER@DOMAIN
Send SASL Kafka frame to broker (71 bytes)
Authenticated
Sent SaslAuthenticateRequest (v0, 96 bytes @ 0, CorrId 11)
Sent MetadataRequest (v2, 25 bytes @ 0, CorrId 12)
Received SaslAuthenticateResponse (v0, 8 bytes, CorrId 11, rtt 14.48ms)
Received SASL frame from broker (0 bytes)
failed: err: Local: Authentication failure: (errno: No error)
SASL authentication error: Decrypt message failed: Invalid token (0x80090308) (after 16ms in state UP)
SASL authentication error: Decrypt message failed: Invalid token (0x80090308) (after 16ms in state UP)
2/2 brokers are down
Metadata request failed: connected: Local: Authentication failure (148784150ms): Permanent

librdkafka log v1.1.0

Selected provider Win32 SSPI for SASL mechanism GSSAPI
librdkafka v1.1.0 (0x10100ff) rdkafka#producer-1 initialized (builtin.features gzip,snappy,ssl,sasl,regex,lz4,sasl_gssapi,sasl_plain,sasl_scram,plugins,zstd,sasl_oauthbearer, SSL ZLIB SNAPPY SASL_SCRAM PLUGINS HDRHISTOGRAM, debug 0x280)
Connected (#1)
Updated enabled protocol features +ApiVersion to ApiVersion
Sent ApiVersionRequest (v0, 25 bytes @ 0, CorrId 1)
Received ApiVersionResponse (v0, 234 bytes, CorrId 1, rtt 2.51ms)
Auth in state APIVERSION_QUERY (handshake supported)
Sent SaslHandshakeRequest (v0, 29 bytes @ 0, CorrId 2)
Received SaslHandshakeResponse (v0, 14 bytes, CorrId 2, rtt 0.69ms)
Broker supported SASL mechanisms: GSSAPI
Auth in state AUTH_HANDSHAKE (handshake supported)
Initializing SASL client: service name kafka, hostname HOSTNAME, mechanisms GSSAPI, provider Win32 SSPI
Acquired Kerberos credentials handle (expiry in 2147483471.-85731329s)
Send SASL frame to broker (4686 bytes)
Received SASL frame from broker (107 bytes)
Initialized security context
Send SASL frame to broker (0 bytes)
Received SASL frame from broker (54 bytes)
Validated server token
Sending response message for user: USER@DOMAIN
Send SASL frame to broker (71 bytes)
Authenticated
Sent MetadataRequest (v2, 25 bytes @ 0, CorrId 3)
Received MetadataResponse (v2, 94 bytes, CorrId 3, rtt 40.50ms)

Broker log

INFO Successfully authenticated client: authenticationID=USER@DOMAIN; authorizationID=USER@DOMAIN. (org.apache.kafka.common.security.authenticator.SaslServerCallbackHandler)

[sebwills]

I am also seeing this when testing upgrade from Confluent.Kafka 1.1.0 -> 1.2.0, on Windows.

[yburke94]

We are also repeatedly seeing this error after upgrading to 1.2.0 from 1.1.0:
Error: sasl_ssl://xxxxxx:6668/bootstrap: SASL authentication error: Decrypt message failed: Invalid token (0x80090308) (after 39ms in state UP)

Also using security.protocol=sasl_plaintext, sasl.mechanisms=GSSAPI on windows 10 x64. Running our apps on linux works fine.

@edenhill Do you know of any change that could have broken this?

[edenhill]

It may be the support for KIP-152 (authentication errors) that broke GSSAPI on Windows.
We're looking into it.

[srsrsrsrsrsrsr]

Is there any workaround that would allow one to still use 1.2.0 on Windows?

[edenhill]

There is no workaround, we recommend downgrading to v1.1.0 for the time being.

[edenhill]

Please try the v1.2.1-RC1 release candidate which contains a fix for this issue.

Reporting your findings.

[edenhill]

v1.2.1, which includes the fix, is now released. Please upgrade.