Add kerberos support #1366

rubenvp8510 · 2019-04-29T21:22:49Z

Signed-off-by: Ruben Vargas ruben.vp8510@gmail.com

This is a pure go solution.

mieczkowski · 2019-04-30T07:43:46Z

Hi. I've just created my solution for kerberos auth, but I will be happy to test your solution (native is better than libsasl2 <3 ).

mieczkowski

Nice work! It works. Dynamic SPN (for kafka cluster) should be supported. And of course error handling :) I will cancel my PR and I'm waiting for this to merge!

gssapi_kerberos.go

rubenvp8510 · 2019-05-02T14:44:18Z

@mieczkowski I did some changes, also handled errors, I'm starting in go and kafka, so accept suggestions WDYT?

mieczkowski

Looks good (and it works with kafka cluster :) ). One missing return, one suggestion about config, and some missing error checks. We should always check and handle errors from external libraries, even if we are pretty sure that it will not happen ;)

mieczkowski · 2019-05-06T07:59:07Z

gssapi_kerberos.go

+	if err != nil {
+		return err
+	}
+	krbAuth.client = krb5client.NewClientWithKeytab(krbAuth.config.Username, krbAuth.config.Realm, kt, cfg)


How about move loading kt and cfg into (c *Config) Validate()? Keytab and KerberosConfig files should be validated in client creation, not in runtime

I understand that could be semantically better but..

After thinking on this, I don't see the difference, now the (c *Config) Validate() happens on the Open method, same as kerberos client creation (where the keytab parsing happens), same place where authentication happens.

Also, If I follow the same logic kerberos config validation, which happens when the kerberos client instance is created, I will need to move the kerberos client instantiation to (c *Config) Validate().

gssapi_kerberos.go

rubenvp8510 · 2019-05-13T01:02:42Z

@mieczkowski @eapache @bai Could you review this one please?

Thanks

rubenvp8510 · 2019-05-22T22:40:24Z

Any updates on this?

Thanks

bai · 2019-05-27T09:17:41Z

Thanks for your contribution!

objectiser · 2019-05-29T08:22:13Z

@bai Is there a release planned soon to include this feature?

bai · 2019-05-29T09:01:38Z

Yp, I'll cut a release in the next few days.

objectiser · 2019-05-29T09:39:53Z

@bai Thanks!

objectiser · 2019-06-18T14:44:07Z

Hi @bai, any update on when a release will be cut?

shriram1993 · 2019-07-04T06:12:09Z

@rubenvp8510 I am trying to use kerberos authetication but getting error "wrong Token ID. Expected 0504, was 6030".
file name: /gssapi_kerberos.go
line no: 181
if err := wrapTokenReq.Unmarshal(bytes, true); err != nil {return nil, err}
Not getting any clue what i am doing wrong?
Please help.

rubenvp8510 · 2019-07-04T19:10:47Z

@shriram1993 Could you please put information about your environment? version of kafka, how is it configured? how Kerberos is configured?

I see similar errors on other libraries that relies on gokrb5 for handling Kerberos authentication messages colinmarc/hdfs#145 , I think it could be something related to encryption.

Thanks.

shriram1993 · 2019-07-05T06:26:55Z

Kafka version: 0.11.03
Kerberos Version: 1.15.1
Kerberos auth type: keytab file based
Kerberos server only supports SASL_PLAINTEXT.

The configuration set in krb5.conf
ticket_lifetime = 1d
renew_lifetime = 7d
forwardable = true
default_tgs_enctypes = rc4-hmac
default_tkt_enctypes = rc4-hmac
permitted_enctypes = rc4-hmac
udp_preference_limit = 1
kdc_timeout = 3000
proxiable = true
dns_lookup_realm = true
dns_lookup_kdc = false
allow_weak_crypto = false

yzpnet · 2019-07-08T07:43:11Z

How to support SASL_PLAINTEXT?

shriram1993 · 2019-07-11T10:02:41Z

@rubenvp8510 I have the same question. My Kerberos server is expecting security protocol 'SASL_PLAINTEXT' but there is no way to set same in configuration. Please suggest.

cfg := sarama.NewConfig()
cfg.Producer.Return.Successes = true
cfg.Metadata.Retry.Max = 0
cfg.Net.SASL.Enable = true
cfg.Net.SASL.Mechanism = sarama.SASLTypeGSSAPI
cfg.Net.SASL.GSSAPI.ServiceName = "kafka"
cfg.Net.SASL.GSSAPI.AuthType = sarama.KRB5_KEYTAB_AUTH
cfg.Net.SASL.GSSAPI.KeyTabPath = "KEY TAB FILE PATH"
cfg.Net.SASL.GSSAPI.KerberosConfigPath = "KRB5 CONFIG PATH"
cfg.Net.SASL.GSSAPI.Username = "PRINCIPAL NAME"
cfg.Net.SASL.GSSAPI.Realm = "REALM"

yzpnet · 2019-07-16T08:19:18Z

@rubenvp8510
I have the same question too.
Kerberos auth need support SASL_PLAINTEXT.

yzpnet · 2019-08-08T06:35:05Z

SASL_PLAINTEXT

rubenvp8510 · 2019-08-09T01:26:36Z

I'm going to start looking at how to implement SASL_PLAINTEXT support .@yzpnet @shriram1993 Could you provide me your kafka configuration? That would help a lot.

I'm a little bit confused, In one comment I read "Kerberos server only supports SASL_PLAINTEXT." but as far as I know the problem is not in the communication with the KDC but with Kafka

I'm assuming that the implementation needs to support Kafka SASL_PLAINTEXT + Kerberos. correct me if I misunderstood please.

Thank you!

shriram1993 · 2019-08-09T10:19:13Z

Kafka config:
properties.setProperty("bootstrap.servers", "kafka-hotels.example.com:9092");
properties.setProperty("key.serializer.class", "kafka.serializer.StringEncoder");
properties.setProperty("serializer.class", "kafka.serializer.DefaultEncoder");
properties.setProperty("request.required.acks", "1");
properties.setProperty("retries", "3");
properties.setProperty("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
properties.setProperty("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
properties.setProperty("compression.type", "snappy");
properties.setProperty("request.timeout.ms", "300000");
properties.setProperty("metadata.fetch.timeout.ms", "100000");
properties.setProperty("security.protocol", "SASL_PLAINTEXT");
properties.setProperty("sasl.kerberos.service.name", "kafka");
System.setProperty("java.security.krb5.conf", "/etc/kerberos/conf/krb5.conf");
System.setProperty("java.security.auth.login.config", "path of jaas-cache.conf");

jaas-cache.conf:
KafkaClient {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="keytab.keytab"
storeKey=true
useTicketCache=false
serviceName="kafka"
principal="mirroruser@EXAMPLE.COM";
};

Krb5.conf:
default_realm = EXAMPLE.COM
clockskew = 300
ticket_lifetime = 1d
renew_lifetime = 7d
forwardable = true
default_tgs_enctypes = rc4-hmac
default_tkt_enctypes = rc4-hmac
permitted_enctypes = rc4-hmac
udp_preference_limit = 1
kdc_timeout = 3000
proxiable = true
dns_lookup_realm = true
dns_lookup_kdc = true
allow_weak_crypto = true

shriram1993 · 2019-08-09T10:24:32Z

I'm a little bit confused, In one comment I read "Kerberos server only supports SASL_PLAINTEXT." but as far as I know the problem is not in the communication with the KDC but with Kafka

I'm assuming that the implementation needs to support Kafka SASL_PLAINTEXT + Kerberos. correct me if I misunderstood please.

Thank you!

Yes!! You are right. The issue is not with KDC it is with Kafka.
Kerberos only support SASL_PLAINTEXT means we are using Kerberos + Kafka with this security protocol SASL_PLAINTEXT.

rubenvp8510 · 2019-08-09T22:00:56Z

well, this should work with SASL_PLAINTEXT by default, I'm reviewing what would cause the error you are seeing.

rubenvp8510 · 2019-08-15T18:58:32Z

Hi @shriram1993 @yzpnet

After reviewing this in detail, I was managed to reproduce the issue, it seems like the actual implementation of Kerberos authentication does not support rc4-hmac.

The thing here is that the RFC-4121 (or at least the JAVA implementation) only support the use of aes128-cts-hmac-sha1-96 and aes256-cts-hmac-sha1-96 as a method for encrypt and integrity check for tkt and tgs. For work with rc4-hmac we would need to implement RFC-1964.

So, I would say this should work if you configure your Active Directory[1] to support those encrypt types and your krb5.conf to use it (not sure but I think you need to regenerate your keytab files if you are using). You can also need to disable FAST negotiation, I'll send a PR to expose that option for sarama.

Let me know if this works for you so I can close the corresponding issue. I'll test it and post my configs here.

1)https://blogs.msdn.microsoft.com/openspecification/2011/05/30/windows-configurations-for-kerberos-supported-encryption-type/

rubenvp8510 · 2019-08-15T19:57:38Z

Hi again

Confirmed that the issue is with rc4-hmac

This is my configuration:

[logging]
    default = FILE:/var/log/krb5libs.log
    kdc = FILE:/var/log/krb5kdc.log
    admin_server = FILE:/var/log/kadmind.log

[libdefaults]
    default_realm = EXAMPLE.COM
    clockskew = 300
    ticket_lifetime = 1d
    renew_lifetime = 7d
    forwardable = true
    default_tgs_enctypes = aes256-cts-hmac-sha1-96
    default_tkt_enctypes = aes256-cts-hmac-sha1-96
    permitted_enctypes = aes256-cts-hmac-sha1-96
    udp_preference_limit = 1
    kdc_timeout = 3000
    proxiable = true
    dns_lookup_realm = true
    dns_lookup_kdc = true
    allow_weak_crypto = true

[realms]
EXAMPLE.COM = {
     kdc = 192.168.58.3
     admin_server = 192.168.58.3
}

[domain_realm]
 .kerberoswin.com = EXAMPLE.COM
  kerberoswin.com = EXAMPLE.COM

Followed the guide mentioned in my previous comment to configure AES encrypt types for my user accounts and regenerate keytabs for my services.

I regenerated my keytabs using this command:

ktpass -princ kafka/kafka@EXAMPLE.COM -mapuser EXAMPLE0\kafka -crypto AES256-SHA1 -ptype KRB5_NT_PRINCIPAL -pass kfk_password -out C:\keytabs\kafka.ktab [2]

https://github.com/edenhill/librdkafka/wiki/Using-SASL-with-librdkafka-on-Windows

virtualsafety · 2020-02-02T06:29:48Z

i don't need kerberos auth,how to disable it ,because i do not want download so many third-party library.

ghost added the cla-needed label Apr 29, 2019

rubenvp8510 closed this Apr 29, 2019

rubenvp8510 reopened this Apr 29, 2019

rubenvp8510 force-pushed the kerberos_support branch from dd3b45e to c127e9b Compare April 29, 2019 23:25

ghost removed the cla-needed label Apr 29, 2019

mieczkowski suggested changes Apr 30, 2019

View reviewed changes

gssapi_kerberos.go Outdated Show resolved Hide resolved

mieczkowski mentioned this pull request Apr 30, 2019

Custom auth handler interface #1365

Closed

rubenvp8510 force-pushed the kerberos_support branch 9 times, most recently from 1d89026 to eefdd51 Compare May 1, 2019 01:02

rubenvp8510 changed the title ~~[WIP] Add kerberos support~~ Add kerberos support May 1, 2019

mieczkowski suggested changes May 6, 2019

View reviewed changes

rubenvp8510 force-pushed the kerberos_support branch 10 times, most recently from 0bc673e to 282954a Compare May 8, 2019 19:54

rubenvp8510 force-pushed the kerberos_support branch from 1862b42 to 3ec45e5 Compare May 12, 2019 16:56

bai merged commit 41f00bc into IBM:master May 27, 2019

objectiser mentioned this pull request Jun 6, 2019

Kafka kerberos authentication support for collector/ingester jaegertracing/jaeger#1589

Merged

rubenvp8510 mentioned this pull request Aug 15, 2019

wrong token id using kerberos authentication #1400

Closed

d1egoaz mentioned this pull request Aug 22, 2019

kerberos auth support #1375

Closed

jrevillard mentioned this pull request Nov 7, 2019

Does support the Kerberos based authentication? linkedin/Burrow#424

Open

chriscasano mentioned this pull request Jan 31, 2020

changefeedccl: support kerberos auth for changefeed kafka sink cockroachdb/cockroach#44603

Open

twmb mentioned this pull request Sep 3, 2021

Disable FAST negotiation Kerberos redpanda-data/kminion#111

Closed

donalhunt mentioned this pull request Sep 21, 2021

SASL handshake: wrong Token ID. Expected 0504, was 6030 colinmarc/hdfs#274

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add kerberos support #1366

Add kerberos support #1366

rubenvp8510 commented Apr 29, 2019 •

edited

Loading

mieczkowski commented Apr 30, 2019

mieczkowski left a comment

rubenvp8510 commented May 2, 2019 •

edited

Loading

mieczkowski left a comment

mieczkowski May 6, 2019

rubenvp8510 May 9, 2019

rubenvp8510 commented May 13, 2019

rubenvp8510 commented May 22, 2019

bai commented May 27, 2019

objectiser commented May 29, 2019

bai commented May 29, 2019

objectiser commented May 29, 2019

objectiser commented Jun 18, 2019

shriram1993 commented Jul 4, 2019

rubenvp8510 commented Jul 4, 2019 •

edited

Loading

shriram1993 commented Jul 5, 2019

yzpnet commented Jul 8, 2019

shriram1993 commented Jul 11, 2019 •

edited

Loading

yzpnet commented Jul 16, 2019

yzpnet commented Aug 8, 2019

rubenvp8510 commented Aug 9, 2019 •

edited

Loading

shriram1993 commented Aug 9, 2019

shriram1993 commented Aug 9, 2019

rubenvp8510 commented Aug 9, 2019

rubenvp8510 commented Aug 15, 2019

rubenvp8510 commented Aug 15, 2019 •

edited

Loading

virtualsafety commented Feb 2, 2020

Add kerberos support #1366

Add kerberos support #1366

Conversation

rubenvp8510 commented Apr 29, 2019 • edited Loading

mieczkowski commented Apr 30, 2019

mieczkowski left a comment

Choose a reason for hiding this comment

rubenvp8510 commented May 2, 2019 • edited Loading

mieczkowski left a comment

Choose a reason for hiding this comment

mieczkowski May 6, 2019

Choose a reason for hiding this comment

rubenvp8510 May 9, 2019

Choose a reason for hiding this comment

rubenvp8510 commented May 13, 2019

rubenvp8510 commented May 22, 2019

bai commented May 27, 2019

objectiser commented May 29, 2019

bai commented May 29, 2019

objectiser commented May 29, 2019

objectiser commented Jun 18, 2019

shriram1993 commented Jul 4, 2019

rubenvp8510 commented Jul 4, 2019 • edited Loading

shriram1993 commented Jul 5, 2019

yzpnet commented Jul 8, 2019

shriram1993 commented Jul 11, 2019 • edited Loading

yzpnet commented Jul 16, 2019

yzpnet commented Aug 8, 2019

rubenvp8510 commented Aug 9, 2019 • edited Loading

shriram1993 commented Aug 9, 2019

shriram1993 commented Aug 9, 2019

rubenvp8510 commented Aug 9, 2019

rubenvp8510 commented Aug 15, 2019

rubenvp8510 commented Aug 15, 2019 • edited Loading

virtualsafety commented Feb 2, 2020

rubenvp8510 commented Apr 29, 2019 •

edited

Loading

rubenvp8510 commented May 2, 2019 •

edited

Loading

rubenvp8510 commented Jul 4, 2019 •

edited

Loading

shriram1993 commented Jul 11, 2019 •

edited

Loading

rubenvp8510 commented Aug 9, 2019 •

edited

Loading

rubenvp8510 commented Aug 15, 2019 •

edited

Loading