Shutdown issue with JMX Auth

[rs017991]

After enabling JMX Auth, the rhel init.d service is no longer able to perform its graceful shutdown commands via nodetool.

It seems like it should be changed to use the credentials defined in node[:cassandra][:jmx]

[xxxxxx@xxxxxxxxxxx ~]$ sudo service cassandra stop

executing nodetool disablethrift ..
nodetool: Failed to connect to 'xx.xxx.xxx.xxx:7199' - SecurityException: 'Authentication failed! Credentials required'.
failed to execute disablethrift ..
executing nodetool disablegossip ..
nodetool: Failed to connect to 'xx.xxx.xxx.xxx:7199' - SecurityException: 'Authentication failed! Credentials required'.
failed to execute disablegossip ..
executing nodetool drain ..
nodetool: Failed to connect to 'xx.xxx.xxx.xxx:7199' - SecurityException: 'Authentication failed! Credentials required'.
failed to execute drain ..
Shutting down cassandra .. [ OK ]

[michaelklishin]

If by "it" in "it should be changed" you mean nodetool, then it makes sense.

[rs017991]

Thanks for the quick response, Michael.

I suppose that I assumed 'it' would be the /etc/init.d/cassandra file which would be changed to include the credentials when it invokes nodetool.

Are you suggesting that the cookbook should instead alter $CASSANDRA_HOME/bin/nodetool file to use those credentials by default?

That would certainly be more convenient(even outside of the context of this issue), though that would have the effect of defeating auth for all local usage of nodetool(which others may have a problem with?)

[cmlicata]

@rs017991 is this issue occurring on Rhel 7 or 6? I have this enabled in my prod cluster and have implemented both local JMX and remote JMX auth in all my environments and I have never run into this issue. Do you know the steps to replicate the problem?

[michaelklishin]

CentOS and RHEL 7 use systemd, which means many if not all things configured via /etc/init.d/cassandra will no longer have effect.

[rs017991]

@cmlicata We're on RHEL 6.

I went ahead and reproduced the issue using this cookbook's test kitchen (just to make sure I'm not crazy).

1. Change .kitchen.yml according to this diff
2. In ChefDK: kitchen converge tarball-centos-68 ; kitchen login tarball-centos-68
3. Once logged in: sudo service cassandra restart

This produces the same output as my original post.