Introduce Kerberos/keytab support

[gczuczy]

In kerberized environments the distribution of keytabs is a very essential task, and it's basically part of the configuration phase of a host. Currently saltstack does not have a mechanism aimed for distributing keytabs. It's not only a binary file, but it has specifics on how to manage them.

In such an environment, a keytab is basically a list of principals (something like security identifiers), which hold authentication information. A keytab is utilized for a service or something to acquire kerberos tickets from the KDC. Many services also require a keytab with service principals in order to be able to authenticate clients using kerberos tickets, for an example, ssh, or kerberized web services or proxies.

A keytab consists of a list of principals. The validity period of a keytab is bound to two things:

• When a principal's key (password) is changed on the KDC, all the previous generated keytabs' such entry will be rendered invalid
• If a keytab is regenerated with an increased KVNO field (which is incremented by every generation per principal) for a principal, then the previously generated keytabs will be invalidated by this.

Therefore during an update process it is very important to not to regenerate a keytab then compare the resulting file's checksum to the deployed one's, because that simply invalidates the deployed keytab, resulting in a service outage.

As mentioned above, a keytab is a list of one or more principals, which can be used for authentication. Using the MIT or heimadal implementation, keytabs can be generated by contacting the KDC with sufficient privileges (privs are defined on the KDC) as a service user. On Microsoft platform a tool like msktutil can be used to generate keytabs, these two ways slightly differ, but do the same task in this regard.

I could imagine a state definition for generating a keytab something like the following:

state-name:
  keytab:
    - managed
    - name: /etc/krb5.keytab
    - require:
      - file: /etc/krb5.conf
      - file: /etc/resolv.conf
      - service: ntp
    - principals:
      - host/$hostname@REALM
      - HTTP/somewebservicename@REALM
    - kvno: 1
    - enctypes:
      - aes256-cts-hmac-sha1-96
      - aes128-cts-hmac-sha1-96

This fields kvno and enctypes are optional ones.

The "name" field states the location of the file. A host might have multiple keytabs for similar principals Keytabs might also be derived from another one, like generating an /etc/krb5.keytab with all the principals the host will need, then from that picking a service principal (SPN) that (for example) apache will need to authenticate, then putting that separately into an apache-accessable directory, with different ownership. Later will detail how this is done.

The require fields are there to show, it needs the kerberos client and the name service to be configured, along with a functioning time synchronization.

The "principals" is a list of principals which would be deployed to the given keytab on the minion.

kvno is optional. If specified, that kvno will be used when generating a keytab. If unspecified then the kadmind will take care of the auto-incrementation.

enctypes are also optional. If specified, then the list will be used, otherwise, kadmin can deal with the defaults.

The generation of keytabs is done on a secure machine, and as a service, with credentials sufficient to do the task. It's prefered to generate the keytabs on the master side, because it needs high privileges on the KDC. Usually this is done by putting a keytab on the master with a principal that has sufficient access, and then the master can use that to authenticate against kadmind to do the task. From the CLI interface such a thing looks like the following:

kadmin -k -t $masterkeytab -p $masterprincipal -q "ktadd -k $minionkeytab $principal_list"

If the minion's keytab has many principals, including ones for services, then once the keytab is shipped to the minon, it can be picked out with the ktutil utility on the minion side, and a couple of principals be put into services' directories, with such access privileges on the filesystem.

When updating keytabs, the minion should read the keytab locally and compare it to the list of specified principals, including the kvno for each. If either there's a change in the list of principals, or any of the principals kvno is changed than the one on the KDC, then the keytab has to be regenerated.

Principals can be queried from the kadmind, and the kvno can be checked on each. Deployed keytabs can be checked on the client side.

The kadmin python module has some of the required functionality, however as I see it's not yet capable of handling keytabs, though the author seems to be helpful to add missing features. So that might be a lot of help.

Probably the deployed keytab's details can be added to grains, it just has to be ensured, that these grains are refreshed after updating/deploying a keytab. Then from the master side, a pillar or something alike can take care of the keytab generation, since it's a highly confidential data. For shipping it, base64/128 can be used

So far that's how I see this, I hope I've explained the issue and the mechanism clearly. If there are any open questions regarding the importance, technical details, or anything, please let me know.

I know not all of the environments are kerberized, but once it is, having your config management system support keytabs is like hitting a jackpot.

[whiteinge]

Good writeup. This would be a neat feature.

A good portion of this could be achieved via the new base64 additions and Salt's event bus. This might make for a cool formula...

[gczuczy]

Thanks. However, I've realized I forgot something.

In same cases, not all principals can be put into keytabs at once, because they are being utilized elsewhere. Examples here are clustered services, like AFS. For an example, all AFS cell member servers need to have the afs/$cellname@REALM principal, so if generating for each individualy, that would invalidate the ones generated earlier. In this case, a keytab has to be generated with such a principal, then distributed into all the minions where it's needed.

It's sometimes getting tricky.

Probably a separate state for generating a keytab with such a principal, then there could be support for managing principals in keytabs on the minions. As this, I mean, reading principals from one or more keytab in a state, then putting them in another one. Something like:

joinedkeytab:
  keytab:
    - managed
    - sources:
      - /etc/krb5.conf
        - host/hostname@REALM
      - /etc/service1.keytab
        - HTTP/webhostname@REALM
    - name: /etc/apache/apache.keytab
...

Along with the usual require, modesetting, watch_in, etc, directives. This could be both utilized for picking a couple of principals from the main (/etc/krb5.keytab) for a service, or picking some from multiple keytabs (main, cluster-service, etc) and making sure that the service actually has its keytab properly set up.

[dmyerscough]

@gczuczy I create a Salt module for kerberos, I will create a Salt State soon too, but if you want to give it a shot and try break it check the PR: #20543 👍

[arthurzenika]

great feature request (content and form), we're interested by this too and can provide testing.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[arthurzenika]

This might be related to #20680 which has been noted as being still relevant. I would argue for keeping it open.

[stale]

Thank you for updating this issue. It is no longer marked as stale.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[stale]

Thank you for updating this issue. It is no longer marked as stale.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.