SPIRE Server should issue unique Subject DN

[georgi-lozev]

• Version: N/A
• Platform: N/A
• Subsystem: Server

Reading https://datatracker.ietf.org/doc/html/rfc2459#section-4.1.2.6 we have the following paragraph.

Where it is non-empty, the subject field MUST contain an X.500
distinguished name (DN). The DN MUST be unique for each subject
entity certified by the one CA as defined by the issuer name field. A
CA may issue more than one certificate with the same DN to the same
subject entity.

Currently SPIRE issues all SVIDs using the following Subject which we think is a violation.

Subject: C=US, O=SPIRE

Based on some internal discussions we got the feedback that setting empty subject will be a breaking change for a lot of (legacy) applications, but we're open for discussion.

Slack discussion: https://spiffe.slack.com/archives/CBNCC2V17/p1652854887619899

Brainstorming below

Our current way of thinking to find a good middle ground is to introduce a configurable subject. Similar to what SPIRE Server has today for the ca_subject and with sane defaults that will not make it breaking change for exiting SPIRE servers.
In addition having the possibility to control the CN based on some templating logic, similar to how the x509pop attestor is generating the agent SPIFFE ID, to ensure the uniqueness.

For example:

SPIRE Server config

subject	Description	Default
country	Array of Country values	["US"]
organization	Array of Organization values	["SPIRE"]
organizational_unit	Array of OrganizationalUnit values	[""]
common_name_template	The CommonName in text/template format.	""

Common Name Template

The common_name_template is a way of customizing the format of generated CNs for SVIDs.
The template formatter is using Golang text/template and can reference values provided by the server or parts of the SPIFFE ID.

Sample config

A sample configuration:

    subject {
        country = ["DE"]
        organization = ["ACME"]
        organizational_unit = ["SPIRE-EU"]
        common_name_template = "{{ .SpiffeID }}" # or others like {{ .TrustDomain }}, {{ .SpiffeID.Path }}
    }

Interested to hear your thoughts on it.
I think if we open this for configuration from the SPIRE Server operators it will help us close this one #1795 as well.

[azdagron]

Would it simply be sufficient to just set the SerialNumber field on the Subject to a string representation of the serial number for the certificate? That seems like it would satisfy the uniqueness requirement without introducing the complexity of a new configuration landscape that operators can still get wrong (e.g. not providing a template that would generate a unique subject).

Is there a strong use case for having a flexible common name? The SPIFFE ID in the URI SAN should be the primary source of identity for the document.

[sebastianGit]

Hello,

for me it sounds like setting the subject to "CN=<certificate's serial number>" satisfies the part that the DN in the subject must be unique for each subject entity certified by one CA (requirement given in [a] - see below), as the serial number must be unique per certificate issued by a CA (see [b] - see below).

• [a] https://datatracker.ietf.org/doc/html/rfc2459#section-4.1.2.6 "Where it is non-empty, the subject field MUST contain an X.500 distinguished name (DN). The DN MUST be unique for each subject entity certified by the one CA as defined by the issuer name field. A CA may issue more than one certificate with the same DN to the same subject entity."
• [b] https://datatracker.ietf.org/doc/html/rfc2459#section-4.1.2.2 "The serial number is an integer assigned by the CA to each certificate. It MUST be unique for each certificate issued by a given CA (i.e., the issuer name and serial number identify a unique certificate).

Yet, from my point of view this approach would be too limited for 2 reasons:

(1) Interoperability: Following this approach means that whenever a new certificate is issued for a subject entity, then the certificate's subject name changes, also in the cases where two (or more) certificates that are issued to the same subject entity. This will limit the interoperability with tools that do not look at SANs to retrieve the subject's identifier (I'm not refering to TLS Web server authentication here, see also #1795). This point could e.g. be addressed by using "CN=< SPIFFE ID >". Thus, "CN=<certificate's serial number>" or "CN=< SPIFFE ID >" would be better defaults than "O=SPIRE,C=US".

(2) Requirements for a certificate's "subject" by enterprises: Enterprises might have requirements how a subject in a certificate looks like, e.g., it might be necessary to include the name of the organization, a country, and the like. As the details here will differ, it would help to support a configurable subject (as indicated by @georgi-lozev above).

The discussion actually triggers a further question on my side (maybe this needs to become an additional issue): How does SPIRE actually make sure that the serial numbers of all issued certificates are unique in a setup where two (or more) SPIRE server instances (scale out) issue certificates with the same issuer name?

[azdagron]

Hmm, yes, using the serial number won't work for the reasons you pointed out.

How does SPIRE actually make sure that the serial numbers of all issued certificates are unique in a setup where two (or more) SPIRE server instances (scale out) issue certificates with the same issuer name?

SPIRE generates random 128-bit serial numbers.

[azdagron]

Requirements for a certificate's "subject" by enterprises

@sebastianGit, you bring up good points, though some of these requirements still feel unclear to me. Do you have specific examples? That would help us make a determination.

In terms of a proposal that makes us RFC compliant and allows some flexibility in the X509-SVID Subject but that stops short of templating, what about the following?

1. X509-SVID's inherits most of same Subject values you can currently configure for the CA (O, Country). We could extend this to also allow OU to be configured.
2. For common name, we use the hash of the SPIFFE ID. This makes the CN unique per subject entity while also not providing something that would distract from the primary source of identity (URI SAN containing SPIFFE ID).

Would that be flexible enough to solve your requirements, @georgi-lozev?

[sebastianGit]

Hi @azdagron ,

thanks for following up. Let me see whether I got your proposal correct:

In short, the CA subject is a configurable DN (built from: 0-or-1 CNs plus an arbitrary number of OUs, Os, Cs); the end entity certificate is derived from the CA's subject DN and the SPIFFE ID using the following pattern:

	end entity DN =   CN={hash of SPIFFE ID} plus "OUs, Os, Cs" as stated in the CA's subject DN

If this understanding is correct, it looks promising to me.

Yet, I have two concerns:

(1) Having a CN in a DN subject (also in a CA subject) is rather common, but the CN of the CA's subject DN is not copied to the end entities subject (as there should be just one CN). In setups with multiple CAs, that differ just by the CA's CN (using the same OUs,Os,Cs), the above pattern will not ensure that the end entity subject DNs are unique across the CAs.

What do you think about constructing the end entity DN as follows:

	end entity DN =   CN={SPIFFE ID or hash of SPIFFE ID} plus (one of O/OU/C = {CN of CA's subject DN}, if CN is present) plus "OUs, Os, Cs" as stated in the CA's subject DN

Ideally the choice between O/OU/C would be configurable; I would consider OU a good default.

(2) W.r.t. to CN=SPIFFE ID vs. CN={hash of SPIFFE ID}, I see arguments for both.

• SPIFFE ID has the risk that some entities might parse the SPIFFE ID out of the subject which the should not do, instead they should look at the complete subject DN, or preferable get the SPIFFE ID from the SANs.

• {hash of SPIFFE} has the following risk: A client that is not capable of using the SANs, treats the whole subject DN (as expected) as the a reference to the subject entity. Thus, it also logs the subject DN. If for any reason (incident handling,...), one now looks up the logs, and one has a hard time to resolve the 'logged subject entry' to the subjects (as I cannot easily convert back from the subject (containing the hashed SPIFFE ID) to the original SPIFFE ID, and also the SPIRE server does not support this reverse look up).

What do you think about the second topic? If it is valid, would SPIRE support a reverse look up? Or could it be an option, to have a choice between SPIFFE ID and hash of SPIFFE ID.

[azdagron]

the above pattern will not ensure that the end entity subject DNs are unique across the CAs.

Is this a requirement? As far as I know, the RFC only states that within a single CA, that the DN uniquely identify an entity, correct? You'd have the same problem today issuing certificates with unique CN's with multiple CA with DNs that are equivalent other than the CN.

end entity DN = CN={SPIFFE ID or hash of SPIFFE ID} plus (one of O/OU/C = {CN of CA's subject DN}, if CN is present) plus "OUs, Os, Cs" as stated in the CA's subject DN

I'm not quite sure I follow what this is proposing. Can you provide a concrete example?

{hash of SPIFFE} has the following risk:

That is an interesting point. I'm a little conflicted. On one hand, providing flexibility here allows the X509-SVIDs to be used across legacy (i.e. SPIFFE unaware, or even SAN unaware) systems, which helps ease gradual adoption.

On the other hand, I think we're walking a dangerous line by placing the SPIFFE ID in the CN. For example, if I were writing software to be deployed in a trust domain where the X509-SVIDs had the SPIFFE ID both in URI SAN and the CN, which one would I prefer? What if they weren't equal? What if I didn't notice the SAN and only picked up the CN? Suddenly the authentication story gets complicated. I don't think we want "complicated" :)

I did think of an alternative to my suggestion above, instead of setting the CN to the hash of the SPIFFE ID, we could instead put the hash in the DN's SerialNumber field. This would sidestep some of the concerns about how the CN gets interpreted by various software.

This isn't the first time folks have wanted to influence fields in the X509-SVID. I'm not saying this is a good idea, but this could be something we could accommodate with our plugin system instead of building all of these knobs inside of SPIRE core to influence the CSR. I'll bring this up with the other maintainers. If we did introduce a plugin to customize the CSR, would that work for you?

[SebRi]

@ end entity subject DNs unique across CAs:

This is a requirement given by the root CA under which we evaluate SPIRE.

@ subject pattern:

I was refering to the following two cases:

Case #1: CA subject without CN

• sample CA subject: O=SPIRE,C=US
• sample end entity subject: CN=spiffe://example.com/123,O=SPIRE,C=US

Case #2: CA subject with CN

• sample CA subject: CN=MyCaName,O=SPIRE,C=US
• sample end entity subject: CN=spiffe://example.com/123,OU=MyCaName,O=SPIRE,C=US

Note: CA subject name needs to be configurable.

@ Complicated

I'm aiming for "simplicity", yet, we need to be aware of the reality as well.

(1) issuing end entity certificates with an empty subject would be the simplest and cleanest solution, but this is blocked by "reality" as some software cannot handle it. Furthermore, it explicitly exclude legacy software.

(2) Issuing all end entity certificates with the same static subject is blocked by the standard, and clients may legitimately interprete those certificates as if they all refer to the same subject entity (although there are different SPIFFE IDs in the SANs).

(3) This is what we try to find: SPIFFE ID provided in the SANs as primary source for the identity, unique value in the subject field which identifies the subject (stable & resolvable identifier => SPIFFE ID as part of the subject) given the constraints from an existing PKI and aiming for support of legacy clients. Note: Here, when looking at the subject then the full subject value will serve as an identifier to the end entity. Whether looking at the subject's CN is sufficient depends on the rules of the PKI. Alternatively, one can of course look at the SPIFFE ID in the SANs. The SPIRE server must ensure that the SPIFFE IDs in the two places are the same.

The example on top of the post would fit for this from my point of view (templating might still provide a little more flexibility, others might still need a DNS entry in the subjects CN)

@ Putting the SPIFFE ID in the DNs serialnumber field.

Sounds like an interesting option. I could look deeper into this next week. I'm not also sure whether the field comes character set or length restrictions.

@ CSR plugin

If you refer to a plugin that allows to set the subject of the to be issued certificate on the server side based on information from the CSR, preliminary fields of the certificate (like SPIFFE ID from SAN), stored SPIFFE entry configuration (e.g. get DNS, if configured) and plugin configuration (e.g. pattern). This sounds promising. It would be a very powerful hook compared to static patterns (e.g. top of the post) or templating.

A plugin in the agent would not be sufficient, as the server would treat most of the information as coming from an untrusted source and thus the server would need to implement additional checks, too.

[azdagron]

Plugin in the agent

Agent's don't mint certificates, only the server. The agent generates a private key and generates a CSR that only contains the public key. Everything else about the certificate is determined by the server, based on the registration entry. A theoretical service side plugin could be fed all the information about the workload the SVID is being minted for.

[sebastianGit]

Hi @azdagron ,

As interoperability with other (potentially legacy) applications was mentioned above a couple of times, I like to share the following OSS-based use case that I came recently across when looking at tooling from the CNCF landscape.

The use case would be to use SPIRE-issued client certificates in TLS-based client authentication when requesting OAuth access tokens from Keycloak using the 'client credentials grant' (no user involved here). The use case seems to be enabled, once SPIRE issues certificates contain a unique, stable identifier in the certificate’s subject (e.g. as discussed above).

As described here (https://www.keycloak.org/docs/latest/server_admin/#_client-credentials, subsection "X509 certificate") Keycloak allows to configure mTLS-based client authentication for OAuth clients. According to the documentation, the client identifier can currently only be read from the subject, but not from the SANs (either full subject DN or regex - how to configure this best will depend on the rules for uniqueness of names defined by the trusted PKI(s)).

Furthermore, with current SPIRE setup where the subject DNs of all client certificates are equal to “O=SPIRE,C=US”, the integration with Keycloak would not properly work as the clients cannot be distinguished by their certificate's subject.

While Keycloak might sooner or later also support reading the client identifier from the SANs, this is just one example of a use case (and application) where it is currently not supported to read information from a certificate's SAN, and where the support of a unique, stable identifier in the certificate's subject would ease the integration with SPIRE.

[azdagron]

That's a good use case! I do agree that providing some flexibility here is important. We've discussed this a bit during our contributor and maintainer calls. We haven't quite come to consensus on the approach but will work on having a more concrete direction by the end of next week.

[sebastianGit]

@azdagron : how is the topic proceeding? could you share some news?

[azdagron]

Thanks for checking in, @sebastianGit! We've kept this issue in discussion over the last few weeks. Right now we are evaluating the various fields in the X509-SVID to understand which of these might be valuable to allow operator customization on. If there are many strong uses cases, then the plugin approach will likely win out. Otherwise we might settle for a configurable (either global, or per-entry) that allows some templating.

Would a global template work in your case or would you need the flexiblity of a per-entry template?

[azdagron]

In either case, we will fix the RFC non-compliance.

[azdagron]

I've opened #3253 to track the discussion on the plugin. Happy to have your thoughts over there!

[sebastianGit]

Let me focus on the subject field here (I'll follow up on other fields as proposed in #3253)

Would a global template work in your case or would you need the flexibility of a per-entry template?

For the subject field of agent and workload certificates global templating would work e.g. see initial proposal by @georgi-lozev above.

Support of "organizational unit" (in addition to "organization" and "country") as RDN in the subject of agents, workloads and CA certificates would be highly appreciated.

[azdagron]

Support of "organizational unit" (in addition to "organization" and "country") as RDN in the subject of agents, workloads and CA certificates would be highly appreciated.

Would replicating these fields from the subject of the CA, which is already customizable (via the ca_subject) configurable, be sufficient?

[sebastianGit]

@azdagron : deriving the end entity certificate name from the CA subject would work, if it is possible to support also the case that the CA uses a CN in its DN:

Case 1: configurable CA subject without CN

sample CA subject: OU=Dev,O=SPIRE,C=US

derived sample end entity subject: CN=spiffe://example.com/123,OU=Dev,O=SPIRE,C=US

Case 2: configurable CA subject with CN

sample CA subject: CN=MyCaName,OU=Dev,O=SPIRE,C=US

derived sample end entity subject: CN=spiffe://example.com/123,OU=MyCaName,OU=Dev,O=SPIRE,C=US

In case 2, the CA subject's CN could be carried over to the end entity subject DN in an RDN like OU (but not CN)

[azdagron]

I've opened #3341 to track this work (there is one outstanding question to be answered). Thanks for opening this issue and the great discussion everybody.

#3253 will continue to track the work for introducing a pluggable interface around the customization of these fields.

[evan2645]

Hi @georgi-lozev and @sebastianGit, happy new year!

We've had lots of issues with the change that we made to address this (the latest is #4755), so I re-opened for further discussion.

We're trying to figure out the best path forward here. The current default behavior of every X509-SVID having a difficult-to-predict subject string has been problematic, so we'd like to address that as a minimum. We also want to make sure any changes we make are not breaking you, so can you please share any/all experiences you've had where non-unique subject string caused a problem for workloads being serviced by SPIRE (or for workloads with which they were communicating)?

There was a time in which we did not set any subject on X509-SVIDs, and I recall an issue with Java validators around this that drove us to set a dummy subject. IMO eliding subject on these leafs is the preferred behavior in the absence of a common name ... we can explore reverting to this behavior if the validators in question are old enough.

[sebastianGit]

@evan2645 Let me follow up on this next week.

[evan2645]

Thank you @sebastianGit! 🙏

[sebastianGit]

Hi @evan2645 ,

let me share some information on our setup, and my point of view how this might related to some of the issues I'm aware of.

We basically use the following pattern:

• SPIRE CA certificates:

  • SPIFFE ID in URI SAN: spiffe://<trust domain>
  • subject DN: C = <country>, O = <org>, OU = <department>, OU = <trust domain>, CN = <CA name>
  • issuer DN: <non-SPIRE intermediate CA/root CA>

• Workload certificates:

  • SPIFFE ID in URI SAN: spiffe://<trust domain>/<opaque-identifier>
  • subject DN: C = <country>, O = <org>, OU = <department>, OU = <trust domain>, OU = <CA name>, CN = sha256(<SPIFFEID>)
  • issuer DN: C = <country>, O = <org>, OU = <department>, OU = <trust domain>, CN = <CA name>

With this setup, the 'subject DN' is as unqiue as the 'SPIFFE ID', i.e., any two certificates with the same subject DN have the same SPIFFE ID and vice versa. The reason for using the sha256 hash of the SPIFFE ID in the workload CN is that the result of the hash function fits the 64 character length restriction of the CN attribute and is collision-resistant. This allows 'consumers' to equally treat the subject DN and the SPIFFE ID as unique identifiers for an entity, and is especially beneficial for integrating with software which can only check for equality of a subject DN (not providing support for pattern matching on the subject DN, not providing support for reading a URI SAN) - which is a case to consider, if interoperability with 'unknown' software is a concern.

On the implementation-side we use the credential composer plugin to achieve this. Assuming the plugin functionality will not change, changing the default setting of the subject DN should not break our setup.

Relation to #4755:
I did not look up what's supported by MySQL, but I would assume MySQL is one more example of software that only can look at the subject DN. I would expect that the current functionality of the credential composer can help in this context, maybe it is even already used here as the subject DN already contains a CN.

Relation to #4415:
As all SPIRE server CA certificates are rooted in a common root CA which is distributed as trust bundle, we did not run into the issue stated in #4415. Note: Each workload provides its workload certificate and the certificate chain (excl. the root CA) in the TLS handshake; SPIRE's CA certificate in the certificate chain is exactly the certificate that was used in this chain; there is only one common root CA certificate that needs to be trusted and distributed.

Relation to #4189:
With current setup, the certificates do not contain explicit attributes about the workload/subject that requested a certificate (e.g., GCP project, tenant identifier,...) -- neither in the SPIFFE ID nor in the subject DN. I'm still trying to find a good way forward here (sorry for not following up on #4189, I'm still looking into this topic). Currently, it looks like including a minimal set of attributes in the SPIFFE ID and in the subject DN is the most interoperable way forward on a short/mid-term timeline (considering standardization efforts as an interesting, but long-term effort).

[evan2645]

Thanks again for all your support here @sebastianGit and @georgi-lozev. This added context has been very helpful.

We decided to roll back the unique identifier change. My understanding is that you will be unaffected by this change since you set your own credential composer. We are additionally shipping a new built-in credential composer to preserve the current behavior if needed. Please see #4755 (comment) for slightly more information. Thanks again for everything and let us know if you have any questions or concerns 🙏

[sebastianGit]

@evan2645 : Thanks reaching out and make us aware of the upcoming change. I also expect, that we won't be effected by the change as we use our own credential composer plugin - still we'll keep an eye on it when updating.

[sebastianGit]

@evan2645 : There is one thing that I like to bring to your attention.

If my understanding of #4862 is correct, then after the change all workload certificates (by default, if no credential composer plugin is used) will have again the following subject DN:

• case 1) no DNS SANs entry provided: "O=SPIRE,C=US"
• case 2) at least 1 DNS SANs entry provided: "CN=<value of first DNS SANs entry>,O=SPIRE,C=US"

As a consequence this means that all workload certificates for which there is no DNS entry specified (which is typical for client certificates) have the same subject DN. For me this is still an unexpected default behavior as I would expect that the subject DN is a unique identifier for the workload (see arguments and discussion in the thread above).

Yet, the credential composer plugin provides us a nice means and the required flexibility to adjust the certificate contents as per our requirements 🙏🙏🙏.

[evan2645]

As a consequence this means that all workload certificates for which there is no DNS entry specified (which is typical for client certificates) have the same subject DN. For me this is still an unexpected default behavior as I would expect that the subject DN is a unique identifier for the workload (see arguments and discussion in the thread above).

Out of curiosity, would you find it more or less surprising if there was no Subject set at all in this case?

[sebastianGit]

@evan2645 : As I've previously stated in this thread - an empty subject DN would be inline with the standard, but sacrifice interoperability, thus, from my point of view it would also not be a good default.

(1) issuing end entity certificates with an empty subject would be the simplest and cleanest solution, but this is blocked by "reality" as some software cannot handle it. Furthermore, it explicitly exclude legacy software.

[evan2645]

Does that mean you'd be less surprised? 😅

The introduction of the unique id in the subject caused significant usability challenges for our community. In my previous comment here, I was trying to understand what the practical impact of identical subjects. I can see why it would be confusing, but fixing it via unique id at the cost of e.g. simplified database auth, seems like a bad trade-off. Preserving the experience in these cases was decided to be more important than conforming to 5280 in this regard.

If we could remove the subject extension entirely without breaking the world, that would be ideal. And if we can't do that, then the next best thing is to roll back x500 unique identifier (which is the route that was chosen). People that use SPIRE for the cases in question DO rely on Subject, however they simply set a DNS name which appears in CN field and thus Subject ambiguity in those cases is a non-issue.

[sebastianGit]

@evan2645

Yes, I'd be less surprised by an empty subject DN, but unfortunately, this does not provide the targeted level of interoperability.

When going with the currently chosen approach (O=SPIRE,C=US or CN=<DNS>,O=SPIRE,C=US), I'm in the situation that when I want to grant one client access to the MySQL DB (see #4755), than this is only possible, if I include a unique DNS entry in the client's certificate, as all certificates without DNS entry have the same subject DN. The situation is similar to the example with Keycloak (see earlier in this thread).

From my point of view making the subject DN as unique as the SPIFFE ID (e.g. by adding in the sha256 hash of the SPIFFE ID into the subject DN either in the CN or in another RDN) would provide the best level of interoperability and flexibility by default from the options discussed. Which drawbacks would you see for this approach?