Fixes #20021 - Optionally generate CA public cert with other CAs certs

[sean797]

Requires Katello/katello-certs-tools#1

[ekohl]

Apologies for the late review.

So the use case for this is that every katello node is a CA and you bundle all the other CAs into it? I think this is a hard part of Katello and I doubt I fully understand this aspect.

[sean797]

@ekohl thanks for your review! Updated 👍

Yes, that's it. This CA cert is also bundled in the smart-proxy-certs.tar when generated. Making sure that the smart proxies install CA certs from all Katello nodes and trust them, currently it'll just trust the one the smart-proxy-certs.tar came from.

[ekohl]

Interesting. I'll need some time to verify this. Maybe @timogoebel could have a look as well since you're looking at this as well.

[ekohl]

Like we discussed on IRC: I'm afraid this pattern won't scale well since every "node" will be a CA. I do like your acceptance tests and I will use it to write some more

[sean797]

So we discussed a little more on IRC, putting it here for a wider audience and allow us to think about it a little more:

I don't think we should require an external CA service like freeIPA or others (although that may be a nice optional feature) I think an option may be:

Candlepin node(s) are the default CA(s) and they trust each other if there is more than one.
Other component nodes (qpidd(s), Katello(s), Smart Proxies) would all be setup using a certs.tar or similar. Server CA certs would be optionally supplied when generating Katello(s) or Smart Proxies certs.tar from a candlepin node, just how foreman-proxy-content-certs-generate works today.

[ekohl]

I could be convinced of merging this even if we're not going to use it as the default multi-server setups but I'd like the opinion of other maintainers because how we deploy certs is still somewhat unfamiliar to me.

[ehelms]

Help remind me -- we need to append because we can't rely on defining a directory of CA certs in all cases to point to vs. a single CA file?

I think it would help me to wrap my brain around the scenarios where multiple CAs would come into play. Feel free to walk me through it like a freshman in college. HA concepts are a weakness of mine that I want to understand better :)

[sean797]

Help remind me -- we need to append because we can't rely on defining a directory of CA certs in all cases to point to vs. a single CA file?

Sorry not sure what you mean here? My goal is to have the clients & Foreman Proxies trust either of the CAs on my Candlepin nodes (right now with our supported monolithic install that also includes app, pulp, qpid, ect..)

I think it would help me to wrap my brain around the scenarios where multiple CAs would come into play. Feel free to walk me through it like a freshman in college. HA concepts are a weakness of mine that I want to understand better :)

Sure, so HA means Highly Available, if a system is acting in an active/passive mode then that is normally considered HA, as one node will take over when the other fails, normally with a outage as the traffic fails over, if fail over is manual then the outage will likely be longer (but this is still considered to be HA)

I'm trying to make it possible to load balance our application(s) active/active not active/passsive. This will allow for there to be no fail over process which will allow users to add extra capacity to the system by adding more nodes, as well as make it natively HA (HA without pacemaker).

At the the heart of things its super simple idea. Be able to add active component nodes independently without sharing anything to increase capacity and resilience. The nodes shouldn't share any resources, if they require shared storage you just move the Single Point of Failure (SPOF) to the storage system.

---

This PR allows me to have 2 CAs that each candlepin node uses. I have tested it with 2 monolithic Katello nodes and a single Smart Proxy & Client.

How does this relate to certificates?
Certificates need to be stored somewhere, today we probably could put the certs on shared storage and scale nodes but that just moves to SPOF somewhere else, doesn't really solve the problem.

Any other approaches?
Yes! I think we could use certmonger and/or freeIPA or some other similar service but that also moves the SPOF somewhere else, though freeIPA can be made active/active (I think?) it adds yet more complexity to HA Katello and I don't know how that would work when candlepin issues certs for clients.

[ehelms]

First off, thanks for the explanation! That helped a ton in wrapping my brain around how to think about this approach.

The last gap of knowledge to help me fill in is "2 CAs that each Candlepin node uses". Are you saying in this case two root CAs, essentially what is katello-default-ca, are generated and given to each Candlepin instance? And that Candlepin is able to take both CAs and anytime a certificate request is made, Candlepin is able to then cut certificates for each CA to hand to clients?

[sean797]

I was thinking:

• CP-node1 would have CA1
• CP-node2 would have CA2

When my client registers or attaches a new sub, one of those nodes would pick up the request and give my client a cert & key. So my client would potentially have:

• client key from CA1
• client cert from CA1
• CA cert contain both CA1 & CA2 (this comes from the katello-ca-consumer RPM)

My Pulp nodes would then provide content to clients with client certs from either CA1 or CA2. Their certs would come from a certs.tar (like Smart Proxies today) or similar (as would Katello app nodes)

But like I said, this need experimentation, other ideas may also work or be a better architecture to go with.

[bkearney]

@ehelms Any update on this?

[sean797]

It needs some work to get the tests green, but functionally this PR is fine. I'm happy to get tests passing if we want to merge this - there was some debate previously regarding this approach.

[zjhuntin]

Here is an RFC being worked on for an issue like this: https://meet.google.com/linkredirect?authuser=1&dest=https%3A%2F%2Fcommunity.theforeman.org%2Ft%2Frfc-redesign-certificate-handling-within-foreman-deployments%2F17933