Regression: Fast validation breaks TLS client cert auth when using TLSContexts

[jwm]

** State as of 1.10 **

(This is @kflynn editing @jwm's original text, in case others land here. Thanks for the help nailing down what's up, @jwm!)

Fast validation causes Kubernetes Secrets of type opaque to not be correctly processed by Ambassador. Nope, I was wrong when I said that. Opaque Secrets are OK: the problem is that Ambassador tries really hard not to load Secrets that it doesn't actually need, and when fast validation is on, 1.10 tries so hard that it would never load the ca_secret of a TLSContext at all. Obviously that makes it hard to use client-cert authentication.

Prior to Ambassador 1.10, all would be well if AMBASSADOR_FAST_VALIDATION was not set. As of 1.10, fast validation is the default, so you will need to set AMBASSADOR_LEGACY_MODE=true to use client-cert auth in 1.10.

---

(@jwm's original text)

Describe the bug
Client cert auth seems broken with fast validation and reconfig (not sure ATM if it's one, the other, or both).

To Reproduce
Steps to reproduce the behavior:

1. Enable fast validation and reconfig.
2. Create a TLSContext that validates client certs:

apiVersion: getambassador.io/v2
kind: TLSContext
metadata:
  name: eligibility
  namespace: ambassador
spec:
  alpn_protocols: h2,http/1.1
  ca_secret: graphql-client-certs-issuers
  cert_required: true
  hosts:
  - eligibility.staging.devoted.com
  redirect_cleartext_from: 8080
  secret: ingress-eligibility-tls

3. Make a request with client cert auth.

Expected behavior
When the supplied client cert is valid, the request should be successful.

Versions (please complete the following information):

• Ambassador: 1.7.3
• Kubernetes environment: EKS
• Version: 1.17

Additional context

Relevant `diagd` debug:

2020-10-05 19:36:44 diagd 0.71.0-7773-gaf9d81def [P30TAEW] DEBUG: resolve_secrets working on: {
    "_active": true,
    "_errored": false,
    "_referenced_by": [
        "eligibility.ambassador.1"
    ],
    "_rkey": "eligibility.ambassador.1",
    "alpn_protocols": "h2,http/1.1",
    "cert_required": true,
    "hosts": [
        "eligibility.staging.devoted.com"
    ],
    "is_fallback": false,
    "kind": "TLSContext",
    "location": "eligibility.ambassador.1",
    "metadata_labels": {
        "ambassador_crd": "eligibility.ambassador"
    },
    "name": "eligibility",
    "namespace": "ambassador",
    "redirect_cleartext_from": 8080,
    "secret_info": {
        "ca_secret": "graphql-client-certs-issuers",
        "secret": "ingress-eligibility-tls"
    }
}
2020-10-05 19:36:44 diagd 0.71.0-7773-gaf9d81def [P30TAEW] DEBUG: TLSContext.resolve_secret ingress-eligibility-tls, namespace ambassador: namespacing is True
2020-10-05 19:36:44 diagd 0.71.0-7773-gaf9d81def [P30TAEW] DEBUG: resolve_secret ingress-eligibility-tls.ambassador: found secret_info
2020-10-05 19:36:44 diagd 0.71.0-7773-gaf9d81def [P30TAEW] DEBUG: SecretHandler (TLSContext eligibility): secret ingress-eligibility-tls in namespace ambassador is still needed
2020-10-05 19:36:44 diagd 0.71.0-7773-gaf9d81def [P30TAEW] DEBUG: resolve_secret ingress-eligibility-tls.ambassador: found secret, asking handler to cache
2020-10-05 19:36:44 diagd 0.71.0-7773-gaf9d81def [P30TAEW] DEBUG: saved secret ingress-eligibility-tls.ambassador: /ambassador/snapshots/ambassador/secrets-decoded/ingress-eligibility-tls/3A25D7116CCB7363C988269764FABE53F635ACEF.crt, /ambassador/snapshots/ambassador/secrets-decoded/ingress-eligibility-tls/3A25D7116CCB7363C988269764FABE53F635ACEF.key, None
2020-10-05 19:36:44 diagd 0.71.0-7773-gaf9d81def [P30TAEW] DEBUG: resolve_secrets: IR returned secret ingress-eligibility-tls as <SavedSecret ingress-eligibility-tls.ambassador -- cert_path /ambassador/snapshots/ambassador/secrets-decoded/ingress-eligibility-tls/3A25D7116CCB7363C988269764FABE53F635ACEF.crt, key_path /ambassador/snapshots/ambassador/secrets-decoded/ingress-eligibility-tls/3A25D7116CCB7363C988269764FABE53F635ACEF.key, user_path None, root_cert_path None, cert_data present>
2020-10-05 19:36:44 diagd 0.71.0-7773-gaf9d81def [P30TAEW] DEBUG: TLSContext eligibility saved secret secret ingress-eligibility-tls in namespace ambassador
2020-10-05 19:36:44 diagd 0.71.0-7773-gaf9d81def [P30TAEW] DEBUG: TLSContext - successfully processed the cert_chain_file, private_key_file, and cacert_chain_file: {'secret': 'ingress-eligibility-tls', 'ca_secret': 'graphql-client-certs-issuers', 'cert_chain_file': '/ambassador/snapshots/ambassador/secrets-decoded/ingress-eligibility-tls/3A25D7116CCB7363C988269764FABE53F635ACEF.crt', 'private_key_file': '/ambassador/snapshots/ambassador/secrets-decoded/ingress-eligibility-tls/3A25D7116CCB7363C988269764FABE53F635ACEF.key'}
2020-10-05 19:36:44 diagd 0.71.0-7773-gaf9d81def [P30TAEW] DEBUG: TLSContext.resolve_secret graphql-client-certs-issuers, namespace ambassador: namespacing is True
2020-10-05 19:36:44 diagd 0.71.0-7773-gaf9d81def [P30TAEW] DEBUG: resolve_secret graphql-client-certs-issuers.ambassador: no secret_info, asking handler to load
2020-10-05 19:36:44 diagd 0.71.0-7773-gaf9d81def [P30TAEW] DEBUG: SecretHandler (TLSContext eligibility): load secret graphql-client-certs-issuers in namespace ambassador
2020-10-05 19:36:44 diagd 0.71.0-7773-gaf9d81def [P30TAEW] ERROR: Secret graphql-client-certs-issuers.ambassador unknown
2020-10-05 19:36:44 diagd 0.71.0-7773-gaf9d81def [P30TAEW] DEBUG: resolve_secrets: IR returned secret graphql-client-certs-issuers as <SavedSecret graphql-client-certs-issuers.ambassador -- cert_path None, key_path None, user_path None, root_cert_path None, cert_data absent>
2020-10-05 19:36:44 diagd 0.71.0-7773-gaf9d81def [P30TAEW] INFO: eligibility.ambassador.1: <RichStatus BAD error='TLSContext eligibility found no validation certificate in secret graphql-client-certs-issuers in namespace ambassador' hostname='ambassador-hp5p7' version='0.71.0-7773-gaf9d81def'>
2020-10-05 19:36:44 diagd 0.71.0-7773-gaf9d81def [P30TAEW] DEBUG: TLSContext eligibility is a termination context, enabling TLS termination

[jwm]

This still happens with 1.8.0.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[jwm]

not stale

[jwm]

This is still happening with 1.10.0.

[djsly]

why was this closed ?