Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: expose ALPN in TLS handshake #422

Merged
merged 1 commit into from
Oct 25, 2024

Conversation

howardjohn
Copy link
Contributor

New versions of gRPC-go are enforcing the h2 ALPN to be presented during the TLS handshake. See
https://pkg.go.dev/google.golang.org/grpc/internal/envconfig#pkg-variables GRPC_ENFORCE_ALPN_ENABLED. The TLS server here isn't automatically getting this set due to usage of GetConfigForClient.

This properly sets it.

Without this, istio-csr will be incompatible with Istio 1.24, which upgrades the gRPC version. Note this can be worked around by setting GRPC_ENFORCE_ALPN_ENABLED=false on the proxy container, which Istio is able to do -- so there is an escape hatch for users.

The Istio logs look like
"transport: authentication handshake failed: credentials: cannot check peer: missing selected ALPN property"

@cert-manager-prow cert-manager-prow bot added the dco-signoff: no Indicates that at least one commit in this pull request is missing the DCO sign-off message. label Oct 8, 2024
@cert-manager-prow
Copy link
Contributor

Hi @howardjohn. Thanks for your PR.

I'm waiting for a cert-manager member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@cert-manager-prow cert-manager-prow bot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Oct 8, 2024
@cert-manager-prow cert-manager-prow bot added dco-signoff: yes Indicates that all commits in the pull request have the valid DCO sign-off message. and removed dco-signoff: no Indicates that at least one commit in this pull request is missing the DCO sign-off message. labels Oct 8, 2024
@howardjohn
Copy link
Contributor Author

cc @inteon

@ThatsMrTalbot
Copy link
Contributor

/ok-to-test

@cert-manager-prow cert-manager-prow bot added ok-to-test and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Oct 21, 2024
@ThatsMrTalbot
Copy link
Contributor

Hiya! Thanks for contributing this fix.

Looks like the golangci-lint check is failing.

Specifically its complaining about the indentation now you have added the comment block, running golangci-lint run --fix gave me this diff:

                Certificates: []tls.Certificate{tlsCert},
                // Advertise ALPN, required in modern gRPC versions
                // Typically gRPC sets this for us, but since this tls.Config ultimately gets returned in GetConfigForClient it doesn't.
-               NextProtos:   []string{"h2"},
-               ClientAuth:   tls.VerifyClientCertIfGiven,
-               ClientCAs:    peerCertVerifier.GetGeneralCertPool(),
+               NextProtos: []string{"h2"},
+               ClientAuth: tls.VerifyClientCertIfGiven,
+               ClientCAs:  peerCertVerifier.GetGeneralCertPool(),
                VerifyPeerCertificate: func(rawCerts [][]byte, verifiedChains [][]*x509.Certificate) error {
                        err := peerCertVerifier.VerifyPeerCert(rawCerts, verifiedChains)
                        if err != nil {

Copy link
Member

@inteon inteon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the bug report and fix.

Could you add a tests/ modify our tests to make sure GRPC_ENFORCE_ALPN_ENABLED now works & will keep working in the future?

Also, do you know where in code this is normally auto-set when we are not using GetConfigForClient?

@howardjohn
Copy link
Contributor Author

Also, do you know where in code this is normally auto-set when we are not using GetConfigForClient?

credentials.NewTLS() from gRPC auto adds it

Could you add a tests/ modify our tests to make sure GRPC_ENFORCE_ALPN_ENABLED now works & will keep working in the future?

Should we just add a test for Istio 1.24 directly? Can use a pre-release in the meantime

@inteon
Copy link
Member

inteon commented Oct 22, 2024

Adding an Istio 1.24 test sounds like a great solution.

howardjohn added a commit to howardjohn/testing that referenced this pull request Oct 22, 2024
See cert-manager/istio-csr#422 for context

Signed-off-by: John Howard <john.howard@solo.io>
@inteon
Copy link
Member

inteon commented Oct 23, 2024

@howardjohn In order to test istio 1.24, we will have to add a file here: https://github.com/cert-manager/istio-csr/tree/main/make/config/istio.

We can locally test the test using the following command:
ISTIO_VERSION="1.24.0-alpha.0" make test-e2e

@cert-manager-prow cert-manager-prow bot added dco-signoff: no Indicates that at least one commit in this pull request is missing the DCO sign-off message. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed dco-signoff: yes Indicates that all commits in the pull request have the valid DCO sign-off message. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Oct 23, 2024
@howardjohn
Copy link
Contributor Author

@howardjohn In order to test istio 1.24, we will have to add a file here: https://github.com/cert-manager/istio-csr/tree/main/make/config/istio.

We can locally test the test using the following command: ISTIO_VERSION="1.24.0-alpha.0" make test-e2e

Thanks, added it and also https://github.com/cert-manager/testing/pull/1066/files

@cert-manager-prow cert-manager-prow bot added dco-signoff: yes Indicates that all commits in the pull request have the valid DCO sign-off message. and removed dco-signoff: no Indicates that at least one commit in this pull request is missing the DCO sign-off message. labels Oct 23, 2024
@ThatsMrTalbot
Copy link
Contributor

Can confirm, on this branch:

Ran 14 of 14 Specs in 109.715 seconds
SUCCESS! -- 14 Passed | 0 Failed | 0 Pending | 0 Skipped

But if I copy the istio config and run the tests on the main branch then they just eventually time out. Whats interesting is I cant see errors in the logs, but I can see a loop of it constantly creating certificate-requests.

@howardjohn
Copy link
Contributor Author

The errors were in the istio-proxy logs in my experience

New versions of gRPC-go are enforcing the `h2` ALPN to be presented
during the TLS handshake. See
https://pkg.go.dev/google.golang.org/grpc/internal/envconfig#pkg-variables
`GRPC_ENFORCE_ALPN_ENABLED`. The TLS server here isn't automatically
getting this set due to usage of GetConfigForClient.

This properly sets it.

Without this, istio-csr will be incompatible with Istio 1.24, which
upgrades the gRPC version. Note this can be worked around by setting
`GRPC_ENFORCE_ALPN_ENABLED=false` on the proxy container, which Istio is
able to do -- so there is an escape hatch for users.

The Istio logs look like
`"transport: authentication handshake failed: credentials: cannot check peer: missing selected ALPN property"`

Signed-off-by: John Howard <john.howard@solo.io>
@cert-manager-prow cert-manager-prow bot added size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Oct 25, 2024
@inteon
Copy link
Member

inteon commented Oct 25, 2024

This worked for me:

ISTIO_VERSION="1.24.0-alpha.0" make test-e2e
...
Ran 14 of 14 Specs in 75.084 seconds
SUCCESS! -- 14 Passed | 0 Failed | 0 Pending | 0 Skipped
PASS

@inteon
Copy link
Member

inteon commented Oct 25, 2024

/approve
/lgtm

@cert-manager-prow cert-manager-prow bot added the lgtm Indicates that a PR is ready to be merged. label Oct 25, 2024
@cert-manager-prow
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: inteon

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@cert-manager-prow cert-manager-prow bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 25, 2024
@cert-manager-prow cert-manager-prow bot merged commit cde2af2 into cert-manager:main Oct 25, 2024
16 checks passed
@SgtCoDFish
Copy link
Member

SgtCoDFish commented Oct 25, 2024

Adding even though this has merged: I saw these errors in istio-proxy in the e2e tests without this change:

2024-10-25T10:13:04.992535Z     error   citadelclient   failed to sign CSR: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: credentials: cannot check peer: missing selected ALPN property"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. dco-signoff: yes Indicates that all commits in the pull request have the valid DCO sign-off message. lgtm Indicates that a PR is ready to be merged. ok-to-test size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants