Client Certificate is no longer being sent on proxied requests

[mikn]

Kibana version: 6.4.0

Elasticsearch version: 6.4.0

Server OS version: alpine 3.8

You know one of those things that you fix because you think it is a bug, but it turns out that you have users that use it as a feature?
Well, this would be the one for me 😄 PR: #17804
We are currently using Searchguard to protect our ES cluster and the connection between ES and Kibana is authenticated using a client certificate issued specifically to the Kibana server.
To authorize the user, we have set up an oauth2_proxy in front of Kibana and we forward the headers from the oauth2_proxy to Kibana, which then forwards it to ES and thus the user is thus individually authorized.

Is there any possibility for this to end up going back into Kibana? This currently means we cannot upgrade to Kibana 6.4 without redoing our authc/authz scheme.

[kobelb]

Hey @mikn, we removed this behavior because sending the certificates and the headers for proxied requests requires Elasticsearch to be configured in a specific manner to ensure that the proper user is authenticated. If the certificates are meant to identify the Kibana server user, and the headers are used to identify the logged in end-user, if we set both for the proxied requests then Elasticsearch must be configured in a specific way to ensure that the headers are used, and we don't fallback to using the certificates for authentication or else we risk authenticating as the Kibana server user for proxied requests.

I'm not too familiar with the way that this works with SearchGuard, it might be worth checking with them to see what they recommend.

[mikn]

If I understand your reasoning, this change was mainly to avoid "mistakes"? Because of the nature of our setup (we run Kibana and ES inside a Kubernetes cluster, so volatile IP addresses), we cannot set a trusted proxy IP address (required for header-only authentication). Instead we force transport authentication with client certificates as well. I'm not sure how this problem is solved within X-Pack, but for our SearchGuard use-case we don't really have any options other than redo our authc/authz, and my understanding is that we need to compromise either convenience (no SSO) or security (cannot require transport authentication). Reference: https://docs.search-guard.com/latest/proxy-authentication

…

On Wed, 5 Sep 2018, 00:13 Brandon Kobel, ***@***.***> wrote: Hey @mikn <https://github.com/mikn>, we removed this behavior because sending the certificates and the headers for proxied requests requires Elasticsearch to be configured in a specific manner to ensure that the proper user is authenticated. If the certificates are meant to identify the Kibana server user, and the headers are used to identify the logged in end-user, if we set both for the proxied requests then Elasticsearch must be configured in a specific way to ensure that the headers are used, and we don't fallback to using the certificates for authentication or else we risk authenticating as the Kibana server user for proxied requests. I'm not too familiar with the way that this works with SearchGuard, it might be worth checking with them to see what they recommend. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#22643 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAIEeC4LUvPNVW9Z1ek8hIYKZg_5-u3jks5uXqa2gaJpZM4WYhqq> .

[kobelb]

@mikn you're correct that with these changes, it's no longer possible to require transport authentication for all connections that are established to Elasticsearch as the proxied requests for end-users won't be presenting the client certificates. This is because we can't authenticate all requests to Elasticsearch using certificates, because in the case of these proxied requests, we don't have the actual end-user's certificate because of the way that TLS works.

Because of the nature of our setup (we run Kibana and ES inside a Kubernetes cluster, so volatile IP addresses), we cannot set a trusted proxy IP address (required for header-only authentication)

I'm not too familiar with SearchGuard, but it appears that they support regular expressions for the internalProxies, is it possible to take advantage of this facility for your use-case?

[mikn]

The IP space inside a regular Kubernetes cluster is not segmented, so any pod from any namespace gets an IP assigned within the 100.64.0.0/10 range. I understand that the certificate itself is not an end user certificate, it is just there to have a CN to validate that it is Kibana connecting and forwarding the headers to prevent "anyone" from connecting internally. I would be very happy indeed if this feature can be brought back. :)

…

On Thu, 6 Sep 2018, 20:19 Brandon Kobel, ***@***.***> wrote: @mikn <https://github.com/mikn> you're correct that with these changes, it's no longer possible to require transport authentication for all connections that are established to Elasticsearch as the proxied requests for end-users won't be presenting the client certificates. This is because we can't authenticate all requests to Elasticsearch using certificates, because in the case of these proxied requests, we don't have the actual end-user's certificate because of the way that TLS works. Because of the nature of our setup (we run Kibana and ES inside a Kubernetes cluster, so volatile IP addresses), we cannot set a trusted proxy IP address (required for header-only authentication) I'm not too familiar with SearchGuard, but it appears that they support regular expressions for the internalProxies, is it possible to take advantage of this facility for your use-case? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#22643 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAIEeOz9JGT8O9QIlfm9xPBDN04MpTZ4ks5uYRLggaJpZM4WYhqq> .

[mikn]

Is there any possibility of an update on this? A decision either way would help us move forward with this issue. :)

[kobelb]

@mikn given my understanding of the impact of this change, I don't think it's sufficient to justify us rolling back this change in behavior. I'd like to keep this issue open for the time being to collect additional input from the community to determine what other use-cases are impacted by this change.

[hs750]

This is also an issue when using a reverse proxy (such as NGINX) to secure/authenticate communication with elasticsearch. In our use case we use mutual authentication between clients (including kibana) and nginx. Because the certificates are no longer sent, nginx now rejects the requests made from kibana.

The lack of documentation about this issue is noted in #18504.

[LukeCross]

I am also experiencing this issue trying to integrate both Searchguard and an Nginx reverse proxy with Elasticsearch 6.4.1 and Kibana 6.4.1. The issue being that using client certificate authentication and Kibana 6.4.1, as the certificates are not sent with every request, both Nginx and Searchguard both reject the requests.

[maglo]

I have seen the same problem. We are using an Auth-scheme where we take care of authenticating requests to Kibana. We then use a client certificate to secure the connection from Kibana to Elasticsearch by way of using a reverse proxy.

We cannot upgrade to 6.4 without redoing our auth scheme.

Consider making it an option to use the certificates even for proxied requests.

[anthonyhaussman]

The same problem via a reverse proxy configures with Nginx.
It is difficult fo to upgrade Kibana without this feature.

To add the option would be great (even if it's disabled by default).

[kobelb]

Thanks everyone for providing your use-cases, I think we have sufficient support and reasoning for adding back in this functionality behind a configuration flag.