From a430e8da8f58381aaccbdb674d428be4047756ba Mon Sep 17 00:00:00 2001 From: Srushti Sapkale Date: Fri, 15 Jul 2022 15:23:20 +0530 Subject: [PATCH 1/6] Proposal: Granular Endpoint Configuration Signed-off-by: Srushti Sapkale --- .../202207-granular-endpoint-configuration | 57 +++++++++++++++++++ 1 file changed, 57 insertions(+) create mode 100644 docs/proposals-accepted/202207-granular-endpoint-configuration diff --git a/docs/proposals-accepted/202207-granular-endpoint-configuration b/docs/proposals-accepted/202207-granular-endpoint-configuration new file mode 100644 index 0000000000..c532b0503f --- /dev/null +++ b/docs/proposals-accepted/202207-granular-endpoint-configuration @@ -0,0 +1,57 @@ +# Granular Endpoint Configuration + +* **Related Tickets:** + + * [Per-Store TLS configuration in Thanos Query](https://github.com/thanos-io/thanos/issues/977) + +## What + +This proposal builds up on the [Unified Endpoint Discovery Proposal](https://thanos.io/tip/proposals-accepted/202101-endpoint-discovery.md/) and [Automated, per-endpoint mTLS Proposal](https://thanos.io/tip/proposals-accepted/202106-automated-per-endpoint-mtls.md/) + +## Why + +The Thanos Querier component supports basic mTLS configuration for internal gRPC communication. Mutual TLS (mTLS) ensures that traffic is both secure and trusted in both directions between a client and server. This works great for basic use cases but it still requires extra forward proxies to make it work for bigger deployments. + +Let’s imagine we have an Observer Cluster that is hosting Thanos Querier along with Thanos Store Gateway. Within the same Observer Cluster, we would like to connect one or more Thanos Sidecars. Additionally, we also want to connect the Querier in the Observer Cluster to several remote instances of Thanos Sidecar in remote clusters. + +In this scenario we need to use a proxy server (e.g. envoy). But, it would probably be simpler to do away with the proxies and let Thanos speak directly and securely with the remote endpoints. + +## Pitfalls of the current solution + +Ideally, we would want to use mTLS to encrypt the connection to the remote clusters. If we would enable the current mTLS, it would be applied to all the storeAPI’s but we don’t want it to be applied on the storeAPI’s of central Thanos instance (Observer cluster) in which Thanos query component is present (for faster communications with storeAPI’s (sidecars) of same cluster, to reduce the pain of provisioning certificates etc.). So it requires extra forward proxies to make it work for bigger (multi-cluster) deployments. + +## Goals + +* To add support for per-endpoint TLS configuration in Thanos Query Component for internal gRPC communication + +## Non-Goals + +## How + +A new CLI option `--endpoints.config`, with no dynamic reloading, which will accept the path to a yaml file is proposed which contains a list as follows : + +```go +- tls_config: + cert_file: "" + key_file: "" + ca_file: "" + server_name: "" + endpoints: [] + endpoints_sd_files: + - files: [] + mode: "" +``` + +The YAML file contains set of endpoints (e.g Store API) with optional TLS options. To enable TLS either use this option or deprecated ones --grpc-client-tls* . + +The endpoints in the list items with no TLS config would be considered insecure. Further endpoints supplied via separate flags (e.g. `--endpoint`, `--endpoint.strict`, `--endpoint.sd-files` (previous `--store.strict`, `--store.sd-files`)) will be considered secure only if TLS config is provided through `--secure`, `--cert`, `--key`, `--caCert`, `--serverName` cli options. User will only be allowed to use separate options or `--endpoints.config`. + +If the mode is strict (i.e. `mode: ”strict”`) then all the endpoints in that list item will be considered as strict (statically configured store API servers that are always used, even if the health check fails, as supplied in `--endpoint.strict`). And if there is no mode specified (i.e `mode: “”`) then all endpoints in that item will be considered normal. + +## Alternatives + +## Action Plan + +* Implement `--endpoint.config`. +* Adding a new `endpoint.config` flag will deprecate the following flags: `store.sd-interval`, `store.sd-dns-interval`, `store.sd-dns-resolver`, `store.sd-files` and all `grpc-client-.*`. +We can mark the current flags as deprecated, and after some time we can remove them. From 0cd9a9ed43fd64c3e09d18226549473857f22144 Mon Sep 17 00:00:00 2001 From: Srushti Sapkale Date: Fri, 15 Jul 2022 16:36:40 +0530 Subject: [PATCH 2/6] update action plan Signed-off-by: Srushti Sapkale --- docs/proposals-accepted/202207-granular-endpoint-configuration | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/proposals-accepted/202207-granular-endpoint-configuration b/docs/proposals-accepted/202207-granular-endpoint-configuration index c532b0503f..c1d7fb3698 100644 --- a/docs/proposals-accepted/202207-granular-endpoint-configuration +++ b/docs/proposals-accepted/202207-granular-endpoint-configuration @@ -55,3 +55,4 @@ If the mode is strict (i.e. `mode: ”strict”`) then all the endpoints in that * Implement `--endpoint.config`. * Adding a new `endpoint.config` flag will deprecate the following flags: `store.sd-interval`, `store.sd-dns-interval`, `store.sd-dns-resolver`, `store.sd-files` and all `grpc-client-.*`. We can mark the current flags as deprecated, and after some time we can remove them. +* Add e2e tests for the changes \ No newline at end of file From 3d7a6073385db767fbf6066ae4330b7246097aea Mon Sep 17 00:00:00 2001 From: Srushti Sapkale Date: Fri, 22 Jul 2022 15:52:55 +0530 Subject: [PATCH 3/6] update proposal Signed-off-by: Srushti Sapkale --- .../202207-granular-endpoint-configuration | 58 ------------------- .../202207-granular-endpoint-configuration.md | 0 2 files changed, 58 deletions(-) delete mode 100644 docs/proposals-accepted/202207-granular-endpoint-configuration create mode 100644 docs/proposals-accepted/202207-granular-endpoint-configuration.md diff --git a/docs/proposals-accepted/202207-granular-endpoint-configuration b/docs/proposals-accepted/202207-granular-endpoint-configuration deleted file mode 100644 index c1d7fb3698..0000000000 --- a/docs/proposals-accepted/202207-granular-endpoint-configuration +++ /dev/null @@ -1,58 +0,0 @@ -# Granular Endpoint Configuration - -* **Related Tickets:** - - * [Per-Store TLS configuration in Thanos Query](https://github.com/thanos-io/thanos/issues/977) - -## What - -This proposal builds up on the [Unified Endpoint Discovery Proposal](https://thanos.io/tip/proposals-accepted/202101-endpoint-discovery.md/) and [Automated, per-endpoint mTLS Proposal](https://thanos.io/tip/proposals-accepted/202106-automated-per-endpoint-mtls.md/) - -## Why - -The Thanos Querier component supports basic mTLS configuration for internal gRPC communication. Mutual TLS (mTLS) ensures that traffic is both secure and trusted in both directions between a client and server. This works great for basic use cases but it still requires extra forward proxies to make it work for bigger deployments. - -Let’s imagine we have an Observer Cluster that is hosting Thanos Querier along with Thanos Store Gateway. Within the same Observer Cluster, we would like to connect one or more Thanos Sidecars. Additionally, we also want to connect the Querier in the Observer Cluster to several remote instances of Thanos Sidecar in remote clusters. - -In this scenario we need to use a proxy server (e.g. envoy). But, it would probably be simpler to do away with the proxies and let Thanos speak directly and securely with the remote endpoints. - -## Pitfalls of the current solution - -Ideally, we would want to use mTLS to encrypt the connection to the remote clusters. If we would enable the current mTLS, it would be applied to all the storeAPI’s but we don’t want it to be applied on the storeAPI’s of central Thanos instance (Observer cluster) in which Thanos query component is present (for faster communications with storeAPI’s (sidecars) of same cluster, to reduce the pain of provisioning certificates etc.). So it requires extra forward proxies to make it work for bigger (multi-cluster) deployments. - -## Goals - -* To add support for per-endpoint TLS configuration in Thanos Query Component for internal gRPC communication - -## Non-Goals - -## How - -A new CLI option `--endpoints.config`, with no dynamic reloading, which will accept the path to a yaml file is proposed which contains a list as follows : - -```go -- tls_config: - cert_file: "" - key_file: "" - ca_file: "" - server_name: "" - endpoints: [] - endpoints_sd_files: - - files: [] - mode: "" -``` - -The YAML file contains set of endpoints (e.g Store API) with optional TLS options. To enable TLS either use this option or deprecated ones --grpc-client-tls* . - -The endpoints in the list items with no TLS config would be considered insecure. Further endpoints supplied via separate flags (e.g. `--endpoint`, `--endpoint.strict`, `--endpoint.sd-files` (previous `--store.strict`, `--store.sd-files`)) will be considered secure only if TLS config is provided through `--secure`, `--cert`, `--key`, `--caCert`, `--serverName` cli options. User will only be allowed to use separate options or `--endpoints.config`. - -If the mode is strict (i.e. `mode: ”strict”`) then all the endpoints in that list item will be considered as strict (statically configured store API servers that are always used, even if the health check fails, as supplied in `--endpoint.strict`). And if there is no mode specified (i.e `mode: “”`) then all endpoints in that item will be considered normal. - -## Alternatives - -## Action Plan - -* Implement `--endpoint.config`. -* Adding a new `endpoint.config` flag will deprecate the following flags: `store.sd-interval`, `store.sd-dns-interval`, `store.sd-dns-resolver`, `store.sd-files` and all `grpc-client-.*`. -We can mark the current flags as deprecated, and after some time we can remove them. -* Add e2e tests for the changes \ No newline at end of file diff --git a/docs/proposals-accepted/202207-granular-endpoint-configuration.md b/docs/proposals-accepted/202207-granular-endpoint-configuration.md new file mode 100644 index 0000000000..e69de29bb2 From f4002b5b4a6aa3a9247f608661c9ddbd2f8e3ef7 Mon Sep 17 00:00:00 2001 From: Srushti Sapkale Date: Fri, 22 Jul 2022 16:07:51 +0530 Subject: [PATCH 4/6] update proposal Signed-off-by: Srushti Sapkale --- .../202207-granular-endpoint-configuration.md | 71 +++++++++++++++++++ 1 file changed, 71 insertions(+) diff --git a/docs/proposals-accepted/202207-granular-endpoint-configuration.md b/docs/proposals-accepted/202207-granular-endpoint-configuration.md index e69de29bb2..a73b13c77e 100644 --- a/docs/proposals-accepted/202207-granular-endpoint-configuration.md +++ b/docs/proposals-accepted/202207-granular-endpoint-configuration.md @@ -0,0 +1,71 @@ +--- +type: proposal +title: Granular Endpoint Configuration +status: approved +owner: SrushtiSapkale +menu: proposals-accepted +--- + +* **Related Tickets:** + + * [Per-Store TLS configuration in Thanos Query](https://github.com/thanos-io/thanos/issues/977) + +## What + +This proposal builds up on the [Unified Endpoint Discovery Proposal](https://thanos.io/tip/proposals-accepted/202101-endpoint-discovery.md/) and [Automated, per-endpoint mTLS Proposal](https://thanos.io/tip/proposals-accepted/202106-automated-per-endpoint-mtls.md/) + +## Why + +The Thanos Querier component supports basic mTLS configuration for internal gRPC communication. Mutual TLS (mTLS) ensures that traffic is both secure and trusted in both directions between a client and server. This works great for basic use cases but it still requires extra forward proxies to make it work for bigger deployments. + +Let’s imagine we have an Observer Cluster that is hosting Thanos Querier along with Thanos Store Gateway. Within the same Observer Cluster, we would like to connect one or more Thanos Sidecars. Additionally, we also want to connect the Querier in the Observer Cluster to several remote instances of Thanos Sidecar in remote clusters. + +In this scenario we need to use a proxy server (e.g. envoy). But, it would probably be simpler to do away with the proxies and let Thanos speak directly and securely with the remote endpoints. + +## Pitfalls of the current solution + +Ideally, we would want to use mTLS to encrypt the connection to the remote clusters. If we would enable the current mTLS, it would be applied to all the storeAPI’s but we don’t want it to be applied on the storeAPI’s of central Thanos instance (Observer cluster) in which Thanos query component is present (for faster communications with storeAPI’s (sidecars) of same cluster, to reduce the pain of provisioning certificates etc.). So it requires extra forward proxies to make it work for bigger (multi-cluster) deployments. + +## Goals + +* To add support for per-endpoint TLS configuration in Thanos Query Component for internal gRPC communication. + +## How + +A new CLI option `--endpoints.config`, with no dynamic reloading, which will accept the path to a yaml file is proposed which contains a list as follows : + +```yaml +- tls_config: + cert_file: "" + key_file: "" + ca_file: "" + server_name: "" + endpoints: [] + endpoints_sd_files: + - files: [] + mode: "" +``` + +The YAML file contains set of endpoints (e.g Store API) with optional TLS options. To enable TLS either use this option or deprecated ones --grpc-client-tls* . + +The endpoints in the list items with no TLS config would be considered insecure. Further endpoints supplied via separate flags (e.g. `--endpoint`, `--endpoint.strict`, `--endpoint.sd-files` (previous `--store.strict`, `--store.sd-files`)) will be considered secure only if TLS config is provided through `--secure`, `--cert`, `--key`, `--caCert`, `--serverName` cli options. User will only be allowed to use separate options or `--endpoints.config`. + +If the mode is strict (i.e. `mode: ”strict”`) then all the endpoints in that list item will be considered as strict (statically configured store API servers that are always used, even if the health check fails, as supplied in `--endpoint.strict`). And if there is no mode specified (i.e `mode: “”`) then all endpoints in that item will be considered normal. + +## TODO + +* To update the filtering of APIs in the querier to return only the endpoints that are needed. + +For example: +The user wants to connect to the store api only.But here, we notice that the store method indiscriminately returns all endpoints ie exemplar api, store api, etc, even if they do not expose store API. + +This is because the query exposes endpoints of all the apis. We can update the filtering logic where the querier endpoint knows which api endpoints it should expose for a particular component. + +## Alternatives + +## Action Plan + +* Implement `--endpoint.config`. +* Adding a new `endpoint.config` flag will deprecate the following flags: `store.sd-interval`, `store.sd-dns-interval`, `store.sd-dns-resolver`, `store.sd-files` and all `grpc-client-.*`. We can also remove the hidden flags `ruleEndpoints`, `metadataEndpoints`, `exemplarEndpoints`. +We can mark the current flags as deprecated, and after some time we can remove them. +* Add e2e tests for the changes From 318f7ed4d07bb09b6c50b5241d58f78bc0f00f5b Mon Sep 17 00:00:00 2001 From: Srushti Sapkale Date: Thu, 28 Jul 2022 12:10:06 +0530 Subject: [PATCH 5/6] Changes according to the suggestions Signed-off-by: Srushti Sapkale --- .../202207-granular-endpoint-configuration.md | 20 ++++++++----------- 1 file changed, 8 insertions(+), 12 deletions(-) diff --git a/docs/proposals-accepted/202207-granular-endpoint-configuration.md b/docs/proposals-accepted/202207-granular-endpoint-configuration.md index a73b13c77e..34d9300eee 100644 --- a/docs/proposals-accepted/202207-granular-endpoint-configuration.md +++ b/docs/proposals-accepted/202207-granular-endpoint-configuration.md @@ -1,7 +1,7 @@ --- type: proposal title: Granular Endpoint Configuration -status: approved +status: Approved owner: SrushtiSapkale menu: proposals-accepted --- @@ -20,7 +20,7 @@ The Thanos Querier component supports basic mTLS configuration for internal gRPC Let’s imagine we have an Observer Cluster that is hosting Thanos Querier along with Thanos Store Gateway. Within the same Observer Cluster, we would like to connect one or more Thanos Sidecars. Additionally, we also want to connect the Querier in the Observer Cluster to several remote instances of Thanos Sidecar in remote clusters. -In this scenario we need to use a proxy server (e.g. envoy). But, it would probably be simpler to do away with the proxies and let Thanos speak directly and securely with the remote endpoints. +In this scenario we can use a proxy server (e.g. envoy). In thanos as we have multiple queriers connected to a central querier, Envoy can efficiently loadbalance gRPC connections, something we currently don't support natively in Thanos. If we used the endpoints config to connect the global to the regional queriers, they will not be able to scale regional queriers due to the persistent nature of gRPC connections. We can use Envoy when we need to scale our regional queries but here having an endpoint configuration is a better option in distributed store architecture as Thanos can communicate securely and directly through remote endpoints. ## Pitfalls of the current solution @@ -29,6 +29,8 @@ Ideally, we would want to use mTLS to encrypt the connection to the remote clust ## Goals * To add support for per-endpoint TLS configuration in Thanos Query Component for internal gRPC communication. +* Adding a new `endpoint.config` flag will deprecate the following flags: `store.sd-interval`, `store.sd-dns-interval`, `store.sd-dns-resolver`, `store.sd-files` and all `grpc-client-.*`. We can mark the current flags as deprecated, and after some time we can remove them. +* Enable token based auth for endpoints ## How @@ -52,20 +54,14 @@ The endpoints in the list items with no TLS config would be considered insecure. If the mode is strict (i.e. `mode: ”strict”`) then all the endpoints in that list item will be considered as strict (statically configured store API servers that are always used, even if the health check fails, as supplied in `--endpoint.strict`). And if there is no mode specified (i.e `mode: “”`) then all endpoints in that item will be considered normal. -## TODO - -* To update the filtering of APIs in the querier to return only the endpoints that are needed. - -For example: -The user wants to connect to the store api only.But here, we notice that the store method indiscriminately returns all endpoints ie exemplar api, store api, etc, even if they do not expose store API. - -This is because the query exposes endpoints of all the apis. We can update the filtering logic where the querier endpoint knows which api endpoints it should expose for a particular component. +Also we noticed that even if the user wants to connect to the store api only, the store method indiscriminately returns all endpoints ie exemplar api, store api, etc, even if they do not expose store API which is confusing. This is because the query exposes endpoints of all the apis. We can update the filtering logic where the querier endpoint knows which api endpoints it should expose for a particular component. ## Alternatives +* Granular endpoint configuration is not a solution for federate querier loadbalncing and scalability. As an alternative can embed Envoy in thanos for loadbalancing. This would work efficiently because endpointset cannot configure federated queriers when they scale due to persistent nature of grpc connections. + ## Action Plan * Implement `--endpoint.config`. -* Adding a new `endpoint.config` flag will deprecate the following flags: `store.sd-interval`, `store.sd-dns-interval`, `store.sd-dns-resolver`, `store.sd-files` and all `grpc-client-.*`. We can also remove the hidden flags `ruleEndpoints`, `metadataEndpoints`, `exemplarEndpoints`. -We can mark the current flags as deprecated, and after some time we can remove them. * Add e2e tests for the changes +* Update the filtering of APIs in the querier to return only the endpoints that are needed. From c9f85b1c5b86e94dbbdd033715b1bd44f2ccbeca Mon Sep 17 00:00:00 2001 From: Srushti Sapkale Date: Wed, 3 Aug 2022 09:38:35 +0530 Subject: [PATCH 6/6] Implement suggestions Signed-off-by: Srushti Sapkale --- .../202207-granular-endpoint-configuration.md | 36 +++++++++++-------- 1 file changed, 21 insertions(+), 15 deletions(-) diff --git a/docs/proposals-accepted/202207-granular-endpoint-configuration.md b/docs/proposals-accepted/202207-granular-endpoint-configuration.md index 34d9300eee..6d6330ed32 100644 --- a/docs/proposals-accepted/202207-granular-endpoint-configuration.md +++ b/docs/proposals-accepted/202207-granular-endpoint-configuration.md @@ -1,5 +1,5 @@ --- -type: proposal +type: Proposal title: Granular Endpoint Configuration status: Approved owner: SrushtiSapkale @@ -12,31 +12,39 @@ menu: proposals-accepted ## What -This proposal builds up on the [Unified Endpoint Discovery Proposal](https://thanos.io/tip/proposals-accepted/202101-endpoint-discovery.md/) and [Automated, per-endpoint mTLS Proposal](https://thanos.io/tip/proposals-accepted/202106-automated-per-endpoint-mtls.md/) +This proposal builds up on the [Unified Endpoint Discovery Proposal](https://thanos.io/tip/proposals-accepted/202101-endpoint-discovery.md/) and [Automated, per-endpoint mTLS Proposal](https://thanos.io/tip/proposals-accepted/202106-automated-per-endpoint-mtls.md/) and to add possibility to have granular setting per endpoint and deprecate old flags. ## Why -The Thanos Querier component supports basic mTLS configuration for internal gRPC communication. Mutual TLS (mTLS) ensures that traffic is both secure and trusted in both directions between a client and server. This works great for basic use cases but it still requires extra forward proxies to make it work for bigger deployments. - -Let’s imagine we have an Observer Cluster that is hosting Thanos Querier along with Thanos Store Gateway. Within the same Observer Cluster, we would like to connect one or more Thanos Sidecars. Additionally, we also want to connect the Querier in the Observer Cluster to several remote instances of Thanos Sidecar in remote clusters. - -In this scenario we can use a proxy server (e.g. envoy). In thanos as we have multiple queriers connected to a central querier, Envoy can efficiently loadbalance gRPC connections, something we currently don't support natively in Thanos. If we used the endpoints config to connect the global to the regional queriers, they will not be able to scale regional queriers due to the persistent nature of gRPC connections. We can use Envoy when we need to scale our regional queries but here having an endpoint configuration is a better option in distributed store architecture as Thanos can communicate securely and directly through remote endpoints. +The Thanos Querier component supports basic mTLS configuration for internal gRPC communication. Mutual TLS (mTLS) ensures that traffic is both secure and trusted in both directions between a client and server. This works great for basic use cases, but it still requires extra forward proxies to make it work for quite common deployments where different servers have different auth, cert or TLS options. +This is similar to cases where we want to have further per-endpoint configuration pieces like what APIs to enable or in future what load balancing groups it belongs to. +We also have to be prepared for more options that will appear as the functionality of Queriers grows per endpoint (e.g limits). ## Pitfalls of the current solution -Ideally, we would want to use mTLS to encrypt the connection to the remote clusters. If we would enable the current mTLS, it would be applied to all the storeAPI’s but we don’t want it to be applied on the storeAPI’s of central Thanos instance (Observer cluster) in which Thanos query component is present (for faster communications with storeAPI’s (sidecars) of same cluster, to reduce the pain of provisioning certificates etc.). So it requires extra forward proxies to make it work for bigger (multi-cluster) deployments. +Current users cannot use different certifcates, auth or TLS options for different endpoints. The current workarounds is to use layered queries for this which is much more complex and confusing deployment. +Second, quite viable option is to use forward proxy like envoy. The problem with this approach is that any Thanos adopter have to learn yet another configuration language/options, deployment (build, release, support) models and operational (metrics, migration) models which is not trivial and hinders adoptions. ## Goals -* To add support for per-endpoint TLS configuration in Thanos Query Component for internal gRPC communication. -* Adding a new `endpoint.config` flag will deprecate the following flags: `store.sd-interval`, `store.sd-dns-interval`, `store.sd-dns-resolver`, `store.sd-files` and all `grpc-client-.*`. We can mark the current flags as deprecated, and after some time we can remove them. -* Enable token based auth for endpoints +* To add support for per-endpoint granularity configuration. For example: + * TLS configuration in for Querier gRPC communication. + * Basic/Token Auth + * Strict mode + * Enabled APIs (specifying that server which can serve Exemplars and Store should only be asked for Exemplars) + * (perhaps more in future) +* Unify endpoint configuration. Which means deprecating all non `endpoint` flags for endpoint configuration like `rule`, `metadata`, `exemplar`, `store.sd-interval`, `store.sd-dns-interval`, `store.sd-dns-resolver`, `store.sd-files` and all `grpc-client-.*`. + +## Non Goals + +* Granular configuration & implementation for load balancing groups. This can be improved in the next proposals and is currently fulfilled using any forward or reverse proxy like `envoy`, `caddy` or `nginx`. ## How -A new CLI option `--endpoints.config`, with no dynamic reloading, which will accept the path to a yaml file is proposed which contains a list as follows : +A new CLI option `--endpoints.config`, with no dynamic reloading, which will accept the path to a yaml file is proposed which contains a list as follows: ```yaml +endpoints: - tls_config: cert_file: "" key_file: "" @@ -58,10 +66,8 @@ Also we noticed that even if the user wants to connect to the store api only, th ## Alternatives -* Granular endpoint configuration is not a solution for federate querier loadbalncing and scalability. As an alternative can embed Envoy in thanos for loadbalancing. This would work efficiently because endpointset cannot configure federated queriers when they scale due to persistent nature of grpc connections. - ## Action Plan * Implement `--endpoint.config`. -* Add e2e tests for the changes +* Add e2e tests for the changes. * Update the filtering of APIs in the querier to return only the endpoints that are needed.