feat: query forward

[BohuTANG]

Summary

Assuming we have a Databend cluster with three nodes, each serving as a service:
Node-1: Create a query ID X. If the query is sent to Node-2 but not found, forward it to Node-1.

Benefits:

• Flexible Routing: Avoids the necessity to route queries strictly to their origin node.
• Statelessness: Similar to k8s deployments, the system doesn't require stateful management.

[yufan022]

Do we need to consider failure? sql callers need to know clearly whether sql execution succeeds or fails.

If forwarding fails and returns execution fails, but the actual sql is still executing, it will cause confusion.

[BohuTANG]

@yufan022

Yes, we should.

Several considerations remain:

• Handling forward failures.
• Determining the layer for forwarding: Should it occur in the HTTP handler or at a lower level?

[yufan022]

some sugesstion:

The forwarding path should not change if the number of query members changes(in process of scale-in or scale-out).

If the client side gets SQL execution failure, then it must really fail, no matter on any node.

[BohuTANG]

Here's a condensed version of the protocol:

1. Client sends a request to query-1 (warehouse-id: w1).
2. Client sends the next request to query-2 (warehouse-id: w2), including a header with the previous warehouse-id (w1).
3. Query-2 responds with a forward protocol(reponse with query-1 host?), instructing the client to redirect the query to query-1.
4. Client redirects the query to query-1.
5. Client receives the result from query-1.

This protocol ensures that the client interacts with the correct query node based on the previous request's warehouse-id, maintaining consistency and safety in a 3-node cluster.

I am not sure this works in k8s, please clarify, @ZhiHanZ

[ZhiHanZ]

It makes sense, basically in kubernetes, if we contain any state on header using deployment as databend onboarding app, the query forward have to be handled by several workaround. Either using a customized gateway/nginx/istio/ingress controller, such kind of workarounds are still fragile to changes, if we could implement the feature on kernel, we could have more simple and easy way to deploy databend on any environment.

cc @BohuTANG , I think we need to prioritize the issue.