[Multistage] Improve Observability for Diagnosing Performance of Multistage Engine Queries

[satwik-pachigolla]

Observability to diagnose performance of the multistage engine queries is lacking. On separate occasions, we observed latency spikes and timeouts for multistage engine queries for a brief period of time. There aren't enough useful logs or metrics to pinpoint which instance(s) is taking longer retrospectively.

Some specific gaps:

• Query failures without metrics
• No logs or stats in response metadata that can be used to identify slow instances in any stage of executing a query. Or no way of correlating broker request IDs to the logs or stats.
• Timeouts are difficult to diagnose without taking an approach such as increasing the timeout, rerunning, and then profiling the query but once again that does not enable retrospective debugging. All that is available retrospectively are a high volume of logs across many instances such as:

Caught exception while processing query
[2025-02-01 10:33:42.386841] java.util.concurrent.TimeoutException: Timed out while offering data to mailbox

these logs do not provide further useful information.

While all the above observations seem to be unchanged in the latest version as of this posting, these observations were from Pinot version 17332de which is very slightly ahead of the 1.2 release version.

[Jackie-Jiang]

@gortiz Can you help take a look at this? Have you already made some improvements in 1.3 release?