[Feature] Hand over downsampling(hour/day) metric processes to BanyanDB

[wu-sheng]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

With a deep discussion in #12290, we will face the alarm feature unfunctional.

Alternatively, we would like to do downsampling data for hour and day dimensionalities. These data would not need to worry about alarm features.

So, L1 and L2 will still exist with the alarm feature. But we will disable the downsampling mechanism in BanyanDB storage and hand over it to BanyanDB.

Note, BanyanDB operation, OAP will be required to build this server side row-level aggregation when it is ready, ref #8720

Use case

No response

Related issues

No response

Are you willing to submit a pull request to implement this on your own?

• Yes I am willing to submit a pull request on my own!

Code of Conduct

• I agree to follow this project's Code of Conduct

[wu-sheng]

Note, even with this solution, the minute dimension is still being pushed as an aggregated result, we will have to decide should we push delta data for downsampling aggregation.

• If we don't, this feature seems impractical too, as BanyanDB pipeline is hard to tell the delta, unless it does the calculation reversedly, new aggregated minus(-) pre-data.
• If we do, we only optimize the memory cache of hour/day, and can't reduce the traffic. Because we have a much lower flush period for hour/day compared with minute, the minute delta has to keep the same flush period as minute data.

[wu-sheng]

From this deep discussion and thinking, I have doubts that this server-side aggregation is feasible and necessity.

@hanahmily Let's re-think about this. Maybe we should remove this in the milestone in the short-term, until the all doubts have solutions.

FYI @wankai123

[wu-sheng]

Summary of downsides for server-side aggregation

1. Removing L2 would impact the alerting. Because minute dimension eventual(aggregated) data will be lost from OAP.
2. Keeping minute dimension aggregation makes the delta data lost, and causes another extra round delta data in minute dimension flushing.
3. Clearly, this server-side aggregation increases the server-side payload(CPU cost), but wouldn't tradeoff for IOPS.
4. The process pipeline would be more complex.

---

The only positive part is the cache of hour/day dimension metrics could be removed.

@hanahmily @wankai123 Let me know what do you think.

[wu-sheng]

We have an agreement, this feature benefit is too limited. I am closing this for now.