max-points-per-req-soft / max-points-per-req-hard misleading

[Dieterbe]

# lower resolution rollups will be used to try and keep requests below this number of datapoints. (0 disables limit)
max-points-per-req-soft = 1000000
# limit of number of datapoints a request can return. Requests that exceed this limit will be rejected. (0 disables limit)
max-points-per-req-hard = 20000000

"request" is ambiguous whether it's a client render request, or an internal models.Req

"soft" is a limit applied on the number of points fetched (models.Req) only. the documentation is not clear on this.
"hard" is a limit on the number of points returned to the client. points at this interval here may be fetched natively, or it may be the result of runtime consolidation (meaning: we allow to fetch substantially more points than the limit, as long as they are runtime consolidated, they "fly under the radar")

These subtleties, and the inconsistency between soft and hard makes using these parameters hard. For the hard parameter, it makes sense to have both:

• a cap on points returned to the client (basically limiting the size of the http response)
• a cap on points fetched (as with our current architecture where we load everything up front, that's the limiting factor to what we can handle. points fetched corresponds directly to RAM used), until we do more JIT-decoding of chunks, that is.

Note also that to address #926 we will no longer align all models.Req for the entire http request all at once. We will support different series with different intervals. To honor soft, we'll need to come up with some sort of heuristic (e.g. try to load coarser data for series that are at the highest resolution, as long as they are not processed by processing functions such as summarize())

Because of these upcoming changes, addressing this ticket is low priority. mainly wanted to document this issue.

[shanson7]

"hard" is a limit on the number of points returned to the client

Is this true? My understanding is that hard is a circuit breaker before functions are even run.

[Dieterbe]

yes, the time of the check/abort is during alignRequests, before any data is fetched or processing is run, however it does the check based on the settings it calculates during alignRequests. if it knows that num outbound targets * points per target is higher than max-points-per-req-hard, it aborts the request. (note: it is not aware of functions such as summarize() that may lengthen the interval and reduce the amount of points)

[shanson7]

I just don't see this. Looking at the code for alignRequests it seems like it's doing the same check as soft. That is, it's checking based on the number of series fetched not that it plans to return.

This makes sense, because it's also not aware of aggregation functions that may be applied.

[Dieterbe]

You're right for the number of targets: it uses number of requested targets.
But with the current code, the hard limit is checked after interval := util.Lcm(listIntervals)
this one line may cause normalization (runtime consolidation) (e.g. if you have one series with interval 1 and another with 120, it will normalize the first to 120, or if you have 10 and 15 it'll normalize both to 30), so many more points may be fetched than what it thinks. also, fewer targets may be returned than what it thinks (due to aggregations).
So basically, it checks some number that could be points fetched, points returned, or anywhere in between.

[shanson7]

Specifically, though, this doesn't account for runtime consolidation due to MDP (which is what I think of primarily for runtime consolidation). So, in its current form, I don't think saying it's "a limit on the number of points returned to the client" is correct at all.

IMO, such a limit is not really that useful, since MDP already exists and it would be difficult to enforce before applying functions. More importantly, I think that there is utility in a hard limit that behaves similar to as it does today (to protect read/query nodes).

[Dieterbe]

Specifically, though, this doesn't account for runtime consolidation due to MDP (which is what I think of primarily for runtime consolidation). So, in its current form, I don't think saying it's "a limit on the number of points returned to the client" is correct at all.

You're right, I forgot to account for MDP. So, there's 3 factors that make the hard setting unintuitive (or more specifically, not exactly match either points returned, nor points fetched):

• MDP consolidation (means points returned can be less than what we check against)
• normalization (means points fetched can be more than what we check against)
• target reduction due to aggregation functions (means points returned can be less than what we check against)

If it weren't for normalization, "hard" would check accurately against points fetched. Thus we could fix this and make it a more accurate check against points fetched by looking at how many points we would actually fetch (looking at the archive point counts)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Dieterbe]

This is fixed as of #951