Time series fill/extend problem consulting in Promxy integration with VictoriaMetrics

[SRE-Rick]

Dear Tom Jackson,

I am an engineer currently utilizing Promxy in our monitoring stack, and I've come across an issue regarding its integration with VictoriaMetrics (vmselect) that I hope to get your insights on.

We have observed a phenomenon where Promxy appears to "fill" or "extend" time series data when VictoriaMetrics stops reporting new data. Specifically, Promxy seems to continue using the last data point from VictoriaMetrics for several minutes before acknowledging the data's staleness. This behavior, commonly referred to as "Staleness Handling" in Prometheus terminology, is causing some confusion in our monitoring reports.

Could you please clarify if this is an expected behavior of Promxy in this scenario? If not, are there any recommended configurations or practices that we can implement to address this issue? Our goal is to accurately represent the real-time status of our data and prevent any misinterpretations caused by stale data.

I am more than willing to provide further configuration details or logs if necessary.

Thank you very much for your contributions to the Promxy project, and I look forward to your guidance on this matter.

Best regards,
rick

[SRE-Rick]

I hope to solve the promxy Staleness Handling problem, subject to the result of vmselect

[jacksontj]

Thanks for the question! This is actually a fairly common question in the prometheus world -- without getting into too much detail, this is due to the QueryLookbackDelta -- which is the same option in prometheus. Basically what it does is configure the promql engine with "how far back" it is allowed to use datapoints. So you can mess with reducing that time interval -- which (to mirror upstream) defaults to 5m (meaning it will allow reading back up to 5m for a datapoint).

In most cases you'll want this to be relatively short (maybe 0-2x your collection interval).

[SRE-Rick]

I fixed this issue after upgrading from v0.0.81 to v0.0.84

[SRE-Rick]

Can you tell us which code handles this

[jacksontj]

If your specific issue was resolved with that upgrade; I assume your query included subqueries (as there was a fix in that range -- #609 )

[SRE-Rick]

如果您的具体问题通过该升级得到解决；我假设您的查询包含子查询（因为该范围内有一个修复 - #609）

That's right. Thank you