cortex_ingester_inflight_push_requests for ingesters doesn't show the correct value of the metric.

[sarthaktyagi-505]

Describe the bug

Ingesters keep showing the following error err-mimir-ingester-max-inflight-push-requests in the logs, When i check the cortex_ingester_inflight_push_requests metric for that particular ingester, The value of ingester.instance-limits.max-inflight-push-requests shows up to never hitting the limit which we have set to 60k per ingester. We are not able to get the metric correctly.

To Reproduce

Steps to reproduce the behavior:

1. Check the logs of ingesters with enough write load . You will see the following error
   the write request has been rejected because the ingester exceeded the allowed number of inflight push requests (err-mimir-ingester-max-inflight-push-requests). To adjust the related limit, configure -ingester.instance-limits.max-inflight-push-requests, or contact your service administrator.
2. Check the metric cortex_ingester_inflight_push_requests for the same ingester where you see the error.
3. You will observe the metric is not accurate.

Expected behavior

We should get the right metric so we can see how many inflight push requests our ingesters are going through so we can properly scale it to the right demands.

Environment

• Infrastructure: Kubernetes
• Deployment tool: jsonnet

Additional Context

1. I am part of the mimir slack channel and an Engineer advised me to check if the ingesters are cpu starved which i checked and seems like our ingesters have enough CPU and do not have any issues with core assignment.
2. I also reduced the scrape interval metrics of the ingesters to 2 seconds to get enough data points to get the right value of the metric and was still unable to get that.
3. Our ingester resides on a pod which has 16 cores, we have set the request to 7 cores but we havent set any limits. At the time the screenshots were taken the ingester was using 6.5 cores.
4. Attaching screenshots of logs seen on loki and metric value for the same time interval.
   NOTE: Please note we have configured the ingester.instance-limits.max-inflight-push-requests property to 60k per ingester
   
   

[bboreham]

Whilst I agree that it should report the correct value, the gap between your 60,000 limit and the 1 or 2 shown in the metric at the time of the error message is so large that I have to suspect something else is going on.

What does the go_goroutines metric report at the same time?

[sarthaktyagi-505]

I checked that for the same ingester the value is 1900 for the moment, which seems quiet high to me, what do you think should be the solution to this @bboreham ?

[sarthaktyagi-505]

@bboreham for a ingester ingesting 40k samples per second for example, is there a Math which we can use to derive the value of ingester.instance-limits.max-inflight-push-requests to set, I am asking that because I think 60k limit for the value might be overwhelming the ingester itself. It will be good to know the Math to tune this parameter?