Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Slow ingester Push operations #3171

Closed
bboreham opened this issue Sep 14, 2020 · 2 comments · Fixed by #3191
Closed

Slow ingester Push operations #3171

bboreham opened this issue Sep 14, 2020 · 2 comments · Fixed by #3191

Comments

@bboreham
Copy link
Contributor

This screengrab is from Jaeger, requesting all Push operations longer than 100ms on a single ingester in an hour:

image

The vertical clustering suggests to me that this is caused by blocking, in one case over a second.
I'm not yet sure of the cause, but my preferred theory is a lock held by a query. Hence #3169

Considering #3093 as a possible cause, I added a span for userState.createSeriesWithFingerprint() but it hasn't showed up in any of the slow traces I've looked at.

@bboreham
Copy link
Contributor Author

This symptom was considerably improved by #3177 when I tried it.
Suggests that Go is blocking until memory is made available by GC: although Go's stop-the-world pauses are very short, individual goroutines can get hit.

@bboreham
Copy link
Contributor Author

OK now I think I've got it. Result of rolling out #3191:

image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant