Time-outs/outage/errors on uncached images (27-09-2024)

[adamliuio]

What I've noticed since this morning is that for certain images (most likely the new, not cached ones), the connections always time out.

[andrieslouw]

We had some scheduler/threading issues from 27-09-2024 21:00 (CET) untill 28-09-2024 09:00 (CET) leading to high error-rates (~25%) on uncached images.

The automatic reloading of nginx config every few hours seems to have worsened the problem, leading to stalled or unresponsive threads.

We didn't catch this issue in earlier testing of some libvips code changes, as the reloading nginx under load is not part of automated tests.

We're sorry for the troubles caused, the partial outage surprised us as well.

[andrieslouw]

Some graphs from one of our servers show what the impact was past night: