Excessive tenant initial load times on large pageserver after restart

[koivunej]

Internal slack thread: https://neondb.slack.com/archives/C033RQ5SPDH/p1681409913535909 by @arssher who brought this up.

In general load is assumed to be fast, but recently two tenants went over multiple times the page_service.rs's get_active_tenant_with_timeout from:

neon/pageserver/src/page_service.rs

Line 347 in b6c7c32

let tenant = get_active_tenant_with_timeout(tenant_id, &ctx).await?;

Quick log analysis revealed no obvious reason. Working assumption is that we were busy, perhaps with even blocking work as earlier initial logical size calculations most likely had started at this point (re: #2975).

Post-analysis ideas in general so we'd learn these early:

• @koivunej thought of a "load takes more than hardcoded seconds => warn!"
• @problame suggested an alert on long times

[koivunej]

Aside note: we do log so much during startup and shutdown it's probably something to look into as well.

[koivunej]

Put the storage operation time histograms on https://neonprod.grafana.net/d/b7a5a5e2-1276-4bb0-9e3a-b4528adb6eb6/storage-operations-histograms-in-prod?orgId=1&var-datasource=victoria-metrics-aws-prod&var-instance=All&var-operation=All

[koivunej]

There seems to be a lot of fluctuation; a tenant with 10 timelines took over 100s to activate, this time only 70s.

[problame]

I know you believe it's not the concurrency limiter semaphore, but, we could rule it out by introducing metrics that measure it.

I think you believe the root cause for this is the synchronous IO that we do. My #4215 might fix this.

[koivunej]

I at least in the past weeks have been thinking it could be the s3 semaphore which is why I had hoped to continue on that, one or two weeks. I did not deprioritize it.

I still don't agree we can just add permits, because it's trivial to go over the rps limits and recently all aws limit exceeding has resulted in unclear failures.

Would very much want to add the task metrics but it will be requiring a different set of buckets if we can get the spawn_blocking.

[koivunej]

There's a more fresh duplicate: #4183 and an maybe an epic as well.

Fixes are in place: #4399 and are working, at best giving us 1ms per timeline, but at worst much more. Added #4892 for us to get understanding how long do things take.

Closing this to focus on #4183.