[QA][Serverless-observability] Remove Defer loading panels below "the fold" setting from under presentation labs in advanced settings

[bhavyarm]

Kibana version: main/serverless observability QA project

Browser version: chrome latest

Original install method (e.g. download page, yum, from source, etc.): from QA

Describe the bug: Defer loading panels below "the fold" setting is a lab setting and there is a planned removal of it from the presentation team. Meanwhile its causing bugs like this in serverless - #183106.
Can we please remove this setting from serverless deployments? Thanks!

[elasticmachine]

Pinging @elastic/kibana-management (Team:Kibana Management)

[lukeelmers]

One thing we need to keep in mind when it comes to removing advanced settings from serverless is that we run the risk of "snowflake" projects floating around with different settings applied. For example, if someone found this "below the fold" setting and turned it on, then once we remove it from the ui, it will be permanently enabled for that user's project with no easy way to disable it other than us taking manual intervention.

So we should think through a strategy for how we handle removal of settings to ensure that our serverless projects all share consistent behavior. The main solution that comes to mind for this is applying a uiSettings override via the yml, forcing the setting to the value we expect in serverless.

Off the top of my head I'm not sure if a uiSettingsOverride is going to update the config SO where these settings live... I'm pretty sure that the overrides are only applied on read, which means that we'd need to permanently keep this override in place until we decide to (re)release the setting to the public. At that point, folks who had turned it on previously would have it suddenly turned back on (unless we were to make a plan for migrating the config SO to remove the setting). @elastic/kibana-core Do I have this right or am I overlooking something?

[rayafratkina]

It's a valid question but I wonder if we know how many projects are actually using this at the moment? If we can remove it quickly without having to do a bunch of work, it might be best

[pgayvallet]

Off the top of my head I'm not sure if a uiSettingsOverride is going to update the config SO where these settings live... I'm pretty sure that the overrides are only applied on read, which means that we'd need to permanently keep this override in place until we decide to (re)release the setting to the public At that point, folks who had turned it on previously would have it suddenly turned back on

Yeah, that's absolutely correct.

Also as you mentioned, one possible work around would be to add a migration to delete this specific setting's value from the config object. However I assume we would want to do that only for serverless, and atm there's no way to add conditional statements on SO model versions / migrations based on a serverless/traditional distinction (and we would need to even discuss if we do want to do such thing first...).

So for that specific case, I think I would agree with Raya: if this setting is causing issues on serverless, I feel like just hiding the setting and adding the override may be good enough. It's not like it's a setting that completely changes some specific user-facing behavior (right?), so even if we were to remove the override later and some customers were to have the setting suddenly get back on, it might be ok?

Of course, if we think that we may face more similar issues in the short term, we could instead try to find a more concrete and robust solution.

[alisonelizabeth]

It's a valid question but I wonder if we know how many projects are actually using this at the moment?

I do not see any users change this setting on serverless in the last 90 days; it's also very low on stateful (3 users).

If everyone agrees, we will move forward with Pierre's recommendation:

if this setting is causing issues on serverless, I feel like just hiding the setting and adding the override may be good enough.

[rayafratkina]

+1 to hiding and leaving in place from me.
I don't anticipate us having a lot of issues like this, so don't think we need to create a pattern to follow

[alisonelizabeth]

@sabarasaba can you help pick this one up?

[lukeelmers]

+1 to hiding and leaving in place from me.
I don't anticipate us having a lot of issues like this, so don't think we need to create a pattern to follow

Considering how low usage is, I agree that removing and applying the uiSettings override should be sufficient. I just wanted to be clear that when we do run into situations like this where a ui setting is "leaking" to serverless, we consider it a bug and apply a uiSettings override as a best practice so that we don't have projects floating around with differing behaviors, which is my primary concern.

And if a leaked setting appears to have had really high usage, then we can have a broader discussion about whether it warrants a migration or other more complex mitigations.