[FEATURE] index pattern refresh with high field count impossible in managed cluster

[pefischerhro]

Describe the bug
We run OpenSearch in an AWS managed cluster with drastically reduced configuration options. Currently it is not possible to create/refresh index patterns for indices with a high field count via OpenSearch Dashboards because an internal POST request to /_dashboards/api/saved_objects/index-pattern fails with HTTP status code 413. In a self-hosted environment I would set the environment variable 'server.maxPayloadBytes'. However, in the AWS managed cluster I do not have access to this.

With the updates to index pattern management in ES 7.11.0 (see #82223, #83576) transferring the full field list can be avoided. Can a similar solution be implemented in OpenSearch dashboards?

To Reproduce
Steps to reproduce the behavior:

1. create an index
2. index a document with a low field count
3. create an index pattern
4. index large documents until index field count reaches ~60k
5. refresh index pattern via UI

Expected behavior
Index pattern refresh should not fail because of an arbitrary, inaccessible configuration value. Preferably, manual index pattern refresh should not be necessary at all. see #82223

OpenSearch Version
1.2

Dashboards Version
1.2.0

Host/Environment

• AWS OpenSearch domain

[kavilla]

Hello @pefischerhro,

Thanks for opening this!

If you are currently blocked or experiencing productivity issues, I would suggest reaching out to your cloud provider to assist.

I do believe we can repurpose this issue to create an API for index-pattern as suggested in this issue. Otherwise, @ananzh if you feel like it's best to recreate an feature request focusing on just API then please feel free. @opensearch-project/opensearch-ux , @ahopp, do you have any opinions about this feature? Right now index-patterns can be created, deleted, and removed via the OpenSearch Dashboards UI. Any opinion on priority?

[kavilla]

Removing bug since this is due to working as expected and replacing with enhancement.

[pefischerhro]

UPDATE: we created a ticket with AWS support and apparently it is now possible to have support set the environment variable 'server.maxPayloadBytes' to a custom value for a selected OpenSearch domain. This provides a temporary solution.

[ananzh]

@kavilla will create a feature request regarding index-pattern api

[Cbeck527]

Adding a +1 to this— our production cluster just got bit by this issue as well. Thankfully, AWS support was able to raise the limit for us but we were unable to view data until the issue was resolved.