Unable to update index pattern

[rlevytskyi]

Describe the bug
At some point, our users started complaining they are unable to update their index patterns.
Even I as cluster_admin unable to update them, getting this at kibana log:

{"type":"log","@timestamp":"2024-02-29T07:56:52Z","tags":["error","opensearch","data"],"pid":1,"message":"[security_exception]: Update is not supported when FLS or DLS or Fieldmasking is activated"}
{"type":"error","@timestamp":"2024-02-29T07:56:52Z","tags":[],"pid":1,"level":"error","error":{"message":"Internal Server Error","name":"Error","stack":"Error: Internal Server Error\n at HapiResponseAdapter.toError (/usr/share/opensearch-dashboards/src/core/server/http/router/response_adapter.js:127:19)\n at HapiResponseAdapter.toHapiResponse (/usr/share/opensearch-dashboards/src/core/server/http/router/response_adapter.js:83:19)\n at HapiResponseAdapter.handle (/usr/share/opensearch-dashboards/src/core/server/http/router/response_adapter.js:79:17)\n at Router.handle (/usr/share/opensearch-dashboards/src/core/server/http/router/router.js:175:34)\n at processTicksAndRejections (node:internal/process/task_queues:95:5)\n at handler (/usr/share/opensearch-dashboards/src/core/server/http/router/router.js:140:50)\n at exports.Manager.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/toolkit.js:60:28)\n at Object.internals.handler (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/handler.js:46:20)\n at exports.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/handler.js:31:20)\n at Request._lifecycle (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:371:32)\n at Request._execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:281:9)"},"url":"http://logs.company.com/api/saved_objects/index-pattern/178bc3f0-0d28-11ea-860b-8fae5a22bf63","message":"Internal Server Error"}
{"type":"response","@timestamp":"2024-02-29T07:56:52Z","tags":[],"pid":1,"method":"put","statusCode":500,"req":{"url":"/api/saved_objects/index-pattern/178bc3f0-0d28-11ea-860b-8fae5a22bf63","method":"put","headers":{"connection":"upgrade","host":"logs.company.com","x-forwarded-for":"10.139.128.40","content-length":"406094","sec-ch-ua":""Chromium";v="122", "Not(A:Brand";v="24", "Google Chrome";v="122"","content-type":"application/json","osd-xsrf":"osd-fetch","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36","osd-version":"2.12.0","sec-ch-ua-platform":""Linux"","accept":"/","origin":"https://logs.company.com","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://logs.company.com/app/management/opensearch-dashboards/indexPatterns/patterns/178bc3f0-0d28-11ea-860b-8fae5a22bf63","accept-encoding":"gzip, deflate, br, zstd","accept-language":"en-US,en;q=0.9","securitytenant":""},"remoteAddress":"127.0.0.1","userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36","referer":"https://logs.company.com/app/management/opensearch-dashboards/indexPatterns/patterns/178bc3f0-0d28-11ea-860b-8fae5a22bf63"},"res":{"statusCode":500,"responseTime":65,"contentLength":9},"message":"PUT /api/saved_objects/index-pattern/178bc3f0-0d28-11ea-860b-8fae5a22bf63 500 65ms - 9.0B"}

And this at coordinating node log:

[2024-02-29T07:56:52,087][WARN ][r.suppressed ] [v48-coordinator.company.com] path: /.kibana/_update/index-pattern%3A178bc3f0-0d28-11ea-860b-8fae5a22bf63, params: {if_seq_no=925, if_primary_term=1, refresh=wait_for, index=.kibana, _source_includes=namespace,namespaces,originId, id=index-pattern:178bc3f0-0d28-11ea-860b-8fae5a22bf63}
org.opensearch.OpenSearchSecurityException: Update is not supported when FLS or DLS or Fieldmasking is activated
at org.opensearch.security.configuration.DlsFlsValveImpl.invoke(DlsFlsValveImpl.java:260) [opensearch-security-2.12.0.0.jar:2.12.0.0]
at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:390) [opensearch-security-2.12.0.0.jar:2.12.0.0]
at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:165) [opensearch-security-2.12.0.0.jar:2.12.0.0]
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.12.0.jar:2.12.0]
...
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.106.Final.jar:4.1.106.Final]
at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]

To Reproduce
Steps to reproduce the behavior:

1. Use OpenSearch as usual with Keycloak SAML
2. At some point, note that some sysadmins don't have access to all indices as they should
3. Waste a lot of time to find that is the case when sysadmin is also a member of some Dev team
4. Waste a lot of time to find that there was a similar issue for user that has several roles with no solution but with workaround, https://forum.opensearch.org/t/how-is-dls-applied-when-user-has-multiple-roles/2946
5. Implement a workaround and make sysadmins able to browse all indices again
6. After some while, receive complain at index pattern update
7. Check yourself and see error as described above
8. Temporarily remove this from the SA role:
   "dls": "{ \"bool\": { \"must_not\": [ { \"match_phrase\": { \"host\": \"v000.company.com\" } } ] } }"
   and see index pattern can now be updated.
9. But now SA cannot see all indices again because of overlapping roles.

Expected behavior
The initial reason is that roles got overlapped when one has empty DSL.

OpenSearch Version
OpenDistro 1.0 through OpenSearch 2.12

Dashboards Version
OpenDistro 1.0 through OpenSearch 2.12

Plugins

• Security plugin with Keycloak SAML

Screenshots
Not applicable, see logs.

Host/Environment (please complete the following information):

• OS: Ubuntu 23.10
• Chrome (doesn't matter acutally)

Additional context
Will be added upon request.

[ananzh]

@opensearch-project/admin could we route to security plugin? a first step is to add more doc to clearly let cx aware of the issue.

[rlevytskyi]

Hi all!
We've just noticed that workaround has another collateral damage.
When you have index where "host" is "mapped" as IP, you cannot see the data in Kibana:

[manasvinibs]

@opensearch-project/admin Can this issue be routed to security plugin team to take a look?

[cwperks]

[Triage] @peternied Can you take a look at this issue and ask any questions that would help with a reproduction to determine if this is a bug or config related issue that documentation can alleviate?

[peternied]

@rlevytskyi Thanks for opening this issue - as this behavior can be confusing. For your sysadmin accounts/clients make sure that no roles are applied that include DLS/FLS rules [1] - including roles that are included by network policies.

We have been consider investing into Views, which could help facilitate better DLS/FLS rules organization [2] [3] that might avoid these scenarios. Thoughts on this would be greatly appreciated.

• Multiple role calculations [1] https://opensearch.org/docs/latest/security/access-control/document-level-security/#dls-and-multiple-roles
• [2] Data projection with views OpenSearch#6181
• [3] [RFC] Aligning Access and Visibility in OpenSearch #4069

[peternied]

Closing this out as there isn't a specific bug here.