[CAMPAIGN] Remove references to authz of users outside of security index and request most up-to-date authz from security when needed

[cwperks]

There are potentially many dangling references to users and user attributes including roles and backend roles throughout many different indices in an OpenSearch cluster. When scheduling jobs, plugins that schedule jobs will often read the User information from the threadcontext that was populated by the security plugin and store the user alongside the job details in an index owned by the plugin. This tightly couples plugins to the security plugin and also creates a point-in-time reference to authorization of the user outside of the security index. If the authorization or the user changes then the entry in the plugin's job index will be stale and the job can potentially execute with incorrect authz. In addition to running with incorrect authz, it is also possible to delete a user and keep a dangling reference to the user outside of the security index.

Ideally, there is referential integrity for users and any references can be handled gracefully.

The primary challenge of keeping track of the most up-to-date authz information about a user is that there generally is no way of linking a user back to the auth domain that the user was originally authenticated from (internal or external). In the case of external authentication backend, it is not clear if its possible to retrieve the most up-to-date authz at any time or only keep track of the latest authz information seen from the last successful login of the user.

To summarize:

1. There are dangling references to users and associated authz in many disparate indices of a cluster
2. Because of the lack of referential integrity, deleting of a user or updating the roles does not propagate to all references
3. It is not possible to discern from a user object stored in an index if its an internal user or otherwise

Challenges:

1. Linking a user to the auth domain that authenticated the user
2. Track last logged in time and keep track of latest authz information seen by auth domain
3. Create deprecation path for job scheduling to remove references to authz outside of the security plugin

[cwperks]

@peternied I tried to encapsulate our discussion in an issue to highlight the problems and challenges with the security model

[cwperks]

One way to move towards a solution is to keep better track of which realm (authc domain) backend that a user comes from. Currently, most plugins only store username in their jobs index and that is not strong enough to link back to which Realm originally authenticated the user.

The security plugin can track in an index when the last successful authentication was for all users and keep track of the attributes of the user at that point-in-time to represent the most up-to-date version of the user that the security plugin has seen.

[stephen-crawford]

[Triage] Hi @cwperks, thank you for filing this issue for a campaign. This seems like it will be an ongoing process that could take some time but it does sound like an eventual change will be required. I am going to mark this as triaged for the time being.

[cwperks]

@davidlago This issue captures the issue of looking up authz information adhoc for a user. Looking up authz adhoc can be done for an internal user (apart from host mapping, but that's another issue...), but it may not be achievable across all identity providers.

Identity providers in particular to look into to see if there's a mechanism for obtaining authz information (or must up-to-date user information) ad hoc:

1. LDAP
2. SAML
3. OIDC

Aside from being able to lookup user info and authz adhoc, a better approach to the current system would be to track that information every time a user authenticates successfully and to keep a latest snapshot from the most recent successful login.