[BUG] Default data source that isn't local breaks Discover and dashboards

[kavilla]

Describe the bug

Discover plugin and Dashboards plugin is based on the current context. The current context has the data source being the local cluster (or the one it started up with). When Discover loads it will retrieve the index patterns from the local cluster.

These index patterns can have references to other data sources but the index patterns that were created and have no external data source ref will still be based on the local cluster.

Then switching the default data source so now the local cluster is not the default cluster. Then any index patterns, that were created with no reference, that are selected will eventually make to the deciding the client with no data source id so it will go with the default client which is NOT the local cluster that has the saved index pattern.

Any search calls that are sourced from the local cluster and did not expect that the default client can be swapped out should break and crash.

Any thing added to a dashboard that doesn't include the data source will break if the default client is not the local client.

Users are stuck because there doesn't seem to be a clear way to set the default data source to the local cluster:

So I would have to delete all my external data sources added.

The Discover plugin data source selector bases off a default Index in Advanced Settings:

This again is based on the default cluster, if the user swaps the default cluster, Discover will do it's initial call to get that value so if the default data source does not have any clue about that ID because it was autopopulated based on a different data source then Discover will crash.

We can add data source selectors every where, but the problem is shared URLs have no context about the data source ref.

If you go into Discover and click "Share", we will save it with the index pattern which will have the data source ref. But if the default cluster is not the local cluster then there is no reference to that for any EXISTING shared URLs for example:

https://playground.opensearch.org/app/data-explorer/discover#?_a=(discover:(columns:!(_source),isDirty:!f,sort:!()),metadata:(indexPattern:ff959d40-b880-11e8-a6d9-e546fe2bba5f,view:discover))&_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-15m,to:now))&_q=(filters:!(),query:(language:kuery,query:''))

Could be bookmarked by users but if some admin updated the default data source to not be the data source that index pattern was originally created then ALL the previous shared urls are broken.

Other information

I would be wary about now pushing down the data source into the search source, instead of referencing it based on the index pattern. Because that could be a lot of work and could break a lot of things.

There appears to be some race condition where I can try to create an index pattern or datasource before the default client is established which kicks me in a state where it thinks I have data but not an index pattern

[seraphjiang]

Local cluster is very confused concept in MDS. we suggest to hide it when enable MDS.

# hide local cluster in the data source dropdown and index pattern creation page.
data_source.hideLocalCluster: true

[joshuali925]

triaging issue..

Users are stuck because there doesn't seem to be a clear way to set the default data source to the local cluster

i do see this problem

Then any index patterns, that were created with no reference, that are selected will eventually make to the deciding the client with no data source id so it will go with the default client which is NOT the local cluster that has the saved index pattern.

I don't see this problem, here is what i did

1. add data_source.enabled: true to dashboards yml
2. run in devtools

POST /a/_doc
{
  "cluster": "local"
}

3. add datasource
4. run in devtools with the datasource selected

POST /a/_doc
{
  "cluster": "remote"
}

5. create index patterns for each
6. search them in discover, the results are correct

i'm on commit f04708e6efb