Elasticsearch: broken cross-cluster search

[GiedriusS]

What happened?

I added a new Elasticsearch data source with index name like foo,othernode:foo. I clicked "Save & Test". This is a completely valid Elasticsearch syntax in the remote clusters feature: https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-cross-cluster-search.html.

What did you expect to happen?

Everything to work.

Did this work before?

Yes, it broke somewhere between 9.5.3 and 10.2.2.

How do we reproduce it?

1. Go to data source settings
2. Add new elasticsearch data source
3. Ensure that the index name field contains a colon character
4. Click "Save & Test"
5. See a call to _mappings that fails

Is the bug inside a dashboard panel?

No response

Environment (with versions)?

Grafana: 10.2.2

Grafana platform?

A package manager (APT, YUM, BREW, etc.)

Datasource(s)?

Elasticsearch

[evan-chaney]

Just upgraded 9.5.1 -> 10.2.3 and was hit by this. @GiedriusS thank you for adding a test case in your PR to hopefully avoid this in the future.

[tonypowa]

tagging both squads for review (unsure who owns this)
@grafana/observability-logs @grafana/plugins-platform-backend

[matyax]

Thanks @tonypowa, we (logs) do.

[ivanahuckova]

@GiedriusS could you please share the error with which the call to _mappings fail? @svennergr created a fix to _mappings the other day #80970 which might be relevant/fix the issue.

Edit: Sorry I just noticed your PR and pinged relevant team to review.

[ivanahuckova]

@GiedriusS Also in the meantime, if you want to temporarily fix the issue for you, you can set enableElasticsearchBackendQuerying feature toggle to false which disables running queries trough Elasticsearch's plugin backend (which is causing the issue). However, we plan to remove this feature toggle, so it is a short term fix. So thank you for reporting the issue and creating a PR to fix it

[ivanahuckova]

I will keep this open until we fix the issue also in ES as it also uses the same url.Parse() which will throw error.

[ivanahuckova]

Fixed by #82327. Thanks again for bringing this to our attention and being proactive and helping with fix. Really appreciated. 🧡