Streamline experience when loading dashboards from modules

[suyograo]

We should make it easier to use modules with Elastic cloud, especially when there are multiple URLs (Kibana and ES) and credentials involved as in the cloud case.

To this end, we should add 2 extra flags for the CLI:

bin/logstash --module netflow --cloud.id=<cloud-id> --cloud.auth:elastic:<password>

cloud.id is of the form: $HOST$ES_UUID$KIB_UUID` in base64 encoded form.

When this is specified, here's what we do:

1. LS will first decode the base64 cloud ID, decompose the different bits as specified above (host, es_uuid and kb_uuid) using the $ delimiter, and then construct the Kibana and ES URL from that. The port for cloud will always be 9243 for both ES and Kibana endpoints.
2. LS will internally overwrite var.elasticsearch.host variable. it will also overwrite var.elasticsearch.password and var.elasticsearch.username using cloud-auth.

Decoding algorithm

From @tsg (elastic/beats#4959)

The cloud-id, to be passed to the --cloud.id flag, looks something like this:

staging:dXMtZWFzdC0xLmF3cy5mb3VuZC5pbyRjZWM2ZjI2MWE3NGJmMjRjZTMzYmI4ODExYjg0Mjk0ZiRjNmMyY2E2ZDA0MjI0OWFmMGNjN2Q3YTllOTYyNTc0Mw==

This can be decomposed using the following algorithm:

1. Everything up to the first : is ignored. This is an optional user friendly token that can be used to more easily identify the cloud ID. If no : is present, then the IS is passed as is to the next step.

2. The rest of the ID is base64 decoded:

   us-east-1.aws.found.io$cec6f261a74bf24ce33bb8811b84294f$c6c2ca6d042249af0cc7d7a9e9625743

3. There are three parts, separated by $, in order:

   host: us-east-1.aws.found.io
   es-id: cec6f261a74bf24ce33bb8811b84294f
   kibana-id: c6c2ca6d042249af0cc7d7a9e9625743

4. The es-id and kibana-id are parsed as URLs, and host is appended to their host parts. The https scheme and port 443 are added:

   es-url: https://cec6f261a74bf24ce33bb8811b84294f.us-east-1.aws.found.io:443
   kibana-url: https://c6c2ca6d042249af0cc7d7a9e9625743.us-east-1.aws.found.io:443

The resulting es-url and kibana-url are used to overwrite the settings var.elasticsearch.host and var.kibana.host

[tsg]

@guyboertje FYI, this is what we're doing on the Beats side: elastic/beats#4964

[guyboertje]

@tsg - Thank you.

[guyboertje]

@suyograo - Can I suggest that we namespace cloud e.g. --cloud.id=<cloud-id> --cloud.auth elastic:<password> it would fit logstash.yml semantics better (and there may other settings down the pike).

[guyboertje]

@suyograo - is it an error to have only one of the pair [id, auth]?

[suyograo]

@suyograo - Can I suggest that we namespace cloud e.g. --cloud.id= --cloud.auth elastic: it would fit logstash.yml semantics better (and there may other settings down the pike).

Yes, we can do that.

is it an error to have only one of the pair [id, auth]?

Yes.

[tsg]

is it an error to have only one of the pair [id, auth]?

Yes.

I initially did the same, but later during the work I decided to allow cloud.id without cloud.auth, but not the other way around. The reason is that in case ES and KB need different users (for the minimal privileges principle), setting them at the output level might be easier to follow.

[guyboertje]

Coded as @tsg suggested ^.

[tbragin]

Pull here: #8059

[guyboertje]

Closed with #8059