Fix: don't miss address scheme

[mtojek]

While working on the Ceph module (#7723) I've prepared the following setup for a new metric set:

modules.d/ceph.yml:

- module: ceph
  metricsets:
    - mgr_cluster_health
  hosts: [ "https://ceph-restful:8003" ]
  username: "demo"
  password: "51dd94b3-f865-4f63-af79-9efb9130cd7f"
  ssl.verification_mode: "none"

module/ceph/mgr_cluster_health:

default: true
input:
  module: http
  metricset: json
  defaults:
    method: POST
    namespace: "json_namespace"
    basepath: "/request"
    headers:
      Content-Type: application/json
    query:
      wait: "1"
    body: '{"prefix": "status", "format": "json"}'

processors:
  - extract_array:
      field: http.json_namespace.finished
      mappings:
        ceph.request: 0
  - decode_json_fields:
      fields: ["ceph.request.outb"]
      process_array: true
      target: "ceph.mgr_cluster_status.output"
  - drop_fields:
      fields: ["http", "ceph.request"]

The problem: Ceph's request API is accessible via HTTPS (by default) and the information about protocol scheme (https://) is getting lost in runtime.

[jsoriano]

Take into account that there can be hosts that don't parse as urls, like some formats for MySQL or PostgreSQL, or in general any metricset that uses a custom host parser, or the PassThruHostParser.

At this point of the code we only want to parse hosts with the HostParser, and we should parse the original setting in the configuration, doesn't base.host contain it? 🤔

[mtojek]

Well, all tests passed here unless I missed something. What I want to do I fix the issue while using http module as base one, so if you know any different workaround, it would be great.

Without this change base.host contained stripped address (e.g. https://ceph-restful:8003 -> ceph-restful:8003, http module picked it up as default http://).

@jsoriano do you suggest to fallback to base.host in case of err here (line 89)?

[jsoriano]

Well, all tests passed here unless I missed something.

Yep, I guess that there are no tests for light modules with configured hosts that cannot be parsed by url.Parse 🙁 Each metricset can implement its own host parser, for their own host strings. We shouldn't assume that they are URLs in all cases.

What I want to do I fix the issue while using http module as base one, so if you know any different workaround, it would be great.

Yes, I see the problem, actually this call to HostParser in this override for light modules was also motivated by some http service. In that moment the problem was that the query defined in a manifest was not being considered, because this is added in the URLHostParserBuilder (same parser as the one used in http module).

I don't know of any workaround at the moment.

@jsoriano do you suggest to fallback to base.host in case of err here (line 89)?

I mean that in the call to HostParser we should use the same host (or an equivalent one) to the one in the configuration file. I was wondering if base.host would contain it, but I see now that base.host is modified in the first call to the HostParser in metricbeat/mb/builders.go.

Maybe we can try using base.hostData.URI, the URI there should be the complete host.

base.hostData, err = registration.HostParser(base.module, base.hostData.URI)

But I am not sure if all hosts parsers actually maintain the complete host.

Other possible source to get the original source is base.module.Config().Hosts, but this can contain all the hosts configured in a module and we may don't know which one is the one for the current metricset.

Maybe we need to keep the original host in some other field in the BaseMetricSet (apart from host that gets modified by some host parsers).

[jsoriano]

As discussed offline, let's do this change, but keeping previous behaviour for hosts that don't parse as URLs. We will think further in other possible cases where this host parser override won't work as expected.

[mtojek]

@jsoriano Thanks, fixed!

[jsoriano]

LGTM, but please add a comment mentioning the purpouse of this new function.

[jsoriano]

Thanks for adding this test!