Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix: don't miss address scheme #16205

Merged
merged 4 commits into from
Feb 11, 2020
Merged

Fix: don't miss address scheme #16205

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
bd79321
Fix: don't miss address scheme
Feb 8, 2020
e689285
Add unit test
Feb 9, 2020
a117e59
Adjust source after code review
Feb 11, 2020
8237e9d
Add comment to method
Feb 11, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.next.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Change lookup_fields from metricset.host to service.address {pull}15883[15883]
- Add dedot for cloudwatch metric name. {issue}15916[15916] {pull}15917[15917]
- Fixed issue `logstash-xpack` module suddenly ceasing to monitor Logstash. {issue}15974[15974] {pull}16044[16044]
- Fix skipping protocol scheme by light modules. {pull}16205[pull]

*Packetbeat*

Expand Down
16 changes: 15 additions & 1 deletion metricbeat/mb/lightmetricset.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,9 @@
package mb

import (
"fmt"
"net/url"

"github.com/pkg/errors"

"github.com/elastic/beats/libbeat/common"
Expand Down Expand Up @@ -83,7 +86,18 @@ func (m *LightMetricSet) Registration(r *Register) (MetricSetRegistration, error
// At this point host parser was already run, we need to run this again
// with the overriden defaults
if registration.HostParser != nil {
base.hostData, err = registration.HostParser(base.module, base.host)
u, err := url.Parse(base.hostData.URI)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Take into account that there can be hosts that don't parse as urls, like some formats for MySQL or PostgreSQL, or in general any metricset that uses a custom host parser, or the PassThruHostParser.

At this point of the code we only want to parse hosts with the HostParser, and we should parse the original setting in the configuration, doesn't base.host contain it? 🤔

Copy link
Contributor Author

@mtojek mtojek Feb 10, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, all tests passed here unless I missed something. What I want to do I fix the issue while using http module as base one, so if you know any different workaround, it would be great.

Without this change base.host contained stripped address (e.g. https://ceph-restful:8003 -> ceph-restful:8003, http module picked it up as default http://).

@jsoriano do you suggest to fallback to base.host in case of err here (line 89)?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, all tests passed here unless I missed something.

Yep, I guess that there are no tests for light modules with configured hosts that cannot be parsed by url.Parse 🙁 Each metricset can implement its own host parser, for their own host strings. We shouldn't assume that they are URLs in all cases.

What I want to do I fix the issue while using http module as base one, so if you know any different workaround, it would be great.

Yes, I see the problem, actually this call to HostParser in this override for light modules was also motivated by some http service. In that moment the problem was that the query defined in a manifest was not being considered, because this is added in the URLHostParserBuilder (same parser as the one used in http module).

I don't know of any workaround at the moment.

@jsoriano do you suggest to fallback to base.host in case of err here (line 89)?

I mean that in the call to HostParser we should use the same host (or an equivalent one) to the one in the configuration file. I was wondering if base.host would contain it, but I see now that base.host is modified in the first call to the HostParser in metricbeat/mb/builders.go.

Maybe we can try using base.hostData.URI, the URI there should be the complete host.

base.hostData, err = registration.HostParser(base.module, base.hostData.URI)

But I am not sure if all hosts parsers actually maintain the complete host.

Other possible source to get the original source is base.module.Config().Hosts, but this can contain all the hosts configured in a module and we may don't know which one is the one for the current metricset.

Maybe we need to keep the original host in some other field in the BaseMetricSet (apart from host that gets modified by some host parsers).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As discussed offline, let's do this change, but keeping previous behaviour for hosts that don't parse as URLs. We will think further in other possible cases where this host parser override won't work as expected.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jsoriano Thanks, fixed!

if err != nil {
return nil, errors.Wrapf(err, "host data URI parsing failed on light metricset factory for '%s/%s'", m.Module, m.Name)
}

var host string
if u.Scheme != "" {
host = fmt.Sprintf("%s://%s", u.Scheme, u.Host)
} else {
host = base.host
}
base.hostData, err = registration.HostParser(base.module, host)
if err != nil {
return nil, errors.Wrapf(err, "host parser failed on light metricset factory for '%s/%s'", m.Module, m.Name)
}
Expand Down
36 changes: 36 additions & 0 deletions metricbeat/mb/lightmodules_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@
package mb

import (
"net/url"
"testing"
"time"

Expand Down Expand Up @@ -287,6 +288,41 @@ func TestNewModuleFromConfig(t *testing.T) {
}
}

func TestLightMetricSet_VerifyHostDataURI(t *testing.T) {
const hostEndpoint = "ceph-restful:8003"
const sampleHttpsEndpoint = "https://" + hostEndpoint

r := NewRegister()
r.MustAddMetricSet("http", "json", newMetricSetWithOption,
WithHostParser(func(module Module, host string) (HostData, error) {
u, err := url.Parse(host)
if err != nil {
return HostData{}, err
}
return HostData{
Host: u.Host,
URI: host,
SanitizedURI: host,
}, nil
}))
r.SetSecondarySource(NewLightModulesSource("testdata/lightmodules"))

config, err := common.NewConfigFrom(
common.MapStr{
"module": "httpextended",
"metricsets": []string{"extends"},
"hosts": []string{sampleHttpsEndpoint},
})
require.NoError(t, err)

_, metricSets, err := NewModule(config, r)
require.NoError(t, err)
require.Len(t, metricSets, 1)

assert.Equal(t, hostEndpoint, metricSets[0].Host())
assert.Equal(t, sampleHttpsEndpoint, metricSets[0].HostData().URI)
}

func TestNewModulesCallModuleFactory(t *testing.T) {
logp.TestingSetup()

Expand Down
4 changes: 4 additions & 0 deletions metricbeat/mb/testdata/lightmodules/httpextended/extends/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
default: true
input:
module: http
metricset: json
3 changes: 3 additions & 0 deletions metricbeat/mb/testdata/lightmodules/httpextended/module.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
name: httpextended
metricsets:
- extends