[libbeat] Change aws_elb autodiscover provider field name to aws.elb.*

[kaiyan-sheng]

What does this PR do?

This PR is to change aws_elb autodiscover provider fields from elb_listener.* to aws.elb.*. For example, the output event will look like below from unit test:

map[aws:{"elb":{"availability_zones":["strVal"],"created":"2020-02-18T13:58:30.569437-07:00","host":"fake.example.net","ip_address_type":"dualstack","listener_arn":"listen_arn","load_balancer_arn":"lb_arn","port":1234,"protocol":"HTTPS","scheme":"internet-facing","security_groups":["foo-security-group"],"ssl_policy":"strVal","state":{"code":"active","reason":"strVal"},"type":"application","vpc_id":"strVal"}} cloud:{"availability_zone":["strVal"],"provider":"aws","region":"strV"} host:fake.example.net id:listen_arn meta:{"aws":{"elb":{"availability_zones":["strVal"],"created":"2020-02-18T13:58:30.569437-07:00","host":"fake.example.net","ip_address_type":"dualstack","listener_arn":"listen_arn","load_balancer_arn":"lb_arn","port":1234,"protocol":"HTTPS","scheme":"internet-facing","security_groups":["foo-security-group"],"ssl_policy":"strVal","state":{"code":"active","reason":"strVal"},"type":"application","vpc_id":"strVal"}},"cloud":{"availability_zone":["strVal"],"provider":"aws","region":"strV"}} port:1234 provider:39021adf-6e56-4c6a-8a3c-b12bf9317c06 start:true]

meta represents what will end up as metadata in the output of modules that get launched. This PR also updated aws_elb provider to have all the meta fields into autodiscover template by adding it to the root of the event. Please see #14823 (comment) for more details.

Why is it important?

With this change, fields from aws_elb and aws_ec2 autodiscover providers will match.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works

Related issues

• Closes Change aws_elb autodiscover provider meta field prefix to aws.elb #16219

[blakerouse]

Why the move out of meta and why now in both places?

[kaiyan-sheng]

Good point, I forgot to add some explanation in PR description for this. This is to update aws_elb provider to have all the meta fields into autodiscover template by adding it to the root of the event. Please see #14823 (comment) for more details. Thanks!

[blakerouse]

Same as above.

[blakerouse]

Okay, thanks for the explanation. Looks good!

[andrewvc]

So, if I understand this patch correctly we're replacing aws_elb with aws.elb in the output documents but not changing the provider name from aws_elb is that correct?

Is it correct that YAML configs should not be affected by this, but the event schema is. Is that correct?

In other words, am I right in my understanding that docs like https://github.com/elastic/beats/blob/master/heartbeat/docs/aws-credentials-examples.asciidoc won't need to change?

[kaiyan-sheng]

@andrewvc Thanks for reviewing! Please see my answer in line.

So, if I understand this patch correctly we're replacing aws_elb with aws.elb in the output documents but not changing the provider name from aws_elb is that correct?

We are changing the output event field from elb_listener.* to aws.elb.*. For example elb_listener.load_balancer_arn to aws.elb.load_balancer_arn. Yes, the autodiscover provider name stays the same to be aws_elb.

Is it correct that YAML configs should not be affected by this, but the event schema is. Is that correct?
In other words, am I right in my understanding that docs like https://github.com/elastic/beats/blob/master/heartbeat/docs/aws-credentials-examples.asciidoc won't need to change?

Yes and no 😄 https://github.com/elastic/beats/blob/master/heartbeat/docs/aws-credentials-examples.asciidoc won't need to change when user is using data.host and/or data. port as a part of the autodiscover template. But if originally, user has a config includes elb meta data fields like meta.elb_listener.listener_arn(probably a bad example that doesn't make sense for heartbeat):

heartbeat.autodiscover:
  providers:
  - type: aws_elb
    period: 1m
    regions: ["us-east-1"]
    credential_profile_name: test-hb
    templates:
    - type: tcp
      hosts: ["${meta.elb_listener.listener_arn}"]
      schedule: "@every 5s"
      timeout: 1s

Then, with this change, user needs to change the config to use data.aws.elb.* instead:

heartbeat.autodiscover:
  providers:
  - type: aws_elb
    period: 1m
    regions: ["us-east-1"]
    credential_profile_name: test-hb
    templates:
    - type: tcp
      hosts: ["${data.aws.elb.listener_arn}"]
      schedule: "@every 5s"
      timeout: 1s

[andrewvc]

LGTM