[metricbeat/aws] When running Metricbeat from GCP cloud.* fields get overwritten

[jamiesmith]

Working with snapshot & beta for 7.0, have ES running on cloud, with the AWS metricbeat module enabled. The dashboards get put out there, and I see one entry. Took me a while, but it turns out that for the cloud.* fields it is picking up the info from the GCP VM that I am running it from, rather than what it's getting for each individual EC2 instance. All of the visualizations are keyed off of cloud.instance.id

It still has the aws info in the aws.ec2.instance.* fields. Note that the key on the legend is 827... (EC2 instances all start with i-)

But the data still has the AWS info:

Compare the first picture with the same snapshot when running the exact same config & start command from my laptop:

(ignore the fact that there aren't lines yet, they take several minutes to show-- just look at the legend)

• Version:
  • metricbeat-7.0.0-beta1-linux-x86_64
  • Elasticsearch on cloud

"version" : {
    "number" : "7.0.0-rc1",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "4834b93",
    "build_date" : "2019-03-13T23:25:26.261953Z",
    "build_snapshot" : false,
    "lucene_version" : "8.0.0",
    "minimum_wire_compatibility_version" : "6.7.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  }

• Operating System: CentOS on a GCP VM sending
• Discuss Forum URL: N/A, slack
• Steps to Reproduce:

1. Install a 7.0 Elasticsearch cluster (or use Elastic Cloud)
2. Download the appropriate metricbeat 7 beta for your system
3. Enable the AWS module ./metricbeat modules enable aws
4. Make sure that your credentials for AWS are set up as per the docs
5. start metricbeat, I used this command: ./metricbeat -E setup.dashboards.enabled=true -e

Using the cloud config, the relevant/modified parts of the metricbeat.yml are:

setup.kibana:
  username: "admin"
  password: "not-the real-admin-password"

cloud.id: "70_RC_1:the-actual-cloud-id="
cloud.auth: "admin:not-the real-admin-password"

I have my AWS credentials sourced in the environment:
env | grep AWS

AWS_DEFAULT_REGION=us-east-2
AWS_SECRET_ACCESS_KEY=40-char-secret
AWS_ACCESS_KEY_ID=20-char-access-key

then start via ./metricbeat -E setup.dashboards.enabled=true -e.

These are the fields which it is overwriting:

• cloud.instance.id
• cloud.instance.name
• cloud.machine.type
• cloud.project.id
• cloud.provider

[jamiesmith]

Side note- I would bet that it has the same problem when running from any provider for which Metricbeat picks up the cloud.* fields automagically (AWS which would be really hard to spot, Azure, etc)

[ruflin]

@jamiesmith I assume in your Metricbeat config you have add_cloud_metadata processor enabled. Could you try to remove it from your config and see what happens?

I think we need to add an option to the add_cloud_metadata that events are not enriched if there are already existing cloud fields.

@webmat That reminds me a lot of our host.* enrichment issue when it's an observer, which is the case here.

[jamiesmith]

Yes, that was the case (it is on by default).

Probably need to have the docs updated, and I will update the AWS blog that went out yesterday to include that tidbit. Thanks!

[ruflin]

Agree, the short term solution is update our docs accordingly. Thanks for updating the blog post.

[jamiesmith]

@ruflin how's this for a disclaimer:

(Note: if you are running Metricbeat from a virtual machine on a cloud instance, such as GCP, AWS, or Azure, you will want to disable the add_cloud_metadata entry in metricbeat.yml)

[ruflin]

@jamiesmith SGTM. Perhaps you could even say remove or comment out? Not sure if readers will now how to disable it.

[kaiyan-sheng]

@jamiesmith I assume in your Metricbeat config you have add_cloud_metadata processor enabled. Could you try to remove it from your config and see what happens?

I think we need to add an option to the add_cloud_metadata that events are not enriched if there are already existing cloud fields.

@webmat That reminds me a lot of our host.* enrichment issue when it's an observer, which is the case here.

@ruflin @jamiesmith Thanks for the temporary fix here. In this case, if we add an option in add_cloud_metadata to not enrich cloud fields, then we will be missing all the cloud.* for GCP. Is it possible to keep both? For example, moving cloud.* data under aws.cloud.* for the instance itself and keep cloud.* for GCP?

[webmat]

Perhaps we could look at the defaults for these processors from a different angle, too.

When Metricbeat is monitoring a host external to itself, perhaps it should by default not run add_host_metadata nor add_cloud_metadata?

[ruflin]

@webmat I think there are cases where the same MB instance monitors a local and remote host. And even if it's separate, how do we tell the difference on startup to magically enabled it? My preference is to have a decision event based.

@kaiyan-sheng I think the challenge you mention here is that Metricbeat is running as an observer. If we need the cloud data for the observer (which I'm not sure we need) we could add it under observer.cloud.*.

[webmat]

Well we could beef up all of the [module].yml.disabled files to include these processors (either commented out or uncommented), with instructions to enable the "host" processor only when monitoring a local thing, and to enable the "observer" processor when monitoring a remote thing.

Or alternately we could try to go the road of detecting whether the monitored address is local or remote (one of the host's IPs => add_host_metadata, a domain or an unknown IP => add_observer_metadata). Although this detection will look like magic, and confuse people when it's not working (e.g. monitoring a virtual IP assigned to the local host, but invisible locally, like an AWS elastic IP).

[ruflin]

Interesting idea about adding it to each module instead of having it globally. But I still think it would need user interaction so we are in the same situation like now.

[exekias]

+1 on changing add_cloud_metadata to skip events with existing cloud fields. I think that should be the default behavior (breaking change / bug fix 😈)