host.name behavior inconsistent across the Elastic stack

[fgabolde]

I'm very confused about how host.name, agent.hostname, observer.hostname etc. are supposed to work.

I have a fairly simple setup where my apps live in containers, Filebeat lives in a different container, and all the apps send JSON-formatted logs to Filebeat over UDP. I have no idea what value should be in host.name.

The ECS states:

ECS host.* fields should be populated with details about the host on which the event happened, or from which the measurement was taken.

So that's pretty clearly the application's host. The APM dashboard seems to agree; when I click "Show host logs", it gives me logs filtered on host.name, with the application's hostname prefilled.

But Filebeat disagrees. I've tried letting it do its own thing, and I've tried setting host.name in the application logs; in both cases, it overwrites host.name with its own hostname, so of course the log dashboard is empty, since all the traces have the application's hostname.

I've come across a bunch of issues related to host.name but I have not been able to understand which behavior is considered a bug, the APM dashboard's or Filebeat's:

#13706
#13589
#12983
#12107
#10698

[jsoriano]

Hi @fgabolde,

Thanks for opening this issue. There are actually some issues with these fields, specially when beats are run in namespaces networks in containers.

A possible workaround for your case is to run filebeat container in the host network (--net=host in docker), then its hostname would be the same as the hostname of the host. Take into account that with this option filebeat will have access to the host network, though this should be safe.

I am going to close this case by now as I think that whatever solution we implement for #13589 would also fix this.

[fgabolde]

@jsoriano , I will subscribe to that issue to follow development, but I don't think it is the same problem. If Filebeat keeps setting host.name to a specific value (however it is obtained), and the rest of the ecosystem keeps using host.name to track where the log originally came from, then the functionality still won't work.

If I understand your workaround, then Filebeat will set host.name to the hostname of the machine that is running the containers, e.g. some AWS instance or my own desktop workstation or whatever. But the APM traces will still have the application containers' host.name value, so they won't match the logs. The issue is worse in my setup, where I have a single Filebeat that is collecting multiple applications' logs. I cannot believe that is not a supported use case, since Filebeat supports autodiscovery of containers.

I guess my question is, who is right? The APM who assumes host.name in logs means the host/container on which the application is running that wrote the logs? Or, Filebeat who assumes host.name in logs should mean the host/container that is collecting the logs?

If Filebeat is right and the host field designates the beat, what is the agent field for?

[jsoriano]

@fgabolde you are right, I was not thinking in cases where the parsed logs are from other machines. I will reopen this for this case. Thanks for the detailed explanations.

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!

[PrplHaz4]

there have been some enhancements but from what I can tell the overall situation here hasn't changed much...

[fgabolde]

I should check if the bug is still present, but in the meantime I've been using container.id more which actually does work as intended.

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!