Opt-out mechanism for libbeat fields

[graphaelli]

Describe the enhancement:

Following on from #11474 (comment), this issues intends to start a discussion around mechanism for selectively inheriting libbeat fields.

305 default libbeat fields as of 692ef9e

@timestamp
agent.ephemeral_id
agent.hostname
agent.id
agent.name
agent.type
agent.version
client.address
client.bytes
client.domain
client.geo.city_name
client.geo.continent_name
client.geo.country_iso_code
client.geo.country_name
client.geo.location
client.geo.name
client.geo.region_iso_code
client.geo.region_name
client.ip
client.mac
client.packets
client.port
client.user.email
client.user.full_name
client.user.group.id
client.user.group.name
client.user.hash
client.user.id
client.user.name
cloud.account.id
cloud.availability_zone
cloud.instance.id
cloud.instance.name
cloud.machine.type
cloud.project.id
cloud.provider
cloud.region
container.id
container.image.name
container.image.tag
container.labels
container.name
container.runtime
counter
destination.address
destination.bytes
destination.domain
destination.geo.city_name
destination.geo.continent_name
destination.geo.country_iso_code
destination.geo.country_name
destination.geo.location
destination.geo.name
destination.geo.region_iso_code
destination.geo.region_name
destination.ip
destination.mac
destination.packets
destination.port
destination.user.email
destination.user.full_name
destination.user.group.id
destination.user.group.name
destination.user.hash
destination.user.id
destination.user.name
docker.container.labels
ecs.version
error.code
error.id
error.message
error.type
event.action
event.category
event.created
event.dataset
event.duration
event.end
event.hash
event.id
event.kind
event.module
event.original
event.outcome
event.risk_score
event.risk_score_norm
event.severity
event.start
event.timezone
event.type
fields
file.ctime
file.device
file.extension
file.gid
file.group
file.inode
file.mode
file.mtime
file.owner
file.path
file.size
file.target_path
file.type
file.uid
geo.city_name
geo.continent_name
geo.country_iso_code
geo.country_name
geo.location
geo.name
geo.region_iso_code
geo.region_name
group.id
group.name
host.architecture
host.containerized
host.geo.city_name
host.geo.continent_name
host.geo.country_iso_code
host.geo.country_name
host.geo.location
host.geo.name
host.geo.region_iso_code
host.geo.region_name
host.hostname
host.id
host.ip
host.mac
host.name
host.os.build
host.os.family
host.os.full
host.os.kernel
host.os.name
host.os.platform
host.os.version
host.type
host.user.email
host.user.full_name
host.user.group.id
host.user.group.name
host.user.hash
host.user.id
host.user.name
http.request.body.bytes
http.request.body.content
http.request.bytes
http.request.method
http.request.referrer
http.response.body.bytes
http.response.body.content
http.response.bytes
http.response.status_code
http.version
jolokia.agent.id
jolokia.agent.version
jolokia.secured
jolokia.server.product
jolokia.server.vendor
jolokia.server.version
jolokia.url
kubernetes.annotations
kubernetes.container.image
kubernetes.container.name
kubernetes.deployment.name
kubernetes.labels
kubernetes.namespace
kubernetes.node.name
kubernetes.pod.name
kubernetes.pod.uid
kubernetes.replicaset.name
kubernetes.statefulset.name
labels
log.level
log.original
message
network.application
network.bytes
network.community_id
network.direction
network.forwarded_ip
network.iana_number
network.name
network.packets
network.protocol
network.transport
network.type
observer.geo.city_name
observer.geo.continent_name
observer.geo.country_iso_code
observer.geo.country_name
observer.geo.location
observer.geo.name
observer.geo.region_iso_code
observer.geo.region_name
observer.hostname
observer.ip
observer.mac
observer.os.family
observer.os.full
observer.os.kernel
observer.os.name
observer.os.platform
observer.os.version
observer.serial_number
observer.type
observer.vendor
observer.version
organization.id
organization.name
os.family
os.full
os.kernel
os.name
os.platform
os.version
process.args
process.executable
process.name
process.pid
process.ppid
process.start
process.thread.id
process.title
process.working_directory
related.ip
server.address
server.bytes
server.domain
server.geo.city_name
server.geo.continent_name
server.geo.country_iso_code
server.geo.country_name
server.geo.location
server.geo.name
server.geo.region_iso_code
server.geo.region_name
server.ip
server.mac
server.packets
server.port
server.user.email
server.user.full_name
server.user.group.id
server.user.group.name
server.user.hash
server.user.id
server.user.name
service.ephemeral_id
service.id
service.name
service.state
service.type
service.version
source.address
source.bytes
source.domain
source.geo.city_name
source.geo.continent_name
source.geo.country_iso_code
source.geo.country_name
source.geo.location
source.geo.name
source.geo.region_iso_code
source.geo.region_name
source.ip
source.mac
source.packets
source.port
source.user.email
source.user.full_name
source.user.group.id
source.user.group.name
source.user.hash
source.user.id
source.user.name
tags
url.domain
url.fragment
url.full
url.original
url.password
url.path
url.port
url.query
url.scheme
url.username
user.email
user.full_name
user.group.id
user.group.name
user.hash
user.id
user.name
user_agent.device.name
user_agent.name
user_agent.original
user_agent.os.family
user_agent.os.full
user_agent.os.kernel
user_agent.os.name
user_agent.os.platform
user_agent.os.version
user_agent.version

Per that discussion, there is a related, but separate, topic for excluding support for libbeat features altogether. For now, that's handled mostly by omitting documentation from APM server for features that don't make sense / aren't supported there.

Describe a specific use case for the enhancement or feature:

Since APM server is built in on libbeat, all of these fields end up in APM's index templates, documentation, etc. While APM currently excludes all of the autoprovider fields from the original issue, it will be unable to selectively include others later, without tricks or actual support from libbeat. I expect the same concern exists for other libbeat users.

[ruflin]

For me the topic of excluding fields and features is directly coupled. We started to have more local fields.go packages which all together in the end build the template. If one of the autodiscovery provider packages is not included in apm-server for example, also the fields for it are not included which I think is as expected.

There is still the issue with the fields.yml but the way I see this moving forward is that we either don't ship it or generate it for the shipping with beat export fields so it will only contain what is actually compiled.

[graphaelli]

For me the topic of excluding fields and features is directly coupled.

Sure, ideally we'd tackle the features part, I was hoping we could start small since we already have a (poor) workaround for the feature support bit.

There is still the issue with the fields.yml but the way I see this moving forward is that we either don't ship it or generate it for the shipping with beat export fields so it will only contain what is actually compiled.

fields.yml is still used for generating documentation so only the latter would be sufficient for APM needs.

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.