Investigate indexing span.context.db.*

[simitt]

Currently span.context.db.* properties are stored but not indexed in ES. This makes it impossible to query/aggregate on them, which might be helpful for multiple use cases (aggregate on them to figure out occurence of queries per type, problematic instances, etc.).

Suggested solution:

attribute	type	description	index type	limit (server)	limit (agent)	note
instance	string	db instance name	keyword	1024	1024	-
user	string	username for accessing db	keyword	1024	1024	-
type	string	database type	keyword	1024	1024	-
statement	string	concrete query	text	none	?	should be sanitized for grouping

[alvarolobato]

We discussed this in the server meeting. Originally these fields were removed from the indexes because we where concerned for the disk space usage. We've decided to document how to add these fields to the index in fields.yml instead of indexing it by default.

[graphaelli]

After some investigation and discussion with the beats team, the fields.yml approach seems too manual and error prone to be the right long-term solution. Most notably, upgrades become very manual, with merging new fields with user-modified fields.yml.

Adding discuss label for brainstorming alternative solutions.

[roncohen]

any chance the discussion is available somewhere?

[roncohen]

there's setup.template.append_fields which allows users to add arbitrary fields to the template: https://www.elastic.co/guide/en/beats/filebeat/6.x/configuration-template.html

seems slightly better than asking people to edit their fields.yml

[simitt]

This is still marked as experimental though (although it landed quite a while ago, see https://github.com/elastic/beats/pull/6024/files). Did you talk to the beats team if there are plans to make this feature stable?

[roncohen]

Related: elastic/beats#8607

[roncohen]

@ruflin @urso any input on the future plans for “append_fields”?

[ruflin]

+1 on making append_fields GA. The main challenge I see around it is that users will add fields after they did run the Beat for the first and will have to overwrite the template to make use of it. We should solve this by docs and think about other options how to raise awareness.

[simitt]

The docs already mention when append_fields can be used, how to overwrite the template and when template changes are applied. With elastic/beats#16576 the experimental flag from append_fields will be removed.

[tobiasstadler]

What is the status? Are their plans to index the fields by default?

[simitt]

@tobiasstadler we don't have any plans to index additional span fields, but the setup.template.append_fields setting has been released as GA for a while. Are you able to achieve what you need by making use of this setting?

[tobiasstadler]

@smith Yes, that works for me. Thank You!

[simitt]

Closing this issue since we don't have plans to work on this.