Question about template usage for ecs

[willemdh]

Hello,

As creator of the following project => https://github.com/OutsideIT/logstash_filter_f5

I'd prefer to follow ECS guidelines in the future and switch current fields if appropriate to ECS fields with dot notation. For now I always used underscores for all my fields, which makes this kind of new.

If I would create a template for some fields, eg.

        "f5_tmm_session_bytes_in": {
          "type": "long"
        },
        "f5_tmm_session_bytes_out": {
          "type": "long"
        },

And switch those to dot notated fields network.inbound.bytes and network.outbound.bytes, would this be the template that I would ideally use for those fields?

        "network": {
          "properties": {
            "inbound": {
              "properties": {
                "bytes": {
                  "type": "long"
                }
              }
            },
            "outbound": {
              "properties": {
                "bytes": {
                  "type": "long"
                }
              }
            },
          }
        }

I saw some examples which also have "type": "object" in the template, but I didn't see that everywhere (not in the beat.* object template for example)

Thanks for confirming the correct or incorrect use of my f5 template.

Another small question, I tend to use the ignore_above parameter alot, which I don't see anywhere in the ECS common field types. Are we 'allowed' to use the ignore_above on ECS fields and can we set them as we want or would this cause mapping conflicts if mixed with data from other indices which have different or no ignore_above parameter for the same field?

Grtz

Willem

[ruflin]

For the example template see here: https://github.com/elastic/ecs/blob/master/template.json#L481 I hope this has the details you are looking for.

Good question about the ignore_above and I must confess I haven't really though about it yet. The current default we use is 1024 as the same template generator is used as we use for beats and that is the default there.

I would expect there is a very small subset of ecs fields where we can expect very long values. There we should probably come up with a good default for ignore_above value that we also recommend to be used. To you have some specific fields in mind?

@willemdh Please let us know if you find fields which are missing in ECS but should be there for your project.

[willemdh]

@ruflin The example template indeed clears things up. Untill now I tried to set ignore_above as small as possible, but 1024 seems like a nice default. I wonder if setting ignore_above smaller for certain fields has any worthwile advantages. Looking for example at agent.version => This should never be larger then let's say 25 characters? Would it prevent us problems setting ignore_above to 25 or would it just make things more complex?

If I ever encounter something that requires an ignore_above higher then 1024 I'll let you know.

About missing fields related to F5, most of them seem pretty specific to F5, I guess it's not worth it creating an ECS field for every possible value? Or would you advise making a separate f5 object in ECS containing all F5 related fields?

Just another question about template usage, do you think it's possible to use https://github.com/elastic/ecs/blob/master/template.json as a default template for all indices containing data which can leverage ECS? Maybe with a high order, making sure it applies? Or would this mean too much unneeded fields are added to each index?
Or do you advise I only use the fields I need from ECS and integrate this into my existing templates, eg https://github.com/OutsideIT/logstash_filter_f5/blob/master/f5.template?

[ruflin]

The main reason we have ignore_above to make sure no field can become massive and would have an affect on Elasticsearch potentially. But in the ECS case I don't think we have such a potential field yet. In general I would prefer not to have the ignore_above but place it only on the values that need it. I think for values that are only expected to contain 25 chars I would not set an ignore_above at all.

For the F5 fields: I don't think we should have a namespace for it in ECS but on your end I would recommend you to put it under f5.* to make sure you don't have any conflicts. Long term if many people will need f5 fields I could see that we have something in an extended version of ECS that covers this.

Elasticsearch 6.x has become much better in handling sparse documents. So I'm not too worried about having more fields defined then actually needed. I definitively like your idea about having ECS there for all indices but must confess I haven't tested this enough yet. But when I created the template I had exactly something like this in mind. Let me know how this goes.

[willemdh]

@ruflin 1024 seems like a good default, i'll revert my smaller ignore_above's to that.

I'll create an f5.* object one of the following months

About the ecs template as default for all indices, let me know if you find the time to test this. It could be a start to use this officially for Beats indices one of the following major release.

For now I'll integrate the used fields into my existing templates.

[webmat]

ignore_above can be seen as a safety feature. It only prevents the keyword (exact match search + aggregations) indexing to take into account values that are longer than 1024. Or more precisely, it stops caring at 1024 chars.

But I don't think it changes anything performance or storage wise on fields where values are all reasonably sized (e.g. 20 to 100 chars).

[willemdh]

I think all my questions in this issue got answered. Tx all. Closing up.