[Filebeat] Select output index based on the source input #14010
Conversation
|
Passing a fmtstr via Meta indeed is hacky. Meta should be treated as a document itself, only storing serializable/final values. We send Meta as About Wrapper. The The I think as of now building the processor pipeline is much too complex, and something I'd like to clean up in the future. Settings like But having processors capabilities, I'm thinking to have the The only limitation would be that global processors follow global processors. Users configuring parsing on a global level might run into issues (but we don't support event fields anyways, and we do not parse timestamps). This is indeed some technical debt, that I hope we can reduce by simplifying the processor setup overall. Just thinking loud (no need to implement this): |
libbeat/idxmgmt/std.go
Outdated
| // This helper mimicks applyStaticFmtstr in ilm.go, creating a placeholder | ||
| // event for the restricted set of fields we allow here. It might be worth | ||
| // making this a shared helper function or otherwise specializing for this case. | ||
| func expandIndexPattern( |
There was a problem hiding this comment.
We have this pattern in a few places. When 7.0 was in the making, devs did forget to adapt one or the other place. Maybe we can provide a common formatter type/instance that can provide these default fields?
| type TimestampFormatString struct { | ||
| eventFormatString *EventFormatString | ||
| fields common.MapStr | ||
| } |
| if idx, ok := tmp.(string); ok { | ||
| return idx | ||
| } | ||
| } |
There was a problem hiding this comment.
hm... where is alias used? The name looks very specific to some use-case. I guess it was added to work around the timestamp addition when index is used. Should we go with 'alias' or 'raw-index' in general?
There was a problem hiding this comment.
I don't know who might be using alias or how, which is why I was reluctant to overload it (that, and because of the potentially confusing name)... my preference is for raw-index or something similar (raw-index-name?) since the functional behavior is independent of whether aliases are being used, though I could be convinced to go with alias instead to avoid proliferating magic Metadata fields...
There was a problem hiding this comment.
Looks like we need to grep the source. I guess this comes from Metricbeat or Heartbeat.
There was a problem hiding this comment.
this is odd I don't remember that?
There was a problem hiding this comment.
I think I copied it from the original ILM PR. It was introduced in 7.0. Not sure if anyone still uses it. Removing could break things. We should at least create a follow up issue (technical debt) to investigate and remove it in the future.
+1 on not relying on alias here.
faec
changed the title
|
Finished the last pieces including new unit tests confirming the order of application of all the processor settings. I think the only question left is whether to keep the metadata name |
Let's not overload "alias". We actually have to figure if it is in use, or some debt. Everything in For events we normally prefer to use namespaces. For example In case ECS does not create a namespace it uses |
|
Ok, all comments addressed, and docs + changelog updated, so I think I'm just waiting on CI now |
|
Looks like the intake job doesn't like the formatting of |
faec
added
Filebeat
(cherry picked from commit 6a03478)
(cherry picked from commit 6a03478)
faec
added
v7.5.0
and removed
needs_backport
(cherry picked from commit 6a03478)
(cherry picked from commit 6a03478)
…d on the source input (elastic#14664) * [Filebeat] Select output index based on the source input (elastic#14010) (cherry picked from commit 82c0bde) * Delete debug file, fix changelog merge
Adds configuration to Filebeat inputs to override the output index of their events (#13255)