timestamp stage with multiple formats

[aminjam]

Is your feature request related to a problem? Please describe.
Yes, We consume data that can have multiple types of timestamp formats (RFC3339Nano, Unix, or "2006-01-02 03:04:05.000000000 +0000 UTC"). All timestamps arrive under the same json key but different formats. Currently we inspect the input beforehand, find out the format, and create a unique promtail config for a given selector.

Describe the solution you'd like
It would be ideal to define multiple formats under timestamp stage. timestamp stage would then process each format and if it errors, it would try the next one on the list. An example timestamp stage would look like the following:

- json:
   expressions:
     timestamp:
- timestamp:
   source: timestamp
   format: RFC339Nano
    fallback_formats:
    - Unix
    - 2006-01-02 03:04:05.000000000 +0000 UTC

Describe alternatives you've considered
Pre-process the input and create unique promtail configs.

[slim-bean]

This is an interesting idea! There would be some overhead of this approach in CPU terms but I think we are balancing that against ease of use. As you describe this can be accomplished now but only in a very verbose and config heavy way.

I think this is a good idea, however we are pretty swamped at the moment, if this is something you are able to help contribute a PR for that would be great, else I'm happy to keep the issue open and see if anyone else comes along or we have some time to take a look!

[aminjam]

Yes, I'd be happy to PR this feature. As far as the UX goes for this feature, which one of the following options would you prefer?

1. format field separated by new line characters (original proposal)
   • Pros: Backward compatible
   • Cons: New line might not be the best way of defining multiple options. On the surface, it looks like the following logic would break. Also there are many ways of having multi-line strings in YAML and hard to support all possible ways.
2. Introduce a new optional array field for timestamp stage (e.g. backup_formats)
   • Pros: Backward compatible.
   • Cons: It feels like format field is not needed and could be optional
3. Convert the format field to an array of strings instead of string
   • Pros: It's more natural to think of the following feature as a list of formats to try
   • Cons: Backward Incompatible
4. Combine Option 2 & 3. Mark format field as optional and add formats field as an array of strings
   • Pros: Backward compatible and it's more natural to think of the following feature as a list of formats to try
   • Cons: ?
5. Others?

[cyriltovena]

I'm in favor of 2. and name the array fallback_formats.