Direction advice for adding additional data types to awsfirehosereceiver

[skandragon]

I want to add log receiving initially, and eventually events and other types of firehose-sent options. However, the way the awsfirehose receiver is structured right now, this is somewhat difficult.

Currently, there is one data type that is expected to be present in the record, and that means for additional types I would need additional receiver instances. This feels like it complicates things as I need to have different ingress DNS names or paths to handle
this externally.

What I'm thinking about here is to take the current config and add a top level "metrics" and "logs" stanza, where I can then list a URL path to match, the data type expected, and any type-specific options. Example:

receivers:
  awsfirehose:
    endpoint: "0.0.0.0::4321"
    tls: ...
    access_key: "howdy"
    metrics:
      - url_path: /metrics
        record_type: cwmetrics
    logs:
      - url_path: /logs
        record_type: cwlogs

The other option, which I am not so keen on, is to auto-detect each record in the firehose payload, and send this differently based on that auto-discovery. This is not great as there is no required format to be present, so searching for keys to try to auto-detect is not only slow due to multiple JSON decoding steps, but inherently unreliable excepting for a few well defined types.

There are headers included in the firehose POST that will have names and such, but these are user-specified, and as I want this receiver to be multi-source, this would not work as well as the URL path method I feel.

I intend to go down the splitting config path, but if someone wants to throw a wrench in that, now would be a great time. I do not intended to maintain backward compatibility unless it is easy to do so. If I do add backward compatibility, it will define a metric source with path / unless the metrics or logs are specified, in which case it will ignore top-level type configuration. It would also emit a warning when doing this.

Advice welcome, especially from @Aneurysm9 as they are the code owner for this receiver.

[skandragon]

I have since implemented CloudWatch log receive using something similar to what I have above. Backward compatibility was not really easy, as I want out of the box the system to support both (if used in the proper pipeline). I will make a PR for this, but it would be best to merge #34927 if possible.