New component: File Receiver

[pmcollins]

The purpose and use-cases of the new component

The File Receiver reads the output of a File Exporter and replays it into a collector.

The intent is for this receiver to be a counterpart to the File Exporter. Like the File Exporter, it should ultimately support:

• JSON and Protobuf
• metrics, logs, and traces
• compression
• automatically reading from rotated files as written by a File Exporter

However, initial versions can support just a subset of these features and still be useful.

When it replays telemetry, it should read the timestamps of the next telemetry message and wait for the appropriate amount of time before sending, while setting telemetry timestamps to be current (this can be a setting if preferable). Users should be able to set a multiplier in the config so that telemetry can be sent faster or slower than initially received.

This functionality could be used for development, demos, and for reproducing issues.

Example configuration for the component

file:
  path: metrics.json
  format: json
  compression: zstd
  duration_multiplier: 0.5

Telemetry data types supported

Traces, metrics, and logs.

Is this a vendor-specific component?

• This is a vendor-specific component
• If this is a vendor-specific component, I am proposing to contribute this as a representative of the vendor.

Sponsor (optional)

No response

Additional context

@ccordi and I have written an early implementation that can read metrics in JSON format.

[atingchen]

Hi, there is already a otlpjsonfile receiver that may work for the json format. We should talk about how to adapt or merge with that one

[atingchen]

Can we import data using fileconsume?

[atingchen]

Can we rename 'otlpjsonfile receiver' to 'file receiver'? This will cause a breaking change. @djaglowski @atoulme

[djaglowski]

If we can sufficiently generalize the otlpjsonfile receiver, then I think it could reasonably be "renamed". However, I think we might as well just start by copying the codebase. If this component progresses far enough, then we can probably deprecate the otlpjsonfile receiver.

[atingchen]

I feel like this might be a better transition to the new receiver.

[atoulme]

The otlpjsonfile receiver was named as such because it served the purpose of reading from specific files. I don't think this is the same as this receiver, which aims to read from a file, but also replay and do more with it.
The feedback I had around otlpjsonfile was that the file exporter should be renamed otlpjsonfile, not the other way around.
In any case, that's not related to this proposal. Maybe this receiver should be named "replay" receiver?

[atingchen]

@atoulme Thanks for your opinion.
File exporter now not only exports data in json format, but also supports data in proto format and compressed data. So I think it might be inappropriate to rename the file exporter to otlpjsonfile.
Let's keep the parallel development of the two receivers. I think the code of otlpjsonfile can be reused by the new receiver, because the new receiver also needs to read the json format data exported by the file exporter by default.

[djaglowski]

The feedback I had around otlpjsonfile was that the file exporter should be renamed otlpjsonfile, not the other way around.

If I remember correctly, the rationale was that the a receiver focused on ingesting only one specific format should be named accodingly.

What is proposed here is a receiver that would ingest all formats emitted by the corresponding fileexporter. I believe this would be sufficient to warrant the more generic name of filereceiver.

this receiver, which aims to read from a file, but also replay and do more with it

This is an important point. The replay use case makes sense in its own right, but the primary purpose of the filereceiver should be to read signals from files without altering them. In my opinion, the replay use case would best be served by a separate receiver that shares much of the code from this one.

[atingchen]

@djaglowski Can we accept this receiver?

[djaglowski]

I'm willing to sponsor this component without the replay functionality and would be open adding replay oriented features later.

[atingchen]

Hi @pmcollins. Do you still have time to support this recently?

[pmcollins]

Hi @atingchen, I'd be happy to provide an initial implementation (JSON, metrics, no compression) based on the work we have already done here. Additional features would need to be done separately.

Hi @djaglowski, regarding replay functionality, I'm envisioning making available: 1) reading the timestamps of the metrics at the playhead and waiting the delta duration to send them and 2) setting the metrics timestamp to be current. Both of these could be enabled or disabled via a configuration setting. Can these features be added to this receiver out of the gate?

[djaglowski]

I'm envisioning making available: 1) reading the timestamps of the metrics at the playhead and waiting the delta duration to send them and 2) setting the metrics timestamp to be current. Both of these could be enabled or disabled via a configuration setting. Can these features be added to this receiver out of the gate?

The functionality seems reasonable to me, but for the sake of being able to review the code, it would be very helpful to me if we have a fully functioning receiver first. We can add replay immediately afterwards though.

[pmcollins]

Ah, got it. Will do it like that then, thanks.

[pmcollins]

Sorry for the delay. I'm starting on this now.

[MovieStoreGuy]

@djaglowski are you happy being the sponsor for this? I assume your answer hasn't change :)

[djaglowski]

Yes, thanks for updating the labels accordingly.

[rboada04]

@pmcollins Hi, is there an issue to track the support for reading trace files and rotated files?

[pmcollins]

Hi @rboada04, there is not as of yet. Since those capabilities are mentioned above, I thought we could just use this issue. Do you think we should capture them separately?

[rboada04]

It may be good to track in a different issue if the support for each of the telemetry data type (e.g. traces, logs) will be done separately.

At a minimum, I think we need to keep a checklist on this issue of the open items that have been called out. Support for traces & logs, and use of the fileconsumer package come to mind.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

[remram44]

Not inactive, work is happening in #20440

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

[remram44]

I actually don't know what the next step is. The feature is still wanted though!

[crobert-1]

This receiver has been merged and is included in releases of the contrib collector now.

We can close this issue as completed by #20440 and this PR now, right?

[atoulme]

Correct. Closing.