Add support for logs (and traces from access logs -aka RTR logs- ) on cloudfoundryreceiver

[jriguera]

Component(s)

receiver/cloudfoundry

Is your feature request related to a problem? Please describe.

Currently the Cloudfoundry receiver supports only metrics, but the Loggregator-firehose subsystem (rlp_gateway endpoint) also process all system and app logs. We would like to be able to read those logs with the OTEL collector (adding a tag by type, eg RTR, CELL, SSH ...). Additionally, if the RTR logs have the tracing fields, add support to export spans.

Describe the solution you'd like

Extend the converter in https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/cloudfoundryreceiver/converter.go#L22 to generate log and traces from logs of type RTR (access logs) in the same way as is done for metrics. Because a log line can have different sources depending on where it was generated (app,container, gorouter, api ...) it makes sense to add flags to select the interested ones. There is also to get events like app crashes, scaling, etc ... More information: https://pkg.go.dev/code.cloudfoundry.org/go-loggregator/rpc/loggregator_v2#pkg-types

Describe alternatives you've considered

This is the best option. This component is already in charge of processing metrics and it makes sense to extend it to receive logs and traces from the same source (loggregator-firehose) . The traditional old way is receiving those logs from a component like this one and emit them using syslog (https://github.com/cloudfoundry-community/firehose-to-syslog) but it adds an unneeded jump and it does not allow to process traces.

Additional context

Loggregator-firehose subsystem is in charge of managing metrics, logs and events in Cloudfoundry.

CloudFoundry http routing layer, also known as gorouter component supports tracing, but it does not emit traces by itself. It decorates the access logs (RTR logs) with tracing information:
my.host.name - [2024-04-24T14:09:44.621845452Z] "GET /health HTTP/1.1" 200 0 15 "-" "NING/1.0" "X.X.X.X:46312" "X.X.X.X:61187" x_forwarded_for:"X.X.X.X, X.X.X.X, X.X.X.X" x_forwarded_proto:"https" vcap_request_id:"07212f87-6146-4207-6d63-84fd1e8cdbc4" response_time:0.003175 gorouter_time:0.000298 app_id:"0e761239-7c71-48e8-bd41-8314d67e74dc" app_index:"1" instance_id:"eb698264-6bc3-4168-58b8-80f6" x_cf_routererror:"-" x_forwarded_host:"my.host.name" x_b3_traceid:"07212f87614642076d6384fd1e8cdbc4" x_b3_spanid:"6d6384fd1e8cdbc4" x_b3_parentspanid:"-" b3:"07212f87614642076d6384fd1e8cdbc4-6d6384fd1e8cdbc4" traceparent:"00-07212f87614642076d6384fd1e8cdbc4-6d6384fd1e8cdbc4-01" tracestate:"gorouter=6d6384fd1e8cdbc4"

[github-actions]

Pinging code owners:

• receiver/cloudfoundry: @pellared @crobert-1

%s See Adding Labels via Comments if you do not have permissions to add labels yourself.

[crobert-1]

Thanks for filing @jriguera, I agree that log support for this receiver would be great. Any help is appreciated if you're interested in contributing!

This is a duplicate of #10025, but we can leave this one open as the other went stale 👍

[jriguera]

Ops, sorry, for some reason I did not find the old issue :-/.
I will evaluate with my team the contribution to extend the functionality.
Thanks!

[jriguera]

Nice!, we will implemented and create a PR. We are also interested in these features:

1. Configuration parameter to enable/disable each telemetry type: metrics (current implementation), logs and traces (only from RTR logs, see 3).
2. Configuration flag to filter log by type (source), eg. ingest only logs source RTR or from app (stdout/sdterr).
3. Ingest RTR logs (aka access logs) as spans instead of logs (if they have the proper tracing attributes).

Regarding the point 1, probably is not needed a configuration parameter per-se. Each telemetry type can be implemented which their own class from the Factory creator which will be automatically instantiated depending on the pipeline type (logs, metrics, traces). We will need to figure out the implementation details of ingesting envelopes in this way, if is not needed, no new configuration parameter will be added.

Regarding the point 2, the reason to filter at this stage instead of using a processor later in the pipeline is only because of trying to reduce resource usage. We need to deal with a lot of logs and for us makes sense to discard the logs we do not need as soon as possible, instead of letting them flow on the pipeline consuming resources.

What do you guys think?

[crobert-1]

Point 1 shouldn't be necessary, as you pointed out the same receiver configuration would just be placed in each telemetry pipeline that a user wants to ingest.

Point 2 makes sense, it's good functionality for the receiver to have if necessary. 👍

[jriguera]

We have created a draft PR: #33044, is still a WIP because the 2nd and 3rd point were not implemented.
We still need to sign the CLA, but I hope you can have a look and tell us what do you think.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• receiver/cloudfoundry: @crobert-1

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[crobert-1]

Resolved by #33044